Add support for configuration of priorityClassName

[Bloodyludi]

Motivation

Allow users to configure custom pod priority and preemption by adding the option to set priorityClassName.

https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/

Modifications

Added priorityClassName to all deployment and statefulset templates.

Verifying this change

• Make sure that the change passes the CI checks.

[lhotari]

@Bloodyludi Please provide more context of why a user would want to twek pod priority and preemption? Does it solve a problem with Pulsar?

[scmi-c8y]

Hi, I'm not the OP but our use case is that we define some components of our application (including Pulsar) as "critical", because the application will not operate without them. Setting a priority class would allow us to prioritise scheduling of Pulsar (and other critical component) pods ahead pf optional components and make our deployments more robust during upgrades and configuration changes. So, not really a "Pulsar problem" but more about prioritising some important components ahead of others. We would certainly welcome a change like this.

[lhotari]

@scmi-c8y Please rebase the changes since there are conflicts. It might be easier to redo the changes since there are a lot of merge conflicts.