diff --git a/docs/docs/backup.md b/docs/docs/backup.md deleted file mode 100644 index bd931b3..0000000 --- a/docs/docs/backup.md +++ /dev/null @@ -1,73 +0,0 @@ ---- -sidebar_position: 5 ---- - -import CodeBlock from '@theme/CodeBlock'; -import Backup from '!!raw-loader!../../examples/backup.yaml'; -import ScheduleBackup from '!!raw-loader!../../examples/schedule_backup.yaml'; - -# Backuping an instance - -There are two ways to backup a PostgreSQL Cluster with this plugin through the CloudNativePG operator : - -* One shot backup, equivalent to running it by hand but through a `Backup` - object definition ; -* With `Scheduled Backup` object, equivalent to defining a crontab entry to run a - backup periodically. - -Whatever the kind of backup, users can list and see them with the -appropriate `kubectl` command : - -```console -kubectl get backups.postgresql.cnpg.io -``` - -## One shot backup - -Backup can be requested through a `Backup` object, using the default -CloudNativePG CRD `Backup` definition. The pgbackrest plugin can be specified -when declaring the `Backup` object. The `method` should be set to -`plugin` and the `pluginConfiguration.name` field to -`pgbackrest.dalibo.com`. - -Here is a full example of a backup definition using the pgBackRest -plugin : - -{Backup} - -It's also possible to use the `cnpg` plugin for `kubectl` to perform -your backup : - -```console -kubectl cnpg backup cluster-demo -m plugin --plugin-name pgbackrest.dalibo.com -``` - -When performing a backup, you can choose the repository to which to push -it. To do this, you need to define the `selectedRepository` key using -the number of the repository, according to its position in the list of -configured repositories. For example, to use the first repository: - -```yaml title="backup.yaml" -[...] - pluginConfiguration: - name: pgbackrest.dalibo.com - parameters: - selectedRepository: "1" -``` - -Or with the `cnpg` plugin: - -```console -kubectl cnpg backup cluster-demo -m plugin --plugin-name pgbackrest.dalibo.com \ - --plugin-parameters selectedRepository=1 -``` - -## Scheduled backup - -A scheduled backup uses almost the same definition as a one-shot backup. Only the kind should be changed to `ScheduledBackup`. When using this object, the schedule field (with a `crontab`-like annotation) should -also be defined under the specification (`spec`). - -Here is a full example of a scheduled backup definition using the -pgbackrest plugin : - -{ScheduleBackup} diff --git a/docs/docs/configuration.md b/docs/docs/configuration.md index 953128e..478d976 100644 --- a/docs/docs/configuration.md +++ b/docs/docs/configuration.md @@ -1,16 +1,18 @@ --- -sidebar_position: 6 +sidebar_position: 3 --- import CodeBlock from '@theme/CodeBlock'; import StanzaCustom from '!!raw-loader!../../examples/stanza_custom.yaml'; import PluginConfig from '!!raw-loader!../../examples/plugin_config.yaml'; +import StanzaS3 from '!!raw-loader!../../examples/stanza.yaml'; +import StanzaAzure from '!!raw-loader!../../examples/stanza_azure.yaml'; # Configuration This pgBackRest plugin supports three levels of configuration: -- **Managed configuration:** Options defined via the Stanza or +- **Managed configuration:** Options defined via the `Stanza` or `PluginConfig` CRD. - **Environment variables:** Free-form variables that can be specified using `CustomEnvVars`. `CustomEnvVars` is a list of variables. @@ -76,6 +78,29 @@ PGBACKREST_MY_CUSTOM=CNPG_ROCKS To run pgBackRest with parameters not directly managed by this plugin, the `CustomEnvVar` option can be used. +## Supported repositories type (S3 and Azure) + +The pgBackRest plugin enables PostgreSQL backup files to be stored in: + +- Amazon s3 +- Microsoft Azure Blob Storage + +The plugin relies on the protocols that pgBackRest supports natively. To +configure the repositories for pgBackRest, you must define a `Stanza` +object, which establishes the link between your PostgreSQL `Cluster` and +the repository or repositories. + +Below are a few examples of how to use the supported backup storage +backend. + +### Amazon S3, or S3 compatible solutions + +{StanzaS3} + +### Azure Blob Storage + +{StanzaAzure} + diff --git a/docs/docs/contribute.md b/docs/docs/contribute.md index a2a1042..ea11ed9 100644 --- a/docs/docs/contribute.md +++ b/docs/docs/contribute.md @@ -4,6 +4,41 @@ sidebar_position: 7 # Contribute +## Basic info on how a CNPG plugin works + +Here are some basic informations about how this plugin should work: + +- When installing the plugin, a new deployment is created to run a Pod + for the controller (`pgbackrest-controller`) of our plugin in the same + namespace as the CNPG operator. + +- The CNPG operator detects the plugin when a dedicated Kubernetes + Service (with some specific annotations) is created. + +- Our specialized controller exposes the supported capabilities (at + least those required to manage the + [lifecycle](https://pkg.go.dev/github.com/cloudnative-pg/cnpg-i@v0.1.0/pkg/lifecycle) + of our CNPG instances) to the CNPG operator. + +- When initializing a new Cluster configured with our plugin, the + pgBackRest controller will be called by the CloudNativePG operator. + +- The plugin controller modifies the resources (Deployment / Pods / + Jobs) requested by the CNPG operator (this is done before requesting + the Kubernetes API), and inject some configuration if needed. + + For our pgbackrest plugin, the controller inject a sidecar container + for `pgBackRest` within the PostgreSQL Pods. This sidecar container + executes a manager dedicated to `pgBackRest` (which expose the + required capabilities archive the WAL and backup the PostgreSQL + instance). + +- Our newly created PostgreSQL instance will call the dedicated + `pgBackRest` manager (on the side container) when the archive command + is triggered. + +https://github.com/cloudnative-pg/cnpg-i/blob/main/docs/protocol.md#cnpgi-wal-v1-WALCapability-RPC + ## Deploy CloudNativePG on kind To contribute and test the pgBackRest plugin, a dedicated Kubernetes diff --git a/docs/docs/howitworks.md b/docs/docs/howitworks.md deleted file mode 100644 index 26b632c..0000000 --- a/docs/docs/howitworks.md +++ /dev/null @@ -1,39 +0,0 @@ ---- -sidebar_position: 4 ---- - -# How it works -## How it works - -Here are some basic informations about how this plugin should work: - -- When installing the plugin, a new deployment is created to run a Pod - for the controller (`pgbackrest-controller`) of our plugin in the same - namespace as the CNPG operator. - -- The CNPG operator detects the plugin when a dedicated Kubernetes - Service (with some specific annotations) is created. - -- Our specialized controller exposes the supported capabilities (at - least those required to manage the - [lifecycle](https://pkg.go.dev/github.com/cloudnative-pg/cnpg-i@v0.1.0/pkg/lifecycle) - of our CNPG instances) to the CNPG operator. - -- When initializing a new Cluster configured with our plugin, the - pgBackRest controller will be called by the CloudNativePG operator. - -- The plugin controller modifies the resources (Deployment / Pods / - Jobs) requested by the CNPG operator (this is done before requesting - the Kubernetes API), and inject some configuration if needed. - - For our pgbackrest plugin, the controller inject a sidecar container - for `pgBackRest` within the PostgreSQL Pods. This sidecar container - executes a manager dedicated to `pgBackRest` (which expose the - required capabilities archive the WAL and backup the PostgreSQL - instance). - -- Our newly created PostgreSQL instance will call the dedicated - `pgBackRest` manager (on the side container) when the archive command - is triggered. - -https://github.com/cloudnative-pg/cnpg-i/blob/main/docs/protocol.md#cnpgi-wal-v1-WALCapability-RPC diff --git a/docs/docs/installation.md b/docs/docs/installation.md index d758b89..f325004 100644 --- a/docs/docs/installation.md +++ b/docs/docs/installation.md @@ -4,16 +4,10 @@ sidebar_position: 2 # Installation -To install and use this plugin, Kubernetes and CloudNativePG users should : +## Install prod-ready version -1. Install the plugin by applying the manifest located in the -`kubernetes` directory : - -``` console -kubectl apply -k ./kubernetes/prod -``` - -Users can also use the manifest file from GitHub: +This plugin can be installed and used by Kubernetes and CloudNativePG +users via the manifest located at the root of the repository. ``` console $ kubectl apply -f \ @@ -22,71 +16,76 @@ $ kubectl apply -f \ This will install the plugin in the same `Namespace` as the operator. -# Verifications +### Verifications -The installation can be verified by checking the presence and status -of the `pgbackrest-controller` deployment in the namespace used by the +The installation can be verified by checking the presence and status of +the `pgbackrest-controller` deployment in the namespace used by the CloudNativePG operator (e.g., `cnpg-system`). -```console +``` console kubectl get pod -n cnpg-system NAME READY STATUS RESTARTS AGE cnpg-controller-manager-65bfdb64c9-hwvpd 1/1 Running 0 9m4s pgbackrest-controller-fc9f4d5f4-5w6wt 1/1 Running 0 15s ``` -And also by confirming -that the `Custom Resource Definition` `stanza.pgbackrest.dalibo.com` -is installed. +And also by confirming that the `Custom Resource Definition` +`stanza.pgbackrest.dalibo.com` is installed. -```console +``` console kubectl api-resources --api-group pgbackrest.dalibo.com NAME SHORTNAMES APIVERSION NAMESPACED KIND stanzas pgbackrest.dalibo.com/v1 true Stanza ``` -# Testing versions +## Testing versions -To use the latest testing or unstable version of this plugin, -apply the `test` kustomize overlay. It is configured to pull -the latest alpha/beta images from Docker Hub. You can simply run : -```console +To use the latest testing or unstable version of this plugin, apply the +`test` kustomize overlay. It is configured to pull the latest alpha/beta +images from Docker Hub. You can simply run : + +``` console kubectl apply -k kubernetes/test ``` ## Dev version -1. Build locally the Docker images and load them to a registry that is accessible -by your Kubernetes cluster. +1. Build locally the Docker images and load them to a registry that is + accessible by your Kubernetes cluster. You can build them locally with the command : -```console + +``` console make build-images ``` It will execute the appropriate `docker build` commands. -2. Install the plugin by applying the manifest located in the -`kubernetes` directory : +2. Install the plugin by applying the manifest located in the + `kubernetes` directory : ``` console kubectl apply -k ./kubernetes/dev ``` :::note -Kustomize layers and overlays are available in the `kubernetes` directory. -You can add your own customisation to patch the resources provided by default. + +Kustomize layers and overlays are available in the `kubernetes` +directory. You can add your own customisation to patch the resources +provided by default. + ::: -# Customisation +## Customisation -The image used by the CloudNativePG instance sidecar container can be customised -by adding the `SIDECAR_IMAGE` environment variable to the pgbackrest plugin -controller container. +The image used by the CloudNativePG instance sidecar container can be +customised by adding the `SIDECAR_IMAGE` environment variable to the +pgbackrest plugin controller container. -For example, this patch can be used to add the `SIDECAR_IMAGE` variable : +For example, this patch can be used to add the `SIDECAR_IMAGE` variable +: -```yaml +``` yaml apiVersion: apps/v1 kind: Deployment metadata: diff --git a/docs/docs/usage.md b/docs/docs/operations.md similarity index 53% rename from docs/docs/usage.md rename to docs/docs/operations.md index 0c907ed..5c483af 100644 --- a/docs/docs/usage.md +++ b/docs/docs/operations.md @@ -1,5 +1,5 @@ --- -sidebar_position: 3 +sidebar_position: 4 --- import CodeBlock from '@theme/CodeBlock'; @@ -8,10 +8,12 @@ import Stanza from '!!raw-loader!../../examples/stanza.yaml'; import Cluster from '!!raw-loader!../../examples/cluster.yaml'; import ClusterRestored from '!!raw-loader!../../examples/cluster_restored.yaml'; import ClusterRestoredBackupID from '!!raw-loader!../../examples/cluster_restored_backupid.yaml'; +import Backup from '!!raw-loader!../../examples/backup.yaml'; +import ScheduleBackup from '!!raw-loader!../../examples/schedule_backup.yaml'; -# Using the plugin +# Operations -## Create an instance with pgBackRest +## Create a Cluster with pgBackRest The `examples` directory contains several pre-configured manifests designed to work with [`kind`](https://kind.sigs.k8s.io/) (Eg: the pull @@ -24,33 +26,34 @@ To use this plugin with a `Cluster`, CloudNativePG users must : the PostgreSQL `Cluster`. This secret must contain the `key` and `secret-key` of the `s3` bucket. -Example: + Example: -{Secret} + {Secret} -2. Create a pgBackRest `stanza` : -Example: +2. Create a pgBackRest `stanza` -{Stanza} + {Stanza} -:::note The `s3Repositories` variable is a list. You can configure -multiple repositories. You can then select the repository to which your -backup will be performed. By default : + :::note -- the first repository is selected for backup ; -- WAL archiving always occurs on all repositories. -::: + The `s3Repositories` variable is a list. You can configure + multiple repositories. You can then select the repository to which your + backup will be performed. By default : + + - the first repository is selected for backup ; + - WAL archiving always occurs on all repositories. + ::: 3. Create the PostgreSQL `Cluster` and adapt the manifest by : -- adding the plugin definition `pgbackrest.dalibo.com` under the - `plugins` entry; -- referencing the pgBackRest `stanza` resource with `stanzaRef`. + - adding the plugin definition `pgbackrest.dalibo.com` under the + `plugins` entry; + - referencing the pgBackRest `stanza` resource with `stanzaRef`. -Example: + Example: -{Cluster} + {Cluster} If it runs without errors, the `Pod` dedicated to the PostgreSQL `Cluster` should have now two containers. One for the `postgres` service @@ -59,41 +62,74 @@ named `pgbackrest-plugin`. The injected container now holds the responsibility for archiving the WALs and triggering backups when a backup request is made. -## Stanza Consideration +## Backup a Cluster -We chose to adhere to the concepts of the pgBackRest project, especially -regarding the scope of a `Stanza` object. +There are two ways to backup a PostgreSQL Cluster with this plugin +through the CloudNativePG operator : -As stated in the -[documentation](https://pgbackrest.org/user-guide.html#quickstart/configure-stanza), -a *stanza* is specific to a PostgreSQL instance cluster. +- One shot backup, equivalent to running it by hand but through a + `Backup` object definition ; +- With `Scheduled Backup` object, equivalent to defining a crontab entry + to run a backup periodically. -> A stanza is the configuration for a PostgreSQL database cluster that -> defines where it is located, how it will be backed up, archiving -> options, etc. +Whatever the kind of backup, users can list and see them with the +appropriate `kubectl` command : -Therefore, you will need to create as many `Stanza` objects as you have -deployed `Cluster`. +``` console +$ kubectl get backups.postgresql.cnpg.io +``` -### Stanza Initialization (or create-stanza operation) +### On-demand backups -Stanzas are initialized when archiving the first WAL. Since the stanza -initialization state is tracked internally, restarting the sidecar -container will cause the `pgbackrest create-stanza` command to run -again. +Backup can be requested through a `Backup` object, using the default +CloudNativePG CRD `Backup` definition. The pgbackrest plugin can be +specified when declaring the `Backup` object. The `method` should be set +to `plugin` and the `pluginConfiguration.name` field to +`pgbackrest.dalibo.com`. -Adding a new repository to a stanza can require running the -`create-stanza` command again. Currently, this is not done -automatically. Restarting the `pgbackrest-plugin` container will launch -the create-stanza command. +Here is a full example of a backup definition using the pgBackRest +plugin : -## WAL Archiving +{Backup} -WAL archiving can be customized through the `pgbackrest` CRD. It is -possible to define the WAL archiving strategy (e.g. [using the -`asynchronous` -mode](https://pgbackrest.org/configuration.html#section-archive/option-archive-async)) -as well as configure the `pgbackrest` queue size. +It's also possible to use the `cnpg` plugin for `kubectl` to perform +your backup : + +``` console +kubectl cnpg backup cluster-demo -m plugin --plugin-name pgbackrest.dalibo.com +``` + +When performing a backup, you can choose the repository to which to push +it. To do this, you need to define the `selectedRepository` key using +the number of the repository, according to its position in the list of +configured repositories. For example, to use the first repository: + +``` yaml +[...] + pluginConfiguration: + name: pgbackrest.dalibo.com + parameters: + selectedRepository: "1" +``` + +Or with the `cnpg` plugin: + +``` console +$ kubectl cnpg backup cluster-demo -m plugin --plugin-name pgbackrest.dalibo.com \ + --plugin-parameters selectedRepository=1 +``` + +### Scheduled backup + +A scheduled backup uses almost the same definition as a one-shot backup. +Only the kind should be changed to `ScheduledBackup`. When using this +object, the schedule field (with a `crontab`-like annotation) should +also be defined under the specification (`spec`). + +Here is a full example of a scheduled backup definition using the +pgbackrest plugin : + +{ScheduleBackup} ## Restoring a Cluster @@ -113,3 +149,39 @@ If no specific backup (BackupID) is specified, the plugin lets pgBackRest automatically choose the optimal backup using its standard algorithm. For more details, see the [pgBackRest restore documentation](https://pgbackrest.org/command.html#command-restore). + +## WAL Archiving + +WAL archiving can be customized through the `pgbackrest` CRD. It is +possible to define the WAL archiving strategy (e.g. [using the +`asynchronous` +mode](https://pgbackrest.org/configuration.html#section-archive/option-archive-async)) +as well as configure the `pgbackrest` queue size. + +## Stanza Consideration + +We chose to adhere to the concepts of the pgBackRest project, especially +regarding the scope of a `Stanza` object. + +As stated in the +[documentation](https://pgbackrest.org/user-guide.html#quickstart/configure-stanza), +a *stanza* is specific to a PostgreSQL instance cluster. + +> A stanza is the configuration for a PostgreSQL database cluster that +> defines where it is located, how it will be backed up, archiving +> options, etc. + +Therefore, you will need to create as many `Stanza` objects as you have +deployed `Cluster`. + +### Stanza Initialization (or create-stanza operation) + +Stanzas are initialized when archiving the first WAL. Since the stanza +initialization state is tracked internally, restarting the sidecar +container will cause the `pgbackrest create-stanza` command to run +again. + +Adding a new repository to a stanza can require running the +`create-stanza` command again. Currently, this is not done +automatically. Restarting the `pgbackrest-plugin` container will launch +the create-stanza command. diff --git a/docs/docs/repositories.md b/docs/docs/repositories.md deleted file mode 100644 index 4d54277..0000000 --- a/docs/docs/repositories.md +++ /dev/null @@ -1,34 +0,0 @@ ---- -sidebar_position: 8 ---- - -import CodeBlock from '@theme/CodeBlock'; -import StanzaS3 from '!!raw-loader!../../examples/stanza.yaml'; -import StanzaAzure from '!!raw-loader!../../examples/stanza_azure.yaml'; - -# Supported repositories type - -The pgBackRest plugin enables PostgreSQL backup files to be stored in: - -- Amazon s3 -- Microsoft Azure Blob Storage - -The plugin relies on the protocols that pgBackRest supports natively. To -configure the repositories for pgBackRest, you must define a `Stanza` -object, which establishes the link between your PostgreSQL `Cluster` and -the repository or repositories. - -Below are a few examples of how to use the supported backup storage -backend. - -## Amazon S3, or S3 compatible solutions - -{StanzaS3} - -## Azure Blob Storage - -{StanzaAzure} - -