Backup Storage Locations and Volume Snapshot Locations

Overview

Velero has two custom resources, BackupStorageLocation and VolumeSnapshotLocation, that are used to configure where Velero backups and their associated persistent volume snapshots are stored.

A BackupStorageLocation is defined as a bucket or a prefix within a bucket under which all Velero data is stored and a set of additional provider-specific fields (AWS region, Azure storage account, etc.). Velero assumes it has control over the location you provide so you should use a dedicated bucket or prefix. If you provide a prefix, then the rest of the bucket is safe to use for multiple purposes. The API documentation captures the configurable parameters for each in-tree provider.

A VolumeSnapshotLocation is defined entirely by provider-specific fields (AWS region, Azure resource group, Portworx snapshot type, etc.) The API documentation captures the configurable parameters for each in-tree provider.

The user can pre-configure one or more possible BackupStorageLocations and one or more VolumeSnapshotLocations, and can select at backup creation time the location in which the backup and associated snapshots should be stored.

This configuration design enables a number of different use cases, including:

• Take snapshots of more than one kind of persistent volume in a single Velero backup. For example, in a cluster with both EBS volumes and Portworx volumes
• Have some Velero backups go to a bucket in an eastern USA region, and others go to a bucket in a western USA region, or to a different storage provider
• For volume providers that support it, like Portworx, you can have some snapshots stored locally on the cluster and have others stored in the cloud

Limitations / Caveats

• Velero supports multiple credentials for BackupStorageLocations, allowing you to specify the credentials to use with any BackupStorageLocation. However, use of this feature requires support within the plugin for the object storage provider you wish to use. All plugins maintained by the Velero team support this feature. If you are using a plugin from another provider, please check their documentation to determine if this feature is supported.

• Velero supports multiple credentials for VolumeSnapshotLocations, allowing you to specify the credentials to use with any VolumeSnapshotLocation. However, use of this feature requires support within the plugin for the object storage provider you wish to use. All plugins maintained by the Velero team support this feature. If you are using a plugin from another provider, please check their documentation to determine if this feature is supported.

• Volume snapshots are still limited by where your provider allows you to create snapshots. For example, AWS and Azure do not allow you to create a volume snapshot in a different region than where the volume is. If you try to take a Velero backup using a volume snapshot location with a different region than where your cluster’s volumes are, the backup will fail.

• Each Velero backup has one BackupStorageLocation, and one VolumeSnapshotLocation per volume provider. It is not possible (yet) to send a single Velero backup to multiple backup storage locations simultaneously, or a single volume snapshot to multiple locations simultaneously. However, you can always set up multiple scheduled backups that differ only in the storage locations used if redundancy of backups across locations is important.

• Cross-provider snapshots are not supported. If you have a cluster with more than one type of volume, like EBS and Portworx, but you only have a VolumeSnapshotLocation configured for EBS, then Velero will only snapshot the EBS volumes.

• File System Backup data is stored under a prefix/subdirectory of the main Velero bucket, and will go into the bucket corresponding to the BackupStorageLocation selected by the user at backup creation time.

• Velero’s backups are split into 2 pieces - the metadata stored in object storage, and snapshots/backups of the persistent volume data. Right now, Velero itself does not encrypt either of them, instead it relies on the native mechanisms in the object and snapshot systems. A special case is File System Backup, which backs up the persistent volume data at the filesystem level and send it to Velero’s object storage.

• Velero’s compression for object metadata is limited, using Golang’s tar implementation. In most instances, Kubernetes objects are limited to 1.5MB in size, but many don’t approach that, meaning that compression may not be necessary. Note that File System Backup has not yet implemented compression, but does have de-deduplication capabilities.

• If you have multiple VolumeSnapshotLocations configured for a provider, you must always specify a valid VolumeSnapshotLocation when creating a backup, even if you are using File System Backup for volume backups. You can optionally decide to set the --default-volume-snapshot-locations flag using the velero server, which lists the default VolumeSnapshotLocation Velero should use if a VolumeSnapshotLocation is not specified when creating a backup. If you only have one VolumeSnapshotLocation for a provider, Velero will automatically use that location as the default.

Examples

Let’s look at some examples of how you can use this configuration mechanism to address some common use cases:

Take snapshots of more than one kind of persistent volume in a single Velero backup

During server configuration:

velero snapshot-location create ebs-us-east-1 \
    --provider aws \
    --config region=us-east-1

velero snapshot-location create portworx-cloud \
    --provider portworx \
    --config type=cloud

During backup creation:

velero backup create full-cluster-backup \
    --volume-snapshot-locations ebs-us-east-1,portworx-cloud

Alternately, since in this example there’s only one possible volume snapshot location configured for each of our two providers (ebs-us-east-1 for aws, and portworx-cloud for portworx), Velero doesn’t require them to be explicitly specified when creating the backup:

velero backup create full-cluster-backup

Have some Velero backups go to a bucket in an eastern USA region (default), and others go to a bucket in a western USA region

In this example, two BackupStorageLocations will be created within the same account but in different regions. They will both use the credentials provided at install time and stored in the cloud-credentials secret. If you need to configure unique credentials for each BackupStorageLocation, please refer to the later example.

During server configuration:

velero backup-location create backups-primary \
    --provider aws \
    --bucket velero-backups \
    --config region=us-east-1 \
    --default

velero backup-location create backups-secondary \
    --provider aws \
    --bucket velero-backups \
    --config region=us-west-1

A “default” backup storage location (BSL) is where backups get saved to when no BSL is specified at backup creation time.

You can change the default backup storage location at any time by setting the --default flag using the velero backup-location set command and configure a different location to be the default.

Examples:

velero backup-location set backups-secondary --default

During backup creation:

velero backup create full-cluster-backup

Or:

velero backup create full-cluster-alternate-location-backup \
    --storage-location backups-secondary

For volume providers that support it (like Portworx), have some snapshots be stored locally on the cluster and have others be stored in the cloud

During server configuration:

velero snapshot-location create portworx-local \
    --provider portworx \
    --config type=local

velero snapshot-location create portworx-cloud \
    --provider portworx \
    --config type=cloud

During backup creation:

# Note that since in this example you have two possible volume snapshot locations for the Portworx
# provider, you need to explicitly specify which one to use when creating a backup. Alternately,
# you can set the --default-volume-snapshot-locations flag on the `velero server` command (run by
# the Velero deployment) to specify which location should be used for each provider by default, in
# which case you don't need to specify it when creating a backup.
velero backup create local-snapshot-backup \
    --volume-snapshot-locations portworx-local

Or:

velero backup create cloud-snapshot-backup \
    --volume-snapshot-locations portworx-cloud

Use a single location

If you don’t have a use case for more than one location, it’s still easy to use Velero. Let’s assume you’re running on AWS, in the us-west-1 region:

During server configuration:

velero backup-location create backups-primary \
    --provider aws \
    --bucket velero-backups \
    --config region=us-west-1 \
    --default

velero snapshot-location create ebs-us-west-1 \
    --provider aws \
    --config region=us-west-1

During backup creation:

# Velero will automatically use your configured backup storage location and volume snapshot location.
# Nothing needs to be specified when creating a backup.
velero backup create full-cluster-backup

Create a storage location that uses unique credentials

It is possible to create additional BackupStorageLocations that use their own credentials. This enables you to save backups to another storage provider or to another account with the storage provider you are already using.

If you create additional BackupStorageLocations without specifying the credentials to use, Velero will use the credentials provided at install time and stored in the cloud-credentials secret. Please see the earlier example for details on how to create multiple BackupStorageLocations that use the same credentials.

Prerequisites

• This feature requires support from the object storage provider plugin you wish to use. All plugins maintained by the Velero team support this feature. If you are using a plugin from another provider, please check their documentation to determine if this is supported.
• The plugin for the object storage provider you wish to use must be installed.
• You must create a file with the object storage credentials. Follow the instructions provided by your object storage provider plugin to create this file.

Once you have installed the necessary plugin and created the credentials file, create a Kubernetes Secret in the Velero namespace that contains these credentials:

kubectl create secret generic -n velero credentials --from-file=bsl=</path/to/credentialsfile>

This will create a secret named credentials with a single key (bsl) which contains the contents of your credentials file. Next, create a BackupStorageLocation that uses this Secret by passing the Secret name and key in the --credential flag. When interacting with this BackupStorageLocation in the future, Velero will fetch the data from the key within the Secret you provide.

For example, a new BackupStorageLocation with a Secret would be configured as follows:

velero backup-location create <bsl-name> \
  --provider <provider> \
  --bucket <bucket> \
  --config region=<region> \
  --credential=<secret-name>=<key-within-secret>

The BackupStorageLocation is ready to use when it has the phase Available. You can check the status with the following command:

velero backup-location get

To use this new BackupStorageLocation when performing a backup, use the flag --storage-location <bsl-name> when running velero backup create. You may also set this new BackupStorageLocation as the default with the command velero backup-location set --default <bsl-name>.

Modify the credentials used by an existing storage location

By default, BackupStorageLocations will use the credentials provided at install time and stored in the cloud-credentials secret in the Velero namespace. You can modify these existing credentials by editing the cloud-credentials secret, however, these changes will apply to all locations using this secret. This may be the desired outcome, for example, in the case where you wish to rotate the credentials used for a particular account.

You can also opt to modify an existing BackupStorageLocation such that it uses its own credentials by using the backup-location set command.

If you have a credentials file that you wish to use for a BackupStorageLocation, follow the instructions above to create the Secret with that file in the Velero namespace.

Once you have created the Secret, or have an existing Secret which contains the credentials you wish to use for your BackupStorageLocation, set the credential to use as follows:

velero backup-location set <bsl-name> \
  --credential=<secret-name>=<key-within-secret>

Create a volume snapshot location that uses unique credentials

It is possible to create additional VolumeSnapshotLocations that use their own credentials. This may be necessary if you already have default credentials which don’t match the account used by the cloud volumes being backed up.

If you create additional VolumeSnapshotLocations without specifying the credentials to use, Velero will use the credentials provided at install time and stored in the cloud-credentials secret.

Prerequisites

• This feature requires support from the volume snapshotter plugin you wish to use. All plugins maintained by the Velero team support this feature. If you are using a plugin from another provider, please check their documentation to determine if this is supported.
• The plugin for the volume snapshotter provider you wish to use must be installed.
• You must create a file with the object storage credentials. Follow the instructions provided by your object storage provider plugin to create this file.

Once you have installed the necessary plugin and created the credentials file, create a Kubernetes Secret in the Velero namespace that contains these credentials:

kubectl create secret generic -n velero credentials --from-file=vsl=</path/to/credentialsfile>

This will create a secret named credentials with a single key (vsl) which contains the contents of your credentials file. Next, create a VolumeSnapshotLocation that uses this Secret by passing the Secret name and key in the --credential flag. When interacting with this VolumeSnapshotLocation in the future, Velero will fetch the data from the key within the Secret you provide.

For example, a new VolumeSnapshotLocation with a Secret would be configured as follows:

velero snapshot-location create <vsl-name> \
  --provider <provider> \
  --config region=<region> \
  --credential=<secret-name>=<key-within-secret>

To use this new VolumeSnapshotLocation when performing a backup, use the flag --volume-snapshot-locations <vsl-name>[,<vsl-name...] when running velero backup create, supplying at most one VSL per provider.

Modify the credentials used by an existing volume snapshot location

By default, VolumeSnapshotLocations will use the credentials provided at install time and stored in the cloud-credentials secret in the Velero namespace. You can modify these existing credentials by editing the cloud-credentials secret, however, these changes will apply to all locations using this secret. This may be the desired outcome, for example, in the case where you wish to rotate the credentials used for a particular account.

You can also opt to modify an existing VolumeSnapshotLocation such that it uses its own credentials by using the snapshot-location set command.

If you have a credentials file that you wish to use for a VolumeSnapshotLocation, follow the instructions above to create the Secret with that file in the Velero namespace.

Once you have created the Secret, or have an existing Secret which contains the credentials you wish to use for your VolumeSnapshotLocation, set the credential to use as follows:

velero snapshot-location set <vsl-name> \
  --credential=<secret-name>=<key-within-secret>

Additional Use Cases

1. If you’re using Azure’s AKS, you may want to store your volume snapshots outside of the “infrastructure” resource group that is automatically created when you create your AKS cluster. This is possible using a VolumeSnapshotLocation, by specifying a resourceGroup under the config section of the snapshot location. See the Azure volume snapshot location documentation for details.

2. If you’re using Azure, you may want to store your Velero backups across multiple storage accounts and/or resource groups/subscriptions. This is possible using a BackupStorageLocation, by specifying a storageAccount, resourceGroup and/or subscriptionId, respectively, under the config section of the backup location. See the Azure backup storage location documentation for details.