Understanding “versions” and “staging labels” in Secrets Manager

On the surface AWS Secrets Manager is a straightforward service. You create secrets, store values inside them, and can then securely retrieve those values later on.

Beneath the simplicity of the GetSecretValue and PutSecretValue APIs, however, is a service with rich version management capabilities. Understanding the less obvious components of Secrets Manager enables you to build more resilient and cost-effective architectures without a corresponding increase in engineering time.

In this post we’ll step through the two most common Secrets Manager operations and examine what’s happening under the hood. From there, we’ll explore how concepts like “staging labels” can be leveraged to build better cloud systems.

What happens when you store a value?

When you call the PutSecretValue API, a new “version” is created for the secret you specify. Secrets Manager data is immutable which means that values stored in Secrets Manager cannot be modified; only created or deleted. Immutability of secret values is an important feature that differentiates Secrets Manager from other offerings like Parameter Store as it makes losing secret values far harder.

Another concept that comes into play when you store values are “staging labels”—sometimes called “version labels.” A staging label is an arbitrary string that you can associate with a particular version stage.

Secrets have many versions, and each version has one value. Each version can have many staging labels.

In addition to staging labels that you assign yourself, Secrets Manager automatically applies two special staging labels named AWSCURRENT and AWSPREVIOUS. Whenever you create a new secret version it is automatically given the AWSCURRENT staging label, and the previous version with the AWSCURRENT label is given the AWSPREVIOUS label.

What happens when you retrieve a value?

The GetSecretValue API can be used in a few different ways. The most common pattern is to simply provide the name of the secret you want to read the value of. In this case where we don’t provide any additional parameters, Secrets Manager will return the value stored in the version labelled AWSCURRENT.

This works because staging labels are unique within the bounds of an individual secret. A particular staging label can only be assigned to one version, which means that assigning a staging label can be more accurately thought of as “moving” the label. Assigning a label to a version that is already attached to another version results in the label being automatically removed from the previous version.

This results in the combination of a secret name and staging label being sufficient to uniquely identify a particular secret version. While it is possible to retrieve a secret value using a combination of a secret’s name and version ID, it is almost always more ergonomic to use the secret’s name and a staging label instead.

This clears up one misconception a lot of people have about Secrets Manager: the default behavior of GetSecretValue doesn’t give you the newest version! It just gives you the value of the version labelled AWSCURRENT.

The AWSCURRENT label is special in that:

1. If you don’t provide a staging label when calling PutSecretValue, then your new value will receive the AWSCURRENT label
2. It cannot be removed from a version using the UpdateSecretVersionStage API
3. Moving it to a different version results in its previous version receiving the AWSPREVIOUS label

While point #3 almost always happens because of a PutSecretValue call automatically triggering the move of AWSCURRENT, there’s actually nothing stopping you from moving the AWSCURRENT label manually. In cases where you move AWSCURRENT to an older version, a call to GetSecretValue that doesn’t specify a version ID or stage will return that older version’s value.

Manually moving AWSCURRENT is extremely useful for rolling back secrets in response to bad changes. Here’s some sample code which does just that:

async function rollbackSecret(secretName: string) {  const previous = await secretsManager.getSecretValue({    SecretId: secretName,    VersionStage: 'AWSPREVIOUS',  });   await secretsManager.updateSecretVersionStage({    MoveToVersionId: previous.VersionId,    VersionStage: 'AWSCURRENT',  });}

Using staging labels to save cost

A use case I’ve seen for staging labels at companies trying to keep cloud costs to a bare minimum is to group related secret values. An example of this in action is storing an NPM access token for use by a build agent. Some pipelines which build libraries require publish access to NPM, but application libraries only require read access. The principle of least privilege dictates that we should create two tokens for each of these use cases.

The obvious method for achieving this would be to create two separate secrets in Secrets Manager, each storing one token. Another option which saves a bit of cost is to instead create a single secret to store both tokens. Assigning a read or write staging label to each token is enough

IAM supports restricting access to individual staging labels through use of the secretsmanager:VersionStage condition key which means that this configuration can be just as secure as the one using two secrets.

The cost savings in this example are minor; we’ve created one less secret which means we’ll spend US$0.40/month less. But in situations where you’re grouping multiple values—a secret can have up to 20 staging labels, including the special AWS labels—or are replicating secrets to multiple regions, the savings can certainly add up.

I haven’t seen this strategy for cost optimization documented elsewhere. The closest thing I can find are the Parameter Store docs discussing the use of parameter labels to store test and production database credentials within a single parameter. This is fairly similar, so I assume this way of leveraging staging labels isn’t an anti-pattern. In any event, it works.

I think one of the biggest drawbacks to this strategy is that it feels a lot easier to make a mistake when authoring IAM policies. In the case of our NPM tokens it’s important that pipelines that only need read access can’t access our write token, and it can be easy to forget to include the necessary condition key in your policy because this use of Secrets Manager is unusual. Provided you are diligent, however, this is a decent way of saving some dollars on your cloud bill and keeping related secret values collocated.

Deprecated secret versions and quota management

When a secret version has no staging labels, Secrets Manager considers the version deprecated. This most commonly occurs when the AWSPREVIOUS label gets automatically removed from your last secret value after saving a new one. Deprecated secret versions get automatically cleaned up by Secrets Manager after they remain in that state for at least 24 hours.

This automatic cleanup process permanently deletes the version and its value from your AWS account. Secret versions with values that might be useful should always have at least one staging label to avoid being cleaned up.

Cleaning up deprecated secret versions is why the limit of 100 secret versions per secret is usually not restrictive. Most of the time you’ll be updating secret values slower than AWS cleans up old versions, which means that you’ll never hit the limit. If you’re calling PutSecretValue less frequently than once every 10 minutes you’ll be fine.

One situation where it can be easy to store secret values too quickly is when writing AWS Lambda functions. If access tokens are short lived and your lambda is capable of refreshing the token, then a naive solution will result in concurrently executing lambdas to all refresh the token and update your secret at the same time. In serverless applications such as this you’ll want to either schedule token refreshes with EventBridge—moving the refresh logic outside of your original lambda—or use something like DynamoDB as a mutex around the secret.

Takeaways

• Secrets have many versions. Versions have one value, and many staging labels.
• Versions without any staging labels are considered deprecated, and get cleaned up by Secrets Manager automatically.
• So long as you store new values less frequently than once per 10 minutes, you shouldn’t hit the maximum version quota.
• Staging labels are unique across versions within a single secret.
• The AWSCURRENT and AWSPREVIOUS staging labels are automatically managed by AWS whenever you create a new secret version, unless you provide your own staging labels.
• The AWSCURRENT label can be manually moved, enabling easy rollbacks of secret values.
• Staging labels can be used to combine related values into a single secret to save cost.