AWS Lambda Runtime Credentials in Secrets Manager: Why I Stopped Putting Them in Environment Variables

When you build an AWS Lambda function that needs to call external APIs—Blogger, Slack, a payment provider, anything that requires an API key or OAuth token—you face a small decision that has outsized consequences for your operational security posture. Do you store those credentials in Lambda environment variables, or do you fetch them at runtime from AWS Secrets Manager? The first option is faster and simpler. The second requires a few extra lines of code, adds a network call to every cold start, and costs a few cents per month. I chose Secrets Manager for the blog agent, and I would make the same choice again for any Lambda that touches sensitive credentials.

The Real Decision: Visibility Versus Runtime Fetching

Lambda environment variables are encrypted at rest using AWS-managed keys, and they are decrypted automatically when your function starts. They are visible in the Lambda console to anyone with read permissions on the function. They appear in CloudFormation templates, in CI/CD logs if you are not careful, and in any infrastructure-as-code repository unless you parameterize them carefully. When a developer troubleshoots a failed deployment or reviews a function's configuration, those environment variables are right there.

Secrets Manager, by contrast, stores credentials outside the function configuration. Your Lambda retrieves them at runtime using the AWS SDK, and access is governed by IAM policies and resource-based permissions. The secret value never appears in the Lambda console. It does not travel through your deployment pipeline unless you explicitly print it to a log. Rotation, versioning, and audit trails are built into the service. The tradeoff is latency—typically 50 to 150 milliseconds on a cold start—and cost, currently twenty cents per month per secret plus four cents per ten thousand API calls.

Why I Chose Secrets Manager for the Blog Agent

The blog agent is an ARM64 container-based Lambda that generates draft blog posts and submits them to the Blogger API. It stores two runtime credentials in Secrets Manager: a Blogger API refresh token and a GitHub personal access token used to read repository metadata. Both credentials grant write access to external systems, and both would be valuable to an attacker who gained read access to the Lambda configuration.

I did not choose Secrets Manager because of a compliance requirement or an audit checklist. I chose it because I wanted to enforce least-privilege access at the IAM layer and because I wanted the blog agent's CI/CD pipeline to deploy without ever possessing the production credentials. The Lambda execution role has secretsmanager:GetSecretValue permission for two specific secret ARNs. The GitHub Actions workflow that builds and deploys the container image has no access to those secrets at all. Deployment succeeds or fails based on the function's code and configuration, not on whether the CI/CD environment has been granted sensitive credentials.

The operational benefit is that I can rotate credentials in Secrets Manager without redeploying the Lambda. If I suspect a Blogger token has been compromised, I update the secret value and the next invocation picks up the new token. There is no deployment, no environment variable update, no infrastructure change. The secret rotation is invisible to the function's configuration and to anyone reviewing its deployment history.

How to Retrieve Secrets at Runtime in .NET

Fetching a secret from Secrets Manager in a .NET Lambda is straightforward. You instantiate the AmazonSecretsManagerClient, call GetSecretValueAsync with the secret ARN or name, and parse the response. For the blog agent, I cache the secret values in memory after the first retrieval to avoid repeated API calls within the same Lambda execution environment. Cold starts pay the latency cost once; warm invocations reuse the cached values.

Here is the essential pattern:

using Amazon.SecretsManager;
using Amazon.SecretsManager.Model;

private static string _cachedBloggerToken;

private async Task<string> GetBloggerTokenAsync()
{
    if (_cachedBloggerToken != null)
        return _cachedBloggerToken;

    using var client = new AmazonSecretsManagerClient();
    var request = new GetSecretValueRequest
    {
        SecretId = "blog-agent/blogger-token"
    };
    var response = await client.GetSecretValueAsync(request);
    _cachedBloggerToken = response.SecretString;
    return _cachedBloggerToken;
}

This caching strategy works because Lambda execution environments are reused across invocations when traffic is steady. The static field survives between calls, so the second and subsequent invocations within the same container pay no Secrets Manager latency. When the environment is recycled or a new instance is provisioned, the secret is fetched again on the first invocation.

IAM Policy and Least-Privilege Boundaries

The Lambda execution role grants secretsmanager:GetSecretValue only for the specific secrets the function needs. It does not grant secretsmanager:ListSecrets, secretsmanager:DescribeSecret, or any write permissions. The policy looks like this:

{
  "Effect": "Allow",
  "Action": "secretsmanager:GetSecretValue",
  "Resource": [
    "arn:aws:secretsmanager:us-east-1:123456789012:secret:blog-agent/blogger-token-abc123",
    "arn:aws:secretsmanager:us-east-1:123456789012:secret:blog-agent/github-token-def456"
  ]
}

This design means that even if an attacker gains the ability to execute arbitrary code within the Lambda—through a dependency vulnerability, a code injection bug, or a supply-chain compromise—they can only retrieve the two secrets the function already uses. They cannot enumerate other secrets in the account, read secrets belonging to other functions, or rotate secrets to lock out legitimate users. The blast radius is small and the audit trail is clear.

Common Pitfalls

• Logging secret values during debugging. If you print the secret to CloudWatch Logs for troubleshooting, you have just moved the credential from Secrets Manager to a log stream that may have broader IAM access. Use structured logging and redact sensitive fields.
• Not caching secrets across invocations. Fetching the same secret on every function call multiplies your Secrets Manager API costs and adds unnecessary latency. Use static fields or a singleton pattern to cache values within the execution environment.
• Granting overly broad IAM permissions. Do not grant secretsmanager:GetSecretValue on Resource: "*". Specify the exact secret ARNs your function needs. This limits the damage if the execution role is assumed by an unexpected principal.
• Assuming Secrets Manager is always faster than environment variables. It is not. Cold starts will be slower. If your Lambda is latency-sensitive and you have strong controls on who can view environment variables, the tradeoff may not be worth it. Measure and decide based on your threat model.

The Honest Takeaway

Secrets Manager is not the right choice for every Lambda credential. If your function reads a low-sensitivity configuration value—a feature flag endpoint, a public API base URL—an environment variable is fine. But when the credential grants write access to an external system, or when your deployment pipeline should not possess production secrets, runtime retrieval from Secrets Manager enforces a cleaner separation of concerns. The blog agent pays a small latency cost on cold starts and a few cents per month in service fees. In return, it gets credential rotation without redeployment, least-privilege IAM boundaries, and a deployment process that never touches sensitive tokens. For a production Lambda that calls external APIs, that is a trade I would make every time.