How many times does one failed downstream call actually get attempted? If you cannot answer that for your own services without opening three consoles, you probably have a retry multiplier you did not choose. This guide is a reasoning exercise with documented defaults, not an incident report: the arithmetic the defaults make possible, and the decisions worth making about it.
For every hop in a call chain, four things should be known: who owns the retry, how many attempts it makes, whether it backs off, and whether the work is safe to repeat. If two layers both answer "I do", the attempts multiply instead of adding.
Two reasonable defaults that together form a multiplier
Take a Lambda function invoked asynchronously that calls another service through an AWS SDK client. Both layers have their own retry behavior.
| Layer | Default behavior | Worst-case attempts |
|---|---|---|
| Lambda asynchronous invocation | A failed invocation is retried up to two more times | 3 handler runs |
| AWS SDK client (standard mode) | Typically 3 attempts, with exponential backoff and jitter | 3 calls per handler run |
| Combined | Each handler run repeats the whole SDK retry sequence | Up to 3 × 3 = 9 downstream calls |
Two details make the multiplication real. Lambda's documentation counts function errors, including runtime errors such as timeouts, as failures, so a slow dependency that trips the function timeout triggers a re-run of the entire handler. And the SDK retries happen inside each run, so they are not shared across runs.
Nine is the worst case, not the typical one, but retry storms happen exactly when a system is already struggling, so the worst case is the one that matters then. SDK defaults also differ by language and mode: the standard mode is typically three attempts, while some SDKs still default to a legacy mode with its own, different counts, so check yours. Add a third layer, such as an upstream caller that also retries, and the product grows again. The AWS Builders' Library makes the same point in general terms: when every layer retries, the number of attempts compounds down the chain, and retries should be limited to one layer where possible.
A retry only helps if it lands on a recovering dependency
Retries fix transient and partial failures, such as a request failing briefly or some requests succeeding while others fail. If the dependency is down for minutes, retries mostly add load.
Timeouts feed the same loop. The Well-Architected guidance warns that a timeout set too low can increase backend traffic because too many requests are retried, and in some cases can contribute to a complete outage. It also notes that many frameworks default to very long timeouts. For that reason the timeout and the retry count are one decision: a tight timeout with three attempts at two layers is a different load profile from a generous timeout with one attempt.
The SDK has a brake, and its defaults are changing
The built-in brake is the retry quota. It is a token bucket that tracks how many retries a client is making. When failures persist, the budget depletes and the SDK stops retrying and fails fast, which matters because retry traffic from many clients can delay a dependency's recovery.
AWS announced updated retry behavior for its SDKs, and the details change the arithmetic:
- Quota costs. In the updated standard mode, each retry after a transient error costs 14 tokens (up from 5), and a retry after throttling costs 5, from a 500-token bucket. By that arithmetic a client can fund roughly 35 consecutive transient retries before failing fast. The quota is per client instance, so it does not translate directly into a fleet-wide figure.
- Backoff. Transient errors now use a 50 ms base delay and throttling errors a 1,000 ms base delay, so the first retry after a brief error is much faster than before.
- Legacy mode. Several SDKs (Java, Python, Ruby, PHP, C++ and the AWS CLI) have historically defaulted to a legacy mode that has no standardized retry quota. The update moves them to standard.
- Rollout. The new behavior can be enabled with
AWS_NEW_RETRIES_2026=trueand is announced to become the default in November 2026. Which behavior your service gets today depends on your SDK and version. - Max attempts may drop. Some SDKs previously defaulted to more than three attempts, and DynamoDB clients use their own defaults, so recalculate the product for your own stack.
Adaptive retry mode is a separate option. The documentation describes it as suited to specialized use cases and cautions against it in multi-tenant applications unless tenants are isolated, so it is not a free upgrade.
The practical rule: do not assume defaults, including this article's numbers. Confirm the retry mode, max attempts and quota behavior of the SDK version you actually run, including the .NET one.
Pick one layer to own retries and make the others fail fast
A sensible default is to let one layer retry and configure the rest to give up quickly. In a small Lambda service, the SDK can absorb brief blips within one invocation, while Lambda's asynchronous retry covers only whole-handler failures. Since the handler may run more than once either way, lower both numbers instead of accepting the product of the defaults.
The following configuration is illustrative. The values show where the knobs are and are not tested recommendations:
Resources:
WorkerAsyncConfig:
Type: AWS::Lambda::EventInvokeConfig
Properties:
FunctionName: !Ref WorkerFunction
Qualifier: $LATEST
MaximumRetryAttempts: 1
MaximumEventAgeInSeconds: 900
DestinationConfig:
OnFailure:
Destination: arn:aws:sqs:REGION:ACCOUNT:worker-failures
WorkerFunction:
Type: AWS::Lambda::Function
Properties:
Environment:
Variables:
AWS_RETRY_MODE: standard
AWS_MAX_ATTEMPTS: "2"
What each setting does:
MaximumRetryAttempts: 1lowers Lambda's own retries from the default of two. With the SDK at two attempts, the worst case falls from nine downstream calls to four (2 × 2).MaximumEventAgeInSecondsmatters because asynchronous events can otherwise be retried for up to six hours, for instance when throttled. A short age avoids retrying obsolete work.OnFailuregives discarded events somewhere to go. Without a destination, Lambda drops events that expire or exhaust their attempts. The function's execution role also needs permission to send to that destination.AWS_RETRY_MODEandAWS_MAX_ATTEMPTSset SDK behavior through environment variables. Support for these shared settings depends on the SDK, so verify it in yours.
Fewer retries only work if repeats are harmless
Fewer retries lower load, but they do not remove duplicates. Lambda's documentation says the same event can arrive more than once even when the function does not error, because the asynchronous queue is eventually consistent, and it advises writing code that handles duplicates gracefully. A failed run may also have completed fully, partially or not at all.
That makes idempotency a precondition for any retry policy, not a follow-up task. The AWS Builders' Library has a separate piece on making retries safe with idempotent APIs, which is worth reading before adding retries to anything that writes.
A pre-change checklist for any call chain
- List every hop, including the invoker, the Lambda service and each SDK client or HTTP library, and write down the attempt count at each.
- Multiply them. If the result is surprising, lower the inner counts first.
- Set explicit connection and request timeouts, and check that a timeout does not trigger a retry the dependency cannot absorb.
- Confirm the retry mode, max attempts and quota behavior of your SDK version, since defaults vary by SDK and are changing.
- Set a maximum event age so stale work is not retried.
- Route exhausted events to a failure destination someone reads, and test that replaying from it is safe.
- Make the handler's side effects idempotent before relying on any of the above.
Where to stay cautious
The nine-call worst case here is arithmetic, not a load test, and the real count depends on which errors are retryable and how the quota drains. SDK defaults are moving, so numbers quoted anywhere, including here, need checking against your version. The recommendation is narrow: count attempts before tuning them, let one layer own retries, and treat idempotency as the price of any retry policy. How much extra load a dependency can absorb is specific to it, and only your own measurements can tell you.
Comments
Post a Comment