Bright Life Code

How many times does one failed downstream call actually get attempted? If you cannot answer that for your own services without opening three consoles, you probably have a retry multiplier you did not choose. This guide is a reasoning exercise with documented defaults, not an incident report: the arithmetic the defaults make possible, and the decisions worth making about it. For every hop in a call chain, four things should be known: who owns the retry, how many attempts it makes, whether it backs off, and whether the work is safe to repeat. If two layers both answer "I do", the attempts multiply instead of adding. Two reasonable defaults that together form a multiplier Take a Lambda function invoked asynchronously that calls another service through an AWS SDK client. Both layers have their own retry behavior. Layer Default behavior Worst-case attempts Lambda asynchronous invocation A failed invocation is retried up to two more times 3 handler runs AWS SDK clie...

A request arrives with a valid token, a malformed JSON body and a path the caller has no business touching. Which component should reject it, and what does each rejection cost? In an API Gateway plus Lambda design there are three candidates: the gateway's request validation, the Lambda authorizer and your handler. Each can say "no", but each can only reason about different information. This guide is a design analysis based on AWS documentation, built around four questions: What information does the check need? Header presence, body shape, caller identity and the target resource's owner are four different inputs. What does a rejection cost? A rejection at the gateway never invokes your code. A rejection in the handler pays for an invocation. How widely is the result cached? A cached decision is only correct if it holds for everything it covers. Who changes the rule, and how fast must a change take effect? Revoking access is a different problem from tightening...

A load job dies halfway through at 3 a.m., and someone reruns it. What, specifically, stops the rerun from double-counting every row that already landed? If the honest answer is "whoever reruns it knows to clean up first", the pipeline is safe by discipline, and discipline fails at the worst moments. This guide covers how to make reruns safe by default on Redshift and Spark, against three simple criteria: A rerun must not need a human to remember a cleanup step. It must not depend on the database to reject duplicates. The input set must be the same on the second attempt as on the first. Redshift will accept your duplicate The first trap is assuming a primary key protects you. In Redshift, uniqueness, primary key and foreign key constraints are informational . An insert can succeed even when it violates one, and the query planner still treats the constraint as true when building plans. If your data breaks that assumption, some query results can come back wrong. The be...

A Spark job is slow, so someone doubles the cluster. The job is still slow, and now it costs twice as much. If that sounds familiar, cluster size was probably never the constraint. This guide covers the checks worth running before approving more nodes, and it separates documented Spark behavior from judgment calls. The deciding question is simple: can the stage that is slow actually spread its work across more cores? If it cannot, extra nodes are a tax rather than a fix. A stage is as slow as its slowest task Spark runs one task per partition, and cores only help if there are enough similarly sized tasks to occupy them. The official tuning guide says a cluster will not be fully used unless parallelism is high enough for each operation, and it suggests two to three tasks per CPU core as a starting point. That guidance is a floor for utilization. It says nothing about how evenly the work is spread across tasks. A stage with 800 tasks can still be dominated by one task holding most ...

A pod restarts itself at 2 a.m., the dashboard is green by morning and nobody opens a ticket. Did the system heal, or did it hide a defect? Kubernetes makes automatic recovery so easy that it is tempting to count every restart as a success. This guide is a framework for deciding what a restart can actually fix, how to configure probes so they do not cause outages of their own, and how to make sure every automatic recovery leaves a record someone reads. A restart only fixes problems that live in process memory Before adding any automatic recovery, ask one question: is the broken state disposable? A restart discards everything in the container's memory, so it clears deadlocks, wedged connection pools and corrupted in-process caches. It does nothing for a bad schema, an expired credential, a poisoned message or an upstream outage. If a fresh process will fail the same way, the restart is noise. Three criteria help for any self-healing mechanism: Is the failure local to the proce...

If your pipeline no longer holds an AWS access key, what stops someone else's workflow from assuming your deploy role? Moving CI/CD to OpenID Connect (OIDC) feels like a finished security task. I think it's only half done. The key is gone, but the question of who may assume the role is now answered by a condition in an IAM trust policy. If that condition is loose, you have swapped a secret you could leak for a boundary you can misconfigure. A note on sources: everything below comes from GitHub's and AWS's own documentation, which I re-read on 11 October 2026. The examples are adapted from it, with placeholders. This is not a benchmark or an incident from my own systems, and I'm not publishing any of my own trust policies. Where a point is my inference or my recommendation rather than something the docs state, I say so. What OIDC removes, and what it leaves for you to decide GitHub's documentation says OIDC lets workflows access AWS without storing AWS cre...