Building Resilient Data Connectors When External APIs Fail, Drift or Throttle

External APIs fail in predictable ways. They rate limit you at peak hours. They rename fields without notice. They return HTTP 200 with the real error buried in the JSON body. They cut requests at a fixed timeout, such as the 29 seconds that API Gateway's default integration limit imposes, just before your own timeout would fire. If a connector treats these as rare exceptions, the pipeline breaks the first day the business depends on it.

Resilience is not a retry loop around an HTTP client. It is a set of decisions about who owns each failure, where state lives, and how problems become visible. This article lays those decisions out as a catalog.

A catalog of failures and what to do about them

Failure	How it shows up	Right response
Transient	429, 503, timeouts, connection resets	Retry with backoff and jitter, within a bounded budget
Permanent	401 after refreshing credentials, 404 for deleted resources, 400 for bad input	Do not keep retrying; route to a dead-letter store and alert
Silent	200 with an error object, an empty page where records were expected, a field that vanished	Validate responses and record counts; alert on data freshness
Schema drift	Renamed field, changed type, new required field	Strict validation, raw payload retention, contract tests

The most common mistake is treating every error as transient. Backoff works for a 503. It only delays the fix when an API key has expired, and it will never succeed when an endpoint has been removed.

Detecting silent failures

• Check the body, not just the status. Some APIs return 200 for everything and put success or error in an envelope. Parse the envelope before trusting the payload.
• Do not let deserializers hide drift. A deserializer that silently ignores unknown or missing fields turns an upstream change into missing data. Validate required fields, and log or count unexpected ones.
• Keep the raw payload. Storing the original response (for example in object storage) lets you reprocess after you fix a parsing bug instead of asking the upstream system to resend.
• Alert on freshness. A connector that fails quietly looks healthy in error metrics. A "no new data for N hours" alarm and a periodic reconciliation of counts against the source catch what error rates miss.

Idempotency and state

Retries are only safe if repeating a step does no harm. Suppose a connector fetches a page, writes the records, and crashes before saving its position. The next run fetches the same page again. If the writes are not idempotent you get duplicates, and if the API does not guarantee stable ordering you can also miss records.

The standard remedy is to commit the records and the new cursor together, so they succeed or fail as a unit:

public async Task<int> FetchAndStoreRecords(string connectorId, CancellationToken ct)
{
    var state = await _stateRepository.GetLastCursor(connectorId);
    var response = await _apiClient.GetRecords(
        new ApiRequest { Cursor = state.Cursor, PageSize = 100 }, ct);

    await using var transaction = await _dbContext.Database.BeginTransactionAsync(ct);

    foreach (var record in response.Records)
        await _recordRepository.Upsert(record);   // unique key on the upstream record ID

    await _stateRepository.UpdateCursor(connectorId, response.NextCursor);
    await transaction.CommitAsync(ct);

    return response.Records.Count;
}

This relies on the records and the state living in the same database. If they do not, you need another way to get the same guarantee, such as writing the cursor after a confirmed, idempotent write and accepting that a crash causes a harmless re-fetch.

A few related points:

• Cursor or token pagination is safer than page-number offsets, which shift when records are inserted or deleted during a run.
• For "updated since" queries, subtract a small overlap from the saved timestamp so records that arrive late are not missed, and let the upsert absorb the duplicates.
• For write calls to the API, send an idempotency key when it supports one, so a retried request does not create a second object.

Rate limits are a contract

A 429 is not an unexpected error; it is the API telling you how fast it wants to be called. If the response carries a Retry-After header (a number of seconds or an HTTP date), wait that long. Otherwise use exponential backoff with jitter so many workers do not retry in lockstep.

Know the scope of the limit before designing concurrency. A per-key limit means one connector can starve the others. A per-IP limit means adding workers behind the same address does not help. A per-tenant limit means you need separate queues or credentials per tenant. For tight limits, a token bucket spreads requests evenly instead of bursting and then waiting.

In .NET, a resilience library such as Polly (surfaced through the standard HTTP resilience handlers in recent versions) gives you retries with jitter, per-attempt and overall timeouts, and a circuit breaker. The circuit breaker is worth emphasizing: when an API is down, it stops you from hammering it and filling your own queues with doomed work.

Dead-letter storage you can act on

When a record still fails after the allowed retries, it must land somewhere durable with enough context to diagnose it. A table or an SQS dead-letter queue both work. Store the connector ID, the upstream record ID, the request, the last response, the error type, the retry count and a timestamp. Monitor the depth, alarm on growth, and build a way to replay entries after the cause is fixed. Without this, "why is this record missing?" can only be answered by guessing.

Observability basics

• Log each API call with endpoint, status, duration and record count, in a structured format.
• Track request rate, error rate, retry rate, queue depth and the age of the newest ingested record.
• Correlate logs across runs with a connector ID and run ID so one failure can be followed end to end.

Checklist

1. Classify errors as transient, permanent or silent, and handle each differently.
2. Commit records and cursor together, and make writes idempotent.
3. Honor Retry-After, back off with jitter, and add a circuit breaker.
4. Validate responses, retain raw payloads, and alert on stale data.
5. Send exhausted failures to a dead-letter store with full context, and be able to replay them.
6. Test the unhappy paths: simulate 429, 503, timeouts and schema changes and check that state is preserved and retries are bounded.

You cannot make an unreliable API reliable. You can make the connector fail visibly, recover automatically, and keep processing the data that is good while the rest waits for attention.