Designing a Resumable Deletion Job: Checkpoints, Idempotency and Safe Restarts

Once a deletion job runs past a few hundred items, interruptions become normal. Connections drop, laptops sleep, and long-running processes get reclaimed under memory pressure. A job designed on the assumption that it will not be interrupted is ruined by the first disconnect, with all prior work lost.

Why long jobs always get interrupted

Of the three causes, only one is solved by retrying. Network-level blips respond to backoff and retry. Local causes such as sleep and process reclamation require the job to recover after the machine wakes. Platform rate limits require the job to slow down and wait deliberately. Each needs a different response, and all three demand the same property: job state cannot live only in memory.

That is the dividing line for resumability. When state exists only in memory, the process disappears and takes the progress with it.

State machine and checkpoints

Reduce the job to a small set of states and write the necessary fields to disk on every transition. The table below lists common states with what each needs to persist.

State	Fields to persist	Action on recovery
Ready	Work list, filter definition	Start from the beginning
Running	Processed count, last timestamp and ID, current batch	Continue from the cursor
Rate limited	Limit start time, suggested wait	Wait, then continue
Batch complete	Outcome counts for the batch	Start the next batch
Paused	Pause reason, total processed	Continue after manual confirmation
Complete	Summary, failure list	Move to the verification step

The point of the state machine is that the recovery action becomes unambiguous. No guessing about roughly where you were: read the state file and take the matching action. Fewer states are better, and six cover the common cases.

Idempotency: what happens when an item is processed twice

The delete endpoint is close to idempotent. Re-deleting a post that is already gone usually just returns not-found. The real difficulty is the job's own bookkeeping: if repeated requests all count toward the processed total, progress reads high, and with per-item pricing it can add invoice lines.

The fix is a set of processed IDs maintained at the job layer and persisted to disk. Before each batch, drop items already in the set; after a successful item, add it. The set needs size control, so past 100,000 entries store it sharded by day and load only the relevant date range at startup.

Outcome accounting is the other easy-to-miss idempotency point. Failures and skips need separate counters, otherwise after a restart there is no way to tell whether a mismatch means duplicate work or a genuine gap. Related reading covers deletion under API rate limits.

Working with rate limits

Resumption and throttling have to be designed together. A job that retries immediately after a rate limit will not be saved by good recovery logic, because the requests never leave. The practical approach is making the rate-limited condition an explicit state: write the wait duration when entering it, and read that duration on recovery rather than reading the cursor.

Gaps between batches help as well. They keep activity density closer to a normal pattern and give the job a natural interruption point, so a process reclaimed during a gap can still recover safely.

A minimal implementation skeleton

1. List generation. Export work items from the parsed archive, sort by time, and write a file that supports offset reads.
2. State file. Separate from the list, holding the current state and cursor. Write to a temporary file and replace, so a partial write cannot corrupt it.
3. Cursor advance. Update the cursor after each batch, using a timestamp plus tweet ID pair.
4. Processed set. Sharded storage, loading the relevant shard at startup and skipping items already present.
5. Outcome accounting. Count success, failure and skip separately, giving the verification step its input.

The five modules fit in a few hundred lines, and the payoff is a job that makes progress on every run instead of starting over. The larger the job, the bigger the payoff, with sizing maths in deletion limits and time estimates.

About Digital Footprint Health

Digital Footprint Health (digital-footprint-health.shop) covers only the reading side. Upload your X data archive and it parses every tweet on your own device, returning a score from 0 to 100 and flagged items by category. It is read-only, uploads nothing, and never asks for account access. Results export by category as input for a deletion job, with implementation notes in parsing archives in the browser and building your own deletion script. Scope and pricing are on the pricing page, and you can start free from the homepage.