Parsing a 200MB X Archive in the Browser: Streaming, Memory and Threads

Limit one: the main thread cannot block

Parsing tens of thousands of records and running regex scans over them takes a few hundred milliseconds on a server. On the browser main thread it freezes the page completely. The user sees an unresponsive tab, and the browser may step in with a page-unresponsive prompt. This is not a performance problem to tune but an architectural one: as long as parsing happens on the main thread, no amount of algorithm work avoids it.

The response is to move parsing into a Web Worker. It runs on its own thread while the main thread only passes a file reference across and receives progress messages. The interface stays interactive, the progress bar keeps moving, and the user can cancel at any point.

Two implementation details matter. A file object can be structured-cloned into the worker, so there is no need to read the contents into memory first. Results, on the other hand, are often large, so trim inside the worker and pass back only the fields you need rather than cloning tens of megabytes in one message.

Limit two: peak memory

Read the whole file as a string and run JSON.parse on it: the most obvious approach, and the first one to fail. A 200MB archive becomes roughly 400MB as a string (two bytes per character in UTF-16), then inflates again when parsed into objects, so the peak easily exceeds a gigabyte. A browser tab's memory ceiling is lower than that.

Keeping the peak down means never holding the whole dataset at once:

• Read the file in chunks, process one at a time, and release the reference when done.
• Discard parsed records as you go, keeping only matched items and counters.
• Avoid string concatenation inside loops. Building large strings repeatedly reallocates; collect into an array and join at the end.
• If intermediate results must persist, put them in IndexedDB rather than resident memory.

One counterintuitive point: converting to a streaming approach to save memory raises complexity without necessarily making anything faster. Streaming saves memory, not time. For an archive of a few megabytes, parsing the whole thing at once is both faster and simpler.

Limit three: how the file gets read

Approach	Memory	Where it fits
Read the whole thing as text	High, roughly twice the file size	Small files, under a few tens of megabytes
Chunked reads	Low, set by chunk size	Large files, with a custom parse boundary
Decompress while reading	Low to moderate	When the archive is a compressed container

Archives usually arrive as a compressed container, which adds a decompression layer. Streaming decompression libraries on the browser side expand data block by block as it arrives instead of producing one large file first. Order matters here: decompress fully then chunk means the peak equals the fully expanded size, while decompressing as you read caps the peak at the chunk size.

Chunking raises a boundary problem: where the cut lands. JSON structure cannot be split at an arbitrary offset, so you maintain a carry-over buffer that keeps the tail of one chunk to prepend to the next. This is the easiest part of a streaming parser to get wrong, and the easiest place to write an infinite loop.

What makes tweets.js awkward

The tweets.js file inside an archive is not plain JSON. It opens with an assignment statement, and the whole thing is not guaranteed to parse as one document. The usual approach is to locate the first opening bracket and parse array elements from there rather than handing the file to a JSON parser.

That has a side benefit: you can report as you parse instead of waiting for the full pass before showing anything. On a 200MB file, seeing a running count within seconds is a very different experience from staring at a blank panel for two minutes.

The internal layout is broken down in the tweets.js anatomy, and field meanings are in the archive file reference.

Keeping regex scans from dominating

Sensitive-data scanning leans on regular expressions, and there are many performance traps. A few rules of thumb:

• Compile the pattern once and reuse it instead of rebuilding it inside the loop.
• Pre-filter with the cheapest possible test, such as a character check, before running the full match.
• Avoid nested quantifiers. Backtracking multiplies the cost of a single long string by several times.
• Run the scan inside the worker so the interface stays responsive.

Patterns for phone numbers and emails need international coverage. Writing only one region's format misses a large share of real-world entries and produces false positives at the same time. Handling those is covered in false positives in audit reports.

Progress and cancellation

A long task has to be cancellable, or closing the tab becomes the only exit. In practice, the main thread posts a flag to the worker, the worker checks it before each chunk, and on a hit it stops and reports how much it processed.

Throttle progress messages. Posting one per record makes messaging itself the bottleneck. Reporting per chunk or on a timer, say every 200 milliseconds, still looks continuous in the interface.

When the user cancels, completed work should not evaporate. Letting them continue with partial results, or explicitly restart, beats forcing a full rerun.

What this buys you

Server-side parsing is far simpler to engineer: a fixed environment, ample memory, no threading constraints. Browser-side parsing pays a real cost in threads, memory and read strategy.

What it buys is that the archive never leaves the device. No upload step, no temporary storage, and no "we delete it from our servers" step that requires trusting someone else. For a privacy tool, that property is part of the product rather than an implementation detail.

There is a practical gain too: no server means no bandwidth bill, so processing tens of thousands of records costs nothing. That arithmetic is what lets a tool keep the audit step free. Further comparison in local versus cloud processing.

About digital-footprint-health.shop

That is the model digital-footprint-health.shop runs on: your X archive is parsed in the browser and never passes through a server. Findings, score and risk list are generated on the device, and closing the tab ends it with no server-side copy. To see what is sitting in your account, start with the free audit; when you want to act on it, deletion runs per tweet, described in the bulk deletion walkthrough.