Building a Local Search Index for a 200MB X Archive

Once an X archive is unpacked, the post text sits in a single file called tweets.js. It opens with an assignment statement followed by one enormous JSON array. Nothing is encrypted, but nothing is indexed either, and what you are looking for is usually a single detail: a house number, an email address, the name of a hotel from one trip.

Parsing an archive inside the browser is covered in a separate piece, which solves the "read it without installing anything" problem. This one goes the other way: turn the archive into a reusable local index, built once, queried in milliseconds afterwards.

Why reading it directly exhausts memory

Start with the numbers, because they dictate how the script should be written.

Object	Typical size	Note
Archive ZIP	150-400 MB	Several js files plus media folders
tweets.js	120-300 MB	UTF-8, single-line JSON
Parsed object array	3-5x the source file	Per-object overhead in V8 far exceeds the text
Index file, stopwords dropped	15-40 MB	Depends on post count and token granularity

The third row is the problem. Parsing a couple hundred megabytes of JSON in one call, plus the resident strings after deduplication, reliably hits the default heap ceiling in Node and surfaces as a vague "JavaScript heap out of memory". Raising the limit gets it running, but there is no need to: an index only wants the text, not the object graph.

Streaming the structure away

The approach is to strip the assignment prefix, then split the body at top-level array elements, processing each one and releasing it immediately. Memory then scales with the largest single post rather than with the number of posts.

import fs from 'node:fs';
import readline from 'node:readline';

const src = process.argv[2];
const out = fs.createWriteStream('index.ndjson');

const rl = readline.createInterface({
  input: fs.createReadStream(src, { encoding: 'utf8' }),
  crlfDelay: Infinity,
});

let buf = '';
let count = 0;

rl.on('line', (line) => {
  buf += line;
  for (;;) {
    const end = findObjectEnd(buf, 0);
    if (end < 0) break;
    const chunk = buf.slice(0, end + 1);
    buf = buf.slice(end + 1);
    try {
      const obj = JSON.parse(chunk);
      const t = obj.tweet || obj;
      out.write(JSON.stringify({
        id: t.id_str,
        d: t.created_at,
        s: (t.full_text || t.text || '').replace(/\s+/g, ' '),
      }) + '\n');
      count++;
    } catch (e) { /* one bad element should not stop the run */ }
  }
});

rl.on('close', () => {
  out.end();
  console.log('indexed', count);
});

The companion findObjectEnd does one thing: scan forward from a position while tracking brace depth, returning the index where depth returns to zero. Braces inside string literals have to be skipped, or a quoted parenthesis in someone's post shifts the count and the split goes wrong.

What the index looks like

The output is line-delimited JSON, one post per line, carrying three fields: identity, timestamp, text. Line-oriented storage means you can search with a streaming filter instead of loading the whole thing.

• Identity lets you map a hit back to the original record and deduplicate repeats.
• Timestamp stays in its original form and gets converted only when you need ordering.
• Text is whitespace-collapsed with punctuation preserved, which keeps regex work predictable.

If you want scored full-text retrieval rather than pattern matching, load the ndjson into a SQLite FTS5 table. A single import command gets you there and search becomes an order of magnitude faster than scanning with regex. The structural differences between the archive's export formats are compared in CSV versus JSON exports.

Querying it

With the index built, finding something is a short loop over the ndjson lines against whatever pattern you care about. A few that earn their keep:

• Phone numbers: match eleven consecutive digits, but strip spaces and hyphens first, since posts are inconsistent about both.
• Emails: the usual local-part-at-domain shape, allowing whitespace around the at sign.
• Address fragments: match suffix tokens like street, road, avenue, then confirm each hit by reading the surrounding text.
• Date windows: filter to the range first, then run the patterns inside it, which keeps the scan bounded.

Freeze those patterns into a small script and re-run it after every archive refresh, emitting only the new hits. Paired with the batching logic in a local deletion script, flagged records can feed straight into cleanup without passing through any third-party service.

The index is sensitive data too

Stripping the object graph does not reduce sensitivity, because the full text remains. Placement matters more than build speed: keep it out of any synced folder and out of the temp directory. For long-term storage, encrypt it the way local key management describes, with the key held separately from the data. After a cleanup run, regenerate or destroy the index alongside the archive it came from.

About digital-footprint-health.shop

Rolling your own index suits people who want full control of the pipeline. There is a packaged route if you would rather not write the script. digital-footprint-health.shop turns parsing and scanning into a product: drop the archive into the entry point on the homepage and everything runs locally, producing a 0-100 score and a risk list with nothing uploaded. The check is free and read-only, cleanup scope and cost are on the pricing page, and engineering notes are collected in the blog index.