Counts as the backbone, top posts as the flesh, static JSON as the product.

Architecture

Five stages, all on Cloudflare. The X keys live only in the collector Worker's secrets; everything public is a derived artifact. The site never talks to a backend: it fetches versioned JSON from the CDN and renders tweets via official X embeds client-side.

Storage: three tiers, and no database in the read path

Do we want a relational database? Yes: a small one, in the middle, that we can lose without losing anything. The shape of this product is "append-only events in, windowed aggregates out, static files served": that wants a lake + a hot working set + a CDN, not a big always-on database. The rule that matters most: readers never touch a database. The site is JSON on a CDN; a database outage can degrade freshness, never availability.

The X API in 2026: what changed and what it costs

Verified against current docs (June 2026). The headline: in February 2026, X replaced the Basic/Pro tiers with pay-per-use for new developers (~$0.005 per post read, 2M reads/month ceiling; reads deduplicated within a 24h UTC window). Pro ($5k/mo, 1M reads) survives only for grandfathered subscribers. We run on our Enterprise contract; the design stays within pay-per-use limits so anyone could run it.

Monthly budget, 20 entities

Collection: query design

Each entity's registry entry compiles to numbered queries (the raw keys in the existing blob already follow this pattern). Aliases are OR-grouped, retweets excluded from content pulls but included in counts, and ambiguous names get guard terms.

// registry/entities/fable-5.json — compiled to queries below
{
  "slug": "fable-5", "kind": "model", "name": "Claude Fable 5", "lab": "anthropic",
  "aliases": ["Fable 5", "Claude Fable", "fable-5", "Fable Anthropic"],
  "guards": ["-aesop -disney"],            // disambiguation for generic words
  "official_handles": ["claudeai", "ClaudeDevs", "AnthropicAI"],
  "launch": { "flag": true, "until": "2026-06-12T18:00:00Z" }
}

// counts query (RTs included — volume is volume)
("Fable 5" OR "Claude Fable" OR "fable-5") -aesop -disney lang:en

// content query (RTs excluded — themes/top posts come from originals)
("Fable 5" OR "Claude Fable" OR "fable-5") -aesop -disney -is:retweet lang:en
// → 2 pulls per cycle: sort_order=relevancy (top) + sort_order=recency since last_seen_id

• Relevancy + recency double-pull. Relevancy surfaces what's resonating; recency with since_id guarantees we never miss the long tail. Both land in raw; dedup happens at enrich.
• RT floods are data, not noise. The June 9 capture is dominated by retweets of one @WatcherGuru post. They count toward volume and amplification metrics, are excluded from theme classification and top-post ranking, and the biggest RT chains surface as "amplification events."
• Query strings are public. Published on the methodology page verbatim. Anyone can re-run them.
• New entity bootstrap: one full-archive pull (30 days) seeds history, then the entity joins the normal cadence.

Enrichment: LLM classification

One batched LLM pass per content pull, ~50 posts per call, classification cached by (post ID, prompt version) in D1 so nothing is ever classified twice. Cheap-fast model class (Gemini Flash / Haiku tier); the per-post cost is fractions of a cent and the whole month of classification costs less than one day of X API.

Derived signals: the signature numbers

All four signature metrics (see Product) are computed in the rollup stage as pure, versioned functions of enriched rows. No new collection, no new infrastructure: they are math on data the pipeline already has.

Data contracts

Three public artifacts, all derived, all versioned with a schema_version. The existing fable-5 summary.json is the seed of the entity summary; v2 splits it into a private full variant and a public variant with no tweet text.

// public/index.json — the leaderboard (one file, ~10KB)
{
  "schema_version": 2, "updated_at": "2026-06-09T21:47:52Z",
  "windows": { "24h": { "total_mentions": 41200,
    "entities": [ { "slug": "fable-5", "rank": 2, "mindshare": 0.218, "delta_24h": 0.186, "vibe_score": 62,
      "sentiment": { "pos": 0.11, "neu": 0.87, "neg": 0.02 },
      "spark_7d": [38, 41, 35, 44, 39, 1107, 2890],
      "top_theme": "novelty", "launch_mode": true } ] } }
}

// public/models/fable-5/summary.json — entity page (evolved from today's blob)
{
  "schema_version": 2, "entity": "fable-5", "updated_at": "…",
  "volume_by_hour": [{ "hour": "2026-06-09T21:00Z", "mentions": 783 }],
  "windows": { "24h": {
    "mentions": 1107, "unique_authors": 976,
    "sentiment": { "pos": 119, "neu": 965, "neg": 23 },
    "say_summary": { "text": "People are taken with how different Fable 5 feels…",
      "citations": [{ "phrase": "token limits", "topic": "pricing", "post_ids": ["2064…"] }] },
    "likes_themes":    [{ "topic": "novelty", "count": 198, "post_ids": ["2064…", "…"], "…": "same shape as below" }],
    "dislikes_themes": [{ "topic": "pricing", "count": 74, "post_ids": ["2064…", "…"],
      "rate_per_1k": 67, "field_median_per_1k": 29, "trend": "accelerating",
      "hourly": [{ "hour": "2026-06-09T21:00Z", "count": 22 }],
      "facets": [{ "label": "token limits", "share": 0.7, "post_ids": ["…"] }],
      "voiced_by": { "builder": 0.31, "influencer": 0.12, "media": 0.08, "anon": 0.49 },
      "verdict": { "text": "The complaint is specific: caps hit mid-session…", "validated": true },
      "emerging": false }],
    "top_posts":  [{ "post_id": "2064453497…", "engagement_score": 935, "author_class": "official", "owned": true }],
    "voices": { "community": [{ "author_id": "…", "class": "builder", "voice_score": 94,
        "rising": false, "post_count": 3, "follower_bucket": "100k-1M" }],
      "owned_affiliated": ["…"], "rival": ["…"] },
    "capabilities": [{ "dimension": "coding", "score": 82, "n": 214, "trend": "up", "post_ids": ["…"] }],
    "tags": [{ "tag": "openclaw", "good": 41, "bad": 3, "post_ids": ["…"] }],
    "threads": [{ "conversation_id": "2064…", "root_post_id": "2064…", "replies": 31,
      "participants": 18, "depth": 4, "organic": true, "thread_score": 87 }],
    "relationship_split": { "owned": 212, "affiliated": 31, "rival": 9, "community": 855 }
  } }
}
// note: post_id / author_id only — text, handles, bios stay in the private variant.
// The page renders posts + author cards client-side via the X embed/oEmbed APIs.

Plus four smaller public artifacts: reads.json (editorial manifest: slug, title, dek, live/frozen status), events.json (the change feed, receipts as post IDs), launches.json (one fingerprint per launch: T0, hourly curve T0→T+72h, subscores, final Launch Score, score version), and tags.json (the global capability-tag index behind search and /good-at pages). Private artifacts (full text, handles, raw) live under derived/ and raw/ prefixes that are never CDN-exposed.

Cadence and launch mode

A per-entity state machine, evaluated hourly by the scheduler against fresh counts. Manual override always wins: flip the flag in the registry before a known launch.

Repo strategy: private machine, public methodology

Decided: the pipeline and site stay private, the entity registry becomes a small public repo, and the methodology page stays radically transparent. Two reasons drive it. First, the anti-gaming logic (spam scores, author classification, amplification detection) only works closed: publish the rules and astroturfers route around them. Second, the collector must be swappable in private: we want the freedom to move collection from the official Enterprise API to a cheaper collector without that swap being visible anywhere.

The Collector contract: the swap point

Everything upstream of R2 raw sits behind one interface; everything downstream consumes normalized rows and never learns the transport.

// the only boundary the rest of the pipeline ever sees
interface Collector {
  counts(query, bucket): CountPoint[]      // hourly volume backbone
  posts(query, opts):    NormalizedPost[]  // relevancy + recency pulls
}
// implementations (all private): x-official (Enterprise), x-payg,
// x-3p (twitterapi.io client, already production-proven in records-ingestion-utils).
// Raw lands under raw/<source>/<entity>/… — the source tag never reaches public artifacts.

• Shadow-mode swaps. Run the candidate collector alongside the incumbent on 2-3 entities for a week; compare hourly counts, unique authors, and top-post overlap. Swap when parity clears ~95%. Counts can stay on the official API as the reference series even after content pulls move.
• Methodology promises outputs, not transport. The public page commits to query strings, windows, dedup rules, and downloadable outputs, all independently checkable by re-running the queries. It never names a vendor, so a collector swap changes nothing we've promised.
• Display is transport-independent. Tweets render via official X embeds regardless of how collection happens; the display-side compliance posture never moves.
• The financial point: third-party collection runs orders of magnitude cheaper per post than pay-per-use (verify current pricing at swap time); at our volume that's ~$2.4k/mo equivalent versus double-digit dollars.
• We say it out loud. A public "How we're open" page states the posture rather than leaving it implicit: open data, open methodology, open registry, closed machine, and the anti-gaming reason why. The score formulas additionally ship as tiny public packages (launch-score, vibe-score) for citability; they're published on the methodology page anyway, so code form costs nothing.

Compliance: how we show tweets without storing people

The site itself

TanStack Start (Vite + TanStack Router) deployed static to Cloudflare Pages: public routes are prerendered to real HTML for SEO and OG cards, then the client hydrates and fetches JSON at runtime. Charts are inline SVG generated from JSON (no chart lib, so the React runtime is the only meaningful bundle). Critically, data refreshes don't rebuild the site: pages fetch the latest JSON at runtime, so a 30-minute launch-mode cadence costs zero deploys.

Risks