MARS v3 · rewrite plan · 2026-07-24
Feed + spider, rewritten.
The current feed + spider were built in 2024, before we had a canonical entity layer, before OpenRouter, before Inferno, before Brave Search, before MCP. Every one of those tools invalidates a load-bearing assumption in the current architecture. This is the plan for rebuilding on the new stack — phased so we can keep processing the task list while it happens.
Why v1 exists in its current shape
In 2024, we had no canonical entity substrate. clean_domains was the entity layer — every deal keyed by md5(domain). Every downstream table was a projection of it. Feed classified with Grok because Qwen didn't exist yet at accessible pricing. Spider used Playwright + Bright Data for every domain because "search-first-then-fetch" wasn't obvious as a pattern. Signals categories (lawsuit / appointment / contract / etc.) got classified but never extracted — the extraction pipeline for those was scoped as a separate signals_pipeline/ vertical that never staffed.
All defensible calls at the time. All now obsolete.
What changed underneath
Canonical entity layer
v2 substrate exists. companies_v2 has 286K canonical entities with CIK, ticker, primary_domain, name_variants, integrity_tier. Feed no longer needs to "discover" companies — it looks them up.
Cheap extraction (Qwen)
OpenRouter Qwen 30B ≈ $0.0002/article. Inferno batch is ~$0 per token on our GPU cluster. Grok-4.3 was $0.005/article. That's a 25× cost reduction — enough to run extraction on all 8 previously-skipped signal categories inline.
Search-first crawling
Brave Search API + requests beats Playwright + Bright Data on 95% of sites. bios team proved it today: 15,695 firm sites, ~$40 total, 0 OOM, 0 rate-limit errors, ~14× faster.
v3 architecture (compact)
NEWS SOURCES (GDELT · RSS · Benzinga SQS · Brave discovery)
│
▼
CLASSIFY (Qwen body-aware, ~$0)
│ route by event_type
├─ funding_round → extract → funding_deals_v2 (existing)
├─ merger → extract → merger_deals_v2 (existing)
├─ lawsuit → extract → lawsuit_events_v2 (NEW)
├─ appointment → extract → appointment_events_v2 (NEW)
├─ contract → extract → contract_events_v2 (NEW)
├─ product_announce → extract → product_events_v2 (NEW)
├─ financial_report → extract → financial_events_v2 (NEW)
├─ license_agreement → extract → license_events_v2 (NEW)
├─ government_issues → extract → government_events_v2 (NEW)
├─ dividend → extract → dividend_events_v2 (NEW)
├─ auditor_change → extract → auditor_change_events_v2 (NEW)
└─ dropped → article_meta_v2 only (existing)
SPIDER (crawl for domains we DON'T already have entity substrate for)
│
├─ 1. BRAVE DISCOVERY (find press / funding article URL up front)
├─ 2. REQUESTS FETCH (95% of sites — no browser, no proxy)
├─ 3. QWEN EXTRACT (in-text task, cheap)
└─ 4. JS-TAIL BROWSER (small nightly pass, only for the 5%
that need it — full OOM defense on
this unit only)
ENTITY RESOLUTION — up front, not downstream
companies_v2 lookup by (CIK | ticker | primary_domain | name_variants)
→ 1 lookup replaces the whole clean_domains → v2 ETL propagation lag
The trade to be honest about
Grok vs Qwen on extraction. Round 4 calibration (task #183 PARKED) found Grok beats Qwen on funding-round extraction: composite 47.5% vs ~42%. But the noise concentrated in one field: funding_date, at 30% agreement, dominates the composite. Task #185 — the deterministic funding-date preprocessor — is estimated to lift Qwen composite 80% → 92%+ by regex-extracting dates and feeding them as candidates to the extractor.
v3 depends on shipping #185 first. Without it, we take a real quality hit for the cost savings. With it, Qwen composite should match or beat Grok at 25× lower cost.
Two ways to build this
| Parallel rewrite | Incremental replacement recommended |
|---|
| Shape |
Build mars_feed_v3 + mars_spider_v3 side-by-side, run in parallel, compare, then cutover |
Add Brave tier in front of current spider; add signals-inline inside existing classify.py; migrate clean_domains → v2 direct writes one event type at a time |
| Timeline |
~2-3 months to first cutover |
~1 month per event type, ~6 months total |
| Blast radius |
Higher at cutover |
Small each step, reversible |
| Managerial overhead |
Big project management surface |
Small each week, sustained |
| What breaks if we stall |
Everything — v3 is 60% done and unusable |
Nothing — each step is standalone |
Incremental wins — smaller cost of a stall, matches how we've built everything else recently (bios, flare, lp_data all shipped incrementally on the existing platform).
Phased plan
Phase 0 — Foundations
~1 week · unblocks everything else
- Ship #185 deterministic funding-date preprocessor. Regex-extract dates from article body, pick closest to publish_date, feed as CANDIDATE. Estimated lift 80% → 92%+ composite. Prerequisite for a Qwen-first rebuild.
- Pilot Brave-discovery on 500 spider domains. Use bios's bios_pipeline/brave_crawl.py as reference. Measure hit-rate on funding/deal shapes (bios saw 62% on team pages). Cost + quality vs current spider.
- Retire the signals_pipeline/ scaffold. Never staffed. Extraction lives in v3-feed. One-shot cleanup + memory update.
Phase 1 — Signals-inline in existing feed
~2-3 weeks · biggest single value unlock
- Extend mars_feed/classify.py to route articles by event_type instead of just scoring them
- Add extractor per category — start with lawsuit (highest-value class per PitchBook competitive analysis) and appointment (person-graph feeder)
- Ship mars.lawsuit_events_v2 + mars.appointment_events_v2 tables, wire Qwen extractor
- Downstream consumer for lawsuit_events_v2: hedge fund / activist monitoring; for appointment_events_v2: prospects vertical
- No v1 replacement yet — clean_domains keeps running unchanged, signals are additive substrate
Phase 2 — Brave-tier in front of spider
~3-4 weeks · biggest cost + reliability win
- Build the search-first path as a new spider service (parallel to current)
- Route domain requests through Brave discovery → requests fetch → Qwen extract
- Only spawn Playwright for the JS-tail sites (small isolated unit, keep the 5-layer OOM defense)
- Migrate mars-spider-gdelt.service first (lower risk than corp_info prod)
- Compare hit-rate, quality, cost, OOM incidence over 2 weeks
- If it holds, migrate mars-spider-prod.service
Phase 3 — Feed direct-to-v2 (per event type)
~1-2 months · migrates the "domain-first" primary key to entity-first
- Migrate funding + merger writes: v1 feed still runs, but v3 path also writes directly to funding_deals_v2 / merger_deals_v2 via canonical entity resolution (companies_v2 lookup, not domain match)
- Add remaining 6 event categories from Phase 1 (contract, product, financial, license, government, dividend, auditor_change)
- Dual-write consumer contract: v3 writes what v1 would have, plus better provenance
- Health check: compare v1 vs v3 event counts per day; alert on drift
Phase 4 — Deprecate v1
~2-4 weeks · retire clean_domains + old feed
- Migrate downstream consumers off clean_domains — coordinate with Raul for dataprep-cleaning + dataprep-alerts
- Freeze v1 feed + spider; verify v3 is stable for 30 days
- Retire Bright Data (except Camoufox Cloudflare fallback path)
- Delete signals_pipeline scaffold if not done in Phase 0
- Update all MCP tools to read v3-tagged methodology
What we're NOT doing
Parallel full rewrite
Too much managerial surface for a small team. Every partial-completion state is unusable.
Retire clean_domains without dual-write
Raul's dataprep-cleaning + alerts consume clean_domains as source of truth. Coordination needed before Phase 4.
Ship v3 before #185 lands
Qwen extraction quality gap on funding_date will contaminate substrate. Preprocessor first.
Migrate Camoufox Cloudflare path
Anti-bot sites need the browser + rotating proxy. Keep that path as-is; it's ~5% of spider volume, not the cost driver.
Cost model (rough)
| Pipeline stage | v1 today | v3 target | Δ |
|---|
| Spider proxy (Bright Data residential) | ~$1,000/mo | ~$50/mo | −95% |
| Feed extraction (Grok-4.3) | ~$400/mo | ~$20/mo | −95% |
| Signals extraction (currently: nothing) | $0 (not extracted) | ~$40/mo (Qwen) | +new |
| Brave Search discovery (new) | $0 | ~$30/mo | +new |
| Total substrate cost | ~$1,400/mo | ~$140/mo | −90% |
Plus the substrate-coverage gain: all 8 signal categories (lawsuit / appointment / contract / product / financial / license / government / dividend / auditor_change) become extracted substrate, not just classified metadata. That's material new product surface for MCP consumers.
Success criteria
- Phase 0: Brave pilot hit-rate ≥ 40% on funding-shape queries; #185 lifts extraction composite to ≥ 85%
- Phase 1: First lawsuit_events_v2 row published; article-level lawsuit coverage ≥ 50%
- Phase 2: mars-spider-gdelt v3 hits ≥ 80% v1's coverage at ≤ 20% v1's cost, zero OOM in 14 days
- Phase 3: v3 funding_deals_v2 writes match v1 daily count ± 5% for 30 days
- Phase 4: Raul's dataprep-cleaning + alerts running on v3 substrate for 14 days without drift
Bottom line
v1 was correct in 2024 and is technically obsolete in 2026 because the tools that would have made it simpler didn't exist. The rewrite is real work — ~6 months total, ~$0 in project-cost extras (all the new tools are cheap-to-free by design). Payoff: substrate coverage expands to 8 new event categories, cost drops 90%, OOM class of failures goes away, and consumer-facing MCP responses inherit richer provenance because v3 writes carry canonical entity IDs from the start instead of resolving them downstream.
Doing this incrementally means the task list stays live — every Phase step is 1-3 tasks, individually reviewable, individually reversible.