MARS · Substrate Study
2026-07-16 · commissioned by kee · prompted by Patrick

Linkage coverage study & plan

Which joins are actually joined?

A row-by-row census of every nullable link column across the MARS v2 substrate — the places where the "connect everything" promise silently degrades into disconnected islands — with a ranked plan to close the biggest.

Why this study

Patrick noticed person_work_history_v2.company_id is populated only 64% of the time and asked whether the NULLs are intended. They are — but the observation exposed the class question: how many other consumer-facing joins have quiet gaps we haven't looked at recently? The value proposition of MARS is that everything connects; when connections are missing, downstream tools return short answers instead of complete ones.

This study measures every nullable link column on every v2 table, ranks the gaps by downstream consumer value (not raw row count), and proposes a concrete plan per top gap. It is scoped as a survey — no changes have been applied.

Substrate at rest: 46 v2 tables · 1.4M persons · 284K companies · 2.47M articles · 498K deals. The value-add of any join is measured against a canonical corpus this big — a 5% gap on a hot join is a lot of missed connections.
Nullable link columns audited
42
across 32 tables
≥95% populated
21
structurally sound
50–95% populated
10
closable backlog
<50% populated
11
study focus

Full coverage matrix

Every nullable link-shaped column on every v2 table, ranked by row count. Columns marked tier S or tier A are the study focus. Anything at ≥95% is listed for completeness but not discussed further.

Table Column Rows Populated % Tier Notes
article_meta_v2company_id2,472,26500.0tier SConsumer-blocking. mars_search_articles needs this to answer "articles about X."
form_d_officers_v2company_id1,421,8361,421,836100.0— ok
persons_v2cik1,417,592210,93914.9tier BStructural — only Form 4 slice has CIK by definition.
persons_v2linkedin_url1,417,59245,9973.2tier BBios Phase 4 crawl (planned) closes some.
persons_v2crd1,417,59214,4591.0tier BOnly ADV-registered persons have CRD.
person_bios_v2linkedin_url527,90547,7259.0tier BBios' own identity anchor.
person_company_roles_v2company_id405,048405,048100.0— ok
person_company_roles_v2person_id405,048405,048100.0— ok
form_d_filings_v2company_id355,309354,81599.9— ok (494 orphans)
companies_v2cik284,423186,47265.6Not a strong-ID gap per se; ~99K are Form-D minted with no ticker either.
companies_v2primary_domain284,42363,08322.2tier A176K CIK-minted companies without domain — resolvable via SEC filings.
companies_v2ticker284,4235,5061.9Structural — most companies are private.
companies_v2crd284,4234,1971.5Structural — only ADV-registered operating companies.
person_work_history_v2source_bio_id138,586117,08884.5Provenance link — news_extraction rows won't have a bio_id, by design.
person_work_history_v2company_id138,58688,69564.0tier APatrick's ask. 86% of NULLs are singleton employers; 889 multi-hit resolvable.
form_144_v2issuer_company_id115,26695,96783.3tier AIssuer CIK usually present; unresolved subset is a CIK-mint gap.
form_144_v2seller_person_id115,26626,33622.8tier A88,930 rows have seller_name text. Insider's Form 4 backfill supplies these.
investors_v2crd100,35051,24751.1tier B~30K legit VCs file Form D not ADV. Real gap ≈ 10-15K.
investors_v2parent_investor_id100,3502,2972.3tier CFund family membership. Structural — needs entity-resolution investment.
private_credit_deals_v2sec_form_d_acc_number94,3243080.3tier CCredit team owns; not yet wired.
private_credit_deals_v2lender_investor_id94,32494,01699.7— ok
private_credit_deals_v2company_id94,32494,324100.0— ok
funding_deals_v2company_id78,97678,976100.0— ok
funding_deals_v2sec_form_d_acc_number78,97633,44842.4tier A17,738 unlinked deals have a CIK. Widening cross-link window closes a big chunk.
funding_deals_v2legacy_funding_id78,97678,23799.1Legacy dedup hash — not a consumer join.
person_education_v2institution_id68,81868,818100.0— ok
mergers_resolved_v2acquirer_company_id46,08046,078100.0— ok
mergers_resolved_v2target_company_id46,08046,077100.0— ok
merger_deals_v2legacy_grouped_id36,57536,575100.0— ok
merger_deals_v2target_company_id36,57536,572100.0— ok
merger_deals_v2acquirer_company_id36,57536,573100.0— ok
merger_advisors_v2advisor_id31,53431,534100.0— ok
institutional_positions_v2reporter_investor_id29,93529,935100.0— ok
real_estate_deals_v2sec_form_d_acc_number19,1912,26611.8tier BReit team owns; parallel to funding_deals gap.
real_estate_deals_v2company_id19,19119,191100.0— ok
schedule_13g_v2issuer_company_id13,7738,71163.2tier A5,062 filings have issuer_cik but no company_id — free win via CIK join.
activist_positions_v2reporter_investor_id10,98510,985100.0— ok
institutions_v2parent_institution_id9,0412382.6tier CSchool family map exists (280 families) but most rows unlinked.
institutions_v2ipeds_id9,04100.0tier CNever populated — would need IPEDS ingest.
institutions_v2ror_id9,04100.0tier CNever populated — would need ROR ingest.
schedule_13d_v2issuer_company_id2,7821,73262.3tier A1,050 filings have issuer_cik but no company_id — free win.
company_valuations_v2deal_id115115100.0— ok (tiny table)

Cross-cutting substrate signals

Any-strong-ID coverage

companies_v2: 243,649 / 284,423 (85.7%) carry at least one of cik / crd / ticker / primary_domain. The remaining 14.3% (~40K) are name-only — mostly news-extracted private companies too small to have surfaced via SEC or ADV.

persons_v2: 268,518 / 1,417,592 (18.9%) carry cik / crd / linkedin_url. Insider's incoming Form 4 mint (~199K new CIK-anchored persons) will lift this materially.

Persons by mint origin

mars-v2-persons-v1: 1,205,989 (cik 2.6%, li 3.8%)
insider-form-4-v1: 179,803 (cik 100.0%)
bios-adv-2b-v1: 30,864 (crd 46.1%)
re-gdelt-persons-v1: 905
re-news-persons-v1: 31

The mars-v2-persons-v1 slab is where the low-strong-ID rate lives. News-extracted persons don't come with identifiers unless we go find them.

Ranked gaps — deep dive

Ordered by consumer value, not raw row count. A gap that blocks an MCP tool outweighs a much larger gap on a rarely-joined column. Each entry ships with a concrete action, cost estimate, and prerequisite check.

rank 1 · tier S
article_meta_v2.company_id
0 / 2,472,265 (0.0%)
Every article in the corpus has a title and body, but not a single one is linked to a company. This is the join that mars_search_articles was built to enable — "give me every article about Anthropic" currently requires an ILIKE title scan instead of an indexed join. The pending sweep is task #145 in the backlog.
Plan. Two-path population — deterministic first, LLM second.
Path A
Bridge inheritance — any article whose docid appears in deal_articles_v2 or funding_rounds_v2.source_docid or mergers_resolved_v2.docid inherits company_id from the deal's company. Pure SQL. Closes an estimated 200K–400K rows for free.
Path B
Qwen classification on (title, text_excerpt) against a top-20K company resolver. This is IN-TEXT — Qwen handles alone, no escalation needed for the bulk. Inferno HTTP submit at 100/shard; ~24K shards at ~40s each. Wall-clock ~3–5 days on current fleet.
Cost
Path A: $0 (SQL). Path B: 2M rows × Qwen ≈ $20, plus ~5K ambiguous residue × gpt-oss abstention gate ≈ $0.35, plus ~500 world-knowledge residue × Grok ≈ $40. Total ~$60.
Prereq
Add article_meta_v2.company_id btree if not present; verify inferno HTTP API queue capacity (per [[ref-inferno-http-api]]).
Reversibility
All writes tagged identity_decisions_v2 methodology mars-article-company-link-v1.
rank 2 · tier A
funding_deals_v2.sec_form_d_acc_number
33,448 / 78,976 (42.4%)
Cross-link from news-derived funding deals to SEC Form D filings. Currently at 42.4% via etl_form_d.py --cross-link (Pass 1 CIK-based ±180d, Pass 2 name-based ±90d). Of the 45,528 unlinked, 17,738 (39%) have a CIK on their company — the strong-ID pathway is available and just needs a wider window or a better name-matcher on Pass 2. Every deal linked here gains a leading indicator (Form D often files 3–6 months before news announcement).
Plan. Two-move upgrade to the existing cross-link.
Move 1
Widen Pass 1 CIK window from ±180d to ±365d for high-confidence matches (name+amount agreement). Straight config bump in etl_form_d.py.
Move 2
Pass 2 name-match currently exact-normalized. Replace with trigram candidate finder → Qwen filter → gpt-oss abstention gate → Grok confirm on residue only. Only Grok-web writes confirmed matches; gpt-oss free-closes confident-DIFFERENTS.
Cost
Tier chain: 28K pairs × Qwen ≈ $0.28 → ~4K UNCERTAIN residue × gpt-oss ≈ $0.28 (closes ~40% as confident-different for free) → ~2.4K survivors × Grok ≈ $192. Total ~$193 vs $2,240 all-Grok — 91% saved by the tier chain.
Prereq
Trigram GIN on form_d_filings_v2.entity_name if missing.
Target
65%+ populated post-fix (from 42.4%).
Reversibility
Methodology tag mars-form-d-crosslink-v2.
rank 3 · tier A
schedule_13d_v2 + schedule_13g_v2 · issuer_company_id
10,443 / 16,555 (63.1% combined)
6,112 SEC filings have issuer_cik populated but issuer_company_id NULL. This is a pure CIK-join that never ran — 2,220 distinct CIKs are involved, and we either have those CIKs in companies_v2 already or need to mint them. Fully deterministic, no model calls.
Plan. Two-move SQL job.
Move 1
UPDATE ... SET issuer_company_id = c.company_id FROM companies_v2 c WHERE c.cik = s.issuer_cik AND s.issuer_company_id IS NULL. Free.
Move 2
For CIKs still unresolved, mint shell rows in companies_v2 via Atlas cik_profile (name + city + state). Same pattern as mint_form_d_companies.py.
Cost
$0.
Prereq
None — Atlas access already wired.
Target
90%+ populated post-fix.
Reversibility
Methodology tag mars-13dg-cik-backlink-v1.
rank 4 · tier A
person_work_history_v2.company_id
88,695 / 138,586 (64.0%)
Patrick's original observation. Of the 49,891 NULLs, 85.7% (42,776 rows) are singleton employers — one-off small firms where minting a canonical row would flood the graph without adding join value. The linkable backlog is 4,025 rows across 889 multi-hit employers. Of those 889, 90 have an exact-name match in companies_v2 already (533 rows immediately linkable via a single UPDATE).
Plan. Three-move sweep of the addressable backlog.
Move 1
90 exact-name matches → single SQL UPDATE, closes 533 rows immediately. Free.
Move 2
Remaining 799 multi-hit employers → trigram candidate finder + Qwen verify against companies_v2 with role/date context in the prompt. Free (Qwen).
Move 3
Ambiguous residue → gpt-oss abstention gate first (free-close confident-DIFFERENTS with no companies_v2 write), then Grok-web on the SAMES / UNCERTAINS for shared-table confirm.
Cost
Move 1: $0 (SQL UPDATE). Move 2: 799 pairs × Qwen ≈ $0.008 → ~200 residue × gpt-oss ≈ $0.014 (~40% confident-differents close free) → ~120 survivors × Grok ≈ $9.60. Total ~$10 vs $64 all-Grok.
Prereq
None — bios_pipeline already owns the extension point via link_employers.py.
Target
67–68% populated post-fix (of the ~4K addressable, expect ~3.5K link).
Reversibility
Methodology tag bios-work-history-link-v2.
rank 5 · tier A
companies_v2.primary_domain
63,083 / 284,423 (22.2%)
221,340 companies lack a domain — but 176,417 of those have a CIK, meaning they came in via SEC substrate (mostly Form D minting). SEC filings do surface issuer websites (Part 1A Item 1.I, corporate profile pages, ADV registrations for the operating-company subset). Closing this unlocks the WHERE c.primary_domain = 'anthropic.com' pattern that the funding MCP tools use heavily.
Plan. Layered enrichment.
Move 1
Atlas cik_profile may carry a website column for a subset — pull directly, no LLM. Estimated 20–40K rows.
Move 2
Public tickers (5,506 rows) — look up via issuer investor-relations pages, deterministic if we have a source list. Optional.
Move 3
SEC Form D Item 1.I (executive office address / website) — extract via regex or Qwen from filings XML. Free.
Move 4
The long tail — private companies with no SEC domain field — needs live web lookup ($$$). Defer unless a specific consumer asks.
Cost
Moves 1–3 are $0 (deterministic + Qwen extraction from filings). Move 4 tier-chain estimate on the remaining ~100K: Qwen extract-what-domain-does-this-company-use from any web body ≈ $1; gpt-oss abstains on ambiguity ($7); Grok-web confirm on ~5K residue ($400). Explicitly deferred.
Hazard
Adding wrong domain is worse than no domain (feedback: wrong > missing). Any move that isn't strong-ID-anchored should require Qwen verify.
Target
45–55% populated post-moves-1-3.
Reversibility
Methodology tag mars-company-domain-enrich-v1.
rank 6 · tier A
form_144_v2.seller_person_id
26,336 / 115,266 (22.8%)
Insider-sale notices — 88,930 rows have seller_name text but no person_id. Sellers are executives, board members, or 10%+ owners of the issuer. Insider's incoming Form 4 CIK backfill (~199K new persons_v2 rows) supplies the identity side; this becomes a name+issuer_company_id lookup.
Plan. Coordination + join.
Move 1
Wait for insider's form_4_transactions_v2 mint to populate CIK-anchored persons. Prospects vertical is also waiting on the same milestone.
Move 2
Once minted, deterministic join: match on issuer_company_id + fuzzy name — a Form 144 seller of Apple stock in 2024 should match a Form 4 filer at Apple in 2024.
Cost
$0.
Prereq
Insider ships their Form 4 classified layer (blocks prospects_pipeline too — see [[ref-prospects-pipeline-status]]).
Target
60–70% populated (some name-only matches will remain ambiguous).
Reversibility
Methodology tag mars-form144-seller-link-v1.
rank 7 · tier B
investors_v2.crd
51,247 / 100,350 (51.1%)
51% populated. Of the 49K NULL rows, a large majority are legitimately non-ADV (VC / PE firms that file only Form D, not ADV). This is not a linkage failure but a structural mix. To make the gap actionable, distinguish "should have CRD but doesn't" from "legitimately no CRD."
Plan. Classify then resolve the addressable subset.
Move 1
Add investors_v2.expected_registration_type — deterministic classifier: has ADV filings in sec.acc → expected ADV; only Form D → expected none. Free SQL.
Move 2
Rows classified "expected ADV" but with NULL CRD — resolve via ADV name-match. Trigram → Qwen filter → gpt-oss gate → Grok confirm on the parent-fund-vs-manager-vs-affiliate ambiguities.
Cost
Move 1: $0. Move 2 on ~12K expected-ADV NULLs: Qwen ≈ $0.12 → 2K residue × gpt-oss ≈ $0.14 → ~800 survivors × Grok ≈ $64. Total ~$64 vs $960 all-Grok.
Target
The metric to move is "expected-ADV rows still NULL" — target <5%.
Reversibility
Methodology tag mars-investor-crd-resolve-v1.
rank 8 · tier B
person_bios_v2.linkedin_url
47,725 / 527,905 (9.0%)
480K bios without a LinkedIn URL — mostly ADV Part 2B supplements, which don't include LinkedIn by convention. Closing this materially unlocks person canonicalization: LinkedIn URL is a strong ID, so wherever it lands the row becomes match-first-mint-only-on-no-match.
Plan. Bios Phase 4 crawl (already scoped).

Move 1
Bios Phase 4 portfolio-company deep pulls — Kee's note: 18,491 companies with domain + bioless persons. Scraped bios often include LinkedIn.
Move 2
Optional: Qwen extract-LinkedIn from ADV Part 2B free-text (some sponsors mention LinkedIn in the "Educational Background and Business Experience" section).
Cost
~$40–80 Bright Data + $0 Qwen (bios' existing budget line).
Prereq
Bios Phase 1 completion (in progress).
Target
25–35% populated post-Phase-4.
Reversibility
Methodology tag bios-linkedin-crawl-v1.
rank 9 · tier C
investors_v2.parent_investor_id
2,297 / 100,350 (2.3%)
Fund family membership — "General Catalyst Growth Fund IV" belongs to "General Catalyst." Only 2.3% linked. Prior investment (2026-07-06) built 143 families via first-word clustering + Qwen audit; contamination cleanup and expansion required. High future value for VC-league-table queries but not a today-blocker.
Plan. Full-scale family resolver.
Move 1
Cluster investors_v2 by first-word + trigram similarity to build candidate families.
Move 2
Qwen classify each family as UNIFIED / MIXED / UNCERTAIN. UNIFIED auto-links; MIXED nuked; UNCERTAIN → gpt-oss abstention gate (closes many confident-differents free) → Grok-web on the true "same brand different vehicle" residue.
Cost
Tier chain on ~100K investors clustered into ~5K candidate families: Qwen ≈ $1 → 2K UNCERTAIN residue × gpt-oss ≈ $0.14 → ~800 world-knowledge survivors × Grok ≈ $64. Total ~$65 vs $400 all-Grok. This is the gap where the tier chain saves the most as a percent.
Target
10–15% linked post-sweep (many investors are truly standalone).
Reversibility
Methodology tag mars-family-cleanup-v2-YYYY-MM.
rank 10 · tier C
institutions_v2 · parent / IPEDS / ROR
238 / 9,041 (2.6%) parent · 0.0% for external IDs
Universities and research institutions. Parent linkage exists for schools within the same family (Wharton → Penn); external IDs (IPEDS, ROR) never populated. The immediate consumer value here is limited — bios team resolved the Wharton-vs-Penn case via name_variants, and downstream tools use name-matching rather than external IDs. Deprioritize unless a consumer specifically asks.
Plan. Defer external IDs; extend name_variants.
Move 1
Continue extending institutions_v2.name_variants as bios team surfaces new school families. No IPEDS/ROR ingest needed.
Cost
$0.
Reversibility
Methodology tag bios-school-families-vN.

Themes across the plan

Deterministic first, LLM second, world-knowledge last

Every ranked gap follows the same shape: a deterministic move (CIK join, exact-name match, bridge inheritance) captures the majority of addressable rows for free. Qwen closes the middle band on context-in-the-text tasks. World-knowledge cases (same brand, different real entity) get escalated to Grok-web today — where OpenRouter's larger models slot in as a cheaper substitute once instructions land.

Tier chain — the 3-model doctrine (reit-benchmarked 2026-07-16)

OpenRouter access is live and reit already ran the head-to-head. The load-bearing finding: gpt-oss-120b is NOT a stronger Qwen. On in-text extraction and classification, it ties or loses to Qwen at 2–8× the cost. Its unique value is calibrated abstention — it returns null when it can't know, which Qwen never does. The right role is a gate between Qwen and Grok, not a Qwen replacement.

Tier Model ~$/pair Role Auto-apply?
1 Qwen3-30b (Inferno) ~$0.00001 In-text extraction, classification, gate Yes for classification / gate. NEVER for entity-merge into a shared canonical table.
2 gpt-oss-120b (OpenRouter) ~$0.00007 Calibrated abstention BEFORE Grok. Free-close on confident-differents. Confident-DIFFERENT verdicts only. A wrong "different" only preserves a duplicate — safe. NEVER free-close confident SAMEs into shared canonicals.
3 grok-4-web ~$0.08 World knowledge (same-brand-different-vehicle, same-name-different-person) ONLY tier allowed to CONFIRM a merge into a shared canonical. Yes for SAME_ENTITY verdicts.

Concrete anchor: reit ran a 452-pair company merge_review_needed queue through this chain. Tier 1 gpt-oss closed 140 confident-differents for $0.018. Tier 2 Grok on the 312 residue cost $24.96. All-Grok would have been ~$36 — the gpt-oss gate saved 31% on an adversarially-hard pairs mix. On a typical mix reit measured ~60% Grok reduction. Every per-gap cost estimate below models this chain explicitly instead of treating the top rung as one number.

Operational defaults. gpt-oss is a reasoning model — set max_tokens ≥ 400 or content comes back empty (hidden reasoning burns the budget first). Default reasoning.effort='low'; reit's tests found no verdict change from bumping to high. Client already exists: re_pipeline/llm_openrouter.py reads $OPENROUTER_KEY or ~/.openrouter_key.

Caveats propagated from reit. Their tests were small-N (12–30 per task) and one shape (short structured extraction). gpt-oss may win on shapes not yet tested — long analytical synthesis, math, code. If a specific linkage gap needs a different task shape, re-benchmark before committing.

Mint discipline stays load-bearing

Every action here writes into a canonical entity table. All must respect MINTING_ENTITIES.md (strong-ID match first, sentinel filter at write, log every decision, weekly dedup sweep). Silent minting from a linkage-sweep is the same class of bug as the 2026-06-27 self-merger cleanup that took 1,850 rows to unwind. Every plan entry above ships with a methodology-version tag so identity_decisions_v2 gives us a rollback surface.

Health-check gaps

None of the coverage numbers in the "full coverage matrix" above are currently asserted in health.py. The reason we hadn't noticed article_meta_v2.company_id = 0% as an escalation is that no assertion exists to catch it. Recommendation: add a check_v2_linkage section that alerts when any tier-A column drifts below its post-fix target — treats coverage as an SLA.

Recommended execution order

If MARS has one Claude Code session to spend on linkage this week, spend it on ranks 1, 3, and 4. Ranks 2 and 5 need more coordination or scope; rank 6 waits on insider; ranks 7–10 are lower priority.

  1. Rank 3 (schedule_13d/g CIK backlink) — half a day, $0, closes 6,112 filings deterministically. Warmup win.
  2. Rank 4 (person_work_history_v2 multi-hit) — day-long, $0, closes ~3.5K rows and answers Patrick's original observation with progress.
  3. Rank 1 (article_meta_v2.company_id Path A) — bridge-inheritance pass first (SQL only), then queue Path B onto the inferno HTTP fleet as a multi-day background job.
Delta on the substrate after those three actions: +3,500 new person↔company links, +2,220 resolved SEC positions, +200K to 400K newly-navigable article edges from Path A alone. Total cost through the tier chain: ~$10 (rank 3 free · rank 4 ~$10 · rank 1 Path A free, Path B ~$60 if run to full corpus). Total wall-clock: ~2 days engineer + 3–5 days background inferno if Path B runs.

What this study did not cover

Method. Queried information_schema.columns + pg_catalog.pg_constraint on argos_readonly@172.31.91.195/mars for every table matching %_v2. Coverage numbers are SELECT COUNT(col) / COUNT(*). Consumer-value ranking cross-references the six MCP tool specs at mars_feed/mcp_specs/, MARS_V2_FOR_CLAUDE_CODE.md, and PRODUCTION_READINESS_V2.md. Every plan entry maps to at least one documented consumer join.

Read next. [[ref-consumer-contract-cheatsheet]] for the three filters every consumer must honor. [[ref-canonical-entity-pattern]] for the JSONB→canonical+bridge doctrine and its six failure modes. [[ref-inferno-tier-system]] for the "Qwen where the answer is IN the text; Grok for world knowledge" line — now extended by reit's 2026-07-16 OpenRouter benchmark placing gpt-oss-120b as the abstention gate between them. [[ref-minting-entities-doctrine]] for the mandatory 5-rule discipline any of the plans above must respect.

Revision. v2 (2026-07-16, later same day) folds in reit's OpenRouter head-to-head — every per-gap cost line now models the full Qwen→gpt-oss→Grok tier chain instead of leaving OpenRouter as TBD. Priority ranking unchanged (still driven by consumer value). v1 sections retained in their original position; changes are additive.

Deliverables. Published to kee.staticpipe.com/mars/2026-07-16-linkage-study.html. A companion team-mail memo (~150 lines) is filed to interested vertical slugs — bios, insider, donor, prospects, reit, credit — with the specific gaps that touch their tables highlighted.