Linkage coverage study & plan
A row-by-row census of every nullable link column across the MARS v2 substrate — the places where the "connect everything" promise silently degrades into disconnected islands — with a ranked plan to close the biggest.
Patrick noticed person_work_history_v2.company_id is populated only 64% of the time and asked whether the NULLs are intended. They are — but the observation exposed the class question: how many other consumer-facing joins have quiet gaps we haven't looked at recently? The value proposition of MARS is that everything connects; when connections are missing, downstream tools return short answers instead of complete ones.
This study measures every nullable link column on every v2 table, ranks the gaps by downstream consumer value (not raw row count), and proposes a concrete plan per top gap. It is scoped as a survey — no changes have been applied.
Every nullable link-shaped column on every v2 table, ranked by row count. Columns marked tier S or tier A are the study focus. Anything at ≥95% is listed for completeness but not discussed further.
| Table | Column | Rows | Populated | % | Tier | Notes |
|---|---|---|---|---|---|---|
| article_meta_v2 | company_id | 2,472,265 | 0 | 0.0 | tier S | Consumer-blocking. mars_search_articles needs this to answer "articles about X." |
| form_d_officers_v2 | company_id | 1,421,836 | 1,421,836 | 100.0 | — ok | |
| persons_v2 | cik | 1,417,592 | 210,939 | 14.9 | tier B | Structural — only Form 4 slice has CIK by definition. |
| persons_v2 | linkedin_url | 1,417,592 | 45,997 | 3.2 | tier B | Bios Phase 4 crawl (planned) closes some. |
| persons_v2 | crd | 1,417,592 | 14,459 | 1.0 | tier B | Only ADV-registered persons have CRD. |
| person_bios_v2 | linkedin_url | 527,905 | 47,725 | 9.0 | tier B | Bios' own identity anchor. |
| person_company_roles_v2 | company_id | 405,048 | 405,048 | 100.0 | — ok | |
| person_company_roles_v2 | person_id | 405,048 | 405,048 | 100.0 | — ok | |
| form_d_filings_v2 | company_id | 355,309 | 354,815 | 99.9 | — ok (494 orphans) | |
| companies_v2 | cik | 284,423 | 186,472 | 65.6 | Not a strong-ID gap per se; ~99K are Form-D minted with no ticker either. | |
| companies_v2 | primary_domain | 284,423 | 63,083 | 22.2 | tier A | 176K CIK-minted companies without domain — resolvable via SEC filings. |
| companies_v2 | ticker | 284,423 | 5,506 | 1.9 | Structural — most companies are private. | |
| companies_v2 | crd | 284,423 | 4,197 | 1.5 | Structural — only ADV-registered operating companies. | |
| person_work_history_v2 | source_bio_id | 138,586 | 117,088 | 84.5 | Provenance link — news_extraction rows won't have a bio_id, by design. | |
| person_work_history_v2 | company_id | 138,586 | 88,695 | 64.0 | tier A | Patrick's ask. 86% of NULLs are singleton employers; 889 multi-hit resolvable. |
| form_144_v2 | issuer_company_id | 115,266 | 95,967 | 83.3 | tier A | Issuer CIK usually present; unresolved subset is a CIK-mint gap. |
| form_144_v2 | seller_person_id | 115,266 | 26,336 | 22.8 | tier A | 88,930 rows have seller_name text. Insider's Form 4 backfill supplies these. |
| investors_v2 | crd | 100,350 | 51,247 | 51.1 | tier B | ~30K legit VCs file Form D not ADV. Real gap ≈ 10-15K. |
| investors_v2 | parent_investor_id | 100,350 | 2,297 | 2.3 | tier C | Fund family membership. Structural — needs entity-resolution investment. |
| private_credit_deals_v2 | sec_form_d_acc_number | 94,324 | 308 | 0.3 | tier C | Credit team owns; not yet wired. |
| private_credit_deals_v2 | lender_investor_id | 94,324 | 94,016 | 99.7 | — ok | |
| private_credit_deals_v2 | company_id | 94,324 | 94,324 | 100.0 | — ok | |
| funding_deals_v2 | company_id | 78,976 | 78,976 | 100.0 | — ok | |
| funding_deals_v2 | sec_form_d_acc_number | 78,976 | 33,448 | 42.4 | tier A | 17,738 unlinked deals have a CIK. Widening cross-link window closes a big chunk. |
| funding_deals_v2 | legacy_funding_id | 78,976 | 78,237 | 99.1 | Legacy dedup hash — not a consumer join. | |
| person_education_v2 | institution_id | 68,818 | 68,818 | 100.0 | — ok | |
| mergers_resolved_v2 | acquirer_company_id | 46,080 | 46,078 | 100.0 | — ok | |
| mergers_resolved_v2 | target_company_id | 46,080 | 46,077 | 100.0 | — ok | |
| merger_deals_v2 | legacy_grouped_id | 36,575 | 36,575 | 100.0 | — ok | |
| merger_deals_v2 | target_company_id | 36,575 | 36,572 | 100.0 | — ok | |
| merger_deals_v2 | acquirer_company_id | 36,575 | 36,573 | 100.0 | — ok | |
| merger_advisors_v2 | advisor_id | 31,534 | 31,534 | 100.0 | — ok | |
| institutional_positions_v2 | reporter_investor_id | 29,935 | 29,935 | 100.0 | — ok | |
| real_estate_deals_v2 | sec_form_d_acc_number | 19,191 | 2,266 | 11.8 | tier B | Reit team owns; parallel to funding_deals gap. |
| real_estate_deals_v2 | company_id | 19,191 | 19,191 | 100.0 | — ok | |
| schedule_13g_v2 | issuer_company_id | 13,773 | 8,711 | 63.2 | tier A | 5,062 filings have issuer_cik but no company_id — free win via CIK join. |
| activist_positions_v2 | reporter_investor_id | 10,985 | 10,985 | 100.0 | — ok | |
| institutions_v2 | parent_institution_id | 9,041 | 238 | 2.6 | tier C | School family map exists (280 families) but most rows unlinked. |
| institutions_v2 | ipeds_id | 9,041 | 0 | 0.0 | tier C | Never populated — would need IPEDS ingest. |
| institutions_v2 | ror_id | 9,041 | 0 | 0.0 | tier C | Never populated — would need ROR ingest. |
| schedule_13d_v2 | issuer_company_id | 2,782 | 1,732 | 62.3 | tier A | 1,050 filings have issuer_cik but no company_id — free win. |
| company_valuations_v2 | deal_id | 115 | 115 | 100.0 | — ok (tiny table) |
companies_v2: 243,649 / 284,423 (85.7%) carry at least one of cik / crd / ticker / primary_domain. The remaining 14.3% (~40K) are name-only — mostly news-extracted private companies too small to have surfaced via SEC or ADV.
persons_v2: 268,518 / 1,417,592 (18.9%) carry cik / crd / linkedin_url. Insider's incoming Form 4 mint (~199K new CIK-anchored persons) will lift this materially.
mars-v2-persons-v1: 1,205,989 (cik 2.6%, li 3.8%)
insider-form-4-v1: 179,803 (cik 100.0%)
bios-adv-2b-v1: 30,864 (crd 46.1%)
re-gdelt-persons-v1: 905
re-news-persons-v1: 31
The mars-v2-persons-v1 slab is where the low-strong-ID rate lives. News-extracted persons don't come with identifiers unless we go find them.
Ordered by consumer value, not raw row count. A gap that blocks an MCP tool outweighs a much larger gap on a rarely-joined column. Each entry ships with a concrete action, cost estimate, and prerequisite check.
mars_search_articles was built to enable — "give me every article about Anthropic" currently requires an ILIKE title scan instead of an indexed join. The pending sweep is task #145 in the backlog.docid appears in deal_articles_v2 or funding_rounds_v2.source_docid or mergers_resolved_v2.docid inherits company_id from the deal's company. Pure SQL. Closes an estimated 200K–400K rows for free.(title, text_excerpt) against a top-20K company resolver. This is IN-TEXT — Qwen handles alone, no escalation needed for the bulk. Inferno HTTP submit at 100/shard; ~24K shards at ~40s each. Wall-clock ~3–5 days on current fleet.article_meta_v2.company_id btree if not present; verify inferno HTTP API queue capacity (per [[ref-inferno-http-api]]).identity_decisions_v2 methodology mars-article-company-link-v1.etl_form_d.py --cross-link (Pass 1 CIK-based ±180d, Pass 2 name-based ±90d). Of the 45,528 unlinked, 17,738 (39%) have a CIK on their company — the strong-ID pathway is available and just needs a wider window or a better name-matcher on Pass 2. Every deal linked here gains a leading indicator (Form D often files 3–6 months before news announcement).etl_form_d.py.form_d_filings_v2.entity_name if missing.mars-form-d-crosslink-v2.issuer_cik populated but issuer_company_id NULL. This is a pure CIK-join that never ran — 2,220 distinct CIKs are involved, and we either have those CIKs in companies_v2 already or need to mint them. Fully deterministic, no model calls.UPDATE ... SET issuer_company_id = c.company_id FROM companies_v2 c WHERE c.cik = s.issuer_cik AND s.issuer_company_id IS NULL. Free.companies_v2 via Atlas cik_profile (name + city + state). Same pattern as mint_form_d_companies.py.mars-13dg-cik-backlink-v1.link_employers.py.bios-work-history-link-v2.WHERE c.primary_domain = 'anthropic.com' pattern that the funding MCP tools use heavily.cik_profile may carry a website column for a subset — pull directly, no LLM. Estimated 20–40K rows.mars-company-domain-enrich-v1.seller_name text but no person_id. Sellers are executives, board members, or 10%+ owners of the issuer. Insider's incoming Form 4 CIK backfill (~199K new persons_v2 rows) supplies the identity side; this becomes a name+issuer_company_id lookup.form_4_transactions_v2 mint to populate CIK-anchored persons. Prospects vertical is also waiting on the same milestone.issuer_company_id + fuzzy name — a Form 144 seller of Apple stock in 2024 should match a Form 4 filer at Apple in 2024.[[ref-prospects-pipeline-status]]).mars-form144-seller-link-v1.investors_v2.expected_registration_type — deterministic classifier: has ADV filings in sec.acc → expected ADV; only Form D → expected none. Free SQL.mars-investor-crd-resolve-v1.bios-linkedin-crawl-v1.mars-family-cleanup-v2-YYYY-MM.name_variants, and downstream tools use name-matching rather than external IDs. Deprioritize unless a consumer specifically asks.institutions_v2.name_variants as bios team surfaces new school families. No IPEDS/ROR ingest needed.bios-school-families-vN.Every ranked gap follows the same shape: a deterministic move (CIK join, exact-name match, bridge inheritance) captures the majority of addressable rows for free. Qwen closes the middle band on context-in-the-text tasks. World-knowledge cases (same brand, different real entity) get escalated to Grok-web today — where OpenRouter's larger models slot in as a cheaper substitute once instructions land.
OpenRouter access is live and reit already ran the head-to-head. The load-bearing finding: gpt-oss-120b is NOT a stronger Qwen. On in-text extraction and classification, it ties or loses to Qwen at 2–8× the cost. Its unique value is calibrated abstention — it returns null when it can't know, which Qwen never does. The right role is a gate between Qwen and Grok, not a Qwen replacement.
| Tier | Model | ~$/pair | Role | Auto-apply? |
|---|---|---|---|---|
| 1 | Qwen3-30b (Inferno) | ~$0.00001 | In-text extraction, classification, gate | Yes for classification / gate. NEVER for entity-merge into a shared canonical table. |
| 2 | gpt-oss-120b (OpenRouter) | ~$0.00007 | Calibrated abstention BEFORE Grok. Free-close on confident-differents. | Confident-DIFFERENT verdicts only. A wrong "different" only preserves a duplicate — safe. NEVER free-close confident SAMEs into shared canonicals. |
| 3 | grok-4-web | ~$0.08 | World knowledge (same-brand-different-vehicle, same-name-different-person) | ONLY tier allowed to CONFIRM a merge into a shared canonical. Yes for SAME_ENTITY verdicts. |
Concrete anchor: reit ran a 452-pair company merge_review_needed queue through this chain. Tier 1 gpt-oss closed 140 confident-differents for $0.018. Tier 2 Grok on the 312 residue cost $24.96. All-Grok would have been ~$36 — the gpt-oss gate saved 31% on an adversarially-hard pairs mix. On a typical mix reit measured ~60% Grok reduction. Every per-gap cost estimate below models this chain explicitly instead of treating the top rung as one number.
Operational defaults. gpt-oss is a reasoning model — set max_tokens ≥ 400 or content comes back empty (hidden reasoning burns the budget first). Default reasoning.effort='low'; reit's tests found no verdict change from bumping to high. Client already exists: re_pipeline/llm_openrouter.py reads $OPENROUTER_KEY or ~/.openrouter_key.
Caveats propagated from reit. Their tests were small-N (12–30 per task) and one shape (short structured extraction). gpt-oss may win on shapes not yet tested — long analytical synthesis, math, code. If a specific linkage gap needs a different task shape, re-benchmark before committing.
Every action here writes into a canonical entity table. All must respect MINTING_ENTITIES.md (strong-ID match first, sentinel filter at write, log every decision, weekly dedup sweep). Silent minting from a linkage-sweep is the same class of bug as the 2026-06-27 self-merger cleanup that took 1,850 rows to unwind. Every plan entry above ships with a methodology-version tag so identity_decisions_v2 gives us a rollback surface.
None of the coverage numbers in the "full coverage matrix" above are currently asserted in health.py. The reason we hadn't noticed article_meta_v2.company_id = 0% as an escalation is that no assertion exists to catch it. Recommendation: add a check_v2_linkage section that alerts when any tier-A column drifts below its post-fix target — treats coverage as an SLA.
If MARS has one Claude Code session to spend on linkage this week, spend it on ranks 1, 3, and 4. Ranks 2 and 5 need more coordination or scope; rank 6 waits on insider; ranks 7–10 are lower priority.
integrity_tier assertions live in a separate audit chain.article_meta_v2 has 2.47M rows but the underlying story dedup is a separate exercise.mars schema (Atlas holdings, MySQL sec.*). Those are pull-only for us and stay one-off.Method. Queried information_schema.columns + pg_catalog.pg_constraint on argos_readonly@172.31.91.195/mars for every table matching %_v2. Coverage numbers are SELECT COUNT(col) / COUNT(*). Consumer-value ranking cross-references the six MCP tool specs at mars_feed/mcp_specs/, MARS_V2_FOR_CLAUDE_CODE.md, and PRODUCTION_READINESS_V2.md. Every plan entry maps to at least one documented consumer join.
Read next. [[ref-consumer-contract-cheatsheet]] for the three filters every consumer must honor. [[ref-canonical-entity-pattern]] for the JSONB→canonical+bridge doctrine and its six failure modes. [[ref-inferno-tier-system]] for the "Qwen where the answer is IN the text; Grok for world knowledge" line — now extended by reit's 2026-07-16 OpenRouter benchmark placing gpt-oss-120b as the abstention gate between them. [[ref-minting-entities-doctrine]] for the mandatory 5-rule discipline any of the plans above must respect.
Revision. v2 (2026-07-16, later same day) folds in reit's OpenRouter head-to-head — every per-gap cost line now models the full Qwen→gpt-oss→Grok tier chain instead of leaving OpenRouter as TBD. Priority ranking unchanged (still driven by consumer value). v1 sections retained in their original position; changes are additive.
Deliverables. Published to kee.staticpipe.com/mars/2026-07-16-linkage-study.html. A companion team-mail memo (~150 lines) is filed to interested vertical slugs — bios, insider, donor, prospects, reit, credit — with the specific gaps that touch their tables highlighted.