Signals Pipeline — As Built
The world-wide news firehose → structured company signals · MARS / signals · 2026-08-17
One firehose in, fifteen structured signal types out.
Every article flows through the same spine — a cheap junk gate, one multi-label
classify call, then a branch that routes each doc to every extractor its categories matched.
An article can be a merger and an executive appointment; both fire. The extractors resolve
companies through the shared mint gate and write to per-category canonical tables bound by
company_id to the entity graph.
The flow
firehose → gate → classify → branch → extractors → per-category tables
Firehose
rss · benzinga · gdelt · SEC 8-K · us-courts · sedar — one corpus, any language.
signals.news_corpus_v2 · 22.8M articles
↓
① Gate — exclude junk, never classify
sklearn char n-gram distilled from Qwen. Language-agnostic. Tag-don't-drop: a drop stays in the corpus, recoverable. Tuned for recall.
char_wb(2,5) · AUC 0.932 (36K labels) · t=0.08 · ~$0 CPU · at ingest
↓
② Classify — one call, all categories
Qwen-direct multi-label. Body-aware + multilingual (mars is English-only). Scores stored once; an other residual surfaces the next category to build.
fundingmergerlawsuitappointmentcontractpartnershiplicensefinancial_reportdividendauditor_changegovernment_issuesproductpersonal_windfallbankruptcydata_breach
qwen3-30b · ~$0.00003/doc · classify_scores->'signals'
↓
③ Branch — route to every matched extractor
Each doc goes to all extractors whose category cleared threshold. Partial index over classified rows keeps it fast.
classify_scores ? 'signals' · ncorp_classify_signals
↓
④ Extractors → per-category tables
One Qwen extract per matched category. Resolve companies via the mint gate; harvest people + edges. Write to the category's canonical table.
The fifteen categories — extractor, table, status
"Staging" (extract → resolve → promote) applies only to deals; every other table is the final canonical event table.
| Category | Extractor | Lands in | Rows | Status |
|---|
| funding + merger | extract_deal | signals_deal_events_v2 (staging→canonical) | 53,093 | mature |
| lawsuit | extract_lawsuit | lawsuit_events_v2 (+parties) | 5,862 | mature |
| appointment | extract_appointment | appointment_events_v2 | 31,664 | mature |
| auditor_change | extract_auditor | auditor_change_events_v2 | 308 | new · 8-K Item 4.01 |
| contract·partnership·license | extract_relationship | commercial_relationships_v2 (shared edge/moat) | 105 | new · verification-gated |
| government_issues | extract_regulatory | government_events_v2 | 35 | new |
| dividend | extract_dividend | dividend_events_v2 | 34 | new |
| bankruptcy | extract_bankruptcy | bankruptcy_events_v2 (shared union) | 15 | new |
| data_breach | extract_databreach | data_breach_events_v2 | 8 | new |
| personal_windfall | extract_windfall | personal_windfall_events_v2 | 21 | new · feeds prospects |
| product_announcement | — | — | — | held · low graph value |
| financial_report | — | — | — | deferred · private-co only |
New-extractor rows are from validation + a first whole-pipeline batch, not full production runs — the lever for real volume is classifying more (each classify batch grows every category's pool at once).
Where the data ends up
- Per-category tables, not one staging table. Each category has different fields (a dividend has
amount_per_share, a bankruptcy a chapter, a breach an attacker) — so each gets its own canonical event table.
- Two are shared with other verticals:
commercial_relationships_v2 (the edge/moat table argos traverses) and bankruptcy_events_v2 (a union of credit's SEC lane + mars's news lane + our signals lane).
- One genuinely shared table: the processing ledger
signals_processing_v2 — it records what got processed per (doc, category), the spine every extractor checks so nothing is done twice.
- The binding: every event table carries a
company_id FK to companies_v2. Ten tables, one entity graph — "everything that happened to Company X" (a round, a lawsuit, an auditor swap, a breach) is one join away.
Entity resolution — two timings
| Model | When company_id is resolved | Rate |
|---|
| Deal (staging → promote) | Deferred to a separate resolve stage (link_deal_entities), before the gated promote to canonical funding/merger_deals_v2. | 70% |
| All new extractors | Resolved inline at extract via mars.submit_person / submit_company — they write straight to their final table, no promote step. | ~100% |
Both call the same mint gate; the difference is only timing. People are minted through the gate too — with anchors (employer, role, IDs) — so a bare name never over-merges.
What holds it together
Extraction is a query, not a gate. The corpus is permanent, so any article can be re-asked a new question — the same doc yields a funding round today and an appointment tomorrow. That's why the classify residual can discover new categories (bankruptcy and data_breach both surfaced this way) and we simply promote them: add a category, add a ~150-line extractor, done.
Current state: 22.8M-article corpus · 11,310 classified (growing) · gate + classify + branch live · 12 of 15 categories with working extractors · all by-hand, no timers yet. Cost: gate ~$0 (CPU), classify ~$0.00003/doc, extraction ~$0.0003/doc. Next: scale classify (the volume lever), then automation.