Signals Pipeline

From landing zone → corpus → screened tiers → per-category extractors → the canonical deliverables. A living reference. Snapshot 2026-08-14.

① Ingest — two entry paths

Benzinga runs on its OWN continuous loaders (IR needs real-time; can't wait behind rss on a 2h timer). Everything else rides the shared feeder.

Shared corpus feeder

corpus_writer.py · signals-pipeline.timer (2h)

rssgdelt8k sectionssedarus-courts

Benzinga — its own path

decoupled (SIG-100), both live & continuous

benzinga-ingest parquet sweep loop
benzinga-stream raptor DynamoDB stream → direct

▼ both write to ▼

signals.news_corpus_v2 one corpus · every article gets a relevance_tier

② Screen → the tier split

relevance_tier (deterministic, at ingest). Now language-aware — non-English is screened by its own-language deal-shape patterns, not dumped. Only the business tier flows on to extraction.

business

deal-shaped → extractors ↓

low_signal

stored + searchable, not extracted

spam

our own domain rule

stub

truncated body

unscreened
_nonenglish

unsupported language only (shrinking)

③ Extract — the business tier fans out by category

One screened corpus, many extractors. Each reads the business tier and writes its own event table. (Qwen is the precision layer — it rejects non-deals as "neither".)

DEAL

funding + merger · 88K processed

extract_deal.py
→ signals_deal_events_v2 (52,976)

LAWSUIT

litigation · 60K processed

extract_lawsuit* + us-courts dockets
→ lawsuit_events_v2 (5,862) + parties

APPOINTMENT

exec/board · 26K processed

extract_appointment.py + 8-K 5.02
→ appointment_events_v2 (31,664)

roadmap (SIG-84): contract · auditor_change · financial_report · product · dividend · license · gov-issues

④ The deal spine — extract → resolve → promote

The deliverable everyone wants. The DEAL box above expands here. (Lawsuit & appointment follow the same extract→resolve shape into their own tables.)

3EXTRACTTimer OFF

Qwen → structured deal. Writes a raw event layer (company_id NULL) so a bad extract can't mint a bad canonical. Now drains English and multilingual through the one business path — no --lang branch.

writes signals_deal_events_v2 (STAGING)timer signals-deal-ingest disabled

Hole #1: ~98K English + ~25K newly-visible non-English business articles queued, timer not armed.

▼ 52,976 staged · merger 31K · funding 22K ▼

4RESOLVETimer OFF

company_name → company_id via mars.submit_company() — the gate (dedup-safe). Uncertain → leaves NULL rather than guess.

script link_deal_entities.pytimer signals-resolve disabled68–75% resolved

▼

5PROMOTEGated · held

Staging → mars.funding_deals_v2 / merger_deals_v2 — the canonical deliverable. Gate rejects dups (0 rows). Held on purpose until stages 1–4 are trustworthy.

script load_to_canonical.pystatus SIG-41 → SIG-91

Working as intended — the deliverable is deliberately gated behind confidence.

Confidence status — for "sure before delivery"

Stage	Works?	Automated?	Trust gap
① INGEST	yes	✅ shared 2h + benzinga continuous	language-aware now
② SCREEN	yes	✅ at ingest	one path, all languages
③ EXTRACT	yes	❌ timer off	~123K backlog (EN + non-EN)
④ RESOLVE	yes · via gate	❌ timer off	25–32% unresolved
⑤ PROMOTE	yes · gate dedup	🔒 gated	net-new rate + validation (SIG-41→91)

Coverage holes → status

✔ Multilingual blind spot — being closed. The English-only ingest screen is retired; relevance_tier is language-aware. The 640K unscreened_nonenglish backlog is being re-tiered (~3.9% → business ≈ 25K non-English deals, ru/de/ja/ko-led) so they flow through the same extract path.
✗ Stage ③ extraction backlog. ~98K English + ~25K non-English business articles queued but not extracted — the extract timer isn't armed. Cheap to clear (~$40); the blocker is reliability. Next up.

Supporting: verify_deals.py (LLM re-check) · deal_guards.py (deterministic guards) · fx_capture (currency→USD) · signals_processing_v2 (per-article ledger — how each extractor knows what it already touched). Gate: all canonical writes go through mars.submit_* as signals_writer.

Automation reality: ingest runs on timers (shared 2h + benzinga continuous); the EXTRACT and RESOLVE timers are disabled; PROMOTE is gated by design. That's why the corpus grows while deals advance by hand. — signals, with kee.