Services + cost update — bios pipeline

2 August 2026 · what we turned off, what we switched, what we added, and what it bought

Short version: we retired more than we added. Net infrastructure spend is down substantially, and output is up sharply. Detail below so you can see the reasoning, not just the numbers.

1. Retired — the old spider

We shut down mars_spider (all four units) after measuring what it was actually producing. It was fetching ~21 GB/month through the Bright Data residential proxy plus 10–15K Grok calls, for a 5.9% fetch success rate — and in its last seven days the only thing it wrote to the database was error timestamps. Its eligibility queue had been empty in both databases for some time.

Deal ingestion was never dependent on it — funding and M&A deals come from the news feed, which is untouched and still running (2,559 funding + 438 merger deals in the last week). Fully reversible if we ever want it back.

Bright Data proxy + Grok extraction

−$250 to −$650 / mo

2. Switched — Brave → Serper for search

Team-page discovery needs one search query per company. We A/B tested both providers on the same companies with the same extractor, and measured people found rather than URLs returned. Quality came out equivalent; price did not.

Provider	Per query	Per full pass
Brave	$0.004	~$101
Serper	$0.001	~$25

Worth flagging honestly: our first test showed Serper finding 41% more people. That turned out to be an artifact of a bug on our side, not a real difference. We fixed the bug and re-ran before switching — the decision is a pure cost saving, which is what we wanted to be sure of.

3. Added — Firecrawl, with guardrails

The one thing we genuinely could not do ourselves: roughly a thousand company sites actively block our crawler (HTTP 403) or render their team page in a way plain fetching can't read. Firecrawl handles those. Before spending anything we did three things:

Removed the dead weight first. We probed the whole backlog and retired 1,824 dead sites — domains that no tool can reach. We were not going to pay a service to fail on them.
Targeted only the pages we can't do ourselves. Free path first; Firecrawl only where it earns its keep.
Hard spend caps per run, and we record every failure so we never pay twice for the same dead end.

First production run
Companies processed	2,500
Yielded people	1,182 (47%)
People recovered	14,490
Cost	$20.00
Cost per person	$0.0014

4. What it bought

The measure that matters is how many companies with funding or M&A activity we can actually show people for — the ones anyone is likely to look up.

	Yesterday	Today
Companies with deals that have people	55,552	59,202
Coverage	58.9%	62.8%
Bios added today	—	44,173

Graph totals: 760,431 bios across 1.94M people. We also found and fixed a bug that had been silently discarding the largest team pages entirely — one firm went from 0 people to 77 — so recent yield is not just more crawling, it's crawling that no longer throws work away.

Net position. We turned off $250–650/month of proxy and extraction spend, cut search costs roughly 4×, and added one paid service running at about a tenth of a cent per person recovered. Total ongoing spend is materially lower than a week ago, and we added 3,650 deal-companies worth of people coverage today alone.

Happy to go deeper on any of it. The general approach we're applying: retire anything that isn't earning its keep, buy capability only where we've measured that we can't do it ourselves cheaper, and cap spend per run so nothing can run away.

— Claude (bios), for Kee