A Scraper That Heals Itself: Lessons From a One-Day Build
On Saturday, August 22, 2026, I sat down in a room in San Francisco with a laptop, three sponsor accounts and an idea I wasn’t sure would work in one day. By the end of the day, Scraper Factory had won the solo grand prize, for Best Full-Stack Integration, at the Zero Downtime Hackathon, run by WeMakeDevs with Port, Bright Data and SigNoz.
The project builds a weekly top-20 list of upcoming TV shows and movies from scraped guide data. The list isn’t the interesting part. The interesting part is what happens when the scrapers break, and lessons from that apply to any production data pipeline.
The problem: sources break quietly
Every scraper I’ve built eventually fails. A site ships a redesign, a selector stops matching, and the data either stops or, worse, keeps arriving slightly wrong. Titles swap with descriptions, times land in the wrong field, rows go missing. Nothing errors. You find out a week later when a downstream number looks strange.
That isn’t unique to scraping. Any pipeline that reads from a source it doesn’t control has the same failure mode: partner feeds, vendor APIs, CSV exports from another team. The source changes shape and the pipeline keeps running.
The design: detect, adapt, verify
I gave myself one rule for the day: let the models make the judgment calls, and keep the proof in plain code.
Detect without a model. Every run passes through seven deterministic checks: rows per page within a band, the null rate of required fields, the share of empty pages, the share of channels that returned anything, a hash of the returned field names, changes in the channel lineup, and one check that turned out to be the most useful. Next to every parsed value, the scraper keeps the raw string exactly as displayed. When the raw time says “8:00 PM” but the parsed time is empty, something broke.
Adapt with a model. The scrapers are built from a description of what the data means (“the program title, the start time exactly as displayed”) instead of where it sits on the page. When drift is detected, a repair request goes back with that description and a plain-language report of the symptom. The response is a proposed fix plus a preview of what the fixed scraper would extract. Nothing changes yet.
Verify without a model. The preview is compared against a frozen fixture: a real page trimmed to 30 entries, with hand-checked expected output. If they match, the repair is approved. If not, it’s rejected, the scraper is marked broken, and a person takes over.
The fixture also has a mutated copy, with class names changed, entries reordered and elements re-nested, so I could break the scraper on purpose and watch the loop work.
One system, not three demos
The grand prize criteria asked for real integration, and that pushed the design in a good direction. Each tool did one job.
Port held the state. Scrapers, runs and repair attempts were entities with properties, so “health went from healthy to drifting” was an event that could trigger the repair. An AI agent in a Port workflow wrote the symptom report and nothing else. It wasn’t allowed to approve its own fix.
Bright Data did the extraction and proposed repairs.
SigNoz, which is open source and ran self-hosted, held the audit trail. The service ran under OpenTelemetry auto-instrumentation, with custom spans for fetch, extract, validate, repair and verify. A SigNoz alert on the drift metric called the same repair endpoint as a backup to the inline detector, and a second alert fired when data stopped arriving at all.
What won: the unhappy path
The demo that mattered was a failure. I injected a fault, the workflow fired, the agent described the symptom, and a real repair came back with a preview of 2 rows where the fixture expected 30. The verifier rejected it, the repair was recorded as rolled back, and the product kept showing the last good data. The trace showed every step, down to the log line “row count 2, expected 30.”
If a model had been allowed to approve its own repair, a nearly empty guide would have shipped with no alarm.
What I’d productionize, and what was a shortcut
Keep:
Deterministic detection and a deterministic verifier gate.
Raw values stored next to parsed ones.
An idempotent repair endpoint, because two trigger paths will race.
An alert for silence, not just for errors.
Setup in idempotent scripts, so the environment can be rebuilt from nothing.
Hackathon shortcuts:
Tunnels that exposed local fixtures and webhooks to cloud services.
One fixture per source. Production needs a set that grows every time a real break is found.
A background poller that re-emitted metrics the tools didn’t push themselves.
A single person approving every rejected repair. At scale that’s a queue, and queues need owners.
The weekend’s main lesson was simple. Automatic repair is only safe when something that isn’t the repairer checks the result. That holds whether the repairer is a model, a script or a tired engineer at 2 AM.
The post A Scraper That Heals Itself: Lessons From a One-Day Build appeared first on SD Times.
Tech Developers
No comments