The 5% floor: why we reject scam-priced rows
Why a $17.99 ASUS laptop never makes it into your alerts, even though the scraper said so.
Three weeks into running retail discovery at scale, we noticed a class of bug that kept showing up in user reports: 'Why is this $17.99 ASUS laptop firing? It's a parser error, not a deal.'
The root cause was almost always the same. A SerpAPI google_shopping result picked up an accessory or replacement-part cell as the primary price. The carousel for a $1199 laptop has a sponsored row for a $17.99 charger. The parser took the first numeric value and shipped it. The detector saw a 98%-off drop and auto-fired. Users complained.
The fix
Every scraped price is gated by a parse-guard: if `list_price_usd` is set and the scraped value is below 5% of it, we reject. We log it as a parser error on the target row (not a real failure — no auto-pause), then move on. Real penny glitches on Home Depot don't have a list_price set, so they're unaffected. Real deep-discount deals (60%+ off) clear the floor by a wide margin.
Why 5%
A genuine retail glitch is in the 70–99%-off range. A 60%-off floor is unusually deep but plausible. A scraper picking up an accessory is almost always at < 5% of the list price. The guard sits exactly where the histogram of legitimate deals ends and where the histogram of scraper errors begins.
Audit trail
Each rejection writes a `last_error: parse_guard: rejected $X.XX (< 5% of list $Y.YY)` to the target row. If you ever wonder why a known-good deal didn't fire, that field tells you. (And you can also lower the list_price temporarily, let the cron run, and restore.)