Scraped SaaS: stopping a copycat from arbitraging a free tier

This customer runs a subscription search product. Search is free to try and paid to use properly, the standard model for this kind of business: the free tier is how people discover whether the product works for them.

That model has a specific failure mode. If somebody automates the free tier at scale, they are not a heavy user. They are a competitor whose cost of goods is your marketing budget.

The problem

A competing service was scraping this customer's results and presenting them as its own product. The scrape ran against the API rather than the web form, as it normally does. There is no reason to automate a form when the endpoint behind it is available.

Subscription founders report this attack more than any other, and it rarely appears as a security incident. It appears as a conversion problem, a bandwidth bill, or a competitor whose results look familiar.

What we did

The first job was separating automated search sessions from real ones. That is where most tools stop. A scraper running one search every few seconds from a residential IP does not look unusual on volume alone.

What separated them was an environment that did not hold together. The client's worker thread and main thread reported different environments, and the processor characteristics it reported did not match the rest of the session. Neither is something a scraper can avoid by slowing down.

Once the pattern was established, the anomaly detectors promoted it into standing Access Rules on the ASN and IP ranges the traffic came from. Those rules block on a cheap comparison at the edge and carry an expiry, so they follow the attacker without accumulating stale entries.

Behavioural collection on the client side did the rest of the work on sessions that came through a browser. Real people searching behave differently from a harvester: they hesitate, they scroll results, they refine.

Why the rules matter as much as the detection

The detectors identify the automation. The rules are what makes stopping it cheap: once a pattern is established, an ASN or IP-range rule rejects matching traffic on a comparison at the edge, rather than running the full scoring path on every request from a source already known to be bad.

When the operator moved infrastructure, the ASN rules followed, because the detectors kept generating them from what they were seeing.

Results

The blocked share is the lowest of the three deployments in these case studies, and that is the correct outcome. A consumer search product is mostly real people. The job is not to block a large fraction of traffic, it is to find the automated fraction inside it without taxing everybody else: of the 7.5 million sessions that were not blocked, 79% were cleared by proof of work and never saw a puzzle.

The bottom half of that list is the part that matters against a copycat. A scraper running slowly from residential addresses will not trip a rate limit, and it will not appear in the top two rows until it has already been caught once. What catches it the first time is the client contradicting itself: network behaviour that does not match the operating system it claims, processor speed that does not match the device it claims, a click pattern produced by a script rather than a hand. None of those depend on how fast the requests arrive.

What we would tell a founder in the same position

Look at your API before you look at your forms. If your product has a free tier and a public endpoint, the endpoint is where the abuse is.

Then ask your current vendor what a block tells you. If the answer is a score, you will be able to see that something was stopped and not what it was, so the next attack is harder to recognise than it needs to be.

Related

• Web scraping protection
• Prosopo Protect: site-wide and API bot protection
• Residential proxy and device farm detection