Web Scraping
What is Web Scraping?
Web scraping is extracting data from websites automatically. It covers a search engine building an index, a price-comparison service reading catalogues, a researcher collecting public records, and a competitor taking your entire product list every morning.
The technique is identical in all four cases. What differs is who benefits, which is why "block scraping" is the wrong goal and "decide which scraping to allow" is the right one.
How it is done
| Approach | What it handles | Cost to the operator |
|---|---|---|
| HTTP client plus parser | Static HTML | Minimal. A few lines of code |
| Headless browser | JavaScript-rendered pages | Moderate CPU per page |
| Patched headless browser | Pages with bot detection | Higher, plus maintenance as detection changes |
| Residential proxy network | Per-address rate limits and geographic restrictions | Charged by the gigabyte, and the main cost |
| Your own API | Anything you expose | Lowest of all, if the endpoint is unprotected |
The last row is the one to notice. An undefended JSON endpoint is easier to scrape than HTML, returns cleaner data, and is often forgotten when the website itself is protected.
The legal position is narrower than people claim
Two things are frequently overstated in both directions.
Scraping publicly accessible data is not inherently unlawful in the UK, EU or US, and a terms-of-service prohibition is a contractual matter rather than a criminal one. At the same time, the data taken can be protected independently: database rights in the EU, copyright in the content, and GDPR obligations if it includes personal data — which it does the moment you collect names, reviews or profiles.
The practical summary: the act of fetching is rarely the exposure; what you then hold and do with it usually is.
What works against unwanted scraping
Ranked by how long each survives contact with a determined operator:
- robots.txt — a statement of preference. Honoured by the crawlers you want, ignored by the ones you do not. Still worth maintaining, because it is what the AI crawlers actually fetch.
- IP rate limits — defeated by residential proxies, and they penalise shared corporate and mobile addresses. Useful as a blunt ceiling only.
- Markup obfuscation — rotating class names and rendering text as images. Costs the scraper an afternoon, costs you accessibility and search visibility permanently. Rarely worth it.
- Behavioural and coherence checks — see bot protection. Automation that fetches only HTML, never requests images or scripts, and follows paths in sequence is identifiable regardless of where it connects from.
- Making it cost something — proof of work per request changes the economics of bulk extraction more reliably than trying to identify the extractor.
Deciding what to allow
Keep an explicit position. Allow the crawlers that send you traffic or cite you, and verify them by IP against the ranges their operators publish rather than by user-agent string — user-agents are trivially forged, and impersonating a well-known crawler is a standard technique. Then defend the endpoints where bulk extraction actually hurts: pricing, inventory, and any API that returns a list.