Most "we need a scraper" requests are actually an integration or a paid data feed in disguise. Here is how to tell which bucket you are in before you build.
Software Development Solutions
Looking for a software development partner?
We build domain-led systems tailored to your industry and workflow. 12 years. 2,100+ engagements.
Every custom software team gets the same question at some point. "Can you build us a scraper for this site?" It sounds simple. It almost never is. The reason it almost never is has nothing to do with the site being hard to read; it has to do with the fact that half the people asking for a scraper do not actually need one. They need a real integration, or a paid data feed, or someone to tell them the data they want has been available through an API for years and they were not looking in the right place. Only the last group actually needs the thing they think they are asking for.
So how do you tell which one you are? That is the point of this piece. You will see what web scraping actually is in plain English, the 3 buckets almost every "we need a scraper" request falls into, the 3 things that kill every scraper (usually within a few site releases), the 4 design choices that decide whether your scraper is a quiet worker or a nightly firefight, a simple shape for building one that does not wake up your team at 2 AM, and 3 signs it is time to stop scraping and buy the data feed instead. All of it is written for the buyer, not the scraper engineer, because the buyer is the person who has to make the call.
Why does this matter more now than it did a few years back? Because AI agents, price-intelligence tools, market research platforms, and lead-generation pipelines all need clean, current data from sites they do not own. The buyer economy for that data has grown fast. The people selling scraping as a service have grown faster. And the honest conversation about what scraping is actually good for has not caught up, which is why so many scraping projects end up abandoned inside a year of shipping.
3
Buckets almost every "we need a scraper" request actually falls into. Only one of them is real scraping.
4
Design choices that decide whether your scraper is a quiet worker or a nightly firefight for your team.
3
Kill switches nobody warns you about: site redesigns, bot-detection escalation, and legal-contract creep.
1
Normalizer layer that must always sit between the raw scraper output and your product database.
The rest of the piece walks the answer in the order the questions show up in a real conversation with a buyer. What is scraping? Do you actually need it? What kills it? How do you build a version that survives? When do you stop and buy the feed instead? None of it is exotic; most of it is boring on purpose, because boring is the version that lasts.
What Is Web Scraping, Actually?
Web scraping is a program that visits a website, reads the page the same way your browser would, and pulls out the pieces of information you care about. If you have ever copied a table off a webpage and pasted it into a spreadsheet, you have done web scraping by hand. A scraper does the same job automatically, on a schedule, at scale. That is the whole idea. Everything else is detail.
Why do people build them? Because the data they want is visible on the site, but the site does not offer an easy way to download it. You can see the product prices, but you cannot export them. You can see the job listings, but you cannot get a spreadsheet of them. You can see the reviews, but nothing lets you pull the numbers into your own dashboard. A scraper closes that gap. It turns "I can look at it" into "I can use it in my own system."
What is the difference between a scraper and an API? An API is the front door the site owner opened for you: they invite you in, they hand you the data in a clean format, they tell you the rules. A scraper is you reading the site from the outside without being invited. The site owner may not mind. They may mind a lot. They may change the door tomorrow and you will not know until your scraper stops working. That difference (invited vs uninvited, promised format vs guessed format, stable rules vs whatever changes overnight) is the reason scraping and integration are not the same tool, even when they can pull the same data.
The First Question to Ask
If the site you are trying to read publishes an API or a data export, you are not doing scraping. You are doing integration, and it will be cheaper, cleaner, and more durable than any scraper you could build. Ask that question first, every time. Most buyers who assumed they needed a scraper find out they did not, once they check.
Which Bucket Is Your Scraper Request Actually In?
Which of the 3 buckets does your request fall into? Almost every "can you build us a scraper" conversation resolves into one of these once you push on it a little. Knowing which one you are in changes everything: the cost, the risk, the time to a working version, and whether the thing you build will still be running a year from now.
3 Buckets
What "We Need a Scraper" Almost Always Turns Out to Mean
Bucket 1
You Actually Need a Real Integration
The site publishes an API, a data export, or a webhook. You did not know because nobody told you. Cost is lower, durability is higher, legal risk is close to zero. Almost half of scraper requests belong here. The right move is to spend an hour reading the target site's developer docs before you write a single line of scraper code.
Bucket 2
You Actually Need a Licensed Data Feed
The data exists inside a commercial provider (a market intel service, a listing aggregator, a public-records vendor). You can buy the feed clean, current, and legal. Roughly a quarter of scraper requests belong here. Building a scraper for data somebody else already sells for a fair price is almost always the more expensive path once you count maintenance.
Bucket 3
You Actually Do Need a Scraper
No API, no data feed, and the data is genuinely only available on the public page. This is real scraping territory. It is a smaller share of requests than most buyers assume, but the work here is durable if the design is right. This is the bucket where every choice in the rest of this piece starts to matter.
How to Sort Yourself
Read the target site's developer, partner, or data pages. Search for a paid feed that already covers the source. If both come back empty, you are in bucket 3, and you are building for real.
Why does the sorting matter so much before you write any code? Because a scraper you build for data that is already available through a real API is a scraper you will maintain forever for no good reason. The API version keeps working when the site redesigns. The scraper version does not. You will spend more of your team's attention on rebuilding the scraper every few site releases than you would have spent doing the integration once. That trade shows up in every scraping project that started life as "quick and cheap."
3 Things That Kill Every Scraper (And Why Nobody Warned You)
What actually goes wrong with scrapers over time? The 3 things below show up in almost every scraping project that started clean and slowly turned into a maintenance headache. None of them are avoidable; all of them are survivable if the scraper is built with them in mind from day one. The projects that skip these usually get quiet for a while, then all 3 arrive within a short window, and the team suddenly finds itself doing scraper repair instead of the work the scraper was meant to enable.
01
The Site Redesigns and Every Selector Breaks
Scrapers read the page by pointing at specific spots in the page markup: this class name for the price, that class name for the product title. When the site owner redesigns, those markers move or vanish. Your scraper does not throw an angry error; it quietly starts collecting the wrong field, or nothing at all, and the bad data flows into your dashboard until somebody notices the numbers look off. What can you do about it? Alert on unexpected shape changes and layout drift, so the scraper tells you when the site moved instead of pretending nothing happened.
02
Bot Detection Gets Aggressive
The site notices you are not a human. It starts serving captchas, or IP blocks, or hidden fields it knows only a real browser would fill in. What used to work with a plain request now needs a full browser simulation, then a proxy rotation, then residential proxies, then something more expensive again. Each escalation costs more to run and takes more of your team's attention. The scrape that ran cheaply for a while stops being cheap the moment the site invests in defending itself.
03
The Legal Line Moves While You Were Not Looking
A site's terms of service quietly get updated, a new regulation lands in the jurisdiction where the site is hosted, or the data you are collecting starts being treated as personal data under a rule that did not apply before. What was ambiguous becomes clearly off-limits. The scraper keeps working technically; the business case for keeping it running falls apart legally. Review the target's terms of service, the applicable data-protection rules, and any court rulings in the relevant jurisdiction on a regular cycle. Waiting to be told is expensive.
Why This Feels Sudden
None of these 3 arrive with a warning email. All 3 arrive as a broken dashboard, an unexpectedly large infrastructure bill, or a lawyer's letter, and by the time your team notices, the scraper has been serving bad data or costing too much for weeks. Building for them from day one is the difference between a scraper that quietly keeps working and one that becomes a running story on your engineering channel.
4 Design Choices That Decide Whether Your Scraper Survives
If you have sorted yourself into bucket 3 and you actually need to build a scraper, what separates one that lasts from one that becomes a maintenance treadmill? The 4 choices below are the ones that keep showing up as the difference. Everything else (which language, which framework, which cloud) is second-order. Get these 4 right and the scraper stays a quiet worker. Get them wrong and every site update turns into a fire drill.
01
Look for the Underlying Data Endpoint Before You Parse the Page
Most modern sites fetch their data from a JSON endpoint their own frontend calls; the HTML you see is assembled on top of that. Reading the JSON is far more stable than reading the HTML because the JSON shape does not change every time the design team ships a new layout. Open the network tab in a browser, find the request the page itself is making, and see if you can call it directly. If you can, your scraper just became a lightweight API client instead of a page parser, and its lifespan roughly triples.
02
Rotate Proxies With Discipline, Not With Panic
A single-IP scraper is a scraper waiting to get blocked. Rotate through a pool of residential or mobile proxies, throttle requests per IP, respect a sensible request cadence, and never hammer a site harder than a real user would browse it. The goal is not to hide that you are a bot; the goal is to be a well-behaved bot the site owner has no reason to escalate against. Aggressive scrapers get treated aggressively. Polite scrapers usually get left alone.
03
Detect Shape Changes Before Bad Data Ships Downstream
Every time the scraper runs, check whether the response looks like it did last time. Same fields? Same types? Similar row count? If anything drifts outside a sensible tolerance, halt the pipeline and alert instead of writing the bad rows into your product database. This one guard is the single biggest reason scrapers survive site redesigns; the ones without it usually pollute downstream data quietly for a while before somebody spots the problem.
04
Never Let the Scraper Touch Your Product Database Directly
Raw scraper output lands in a staging store. A normalizer reads that staging store, validates the shape, cleans the fields, and only then writes to your product database. Two things fall out of this. First, a bad scraping run cannot corrupt your production data. Second, when the site changes and the scraper starts producing junk, the normalizer notices and refuses to promote it, so the downstream product keeps behaving even while the scraper is being fixed. This is the layer that turns scraping from a fragile hack into a maintainable pipeline.
The Order Matters
Do these 4 in the order listed. The endpoint check saves you the most work, the proxy discipline decides whether the scraper ever runs, the shape detection decides whether it stays honest, and the normalizer decides whether a scraper failure ever becomes a customer-visible outage. Teams that skip the last one usually re-learn why every scraper design guide recommends it.
A Simple Shape for a Scraper That Does Not Wake You Up at 2 AM
So what does a scraper that lasts actually look like from the outside? Not fancy. The shape below is the arrangement that lets a scraper collect data, normalize it, land it cleanly, and tell your team when something is off, all without the scraper ever touching a piece of your product's live systems. Every layer has one job. The layers do not overlap. When something breaks, you know exactly which layer to look at.
Architecture
A Shape That Keeps a Scraper From Becoming Your Team's 2 AM Problem
Layer 1
Collector
Fetches the raw pages or JSON. Proxy pool, retry logic, polite request rate.
Layer 2
Staging Store
Raw responses land here first. Nothing downstream reads them until they pass validation.
Layer 3
Shape Check
Compares this run against the last one. If anything drifts, alert and halt promotion.
Layer 4
Normalizer
Cleans, types, and shapes the data into your product's schema. Rejects anything malformed.
Layer 5
Product Database
Reads only from the normalizer. Never touched by the collector directly.
↓
Cross-Cutting
Observability and Alerting Across Every Layer
Run Health
Success rate, request latency, proxy failure rate per run.
Shape Drift
Alerts when field counts, types, or row counts move outside tolerance.
Legal Watch
Automated reminder to review target terms and rules on a fixed cycle.
Why This Shape Survives
Every failure has a bounded blast radius. A bad collector run does not corrupt the staging store. A bad staging batch does not reach the normalizer. A bad normalizer promotion does not land in the product database. The scraper can break and your product keeps behaving, which is the point of the whole arrangement.
Why go to this trouble for something that "just reads a webpage"? Because scrapers fail differently from other software: they usually keep running while producing bad data, instead of throwing an obvious error. The staging store plus the shape check plus the normalizer is the arrangement that catches that failure mode before it reaches anywhere it can do harm. Teams that skip this shape usually skip it once, ship a scraping pipeline, and then quietly rebuild it after the first time the scraper served weeks of wrong prices into a dashboard nobody double-checked.
3 Signs You Should Stop Scraping and Buy the Data Feed Instead
When is it time to walk away from a scraper and pay for a proper data feed instead? The 3 signals below are the ones that keep showing up. If you see more than one at a time, the buy-the-feed math has probably already tipped, even if it does not feel like it yet.
01
The Scraper Is Breaking Faster Than Your Team Can Fix It
Every site release breaks selectors, every proxy provider change breaks the pipeline, every anti-bot upgrade forces another round of investment. You are spending more of your team's attention on scraper repair than on the product the scraper was meant to feed. That is the moment the buy-versus-build math flips. A licensed feed you never have to maintain is almost always cheaper than a scraper you have to babysit at that cadence.
02
The Site Owner Has Started Actively Defending the Data
Captchas, IP bans, legal notices, sudden new sign-in walls that were not there last quarter. The site is telling you, without saying it, that scrapers are not welcome. Escalating the fight is expensive and the site owner has home-field advantage. If a licensed feed exists for the same data, this is the moment to switch. If no feed exists yet, this is often the moment to talk to the site owner about a paid partnership instead.
03
The Data Has Started Feeding a Customer-Facing Product
Scraping is one thing when the data ends up in an internal report nobody outside the team sees. It is a very different thing when the data ends up in a dashboard your customers are paying for. The reliability bar goes up, the legal exposure goes up, the compliance conversations start. At that point a licensed feed is not just cheaper in the long run; it is usually the only version that meets the reliability and legal bar your customers now expect.
The Honest Test
If a paid feed exists for the same data at a fair price and any one of the 3 signals above is showing up, buy the feed. The scraper you would keep maintaining costs more than the feed price after you count engineering time, infrastructure, and the risk of a bad run reaching your customers.
Frequently Asked Questions
Is web scraping legal?
It depends on the site, the data, and the jurisdiction. Scraping publicly visible information is generally allowed in many jurisdictions, especially when it does not involve bypassing a login or a paywall. Scraping personal data, copyrighted content, or anything protected by an active terms-of-service prohibition is much riskier and often clearly off-limits. The safe rule is: read the target site's terms of service, check the data-protection rules in the jurisdiction where the site lives and where you operate, and when in doubt, talk to a lawyer before you turn the scraper on. The legal cost of asking is a fraction of the cost of getting it wrong.
Should you build the scraper in-house or hire a specialist team?
Depends on whether scraping is the product or just a means to it. If the data feed is central to what your product does, and you expect to scrape multiple sources for the long term, building the muscle in-house makes sense. If the scraper is a one-off supply line to a product that does something else with the data, a specialist team usually delivers a more durable pipeline and hands it back for your team to run. The wrong choice in either direction ends up expensive; the right one comes down to how central the data collection is to the product itself.
What happens when the target site uses a lot of JavaScript?
You have two options. The first is to look for the underlying JSON endpoint the JavaScript is calling, which is usually available in the browser's network tab and far more stable than the rendered page. The second is to run a headless browser (a real browser without a visible window) so the JavaScript executes and the scraper reads the finished DOM. Option one is cheaper, faster, and more durable when it works. Option two is the fallback when the site actively resists direct calls to its JSON. Always try option one first.
How do you decide between residential, datacenter, and mobile proxies?
Datacenter proxies are cheapest and easiest to detect; use them when the target site does light or no bot detection. Residential proxies look like normal home internet connections; use them when the site has moderate detection and datacenter IPs get blocked. Mobile proxies are the most expensive and the hardest to block; reserve them for sites that actively resist scraping and where the data value justifies the cost. The wrong choice is to start with the most expensive tier out of habit. Start light, escalate only when the response tells you to.
How often should the scraper actually run?
As infrequently as your product allows. Every additional run is another chance to get blocked, another opportunity to trip anti-bot defenses, and another slice of infrastructure cost. If your product refreshes prices once a day, do not scrape every hour. If the target data changes rarely, poll only when a reason exists to expect change. Scrapers that run at the lowest cadence the product needs almost always outlast the ones that hammer sources continuously.
What is the difference between scraping and using an official API?
An official API is a promise. The site owner has agreed to expose the data, agreed to a stable format, and usually agreed to a contract you can enforce. A scraper is a workaround. You are reading the site from the outside without any of those promises. The API costs less to maintain, is legally safer, and does not break every time the design team ships a new version. When an API exists for the data you need, use it, even if the scraper looks faster to build up front. The maintenance savings pay for the extra setup within the first year of running.
Can Entexis build a maintainable scraping pipeline for your product?
Yes. Entexis designs and builds scraping pipelines for products where the data collection is central to what the product does. That work starts with the sorting conversation (are you actually in bucket 3, or is there a better path we should take instead), then moves into the design choices covered above (endpoint discovery, proxy discipline, shape detection, the staging-plus-normalizer layer), and delivers a pipeline your team can run without daily firefighting. We also help with the legal review, the observability wiring, and the licensed-feed comparison when that turns out to be the cheaper answer. Reach out with what data you need, from which source, and what your product does with it, and we can walk through what a durable version looks like for your specific case.
So where does that leave you? Most "we need a scraper" conversations end with a different answer than the one the buyer walked in with. Sometimes the answer is a real integration. Sometimes it is a paid data feed. Sometimes it really is a scraper, and when it is, the choices above (endpoint discovery, polite proxy discipline, shape detection, and the staging-plus-normalizer layer) are what decide whether the scraper stays a quiet worker or turns into a running story on your engineering channel. The honest conversation up front is worth more than any tooling choice downstream. Want to see how a site reads to an automated caller before you decide whether to scrape it or integrate with it? The AI Search Readiness Checker on entexis.tech scores a public site the way an automated reader would parse it, which is a fast way to feel out how friendly the target is before you invest.
Want a Scraping Pipeline That Does Not Break Every Few Site Releases?
At Entexis, we design and build data collection pipelines for products where the data feed is core to what the product does. We start with the sorting conversation to make sure scraping is actually the right answer, review the legal position for your specific target and jurisdiction, design the collector plus staging plus shape-check plus normalizer plus product-database layers, wire the observability so your team hears about a problem before your customers do, and hand you a pipeline that behaves the way the rest of your product does. Your dashboards stay honest, your team stops firefighting, and your data feed becomes a reliable part of the product instead of a fragile one. Start the conversation with Entexis.
Need Custom Software Built?
From web apps to enterprise platforms, we build software that fits your workflow, not the other way around. Tell us what you need.
We'll get back within one business day.
Thank You!
We've received your message and will get back to you within one business day.
