The pricewatch.io crawler
If you found this URL in your server logs, this page explains what the request was, what it collects, and how to stop it. Most of what pricewatch publishes never comes from crawling at all: it comes from official open-data feeds published for reuse.
The exact User-Agent
Every request the fetcher makes carries this header, unmodified. If a request claims to be pricewatch but does not match this string exactly, it is not us.
pricewatch.io/1.0 (+https://pricewatch.io/bot)Click the block to select all of it.
What it does
- Fetches published price data from pages that are open to any visitor.
- Identifies itself on every request with the User-Agent above.
- Reads and obeys robots.txt, including crawl-delay, before requesting anything else.
- Rate-limits per host, with one request in flight at a time and a gap between requests.
- Backs off on 429 and 503 rather than retrying hard, honouring Retry-After and widening the interval on repeats.
What it does not do
- No login-walled content. It has no accounts and holds no credentials for any site.
- No personal data. It records prices, sellers, places and timestamps, nothing about your users.
- No attempt to bypass any access control: no paywall circumvention, no CAPTCHA solving, no rotating IPs or spoofed user agents.
- No form submissions, no checkout flows, no writes of any kind. It issues GET requests only.
How to block it
Add this to your robots.txt. The crawler re-reads robots.txt before each run, so a block takes effect on the next visit, and no appeal or confirmation from us is needed.
User-agent: pricewatch.io Disallow: /Click the block to select all of it.
Blocking by robots.txt is enough. Blocking by User-Agent at your edge also works and we will not route around it.
Reaching a human
If the crawler is causing you load, has picked up something it should not have, or you simply want it gone without touching robots.txt, write to us and a person will read it. We answer every message within two working days, and we will stop fetching a host on request without asking why.
Most data is not crawled
The bulk of what pricewatch holds arrives as bulk open-data feeds published by the source itself under a licence that permits reuse. French fuel prices, for example, come from prix-carburants.gouv.fr under Licence Ouverte 2.0, one scheduled download rather than thousands of page requests. Crawling is the fallback for sources with no feed, and every record keeps its source and licence attached.