About ScrapeMasterBot

What it is

ScrapeMasterBot is the crawler behind ScrapeMaster. It reads public product pages to record prices, so shoppers can see price history and get an alert when a price drops. Its user agent ends with ScrapeMasterBot/1.0 (+https://scrape-master-ai.vercel.app/bot).

How it behaves

• It reads robots.txt before fetching anything and follows it (RFC 9309). Rules for ScrapeMasterBot take priority over rules for all crawlers.
• It waits at least 1.2 seconds between requests to the same site, and most pages are visited at most once every 30 minutes.
• It backs off after errors, and pauses a site for two hours after repeated failures instead of retrying.
• Where a store offers an official data feed (such as the Flipkart Affiliate API), that is used instead of its web pages.
• It reads only public information such as titles, prices and images. It never logs in, adds to carts or places orders, and uses no proxies.

Opting out

To stop ScrapeMasterBot visiting your site, add this to your robots.txt. It takes effect within a day:

User-agent: ScrapeMasterBot
Disallow: /

For anything else, open an issue and we will respond promptly.