The extractr crawler

You probably came here from a user agent in your server logs. This page says what extractr fetches, how slowly, and how to make it stop.

The user agent is:

extractr/<version> (+https://bot.haypin.net; crawler@haypin.net)

What it does

extractr builds a database of the specifications that manufacturers declare for their products, for example a TV's screen size or a camera's weight. It reads public product pages, spec sheets, manuals and sitemaps. Each value keeps a link to the page it came from.

It does not collect prices or reviews.

How it fetches

• Slowly. At most one request every 30 seconds to each host, or slower if your robots.txt sets a longer Crawl-delay.
• Rarely. Most pages are read again only once a week. It sends If-None-Match and If-Modified-Since, so a page that did not change costs you a 304.
• It backs off. On a 429 it follows Retry-After, or waits 1, 4 and then 16 minutes, and it halves its rate on your host for the rest of that run.
• Always with this user agent. Some pages need JavaScript. Those load in a headless browser that sends the same user agent and asks only for what robots.txt allows.
• From one server in Frankfurt, Germany. It does not use proxies or change its IP address.

What it never does

• It never tries to get past bot protection. A CAPTCHA or a challenge page (such as "Just a moment…") makes it stop fetching from that site.
• It never logs in, makes accounts, or accepts terms to get to a page.
• It never publishes your text. Copies stay private and are deleted when they are no longer needed. Only the values and a link to your page are used.

How to make it stop

With robots.txt. extractr obeys the token extractr, and * if no group names it. It reads robots.txt again at least once a day.

User-agent: extractr
Disallow: /

With a text and data mining opt-out. extractr does not keep copies of content that any of these reserve:

• Content-Usage: tdm=n in robots.txt or as a response header
• TDMRep: tdm-reservation: 1 as a header, or in /.well-known/tdmrep.json
• X-Robots-Tag: noai or noimageai

By email. Write to the address below and name your domain. We stop fetching from it and do not start again without your permission. You do not need to give a reason.

Contact

crawler@haypin.net

For questions, problems with the crawler, or to give permission for a site that it does not fetch now.