JobCrawlsBot

How to identify it

Every request we make sends this exact User-Agent string:

Mozilla/5.0 (compatible; JobCrawlsBot/1.0; +https://www.jobcrawls.com/bot)

We do not disguise our crawler as a human visitor, and we do not rotate through residential proxies or spoofed browser fingerprints. If you see traffic claiming to be JobCrawlsBot that behaves differently from what is described on this page, it is not us — please tell us at jobcrawls@gmail.com so we can look into it.

What it collects

JobCrawlsBot reads publicly accessible job listing pages: the role title, description, location, employment type, and salary range where one is published. It uses that to build a searchable index of open positions, and each listing on JobCrawls links back to your own careers site so applications go directly to you.

It does not attempt to log in, does not submit application forms, and does not access anything behind authentication. It does not collect personal data about candidates or employees. Because many career sites reveal listings only after interaction, it may click pagination and “load more” controls, and dismiss cookie banners, in order to see the same listings a visitor would.

How it behaves

• At most one request to a given domain every 10 seconds. We do not run parallel requests against the same site.
• Any individual page is fetched at most once per 24 hours, so repeat visits stay proportional to how often listings actually change.
• We render pages with a headless browser, which means we also request the JavaScript, CSS, and images a normal visitor would.
• We crawl from AWS infrastructure in the EU (eu-central-1), so requests originate from datacenter IP addresses rather than residential ones.

How to slow it down or block it

JobCrawlsBot obeys robots.txt rules addressed to it by name. To stop it visiting your site entirely:

User-agent: JobCrawlsBot
Disallow: /

Please note: a wildcard User-agent: * block will not stop JobCrawlsBot. We treat Disallow rules as binding only when they name JobCrawlsBot specifically, so if you want us gone you have to say so by name using the rule above. This is narrower than what most crawlers do and we would rather state it here than have you assume otherwise. A wildcard Crawl-delay, on the other hand, we do apply.

If you would rather keep your roles listed but see less traffic, ask for a longer gap between requests:

User-agent: JobCrawlsBot
Crawl-delay: 30

We re-read robots.txt at most once an hour, so a change takes effect on the first crawl after that. If you need us to stop immediately, or you want specific listings removed from JobCrawls, email jobcrawls@gmail.com and we will action it by hand — you do not need to justify the request.