JobAwareBot

JobAwareBot is the crawler behind Attention Head, Portrait, Left Join and Level Select. It reads employers’ public job boards so people can find jobs at the source, and every job it finds links back to the employer to apply.

How to recognise it

Its requests carry this user agent:

JobAwareBot/0.1 (+https://jobaware.ai/bot)

In robots.txt it answers to the product token JobAwareBot.

What it reads

  • The public job boards and careers pages of the 12,601 employers in our registry: job lists and individual postings.
  • Each site’s robots.txt.

It doesn’t sign in, submit forms or apply to jobs, and it reads nothing behind a login.

How it limits itself

  • robots.txt. It follows the rules for JobAwareBot, or for * when there are none for it. It keeps a site’s rules for up to 24 hours. If robots.txt answers with a server error or can’t be reached, it reads nothing from that site until it can.
  • One site at a time, gently. It sends at most two requests at a time to an employer’s own site. Shared job board platforms get a small fixed budget, spread across every employer on them.
  • On a schedule. It reads each board on its own schedule, the median board every 8 hours, and tries a failed request at most three times.
  • Pages that need JavaScript are opened in a headless browser that identifies itself the same way and follows the same rules.

How to block or limit it

To stop it reading your site, add this to your robots.txt:

User-agent: JobAwareBot
Disallow: /

To have your company removed from our sites altogether, or to ask about anything else the crawler does, contact us. We act on removal requests within 48 hours. Email [email protected].