We been experiencing an invasion of bots going through our servers and client websites.

Bots – are not customers, but they slow down servers and make websites slower for human visitors.

Do you want your competitors updating their website using your content? That is potentially happening with AI crawler bots.

Some bot traffic is beneficial; Search Engine bots, like those operated by Google.

Fortunately, with most of our clients are using Cloudflare DNS and we have the ability to manage bot traffic going to individual websites.

Bot Categories that we can block include;

SEObots – like those from Ahrefs and SEMRush that continually scrape your site for competitors.

Amazonbots; looking for pricing to make Amazon more competitive.

For most clients we recommend allowing Search bots and Ad bots. All other bot categories can most likely be blocked.

Bots categories we allow:

• Search Engine Crawler (block Yandex – Russian Search Engine)
• Monitoring & Analytics
• Advertising & Marketing
• Security
• Webhooks
• Page Preview
• Academic Research
• Aggregator (this one is questionable)

Bots & Categores that we recommend blocking:

• Amazon Bot – unless you are selling on Amazon
• Yandex Bot – unless you do business in Russia
• AI Crawler Bots – examples ChatGPT, Bard, Claude, etc (huge source of server traffic – but no to minimal benefit to clients)
• SEO Bots – unless you are paying for services from SEMRush, Ahrefs, etc. (2nd highest traffic source)
• Python Bots – typically scrapper bots used by developers/hackers (potentially dangerous traffic)
• Tor Traffic – visitors arriving from the ‘dark web’,

We can also block or challenge traffic from Countries/Continents that are not within your market area.

A ‘challenge’ is similar to a captcha that allows humans through, but stops all bot traffic.

A ‘block’ stops all traffic from reaching your server and website.

To see how we’re managing bots through Cloudflare, schedule a Zoom call with Allen or send an email.