The bot landscape is shifting. Harmful—or “bad”—bots are becoming more sophisticated and human-like in their behavior, while a rising class of AI-driven “gray bots” blurs the line between legitimate and questionable activity.
Barracuda’s security researchers analyzed bot-related traffic and activity targeting web applications and APIs from September 2023 through the end of August 2024. Key findings include:
- Bad bots made up 24% of internet traffic in 2024, down from 39% in 2021.
- The number of distinct bad bot clients has increased, now representing 44% of detected clients versus 36% a year earlier.
- 49% of bots were classified as “advanced bots,” most of which are malicious and designed to mimic human behavior and handle complex online interactions, such as engaging in account takeover attacks.
Bot Landscape 2024
Bots are automated programs built to perform online activities at scale. Good bots include search engine crawlers, SEO bots, and customer support bots that help organizations streamline processes, increase efficiency, improve online visibility, and enhance customer interactions.
Bad bots, on the other hand, are engineered for harmful or malicious online purposes. They can be directed at a variety of targets, including websites, servers, application programming interfaces (APIs), and other endpoints. Bad bots often target e-commerce and login pages to break into accounts, steal personal information, or commit fraud; they exploit website vulnerabilities for access. Bad bot activity can overwhelm targets with traffic, spread spam, skew business analytics, disrupt services for legitimate customers, damage brand reputation, and more.
The Rise of AI “Gray Bots”
Barracuda’s researchers also observed an emerging category of AI-driven bots that can be labeled “gray bots” because they are not overtly malicious but operate in ethically and legally ambiguous ways.
These AI bots are primarily designed to extract or scrape large volumes of data from websites, often to train generative AI models. They can be aggressive in their data collection, removing or copying content without permission and sometimes ignoring embedded robots.txt directives that site publishers use to signal scraping restrictions.
The Changing Dynamics Between Bots and Humans
Traffic Distribution — Bots vs. Humans
From September 2023 through August 2024, good bots accounted for 18% of internet traffic, bad bots accounted for 24%, and human users made up 58%.
The share of bad bot traffic has fallen year over year: in 2023 bad bots made up 30% of internet traffic, down from 39% in 2021.
At first glance this appears to be positive. A closer look shows that although the proportion of bad bot traffic has declined, the number of distinct bad bot clients has risen over the past 12 months. In other words, there’s less volume on the road but many more makes and models of vehicles.
Researchers believe the overall decline in detected bad bot traffic is driven both by heightened awareness of the threat and by reduced demand for mass-automated shopping bots.
More organizations recognize the damage bad bots can inflict on their web applications and are taking steps to detect and block malicious or suspicious bot traffic. This has lowered the success rate for automated bad-bot attacks and made them less attractive to threat actors.
In 2021, bad bot traffic included swarms of shopping bots that targeted e-commerce sites to buy up limited-availability goods for resale at a premium—think the notorious “sneaker bots.” As markets for those products softened during economic downturns, demand for mass shopping bots fell, reducing overall bad-bot volume.
In their place, more advanced and targeted bots have emerged.
Bad Bot Activity in 2024
Researchers also examined the types of bot activity detected between January and the end of August 2024.
Data shows that 2024 bot activity is dominated by “advanced bots,” the majority of which are confirmed malicious:
- Advanced bots: These account for 49% of bot activity, much of it malicious. These bots use sophisticated techniques to mimic human behavior and navigate complex web interactions while evading conventional checks such as traffic rate, error rate, CAPTCHA, and IP-based filters. Examples include account takeover bots that employ “low-and-slow” methods—spreading requests across diverse IPs and regions to stay under detection thresholds. Note that some advanced bot classifications can produce unintentional false positives, where legitimate crawlers or upgraded tools are mistakenly labeled malicious.
- Imitators: A subset of advanced bots designed to impersonate human behavior, often for fraudulent purposes. For example, a bot might spoof a well-known crawler identity to avoid being blocked when scraping content.
- Known offenders: Previously identified clients that have engaged in unwanted or malicious behavior and are blocked by security tools. Known offenders represented 6% of activity during the analyzed period.
- Browser integrity anomalies: Clients showing irregularities in browser configuration that indicate emulation or spoofing. These made up 3% of total activity.
Good Bot Activity Detected
- Crawlers/indexers: Bots used to index web content for search engines.
- Feed fetchers: Bots that retrieve content for feeds, aggregators, or news discovery tools.
- Search engine bots: Bots interacting with site systems to index content for search optimization.
- Social media agents: Automated agents that manage or interact via social platforms.
- Technical partner/commercial bots: Bots run by third-party vendors to integrate services or content.
- Tools: Clients using bots for testing, monitoring, or other operational functions.
Overall, good bots and other non-malicious categories accounted for 42% of internet engagement in 2024.
How to Protect Your Organization
Understanding and managing the threat posed by bad bots is essential to protect the security and integrity of online operations. This includes safeguarding e-commerce sites from price scraping, inventory hoarding, and fraudulent transactions; preventing spam, fake accounts, and disinformation campaigns on social platforms; protecting proprietary data; countering negative SEO tactics that harm search rankings; and ensuring that login attempts are legitimate.
Effective, targeted bot protection detects and defends against automated attacks from malicious bots while allowing known good bots—like search engine crawlers and SEO bots—to access your web application.
Such protection requires a layered approach, including:
- Robust application security. Deploy advanced application protection for web apps and APIs, properly configured with rate limiting and monitoring. This foundational step ensures your application security solution functions as intended.
- Specialized bot protection. Choose an application security solution that includes anti-bot capabilities to effectively detect and stop advanced automated attacks.
- Leverage machine learning. Solutions that use machine learning are better able to detect and block stealthy, nearly human-like bot attacks. Enable login protection to help prevent account takeover attempts as well.
- Don’t forget the basics. Access and authentication controls, including multi-factor authentication, help secure vulnerable entry points like login pages from brute-force and credential attacks.
For guidance on defending your environment against advanced and evolving bot attacks, consult your security provider or vendor documentation to implement the appropriate protections for your applications and APIs.