Bad bots evolve to become more “human”

The bot landscape is changing. Malicious – or bad bots – are evolving to become more advanced and human-like in their behavior, while an emerging category of AI bots, which we can think of as “gray bots,” is blurring the line between legitimate activity.

Barracuda security researchers analyzed bot-related traffic and activity targeting web applications and APIs between September 2023 and the end of August 2024. Among other things, the researchers found that:

• Bad bots will account for 24 % of internet traffic in 2024, down from 39 % in 2021.
• The number of individual bad bots has increased and now comprises 44 % of detected clients, compared to 36 % a year ago.
• 49 % of bots are classified as “advanced bots,” most of which are malicious and designed to mimic human behavior and handle complex online interactions such as engaging with targets in account takeover attacks.

Bot landscape 2024

Bots are automated programs designed to perform online activities on a large scale. Good bots includes search engine bots, SEO bots, and customer service bots that can help organizations streamline processes, increase efficiency, grow their online presence, and strengthen customer interactions.

Advertisement

Bad bots, on the other hand, are designed for malicious or harmful online activities. Bad bots can be deployed against a wide variety of targets, including websites, servers, application programming interfaces (APIs), and other endpoints. Bad bots target e-commerce and login websites, among others, with the aim of breaking into accounts to steal personal information or commit fraud, and they exploit vulnerabilities in websites for access. Bad bots can overload the target with traffic, spread spam, skew business analytics, disrupt services for legitimate customers, damage a company’s reputation, and more.

The rise of AI “gray bots”

Barracuda security researchers also noted an emerging category of AI bots, which can be classified as “gray bots” because they are not overtly malicious, but their approach may be questionable.

These AI bots are primarily designed to extract or scrape large amounts of data from websites, for example to train generative AI modelsThe bots can be aggressive when collecting data and can remove information without permission, possibly ignoring any embedded robots.txt code added by publishers to signal scraper bots not to grab that site's data.

The changing dynamic between bots and humans

Traffic distribution – Bots vs. humans

From September 2023 to the end of August 2024, good bots accounted for 18 % of internet traffic, while bad bots accounted for 24 % and human users 58 %.

The proportion of bad bot traffic is decreasing year on year. In 2023, bad bots accounted for 30 % of internet traffic, down from 39 % in 2021.

On the surface this seems like good news. However, a deeper analysis shows that while the percentage of bad bots has decreased, the percentage of individual bad bots has increased over the past 12 months. In other words, there is less traffic on the road, but many more vehicle brands. 

The researchers believe that the overall decline in detections of bad bot traffic is driven by both increased awareness of the threat and reduced demand for mass-automated shopping bots.

More companies are aware of the damage that bad bots can do to their web applications and are taking steps to detect and block malicious or suspicious bot traffic. This has reduced the success rate of automated bad bot attacks and made them less attractive to attackers. cyber attacker.

In 2021, bad bot traffic included swarms of shopping bots targeting e-commerce websites to obtain affordable consumer goods to resell at a significantly higher price. This included the infamous “sneaker bots” that hunt for limited-edition shoes. As the market for such products collapsed during the economic downturn, demand for mass shopping bots decreased, reducing the volume of bad bot traffic.

In its place, we now have more advanced and targeted bots.

Poor bot activity in 2024

Our security researchers also looked at the types of bot activity detected during the eight months between January and the end of August 2024.

Data shows that bot activity in 2024 is dominated by “advanced bots” and that most of these are confirmed to be malicious:

• Advanced bots : These account for 49% of bot activity, much of it malicious. They malicious bots use sophisticated techniques to mimic human behavior, and they can navigate complex web interactions and bypass standard checks that look at traffic speed, error rates, CAPTCHAs, and IP addresses. Examples include account takeover bots that use multiple methods to perform so-called “low and slow” attacks, which leverage different IP/geographic locations to stay under the radar and avoid detection. It should be noted that some advanced bot classifications may be unintentional false positives, where benign crawlers or other tools that have upgraded their ability to avoid being blocked are inadvertently mislabeled as malicious.
• Imitator : In the case of advanced bots, these bots are designed to imitate human behavior, usually for malicious purposes such as fraud. An example of this would be a bot that attempts to forge GoogleBot to avoid being blocked from scratching.
• Known violator : These are previously identified entities that have engaged in unwanted or malicious activity. For security tools, they represent bots that have been caught previously and are therefore prevented from accessing protected applications. Know-it-alls accounted for 6 % of the activity during the analyzed period.
• Browser privacy anomalies : These are detected clients that have anomalies in their browser configurations that may indicate emulation or spoofing. They accounted for 3% of the total activity.

Good bot activity detected

• Crawler/indexer : These bots are primarily used to index web content for search engines.
• Flow restrictor : Bots that retrieve content for web feeds, aggregators, or news crawlers.
• Search engine bot : Bots that interact with enterprise systems for the purpose of indexing for search engine optimization.
• Social media agent : Automated agents that manage or interact via social media platforms.
• Technical Partner/Commercial Bot : Bots operated by third-party companies to integrate services or content.
• · Tool : Clients using tools for testing, monitoring, or other operational functions.

Good bots and other categories will account for 42% of internet engagement in 2024.

How to protect your organization

Understanding and managing the threat from bad bots is critical to maintaining the security and integrity of online activities. This includes protecting e-commerce sites from price gouging, stock hoarding, and fraudulent transactions, and preventing spam, fake accounts, and disinformation campaigns from targeting social media, as well as protecting proprietary data, guarding against negative SEO tactics that can harm site rankings, and ensuring that login authentication attempts are legitimate.

Effective, targeted bot protection helps detect and protect against automated attacks carried out by malicious bots, while also allowing known good bots, such as search engine bots and SEO bots, to crawl your web application.

Such protection requires a multi-layered approach, including:

• Robust application security. Installing advanced application protection to protect web applications and APIs and ensuring it is properly configured with rate limiting and monitoring in place is an important first step to ensuring your application security solution is working as intended.
• Specialized bot protection. Make sure the application security solution you choose includes anti-bot protection to effectively detect and stop advanced automated attacks.
• Take advantage of machine learning. With a solution that uses the power of machine learning, you can effectively detect and block hidden, near-human bot attacks. Be sure to enable login-killing protection to prevent account takeovers as well.
• Don't forget the basics. Access and authentication controls, including multi-factor authentication, help secure vulnerable access points like login pages from brute force and authentication attacks.

For more information on how to defend your environment against advanced and evolving bot attacks, visit our website.