Bot traffic has surpassed humans this year, now accounting for 59 percent of all site visits, according to a report released today by Distil Networks.
By comparison, last year, bots accounted for 45 percent of all traffic to Distil's customers' websites.
The majority of the bot traffic was "good bots," said Distil CEO Rami Essaid, such as that from search engines and social media sites. These bots comply with "robot.txt" files that Website administrators use to tell bots what to do, and bring value to the websites they visit.
He attributed the growth in bot traffic to more aggressive indexing by Bing, and by new search engines coming online.
"The 'good bots' almost doubled as a category this year," he said.
"Good bots" accounted for 36 percent of traffic this year, up from 21 percent last year. "Bad bots" were responsible for 23 percent of traffic this year, down slightly from 24 percent last year -- not because volumes were down, Essaid repeated, but because the number of "good bots" rose dramatically. Human traffic was just 41 percent, down from 55 percent last year.
The company helps customers keep out bad bots, of which the company saw 23 billion last year.
The company defines "bad bots" as those that don't respect "robots.txt" files and don't provide value to the sites they visit.
The bad bots fall into three categories. The largest one is the unwanted bots that scrape data from websites, especially bots used by business competitors such as those looking for pricing information. Then there are the malicious bots that explore websites looking for vulnerabilities.
The smallest category, which accounts for 10 percent of the bad bots, are the ones engaging in click fraud, making brute-force login attempts or trying to post spam. These actively malicious bots can be identified by their use of "post" requests
Amazon is the bot leader
Amazon led the charts this year. Its cloud computing platform was popular with bot creators -- 78 percent of all traffic from the Amazon cloud was bots this year, accounting for 15 percent of all bots, up from 9 percent last year.
In addition, Amazon itself was a bot creator, with 70 percent of traffic from Amazon.com being bots, accounting for 3 percent of all bots.
"Some of our customers compete with Amazon," said Essaid. "They don't want Amazon bots indexing their pricing, inventory and content."
Traditional hosting companies, like Verizon Business, Level 3, Hosting Solutions International, and the Las Vegas Datacenter dropped in the rankings or fell off the list altogether as defenders learned to block traffic from those sources.
"So the bad guys are now leveraging more and more residential networks, relying more on computers that are part of botnets," he said.
It's harder to tell if traffic that originates from a cable company like Comcast is human or not, he said.
Last year, only 1.8 percent of Comcast's traffic was bots. This year, that share was up more than three-fold, to 6.4 percent.
Bots get mobile
Traffic from mobile devices was also up, he said. Last year, 6 percent of bad bot traffic looked like it came from mobile devices, up from less than 1 percent in 2013.
While most of the this mobile traffic is spoofed, so as to avoid anti-bot defenses, between 20 and 30 percent of it is running on actual mobile devices, Essaid said.
The bot makers either install botnets on infected Android devices, or set up bot farm using cheap Android phones, he said.
Not only is traffic from actual mobile devices more like human mobile traffic, but it also allows the bad guys to interact directly with native mobile applications or custom content that requires a mobile device.
In addition to pretending to originate on mobile devices, some bots also try to mimic human behavior.
In 2014, 41 percent of bad bots used such tools as browser automation software to replicate actual human web browsing, for example.
In addition, in 2013, many bot makers scheduled their bots for the late evening hours, when security staffers were less likely to be at work. Bat bot traffic percentage would jump from less than 2 percent during business hours to as high as 18 percent in the evening.
"This year, we've seen that level out completely," said Essaid. "They continue to try to adapt their behavior based on what's getting them caught."
Some bots even disguise themselves as other bots. Webmasters typically allow the Googlebot to index their pages so that their sites would rank well in search.