Cloudflare launches free tool to block AI crawlers with one click

Since the explosive growth of generative AI tools, copyright disputes between creators and AI-generated works have continued to occur. Many creators worry that their works will be crawled away by AI robots at any time, and now there may be a potential solution.

Cloudflare, a well-known cloud service and information security platform, recently announced that it has launched a new free tool, claiming that users can prevent the data of their hosted websites from being used to train AI models as long as they turn on Bot Fight Mode in the security page.

Image source: Cloudflare Cloudflare launches free tool to prevent AI crawlers with one click

How does Cloudflare monitor AI bot crawlers?

Currently, vendors that provide AI services, such as Google, OpenAI, and Apple, allow website owners to modify robots.txt to prevent their robots from crawling data. However, Cloudflare’s latest report points out that not all websites can successfully block it. AI crawler tools crawl data.

For example, the number one AI crawler robot is Bytespider (developed by China's ByteDance Company), which has a high frequency of crawling website data and being rejected by crawlers. Next is GPTBot, managed by OpenAI, which is specially designed for crawling its product ChatGPT. Get data.

Cloudflare pointed out that although website owners can modify robot.txt to ban GPTBot, CCBot and Google robots, many people may not be aware of unfamiliar robots such as Bytespider and ClaudeBot, so they have not blacklisted them.

To solve this problem, Cloudflare analyzed AI bot and crawler traffic to fine-tune their automatic bot detection model. Among other factors, Cloudflare's model monitors whether AI bots mimic the behavior of real humans in order to evade detection.

Media industry, creators are fighting back against AI crawlers

Research from September 2023 noted that around 26% of the top 100 most trafficked websites globally have blocked GPTBot, while another study noted that more than 600 news outlets have also blocked the bot on behalf of website owners Content producers and content producers are preventing AI crawlers from causing arbitrary use of their works.

Some creators and media have taken more direct actions, that is, directly mentioning AI companies. Foreign media "Axios" broke the news that according to a copy of the letter they saw, "Forbes" seemed to have directly sent a letter to warn the CEO of the AI ​​search company Perplexity, warning him to steal text and images and intentionally infringe copyright.

The Recording Industry Association of America also recently announced that it has filed a lawsuit against the music generative AI tools Udio and Suno, accusing them of large-scale infringement of intellectual property rights.