AI and Web Security

here is the composition of my latest 2025 robots.txt file it should always be placed at the root of your web server. in this new 2025 version, i have integrated all the ai robots to prevent the various ai companies from exploring my site, my content, my images, ideas. also this data crawling technique (scraping your data) you can be compared to a new type of ddos attack because your site will receive a huge overload and will become inaccessible during their visits. also i strongly recommend that you read and install this robot at the root of all your website. 

ROBOTS.TXT > Stop AI Scraping your data at start

https://github.com/zigmoon/robots.txt

User-agent: Googlebot
Allow: /

User-agent: Googlebot-Image
Allow: /

User-agent: bingbot
Allow: /

User-agent: msnbot
Allow: /

User-agent: SiteAuditBot
Crawl-delay: 1
Allow: /

User-agent: Semrushbot-SI
Allow: /

User-agent: Yahoo Pipes 2.0
Disallow: /

User-agent: 008
Disallow: /

User-agent: DeepSeek Bot
Disallow: /

User-agent: ADmantX
Disallow: /

User-agent: AlphaBot
Disallow: /

User-agent: anthropic-ai
Disallow: /

User-agent: AwarioRssBot
Disallow: /

User-agent: AwarioSmartBot
Disallow: /

User-agent: BLEXBot
Disallow: /

User-agent: Buzzbot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

User-agent: claritybot
Disallow: /

User-agent: Claude-Web
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: cohere-ai
Disallow: /

User-agent: Diffbot
Disallow: /

User-agent: FacebookBot
Disallow: /

User-agent: FriendlyCrawler
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: huggingface
Disallow: /

User-agent: ImagesiftBot
Disallow: /

User-agent: img2dataset
Disallow: /

User-agent: magpie-crawler
Disallow: /

User-agent: Meltwater
Disallow: /

User-agent: Neevabot
Disallow: /

User-agent: news-please
Disallow: /

User-agent: NewsNow
Disallow: /

User-agent: Nutch
Disallow: /

User-agent: omgili
Disallow: /

User-agent: omgilibot
Disallow: /

User-agent: panscient.com
Disallow: /

User-agent: Perplexity-ai
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: PetalBot
Disallow: /

User-agent: PiplBot
Disallow: /

User-agent: scoop.it
Disallow: /

User-agent: Scrapy
Disallow: /

User-agent: Seekr
Disallow: /

User-agent: SentiBot
Disallow: /

User-agent: SeznamBot
Disallow: /

User-agent: TurnitinBot
Disallow: /

User-agent: YouBot
Disallow: /

User-agent: ZumBot
Disallow: /

Sitemap: https://YOURSITE.com/sitemap.xml

ZIGMOON DESIGN

concepteur d'interface web

Leave a Reply Cancel reply