Crawler Control

Robots.txt Generator

Generate valid robots.txt files with custom user-agent directives, sitemap links, and one-click AI crawler blocking presets.

General Crawler Policy

Default Policy for All Search Engines (User-agent: *)

Disallowed Paths (1 path per line)

Allowed Exceptions (1 path per line)

AI Crawlers & Model Training Bots

GPTBot

OpenAI crawler for ChatGPT training data

ClaudeBot

Anthropic crawler for Claude training data

PerplexityBot

Perplexity AI search crawler

Bytespider

ByteDance / TikTok crawler

Google-Extended

Google AI / Gemini model training data crawler

XML Sitemap Link

Generated robots.txt Preview

robots.txt

# Robots.txt Generated via Vitals Audit Toolkit

User-agent: *
Disallow: /admin/
Disallow: /private/
Disallow: /api/

# Block AI Crawlers & Model Training Bots
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

User-agent: Bytespider
Disallow: /

# XML Sitemap
Sitemap: https://www.vitalsaudit.com/sitemap.xml

About this tool

Comprehensive documentation & audit utility guide

Verified Utility

Robots.txt is a plain text file placed at the root of a domain that instructs web crawlers (Googlebot, Bingbot, GPTBot) on which URLs they can or cannot request. Blocking AI crawlers helps prevent uncredited model training on your content.

How it works

Set crawler rules

Choose default policies, allowed/disallowed paths, and sitemap URL.

Select AI bot toggles

Easily disallow GPTBot, ClaudeBot, PerplexityBot, and others.

Download robots.txt

Copy the output or download robots.txt directly to your root folder.

What it checks

Syntax compatibility for User-agent, Allow, Disallow, and Sitemap directives

One-click AI training bot blocking presets

Root path normalization and line comment formatting