What is robots.txt?

Every website has a small text file named exactly robots.txt. To see yours: in your browser's address bar, type your domain name followed by /robots.txt and press Enter. For example, if your website is joespizza.com, type joespizza.com/robots.txt.

Think of it as a "who's allowed in" sign for automated visitors. Search engines like Google have always read it. Now AI assistants — ChatGPT, Gemini, Perplexity, and others — send their own automated readers too. If your robots.txt says "stay away," those AI assistants can never read your site. You're invisible to them.

Why this is the #1 finding

Everything else in your report only matters if AI can read your site in the first place. Unblocking it is the single highest-impact fix.

The fastest fix: forward it to your web person

Most business owners don't edit this file themselves — and that's fine. Your CrawlReady report has a "Send to your web person" button on this finding. Click it. It copies a ready-to-send message containing: what the problem is, the exact text to add, and exactly where to put it. Paste it into an email or text message to whoever handles your website and send it.

That's it. They'll know what to do.

Doing it yourself

If you pay for web hosting directly (a company like Bluehost, GoDaddy, HostGator, or SiteGround), you can do this in about two minutes:

  1. See your current file

    In your browser, go to yourdomain.com/robots.txt — using your real domain name. For example: joespizza.com/robots.txt. You'll see what's there now.

  2. Open your hosting file manager

    Log in to your hosting account (the company you pay for web hosting). Look for "File Manager" — in cPanel it's an icon labeled exactly that. Open it, then open your website's main folder (often called public_html). Find the file named exactly robots.txt and click it to edit.

  3. Don't delete what's there

    Your robots.txt may already have rules for Google and other search engines. Leave every existing line alone. You're only adding new lines.

  4. Paste the new lines at the very top

    In your CrawlReady report, click Copy on the text box under this finding. Go back to the robots.txt editor. Click at the very beginning of the file — before the first character — and paste. The new lines must sit above any line that begins with the word Disallow.

  5. Save and verify

    Click Save (or Save Changes). Go back to yourdomain.com/robots.txt in your browser and refresh the page — you should see your new lines at the top.

Why the top?

When two rules conflict, the most specific one wins. Your new "Allow" lines override a general "Disallow" for the AI readers listed — but only if they're placed above it.

What if you're on Wix, Squarespace, or WordPress.com?

Here's the honest answer: on most website-builder platforms, you cannot open robots.txt yourself. The platform controls that file. Here's exactly what to do on each:

  • Wix: Log in at wix.com and open your site's dashboard (not the editor — the dashboard). Click Settings in the left menu, then SEO, then look for a Robots.txt section. If you see it, you can add the lines from your report there. If you don't see it, your plan doesn't include it — use the "Send to your web person" button or contact Wix support at wix.com/contact.
  • Squarespace: Squarespace does not let you edit robots.txt on any plan. Go to squarespace.com/help, click Contact Us, and ask: "Please allow AI crawlers (GPTBot, ClaudeBot, PerplexityBot, and others) in my site's robots.txt." Or forward the issue to your web person using the button in your report.
  • WordPress.com (hosted): On the Free, Personal, and Premium plans, you cannot edit robots.txt. On the Business plan or higher (which allows plugins): install the free Yoast SEO plugin, then in your WordPress dashboard go to Yoast SEO → Tools → File Editor and add the lines there. On a lower plan, your options are upgrading or contacting WordPress.com support.
  • WordPress.org (self-hosted): You can edit it. In your WordPress dashboard, install the free Yoast SEO plugin (Plugins → Add New, search "Yoast SEO", Install, Activate). Then go to Yoast SEO → Tools → File Editor, paste the lines from your report at the top, and click Save changes to robots.txt. Or edit the file directly through your hosting file manager as described above.
  • Shopify: Shopify generates robots.txt automatically and doesn't allow direct editing on any plan. Contact Shopify support or forward to your web person.

Bottom line: if your platform won't let you touch the file, click "Send to your web person" in your report. They deal with platform limitations every day.

After the fix

In your CrawlReady dashboard, run a new scan of the same website. The crawler check reads your robots.txt fresh each time — once the AI readers are allowed, this finding disappears and your score goes up.