Here is a robots.txt we'd flag as High:
User-agent: *
Disallow: /admin
Disallow: /api/internal
Disallow: /backup
Every line is meant for Googlebot. Every line is also readable by anyone who types /robots.txt after your domain, and that's one of the first requests a curious attacker makes. The file you wrote to keep pages out of search results is a list of the pages you most wanted nobody to find.
The robots.txt standard says this itself. RFC 9309, section 3: "Listing paths in the robots.txt file exposes them publicly and thus makes the paths discoverable."
TL;DR
robots.txt is a public file and Disallow is a request to polite crawlers, not access control. The fix has three parts: put real authentication on any path you were hiding, replace the Disallow line with an X-Robots-Tag: noindex header if you also want it out of Google, and delete anything (backups, dumps, .env) that shouldn't be on the server at all. Don't try to fix it with a cleverer robots.txt.
Why This Shows Up in AI-Built Apps
Ask an AI tool to "hide the admin page from search engines" and you'll usually get a robots.txt line, because that's what the request literally asks for. It's the wrong tool for the job, and the code looks right, so it ships.
The other common source is copying the framework docs. The Next.js robots.ts reference uses disallow: '/private/' as its example. Paste that as-is and our scanner reports a Medium finding for a folder you may not even have.
What Disallow Actually Does
Less than people expect. Google's robots.txt introduction puts it bluntly: robots.txt "is not a mechanism for keeping a web page out of Google." Two consequences follow from that page and from Google's noindex documentation:
- A disallowed URL can still be indexed. If another page links to
/admin, Google can list the URL without ever crawling it. It just shows up with no description. - Disallow and noindex cancel out. Google says a noindex rule only works if the page is not blocked by robots.txt, because the crawler has to fetch the page to see the rule. Block it and the noindex is invisible.
And none of it applies to anyone who isn't a well-behaved crawler. A person with a browser ignores robots.txt completely.
If the only thing between the internet and your /admin page is a Disallow line, the page is public. Open it in a private window, logged out, right now. If it loads, that's your real finding, and it's more urgent than the robots.txt one.
How to Fix It
Check each disclosed path while logged out. Open a private window and visit every path the finding lists. Note which ones return the page, which redirect to a login, and which 404. Anything that loads without auth goes to the top of your list.
Put real access control on what's left. An admin area needs a server-side check on every request: a session, a role, a 401 or 403 when either is missing. A hidden link or a client-side redirect doesn't count. Our route protection guide covers this for Next.js, Nuxt and Express.
Delete what shouldn't exist. A Disallow: /backup, /dump, /sql, /.env or /.git line usually means someone knew the thing was there. No header fixes a database dump in your web root. Remove the files, then rotate any credential they contained.
Swap Disallow for noindex where you still want it out of search. Remove the sensitive lines from robots.txt and send an X-Robots-Tag: noindex header on those routes instead. Code for each framework is below.
Re-fetch /robots.txt from production. Builds cache, CDNs cache, and some frameworks generate the file at build time. Check the live file, not your repo.
Replace Disallow With a noindex Header
A header keeps the path out of search results without writing the path into a public file.
module.exports = {
async headers() {
return [
{
source: "/admin/:path*",
headers: [{ key: "X-Robots-Tag", value: "noindex, nofollow" }],
},
]
},
}
If you use app/robots.ts, also remove the disallow entries for those routes. Leaving them in stops Google from crawling the page and seeing the header.
export default defineNuxtConfig({
routeRules: {
"/admin/**": { headers: { "X-Robots-Tag": "noindex, nofollow" } },
},
})
{
"headers": [
{
"source": "/admin/(.*)",
"headers": [{ "key": "X-Robots-Tag", "value": "noindex, nofollow" }]
}
]
}
For a single-page app where /admin is a client-side route, the server only ever sends index.html, so a per-route header won't apply. Put <meta name="robots" content="noindex"> in the admin layout instead. Then put the admin data behind authenticated API calls. Hiding the page from Google doesn't do much if the API behind it answers anyone.
What a Safe robots.txt Looks Like
Short and boring:
User-agent: *
Disallow: /search
Disallow: /cart
Sitemap: https://yourapp.com/sitemap.xml
These lines are fine because they're crawl-budget hints for pages that are public anyway. Nothing here tells a stranger where to look.
What Our Scanner Checks
The finding comes from our robots.txt analyzer, which reads every Disallow line and compares it against 26 path patterns. The severity is the worst one it matched:
| Severity | Paths |
|---|---|
| High | /backup, /secret, /.env, /.git, /phpmyadmin, /sql, /database, /dump |
| Medium | /admin, /administrator, /api/internal, /api/admin, /debug, /staging, /internal, /private, /config, /cpanel, /log |
| Low | /dashboard, /test, /dev, /wp-admin, /webmail, /cgi-bin, /includes |
One thing to know: the patterns match prefixes. /login trips the /log rule, /developers trips /dev and /testimonials trips /test. We'd rather over-report on a file this short than miss /logs-2026/, but read the evidence line before you start changing things. A login page in robots.txt tells nobody anything.
The High rows are the ones to act on today. They're less about the robots.txt line than about what it suggests is sitting on the server. Our exposed .git directory guide covers what to do if one of them turns out to be real.
Is listing /admin in robots.txt a security risk?
On its own it only reveals a path, but that path is exactly the thing an attacker wants to try first. RFC 9309, the robots.txt standard, says listing paths in robots.txt exposes them publicly and makes them discoverable. If /admin has real authentication, the line costs you little. If it doesn't, you've published the address of an open door.
How do I hide an admin page from Google without putting it in robots.txt?
Put it behind a login, and send an X-Robots-Tag: noindex header (or a noindex meta tag) on it. Google's docs say robots.txt is not a mechanism for keeping a page out of Google, and that a noindex rule only works if the page is not blocked in robots.txt, because Google has to crawl the page to see it.
Can a page blocked in robots.txt still show up in Google?
Yes. Google's documentation says a URL blocked by robots.txt can still be indexed if other pages link to it, and it appears in results without a description. Disallow stops crawling, not indexing.
Should I delete my robots.txt entirely?
No. A missing robots.txt is fine for security, but a short one that only lists things like search result pages or a sitemap is better. Remove the sensitive Disallow lines, keep the file.
Why did the scanner flag /login or /developers?
Our check matches path prefixes, so /login trips the /log rule and /developers trips the /dev rule. Read the evidence line in the finding and judge each path. A login page in robots.txt is harmless; a /backup line is not.
See What Your robots.txt Gives Away
We read your live robots.txt, flag every sensitive path it lists, and check the paths themselves for exposed files.