Keeping PDFs Out of Google With an X-Robots-Tag Header (Apache and nginx)
Short answer: A PDF has no head section, so the usual noindex meta tag cannot be used. Instead, the server sends an X-Robots-Tag: noindex header with the file. On Apache you add a short FilesMatch block, on nginx a location block, and then you check the response with a header-inspection tool. Googlebot needs to remain able to download the file, so do not block the same folder in robots.txt while you wait for the listing to drop.
When this is the right tool
Some documents are meant for customers but not for search: supplier price lists, old brochures superseded by a web page, staff handbooks left on a public path, scanned application forms. Putting noindex on them stops them being listed while keeping the links working for anyone you give the address to.
It is not a lock. Whoever holds the address can still download the file. If the document is confidential, place it behind a sign-in. Header-based noindex is for files that are fine to be public but pointless as search results.
How the header works
When a browser or Googlebot requests a file, the server answers with headers before the body. Adding X-Robots-Tag: noindex to that answer carries the same instruction as a robots meta tag in a web page. It works for any file type, including PDF, Word documents, images and plain text.
You can combine values, for example noindex, nofollow, though for a PDF the follow part rarely matters. Since the header is read at fetch time, Google has to crawl the file at least once after you add it.
Step by step on Apache
- Confirm the server has
mod_headersenabled. On many shared hosts it is on by default; if not, ask the host. - Open the
.htaccessfile in the site root, or the relevant virtual host file if you manage the server. - Add this block:
<FilesMatch "\.pdf$">
Header set X-Robots-Tag "noindex"
</FilesMatch>
- Save and reload. With
.htaccessthere is nothing to restart. - To limit the rule to one folder, place the same block in an
.htaccessfile inside that folder, for example/downloads/internal/.
To exclude only certain files, use their names in the pattern, such as "^(price-list-2022|old-menu)\.pdf$".
Step by step on nginx
- Find the server block for your site, usually under
/etc/nginx/sites-available/. - Add a location that matches PDFs:
location ~* \.pdf$ {
add_header X-Robots-Tag "noindex" always;
}
- Test the configuration with
nginx -t, then reload withsystemctl reload nginxor your host's equivalent. - Note that
add_headerinside a location block replaces headers inherited from outer blocks, so repeat any other headers you rely on, such as caching or security headers, inside the same block.
Managed hosting and CMS uploads
On WordPress, files in the uploads folder are served directly by the web server, so the Apache or nginx rule above is what applies. Some SEO plugins offer a switch to send the header for attachments; check the plugin documentation rather than assuming. On site builders where you cannot edit server config, you may have no way to set headers. In that case your options are to host the file somewhere that supports it, replace the PDF with an HTML page that carries a meta tag, or delete the file and let it return a 404 or 410.
Check that it works
Do not trust the config until you have seen the header.
- In a terminal, run
curl -I https://example.com/downloads/old-menu.pdfand look forx-robots-tag: noindexin the output. - In Chrome, open developer tools, the Network tab, click the file request and read the response headers.
- In Search Console, run the URL Inspection tool on the PDF and use the live test. It should report that indexing is not allowed because of the noindex.
Also check that the response is a plain 200. A header on a redirecting address will not help unless the final file carries it too.
Getting the existing listings removed
After the header is live, Google needs to recrawl each file before it drops out. That can take days to weeks. To speed up:
- Paste the PDF address into Search Console and ask for a recrawl so the header is picked up sooner.
- For an urgent case, file a temporary request in the Removals tool while the permanent fix takes effect.
- Leave robots.txt alone for that path until the files vanish from results. A block would stop Google seeing the header.
Choosing between noindex, deletion and a redirect
| Situation | Best option |
|---|---|
| File still useful to people who have the link | X-Robots-Tag noindex |
| File outdated and replaced by a web page | 301 redirect to the new page |
| File should never have been public | Remove it, return 404 or 410, move originals behind a login |
| File sensitive | Access control first, removal request second |
FAQ
Can I add noindex to a PDF without server access?
Not directly. You would need your host or CDN to add the header, or you would swap the PDF for an HTML page with a robots meta tag.
Will noindex stop people downloading the PDF?
No. It only affects search listings. Whoever holds the link can still open it.
Does it matter whether the header is spelled in capitals?
Header names are not case sensitive, but keep the standard spelling to avoid confusion when someone reads the config later.
How do I noindex every file in a folder?
On Apache, place the rule in an .htaccess file inside the folder. On nginx, use a location block matching the folder path.
If you would like us to audit which documents on your site are public and decide what should stay indexed, get in touch through our enquiry page.