I run a small tool site on plain PHP shared hosting, and one of the smallest decisions I made still shapes how I handle every new page: I control crawlers with a <meta name="robots"> tag in the HTML, and I never touch the X-Robots-Tag HTTP header.
Both do the same job. A robots meta tag lives inside the page and tells a search engine whether to index that page, follow its links, or show a snippet. The X-Robots-Tag does exactly the same thing, but it is sent as an HTTP response header by the server. If you can set headers, the header version is more powerful, because it works on files that are not HTML at all — a PDF, an image, a video. For a page, though, the two are interchangeable, and Google says so plainly.
So why pick one and ignore the other? Because on my setup, headers are work and meta tags are not.
My site is plain PHP pages on shared hosting. There is no framework, no middleware layer, and no single place where every response passes through. To send an X-Robots-Tag I would have to call header() in every PHP file, or add rules to .htaccess that match each URL I want to control. The meta tag, by contrast, goes into the one shared header include every page already loads. One line, one place, done:
<meta name="robots" content="index, follow"> And when a page should not be indexed — a thin utility page, a duplicate landing page, a page I only built for a campaign — I change the same line in the same include, or set a variable before the include runs:
<?php $robots = 'noindex, follow'; ?> <?php include 'includes/head.php'; ?> That is the whole system. There is no header logic to keep in sync with the markup, and no risk that a header set in one place contradicts a meta tag set in another. When two rules disagree, Google follows the most restrictive one, which is exactly the kind of silent surprise I do not want. One source of truth, in the file I can actually see when I open the page.
There is a second reason, and it is about debugging. When something looks wrong in search results — a page indexed that should not be, a snippet showing text I hid — I can open the page in a browser, view the source, and read the robots instruction right there. A response header is invisible unless I remember to open the network tab or run curl. On a small site run by one person, the thing you can see is the thing you will actually check.
None of this is an argument that the header is bad. If I served PDFs or images I wanted kept out of the index, the meta tag could not help me at all, because a PDF has no <head> to put one in. And on a bigger setup with a framework that sets headers centrally, the header is arguably the tidier choice. But my site serves HTML pages and almost nothing else, and the one non-HTML thing I care about — the download responses — should never be indexed anyway, which I handle by making sure they are never linked as pages.
The lesson is not really about robots at all. On a small project, the best tool is usually the one that lives where you already work. I already edit the shared head include for every page. Putting the crawler instruction there means it gets reviewed every time I touch the page, it cannot drift away from the content it describes, and I never have to remember a second place to look. The header would have been the more "proper" solution. The meta tag is the one that actually gets maintained.
That is the pattern I keep coming back to while building ramrom.io: pick the boring option that lives in one place, and save the clever option for the day the boring one genuinely cannot do the job.
