What are robot meta tags?

Robots meta directives (sometimes called "meta tags") are pieces of code that provide crawlers instructions for how khổng lồ crawl or index web page nội dung. Whereas robots.txt tệp tin directives give sầu bots suggestions for how to lớn crawl a website"s pages, robots meta directives provide more firm instructions on how to crawl và index a page"s content.

Bạn đang xem: The purpose of meta robots tag

There are two types of robots meta directives: those that are part of the HTML page (lượt thích the meta robotstag) and those that the website server sends as HTTP headers (such as x-robots-tag). The same parameters (i.e., the crawling or indexing instructions a meta tag provides, such as "noindex" & "nofollow" in the example above) can be used with both meta robots and the x-robots-tag; what differs is how those parameters are communicated to crawlers.


Code sample:

While the general tag is standard, you can also provide directives to specific crawlers by replacing the "robots" with the name of a specific user-agent. For example, lớn target a directive sầu specifically to Googlebot, you’d use the following code:

Want lớn use more than one directive on a page? As long as they’re targeted lớn the same "robot" (user-agent), multiple directives can be included in one meta directive – just separate them by commas. Here’s an example:

This tag would tell robots not lớn index any of the images on a page, follow any of the link, or show a snippet of the page when it appears on a SERPhường.

If you’re using different meta robots tag directives for different tìm kiếm user-agents, you’ll need to use separate tags for each bot.


While the meta robots tag allows you khổng lồ control indexing behavior at the page level, the x-robots-tag can be included as part of the HTTP.. header lớn control indexing of a page as a whole, as well as very specific elements of a page.

While you can use the x-robots-tag khổng lồ exeđáng yêu all of the same indexation directives as meta robots, the x-robots-tag directive sầu offers significantly more flexibility and functionality that the meta robots tag does not. Specifically, the x-robots permits the use of regular expressions, executing crawl directives on non-HTML files, & applying parameters at a global cấp độ.

Xem thêm: Cách Xóa Lịch Sử Duyệt, Tìm Kiếm Và Tải Xuống Trên Firefox, Xóa Thông Tin Khỏi Google


To use the x-robots-tag, you’ll need to have access lớn either your website’s header .php, .htaccess, or hệ thống access tệp tin. From there, add your specific hệ thống configuration’s x-robots-tag markup, including any parameters. This article provides some great examples of what x-robots-tag markup looks like if you’re using any of these three configurations.

Here are a few use cases for why you might employ the x-robots-tag:

Controlling the indexation of content not written in HTML (like flash or video)

Blocking indexation of a particular element of a page (lượt thích an image or video), but not of the entire page itself

Controlling indexation if you don’t have access to lớn a page’s HTML (specifically, to lớn the section) or if your site uses a global header that cannot be changed

Adding rules to whether or not a page should be indexed (ex. If a user has commented over trăng tròn times, index their profile page)

SEO best practices with robots meta directives

All meta directives (robots or otherwise) are discovered when a URL is crawled. This means that if a robots.txt tệp tin disallows the URL from crawling, any meta directive on a page (either in the HTML or the HTTP header) will not be seen và will, effectively, be ignored.

In most cases, using a meta robots tag with parameters "noindex, follow" should be employed as a way khổng lồ khổng lồ restrict crawling or indexation instead of using robots.txt file disallows.

It is important khổng lồ note that malicious crawlers are likely to lớn completely ignore meta directives & as such, this protocol does not make a good security mechanism. If you have private information that you don’t want lớn make publicly searchable, choose a more secure approach, such as password protection, to keep visitors from viewing confidential pages.

Xem thêm: Tổng Hợp Điểm Chuẩn Ngành Quản Trị Kinh Doanh Các Trường, Điểm Chuẩn Đại Học Tài Chính Quản Trị Kinh Doanh

You vì chưng not need to lớn use both meta robots và the x-robots-tag on the same page – doing so would be redundant.

Keep learning

Put your skills to work Prolets you run crawls, research và report on keywords ranking, and track your site"s SEO performance, including its accessibility, over time. Try it >>