What is robots.txt file?

Posted Date: 24 May 2008	Resource Type: Articles	Category: ASP.NET/Web Applications
Author: Ravindra Kumar Singh	Member Level: Gold
Rating:	Points: 15

If you are web developer or working with web application then i think all of you aware with this file. Yeah it is text file because extension is .txt. But it is specific also.In this file, you can instruct robots to not crawl and index certain files, folders/subfolders within your site. For example, you may not want search engine to crawl the /admin directory of your site, as it's both meaningless to you. So text file "Robots.txt" lets you tell search engine just that.

How to Create this text file:
1. File name should be the same "robots.txt"
2. It should be in the root of the web application and not any sub folder.

example:
Useragent: *
Disallow: /bin/
Disallow: /content/
Disallow: /counsellor/

Where a USERAGENT line to identify the crawler in question followed by one or more DISALLOW: lines to disallow it from crawling certain parts of your site.

Responses

Author: Raghav 25 May 2008

Member Level: Gold Points : 2

Thank you sir

Feedbacks

Popular Tags

What are tags ?

Search Tags

Robots .

Post Feedback

This is a strictly moderated forum. Only approved messages will appear in the site. Please use 'Spell Check' in Google toolbar before you submit.
You must Sign In to post a response.

	Next Resource: Dynamic XML file for AdRotator through database using SQLXML
	Previous Resource: Export your Crystal Report data to PDF
	Return to Discussion Resource Index
	Post New Resource
	Category: ASP.NET/Web Applications

Post resources and earn money!

More Resources