robots.txt

[EnergyFreak +]

What version of 68 Classifieds are you running?
Example: V4.1.6 Designer

What template are you using?
purple

Please describe in detail the issue you are having:

Hi,

Should we use a robots.txt for our web site? If so, what should it look like, which directories should we protect from being crawled?

Thanks.

[seymourjames +]

Yes - it is a good idea in general to exclude certain pages. Exclude pages on the other other side of login for example. I often also exclude the login, userforgot password and user registration pages too for SEO reasons as they are so similar to other peoples sites and could be regarded as duplicate content. It depends how different you make them from others.

[EnergyFreak +]

Hi,

Sorry but I am sort of noob when it comes to robots.txt.

How would I write my robots.txt?

Something like :

User-agent: *
Disallow: /backup/

and should I block any bots at all?

[bowers01 +]

just block what you dont want to come up on google and yahoo ect.
I block:
Disallow: /searchresults.php - Because if you search for a ad in google it comes up with that page rather than the ad, eg if i search for volvo fm12 it can bring up this page rather than the ad itself
Disallow: /toplistings.php - Same as above
Disallow: /category.php - A lot of my categories are emtpy, i dont want people to be able to coem to my site for a empty page as they will leave and probably never come back
Disallow: /printer.php - Some times comes up instead of the ad

[EnergyFreak +]

Originally Posted by bowers01

just block what you dont want to come up on google and yahoo ect.
I block:
Disallow: /searchresults.php - Because if you search for a ad in google it comes up with that page rather than the ad, eg if i search for volvo fm12 it can bring up this page rather than the ad itself
Disallow: /toplistings.php - Same as above
Disallow: /category.php - A lot of my categories are emtpy, i dont want people to be able to coem to my site for a empty page as they will leave and probably never come back
Disallow: /printer.php - Some times comes up instead of the ad

Thanks will definitely add these in my robots.txt

[EnergyFreak +]

Ignore my previous statements. What sections of my site should I block for security?

[seymourjames +]

Robots.txt does not block anything if it is a malicious robot. Good ones just obey the rule not to index those pages you exclude. When you say block for security what do you actually mean?

[EnergyFreak +]

like files search engines should not get their hands on.

[seymourjames +]

I think Nick has answered that question. Just exclude directories or files you do not want the robot to go to.