Support Forums

robots.txt

This is a discussion on robots.txt within the Technical Support forums, part of the Technical Support Forums category; What version of 68 Classifieds are you running? Example: V4.1.6 Designer What template are you using? purple Please describe in ...


Go Back   68 Classifieds Forums > Technical Support Forums > Technical Support

Reply
 
Thread Tools Display Modes
Old 02-18-2010, 05:12 PM   #1
Customer
 
Join Date: Feb 2010
Location: Montreal, QC
Posts: 162
Rep Power: 6
EnergyFreak is on a distinguished road
Default robots.txt

What version of 68 Classifieds are you running?
Example: V4.1.6 Designer

What template are you using?
purple

Please describe in detail the issue you are having:

Hi,

Should we use a robots.txt for our web site? If so, what should it look like, which directories should we protect from being crawled?

Thanks.
EnergyFreak is offline   Reply With Quote
Old 02-18-2010, 06:44 PM   #2
All Hands On Deck
 
 
Join Date: Mar 2008
Posts: 2,744
Rep Power: 66
seymourjames is a jewel in the rough
Default

Yes - it is a good idea in general to exclude certain pages. Exclude pages on the other other side of login for example. I often also exclude the login, userforgot password and user registration pages too for SEO reasons as they are so similar to other peoples sites and could be regarded as duplicate content. It depends how different you make them from others.
__________________
"The fool doth think he is wise, but the wise man knows himself to be a fool.".

TemplateCodes.com for 68 Classifieds
seymourjames is offline   Reply With Quote
Old 02-18-2010, 07:32 PM   #3
Customer
 
Join Date: Feb 2010
Location: Montreal, QC
Posts: 162
Rep Power: 6
EnergyFreak is on a distinguished road
Default

Hi,

Sorry but I am sort of noob when it comes to robots.txt.

How would I write my robots.txt?

Something like :

User-agent: *
Disallow: /backup/

and should I block any bots at all?
EnergyFreak is offline   Reply With Quote
Old 02-18-2010, 08:00 PM   #4
Genius At Work
 
bowers01's Avatar
 
Join Date: May 2008
Location: Geelong, Victoria, Australia
Posts: 1,050
Rep Power: 31
bowers01 is on a distinguished road
Default

just block what you dont want to come up on google and yahoo ect.
I block:
Disallow: /searchresults.php - Because if you search for a ad in google it comes up with that page rather than the ad, eg if i search for volvo fm12 it can bring up this page rather than the ad itself
Disallow: /toplistings.php - Same as above
Disallow: /category.php - A lot of my categories are emtpy, i dont want people to be able to coem to my site for a empty page as they will leave and probably never come back
Disallow: /printer.php - Some times comes up instead of the ad
__________________
Nick Bowers
68c v4.1.10 Developer Custom Template
bowers01 is online now   Reply With Quote
Old 02-18-2010, 08:09 PM   #5
Customer
 
Join Date: Feb 2010
Location: Montreal, QC
Posts: 162
Rep Power: 6
EnergyFreak is on a distinguished road
Default

Quote:
Originally Posted by bowers01
just block what you dont want to come up on google and yahoo ect.
I block:
Disallow: /searchresults.php - Because if you search for a ad in google it comes up with that page rather than the ad, eg if i search for volvo fm12 it can bring up this page rather than the ad itself
Disallow: /toplistings.php - Same as above
Disallow: /category.php - A lot of my categories are emtpy, i dont want people to be able to coem to my site for a empty page as they will leave and probably never come back
Disallow: /printer.php - Some times comes up instead of the ad
Thanks will definitely add these in my robots.txt
EnergyFreak is offline   Reply With Quote
Old 05-31-2010, 04:12 PM   #6
Customer
 
Join Date: Feb 2010
Location: Montreal, QC
Posts: 162
Rep Power: 6
EnergyFreak is on a distinguished road
Default

Ignore my previous statements. What sections of my site should I block for security?
EnergyFreak is offline   Reply With Quote
Old 05-31-2010, 06:29 PM   #7
All Hands On Deck
 
 
Join Date: Mar 2008
Posts: 2,744
Rep Power: 66
seymourjames is a jewel in the rough
Default

Robots.txt does not block anything if it is a malicious robot. Good ones just obey the rule not to index those pages you exclude. When you say block for security what do you actually mean?
__________________
"The fool doth think he is wise, but the wise man knows himself to be a fool.".

TemplateCodes.com for 68 Classifieds
seymourjames is offline   Reply With Quote
Old 05-31-2010, 06:47 PM   #8
Customer
 
Join Date: Feb 2010
Location: Montreal, QC
Posts: 162
Rep Power: 6
EnergyFreak is on a distinguished road
Default

like files search engines should not get their hands on.
EnergyFreak is offline   Reply With Quote
Old 06-01-2010, 05:32 AM   #9
All Hands On Deck
 
 
Join Date: Mar 2008
Posts: 2,744
Rep Power: 66
seymourjames is a jewel in the rough
Default

I think Nick has answered that question. Just exclude directories or files you do not want the robot to go to.
__________________
"The fool doth think he is wise, but the wise man knows himself to be a fool.".

TemplateCodes.com for 68 Classifieds
seymourjames is offline   Reply With Quote
Reply

Thread Tools
Display Modes


Similar Threads
Thread Thread Starter Forum Replies Last Post
robots and deny acces damiun Off Topic 4 12-16-2009 11:32 AM
Robots.txt and Sitemap module teggen TemplateCodes 3 11-03-2009 06:30 AM
Robots.txt Question - Hide viewmember.php? bgordon Site Marketing 3 11-12-2006 03:53 PM


All times are GMT -4. The time now is 06:43 AM.


Powered by vBulletin® Version 3.8.1
Copyright ©2000 - 2010, Jelsoft Enterprises Ltd.
Search Engine Friendly URLs by vBSEO 3.2.0