Support Forums

Robots.txt and Sitemap module

This is a discussion on Robots.txt and Sitemap module within the TemplateCodes forums, part of the Third Party Support category; I am using Google Webmaster, and it appears the robots.txt file is restricting access to the sitemap file. In the ...


Go Back   68 Classifieds Forums > Third Party Support > TemplateCodes

Reply
 
Thread Tools Display Modes
Old 11-02-2009, 05:29 PM   #1
Junior Member
 
Join Date: Mar 2009
Posts: 12
Rep Power: 12
teggen is on a distinguished road
Default Robots.txt and Sitemap module

I am using Google Webmaster, and it appears the robots.txt file is restricting access to the sitemap file. In the robots.txt, there is among other things "Disallow: /*php"

I am guessing this is the cause since I am using the Templatecodes Sitemap, which has the adress .../modules.php?mod=tc_site_map

Is there a way around to fix this, ie to exclude all .php pages from being indexed- but to include this specific page with the sitemap?

Thanks for your help.

Terje
__________________
Developer version 4.1.6 - TC Fluid template customised
teggen is offline   Reply With Quote
Old 11-02-2009, 11:59 PM   #2
All Hands On Deck
 
 
Join Date: Mar 2008
Posts: 3,433
Rep Power: 87
seymourjames is a jewel in the rough
Default

The robots file is excluding php stuff because you are using the SEO module. All the valuable urls are converted and you don't want any canonical problems either although that is less of an issue nowadays if at all.

URLs like usercheckout.php are being excluded for good reason having very little value in them and indeed they will be identical to many other sites. It will of course exclude the templatecodes site map url as well.

Its a good point you raise because I always viewed this module in a different way. This url is for submitting the contents to google webmaster tools in an XML file (i.e. sitemmap.xml) . Not as a php one. The basic idea is that you copy and paste the contents into an file with an xml extension. If you want a sitemap for your site with clickable links in it that is a different issue and serves a different purpose.

I am unsure what the google indexing engine would make of the sitemmap php version which is full of xml. Whether it would decide these were in fact links or not. You would not really want this page indexed as such. The webmaster tools is set up to read the contents however as an xml file. You could try and rewrite the url into an a file with an xml extension. Then place a link to that file on your site. You would also then not have to maunally generate the xml file for webmaster tools either.
__________________
"The fool doth think he is wise, but the wise man knows himself to be a fool.".

TemplateCodes.com for 68C, Version 4 Templates & Modules
seymourjames is offline   Reply With Quote
Old 11-03-2009, 04:36 AM   #3
Junior Member
 
Join Date: Mar 2009
Posts: 12
Rep Power: 12
teggen is on a distinguished road
Default

Thanks for the reply. I have downloaded the sitemap into an XML-file and uploaded it. But how often should I generate a new sitemap, and upload it again? I have ads coming in every day, and I would like Google to index them also.

Alternatively, how can I do the rewrite of the url modules.php?mod=tc_site_map into a file with .xml extension? This would be, as you point out, the best option cause then I can just upload and forget about it.

Thanks again for your help.
__________________
Developer version 4.1.6 - TC Fluid template customised
teggen is offline   Reply With Quote
Old 11-03-2009, 07:30 AM   #4
All Hands On Deck
 
 
Join Date: Mar 2008
Posts: 3,433
Rep Power: 87
seymourjames is a jewel in the rough
Default

You can't force google to do anything. It will find those adverts naturally when it crawls your site. I would not bother to submit your sitemap more than once every week or so. I used to do it very frequently but now I only do it once every month or so for each of my sites. I personally have seen no really big benefits by submitting too frequently. Google only takes it as advice as such.

Well you would need to try doing a redirect in your .htaccess file. Then you can just press the submit button in google webmaster tools.

There is a danger that you can become obsessed with google indexing. If you really want to improve your ranking you need to focus on great content and some quality inward point links to your site (various pages within it too - see here http://www.68classifieds.com/forums/...imization.html).
__________________
"The fool doth think he is wise, but the wise man knows himself to be a fool.".

TemplateCodes.com for 68C, Version 4 Templates & Modules
seymourjames is offline   Reply With Quote
Reply

Thread Tools
Display Modes


Similar Threads
Thread Thread Starter Forum Replies Last Post
Sitemap Module Does Not Work shanedawg Technical Support 4 02-16-2009 09:24 AM
Sitemap Module Eric Barnes Modification Release 0 02-12-2009 04:30 PM
Sitemap Module - Index Page Not Included bowers01 Technical Support 0 08-11-2008 07:03 AM
Robots.txt Question - Hide viewmember.php? bgordon Site Marketing 3 11-12-2006 04:53 PM


All times are GMT -4. The time now is 05:59 AM.


Powered by vBulletin® Version 3.8.1
Copyright ©2000 - 2011, Jelsoft Enterprises Ltd.
Search Engine Friendly URLs by vBSEO 3.2.0