Robots.txt and Sitemap module

[teggen +]

I am using Google Webmaster, and it appears the robots.txt file is restricting access to the sitemap file. In the robots.txt, there is among other things "Disallow: /*php"

I am guessing this is the cause since I am using the Templatecodes Sitemap, which has the adress .../modules.php?mod=tc_site_map

Is there a way around to fix this, ie to exclude all .php pages from being indexed- but to include this specific page with the sitemap?

Thanks for your help.

Terje

[seymourjames +]

The robots file is excluding php stuff because you are using the SEO module. All the valuable urls are converted and you don't want any canonical problems either although that is less of an issue nowadays if at all.

URLs like usercheckout.php are being excluded for good reason having very little value in them and indeed they will be identical to many other sites. It will of course exclude the templatecodes site map url as well.

Its a good point you raise because I always viewed this module in a different way. This url is for submitting the contents to google webmaster tools in an XML file (i.e. sitemmap.xml) . Not as a php one. The basic idea is that you copy and paste the contents into an file with an xml extension. If you want a sitemap for your site with clickable links in it that is a different issue and serves a different purpose.

I am unsure what the google indexing engine would make of the sitemmap php version which is full of xml. Whether it would decide these were in fact links or not. You would not really want this page indexed as such. The webmaster tools is set up to read the contents however as an xml file. You could try and rewrite the url into an a file with an xml extension. Then place a link to that file on your site. You would also then not have to maunally generate the xml file for webmaster tools either.

[teggen +]

Thanks for the reply. I have downloaded the sitemap into an XML-file and uploaded it. But how often should I generate a new sitemap, and upload it again? I have ads coming in every day, and I would like Google to index them also.

Alternatively, how can I do the rewrite of the url modules.php?mod=tc_site_map into a file with .xml extension? This would be, as you point out, the best option cause then I can just upload and forget about it.

Thanks again for your help.

[seymourjames +]

You can't force google to do anything. It will find those adverts naturally when it crawls your site. I would not bother to submit your sitemap more than once every week or so. I used to do it very frequently but now I only do it once every month or so for each of my sites. I personally have seen no really big benefits by submitting too frequently. Google only takes it as advice as such.

Well you would need to try doing a redirect in your .htaccess file. Then you can just press the submit button in google webmaster tools.

There is a danger that you can become obsessed with google indexing. If you really want to improve your ranking you need to focus on great content and some quality inward point links to your site (various pages within it too - see here http://www.68classifieds.com/forums/...imization.html).