I have this since I don't want members pages, search, help, misc, login, register pages indexed from Google, and only threads and forums.
Robots.txt and sitemaps should always be in synch. In other words: if the xenforo sitemap sends a URL to search engines, then it should not be disallowed in ...
Old Hard to Find TV Series on DVD
I am getting robots blocked and redirection error messages too. When I checked for robots.txt file (ex: https://yourdomain.com/robots.txt ) then ...
Hello, I've wanted ask, what nowadays is recommended to setup robots.txt for XF2.2 installation. Any recommendations are very appreciated, ...
txt file tells search engines what parts of the site they can and cannot explore. These are recommendations to the search engine crawlers which ...
There is no default robots.txt file included in XenForo, but it's very easy to create one. Just upload it to the root of your installation.
Just trying to get some ideas of what anyone is currently using for their robots.txt on their Xenforo site. Here is mine: # robots.txt file ...
I am thinking maybe my robots.txt is not good and blocking search engine from crawling my site. I would appreciate if someone can confirm to me ...
Adding entries to robots.txt doesn't prevent the URLs from being accessed. Some crawlers abide by robots.txt, others don't. ... If you want to ...
I want to block this forum from being crawled. What is the best method? I tried adding the path to the node to my robots.txt file and that did ...