-
Howdy! Welcome to our community of more than 130.000 members devoted to web hosting. This is a great place to get special offers from web hosts and post your own requests or ads. To start posting sign up here. Cheers! /Peo, FreeWebSpace.net
How can I block an domain from being indexed in google.
- Thread starter Subhash
- Start date
Secret
Active Member
first, put a robots.txt on the root of the site
to specfy what to deny on the site, please read this article:
http://www.robotstxt.org/wc/exclusion.html
second, visit the search engine who crawled you (google) and use the 'remove my site' form and enter your site. just wait and your site will be removed form the list.
more information
http://www.google.com/support/webmasters/bin/topic.py?topic=8459
to specfy what to deny on the site, please read this article:
http://www.robotstxt.org/wc/exclusion.html
second, visit the search engine who crawled you (google) and use the 'remove my site' form and enter your site. just wait and your site will be removed form the list.
more information
http://www.google.com/support/webmasters/bin/topic.py?topic=8459
Last edited:
Subhash
New Member
Can you plz explain me what will happen when this is the case?
Suppose I have two domains domain A and domain B. Both are pointing to my single site on a single server(I have parked two domain in single account).
Now both Domains A and B are indexed by google. But I want only Domain A to be exclude and not Domain B..
As you say to place robot.txt, as much I know both Domains A and B will be excuded. That's not I want. How can I use robot.txt to exculde by domain name
Correct me if I am wrong.
arizonadream
Active Member
Subhash
New Member
I am not arguing, but only asking for your generous help. I read those long before and i think it doesn't fits my need.
If you think that my answer is there can you point me exactly to that page or part. Or you can copy that part and paste here in quotes. I read both the article and help, I didn’t find my answer.
arizonadream said:
This is quite helpful, I got some new ideas. Anyways I am finding some easier and effective solution.
Last edited:
OMG... afraid of testing or trying something ?
Block or remove your entire website using a robots.txt file
To remove your site from search engines and prevent all robots from crawling it in the future, place the following robots.txt file in your server root:
User-agent: *
Disallow: /
To remove your site from Google only and prevent just Googlebot from crawling your site in the future, place the following robots.txt file in your server root:
User-agent: Googlebot
Disallow: /
Each port must have its own robots.txt file. In particular, if you serve content via both http and https, you'll need a separate robots.txt file for each of these protocols. For example, to allow Googlebot to index all http pages but no https pages, you'd use the robots.txt files below.
For your http protocol (http://yourserver.com/robots.txt):
User-agent: *
Allow: /
For the https protocol (https://yourserver.com/robots.txt):
User-agent: *
Disallow: /
Have fun testing !
PaulP.
Block or remove your entire website using a robots.txt file
To remove your site from search engines and prevent all robots from crawling it in the future, place the following robots.txt file in your server root:
User-agent: *
Disallow: /
To remove your site from Google only and prevent just Googlebot from crawling your site in the future, place the following robots.txt file in your server root:
User-agent: Googlebot
Disallow: /
Each port must have its own robots.txt file. In particular, if you serve content via both http and https, you'll need a separate robots.txt file for each of these protocols. For example, to allow Googlebot to index all http pages but no https pages, you'd use the robots.txt files below.
For your http protocol (http://yourserver.com/robots.txt):
User-agent: *
Allow: /
For the https protocol (https://yourserver.com/robots.txt):
User-agent: *
Disallow: /
Have fun testing !
PaulP.
Subhash
New Member
Now this was what you were telling me 'answer of my question'. NO, this is not.
You don't understand a simple thing i am telling repeatedly, please read my full posts and try to understand it.
Now if I use two domains pointing to same account, same server space. And if I use the the robot.txt which you specified above and place in server root (which is common for both domains) then both domains will be blocked from google. Right?? This is not what i want.
I only want one domain to be blocked and other to remain as it is (i.e indexed in goolge).
There is no use of testing something which definitely going to give negative result. Test those for which you are not sure. Apart from that robot.txt s are sensitive parts try testing as least as possible because once your site is removes you will find a hard time to re-index in google.
You don't understand a simple thing i am telling repeatedly, please read my full posts and try to understand it.
Now if I use two domains pointing to same account, same server space. And if I use the the robot.txt which you specified above and place in server root (which is common for both domains) then both domains will be blocked from google. Right?? This is not what i want.
I only want one domain to be blocked and other to remain as it is (i.e indexed in goolge).
There is no use of testing something which definitely going to give negative result. Test those for which you are not sure. Apart from that robot.txt s are sensitive parts try testing as least as possible because once your site is removes you will find a hard time to re-index in google.
Hi Subhash,
I fully understand your problem but you only accept a full answer that is suitable for your environment.
Well, read again. You should read/study the robots.txt information and use that knowledge in your network.
I will NOT provide you a fully configured robots.txt and that seems to be the only thing you are willing to see from us.
There is a part that you need to do yourself as each network administrator is having a different environment.
Well, 2 things you can do now, read/study more and do something, or just wait untill some-one gives you a fully configured robots.txt (Which I don't think will happen at all)
I rest my case, I helped enough. :classic2:
Good luck with your server.
Kind regards,
PaulP.
I fully understand your problem but you only accept a full answer that is suitable for your environment.
Well, read again. You should read/study the robots.txt information and use that knowledge in your network.
I will NOT provide you a fully configured robots.txt and that seems to be the only thing you are willing to see from us.
There is a part that you need to do yourself as each network administrator is having a different environment.
Well, 2 things you can do now, read/study more and do something, or just wait untill some-one gives you a fully configured robots.txt (Which I don't think will happen at all)
I rest my case, I helped enough. :classic2:
Good luck with your server.
Kind regards,
PaulP.
Subhash
New Member
Yes PaulP, Helped enough, Thanks for that.
I studied many articles and helps regarding robot.txt and which I can conclude that in ordinary way robot.txt can not be used to work for specific domain (atleast I didn't find), but can be used to specify subdirectories only.
However something similar to fit my need can be done with cgi scripting. I myself planned something to do PHP script.
Anyways Thanks again for your effort,
Regards,
Subhash.
I studied many articles and helps regarding robot.txt and which I can conclude that in ordinary way robot.txt can not be used to work for specific domain (atleast I didn't find), but can be used to specify subdirectories only.
However something similar to fit my need can be done with cgi scripting. I myself planned something to do PHP script.
Anyways Thanks again for your effort,
Regards,
Subhash.
