Comments on: You Need A Custom Robots.txt, So Lets Make One. http://www.realbloggingtips.com/you-need-a-custom-robotstxt-so-lets-make-one/ Blogging for Money - 6 Figure Blogging Tips Sat, 31 Jul 2010 14:50:02 +0000 http://wordpress.org/?v=2.9.2 hourly 1 By: Narzuty http://www.realbloggingtips.com/you-need-a-custom-robotstxt-so-lets-make-one/comment-page-1/#comment-2651 Narzuty Thu, 02 Apr 2009 09:03:40 +0000 http://www.realbloggingtips.com/?p=1420#comment-2651 robots.txt is an extremely important part of every blog. I'm surprised to see so many people in comments that were not aware of its functions. robots.txt is an extremely important part of every blog. I’m surprised to see so many people in comments that were not aware of its functions.

]]> By: Beijinger http://www.realbloggingtips.com/you-need-a-custom-robotstxt-so-lets-make-one/comment-page-1/#comment-2420 Beijinger Sat, 07 Feb 2009 15:55:22 +0000 http://www.realbloggingtips.com/?p=1420#comment-2420 Request-rate: 1/5 Visit-time: 0600-0845 I was having some issue with certain bots which is eating my bandwich , glad to know above commands. Request-rate: 1/5
Visit-time: 0600-0845

I was having some issue with certain bots which is eating my bandwich , glad to know above commands.

]]> By: Ajith Edassery http://www.realbloggingtips.com/you-need-a-custom-robotstxt-so-lets-make-one/comment-page-1/#comment-2313 Ajith Edassery Thu, 23 Oct 2008 19:22:04 +0000 http://www.realbloggingtips.com/?p=1420#comment-2313 Though I was reasonably well-versed with the robot.txt file, I wasn't aware of those extended keyword set. Thanks a lot, Cheers, Ajith Though I was reasonably well-versed with the robot.txt file, I wasn’t aware of those extended keyword set.

Thanks a lot,

Cheers,
Ajith

]]> By: Ben Moreno http://www.realbloggingtips.com/you-need-a-custom-robotstxt-so-lets-make-one/comment-page-1/#comment-2262 Ben Moreno Mon, 13 Oct 2008 00:07:59 +0000 http://www.realbloggingtips.com/?p=1420#comment-2262 Hey this is cool. Thanks for the info man. I need to do this for my site. My name is Ben too! I like your blog. Nice design. Hey this is cool. Thanks for the info man. I need to do this for my site. My name is Ben too! I like your blog. Nice design.

]]> By: Sunday Hyper-Linkage & Updates | The Net Fool http://www.realbloggingtips.com/you-need-a-custom-robotstxt-so-lets-make-one/comment-page-1/#comment-2260 Sunday Hyper-Linkage & Updates | The Net Fool Sun, 12 Oct 2008 07:31:24 +0000 http://www.realbloggingtips.com/?p=1420#comment-2260 [...] Benjamin Patton - You Need a Custom Robots.txt, So Let’s Make One [...] [...] Benjamin Patton – You Need a Custom Robots.txt, So Let’s Make One [...]

]]> By: Armen Shirvanian http://www.realbloggingtips.com/you-need-a-custom-robotstxt-so-lets-make-one/comment-page-1/#comment-2257 Armen Shirvanian Sun, 12 Oct 2008 06:11:36 +0000 http://www.realbloggingtips.com/?p=1420#comment-2257 This is wonderful because it allows me to see the specifics of a file that is used ubiquitously. The internals of the file are quite easily altered, and you have presented them smoothly. This is wonderful because it allows me to see the specifics of a file that is used ubiquitously. The internals of the file are quite easily altered, and you have presented them smoothly.

]]> By: Donace http://www.realbloggingtips.com/you-need-a-custom-robotstxt-so-lets-make-one/comment-page-1/#comment-2255 Donace Sat, 11 Oct 2008 21:46:47 +0000 http://www.realbloggingtips.com/?p=1420#comment-2255 One thing I will chip in here is that following robot.txt and stuff like do-follow no-follow is all 'recommendations' for search engine guidelines. Yahoo is notorious is disobeying robot.txt files and the such. One thing I will chip in here is that following robot.txt and stuff like do-follow no-follow is all ‘recommendations’ for search engine guidelines.

Yahoo is notorious is disobeying robot.txt files and the such.

]]> By: John @ Maryland Real Estate http://www.realbloggingtips.com/you-need-a-custom-robotstxt-so-lets-make-one/comment-page-1/#comment-2253 John @ Maryland Real Estate Sat, 11 Oct 2008 19:27:49 +0000 http://www.realbloggingtips.com/?p=1420#comment-2253 good post Ben, it's also a good idea under the User-agent to disallow a bunch of the site scraping scripts out there. You can really save you a bunch of bandwidth if you have a large site. Just make sure not to disallow any of the important search engine spiders of course. good post Ben,

it’s also a good idea under the User-agent to disallow a bunch of the site scraping scripts out there. You can really save you a bunch of bandwidth if you have a large site. Just make sure not to disallow any of the important search engine spiders of course.

]]> By: Dinh Trung http://www.realbloggingtips.com/you-need-a-custom-robotstxt-so-lets-make-one/comment-page-1/#comment-2248 Dinh Trung Sat, 11 Oct 2008 14:02:13 +0000 http://www.realbloggingtips.com/?p=1420#comment-2248 Hi Benjamin, Sorry to ask you this question, since I do not know to buil a robot.txt for mine !! In my case, and my site (kienthucdulich.info), i faced problem with google sitemp, sometimes. Pls help to clarify ! Thanks and best regards, Dinh Trung www.kienthucdulich.info Hi Benjamin,

Sorry to ask you this question, since I do not know to buil a robot.txt for mine !!

In my case, and my site (kienthucdulich.info), i faced problem with google sitemp, sometimes.

Pls help to clarify !

Thanks and best regards,
Dinh Trung
http://www.kienthucdulich.info

]]> By: Big Ben Patton http://www.realbloggingtips.com/you-need-a-custom-robotstxt-so-lets-make-one/comment-page-1/#comment-2246 Big Ben Patton Fri, 10 Oct 2008 22:46:56 +0000 http://www.realbloggingtips.com/?p=1420#comment-2246 more or less yes one will send the comments one will notify you that their are some comments. more or less yes one will send the comments one will notify you that their are some comments.

]]> By: Kevin@Great Wall of China Facts http://www.realbloggingtips.com/you-need-a-custom-robotstxt-so-lets-make-one/comment-page-1/#comment-2245 Kevin@Great Wall of China Facts Fri, 10 Oct 2008 22:46:03 +0000 http://www.realbloggingtips.com/?p=1420#comment-2245 Thanks for info on this. I need a robots.txt. I will do more research on this! Thanks for info on this. I need a robots.txt. I will do more research on this!

]]> By: Mark Sierra at MeAndMyDrum.com http://www.realbloggingtips.com/you-need-a-custom-robotstxt-so-lets-make-one/comment-page-1/#comment-2243 Mark Sierra at MeAndMyDrum.com Fri, 10 Oct 2008 12:10:58 +0000 http://www.realbloggingtips.com/?p=1420#comment-2243 Hi Ben, thanks for helping me. The link is: http://www.accelerated-nursing.com/ Hi Ben, thanks for helping me. The link is:
http://www.accelerated-nursing.com/

]]> By: Mark Sierra at MeAndMyDrum.com http://www.realbloggingtips.com/you-need-a-custom-robotstxt-so-lets-make-one/comment-page-1/#comment-2242 Mark Sierra at MeAndMyDrum.com Fri, 10 Oct 2008 12:10:27 +0000 http://www.realbloggingtips.com/?p=1420#comment-2242 Thanks for checking! Surprisingly, that wasn't the domain I had an error on. Even Google isn't acknowledging anything wrong with the file, however, it is weird that it's one line though. I'll check into that. The site is http://www.accelerated-nursing.com/ Thanks for checking!

Surprisingly, that wasn’t the domain I had an error on. Even Google isn’t acknowledging anything wrong with the file, however, it is weird that it’s one line though. I’ll check into that.

The site is http://www.accelerated-nursing.com/

]]> By: egk http://www.realbloggingtips.com/you-need-a-custom-robotstxt-so-lets-make-one/comment-page-1/#comment-2241 egk Fri, 10 Oct 2008 08:33:58 +0000 http://www.realbloggingtips.com/?p=1420#comment-2241 BTW Ben, What is the difference between "Subscribe to comments via email" (above comment box) and "Notify me of followup comments via e-mail" (below comment box)? Does the first option send you an email for every comment on BPP, even if you didn't comment on that post? BTW Ben,
What is the difference between “Subscribe to comments via email” (above comment box) and “Notify me of followup comments via e-mail” (below comment box)? Does the first option send you an email for every comment on BPP, even if you didn’t comment on that post?

]]> By: egk http://www.realbloggingtips.com/you-need-a-custom-robotstxt-so-lets-make-one/comment-page-1/#comment-2240 egk Fri, 10 Oct 2008 08:28:16 +0000 http://www.realbloggingtips.com/?p=1420#comment-2240 Hey Ben, Haven't commented in a while, but I do read every post and must say your posts have been getting consistently better over time. Every post now adays is really good! Great job outlining the common components of robots.txt. @Mark Sierra: if you are referring to the site meandmydrum.com that you linked to then the problem is very obvious. The syntax of your robots.txt file is one very long line, with the different components running into each other without spaces. You need to break each entry into its own line. Hey Ben,
Haven’t commented in a while, but I do read every post and must say your posts have been getting consistently better over time. Every post now adays is really good! Great job outlining the common components of robots.txt.

@Mark Sierra: if you are referring to the site meandmydrum.com that you linked to then the problem is very obvious. The syntax of your robots.txt file is one very long line, with the different components running into each other without spaces. You need to break each entry into its own line.

]]> By: Big Ben Patton http://www.realbloggingtips.com/you-need-a-custom-robotstxt-so-lets-make-one/comment-page-1/#comment-2239 Big Ben Patton Fri, 10 Oct 2008 07:51:59 +0000 http://www.realbloggingtips.com/?p=1420#comment-2239 Give me a link to the exact site in question and I will check it out and let you know. Give me a link to the exact site in question and I will check it out and let you know.

]]> By: Mark Sierra at MeAndMyDrum.com http://www.realbloggingtips.com/you-need-a-custom-robotstxt-so-lets-make-one/comment-page-1/#comment-2238 Mark Sierra at MeAndMyDrum.com Fri, 10 Oct 2008 04:52:51 +0000 http://www.realbloggingtips.com/?p=1420#comment-2238 Hi Ben, This is very timely because I've had a question come up recently that I cannot answer regarding one of my robots files, but I'm hopeful you can provide some insight. I'm using Google Webmaster Tools for all my WP blogs and sites. When I go to Web Crawl for one of those sites, there's a topic there for "Errors for URLs in Sitemaps". The details columns says, "robots.txt unreachable". Why is that? I can access it just fine if I type in the path to it. I should point out that it's a text file and reiterate that I'm not using a plugin for generate it. This is a static, one-page HTML site. So I don't know if that has anything to do with it. Any help you can offer is very much appreciated. :) Hi Ben,

This is very timely because I’ve had a question come up recently that I cannot answer regarding one of my robots files, but I’m hopeful you can provide some insight.

I’m using Google Webmaster Tools for all my WP blogs and sites. When I go to Web Crawl for one of those sites, there’s a topic there for “Errors for URLs in Sitemaps”. The details columns says, “robots.txt unreachable”. Why is that? I can access it just fine if I type in the path to it.

I should point out that it’s a text file and reiterate that I’m not using a plugin for generate it. This is a static, one-page HTML site. So I don’t know if that has anything to do with it.

Any help you can offer is very much appreciated. :)

]]> By: MLRebecca http://www.realbloggingtips.com/you-need-a-custom-robotstxt-so-lets-make-one/comment-page-1/#comment-2237 MLRebecca Thu, 09 Oct 2008 21:54:28 +0000 http://www.realbloggingtips.com/?p=1420#comment-2237 This is so helpful! I have always wondered what the Robots.txt was for, and now I know. I'm definitely going to make this a project of mine. I love reading blog posts that teach me something about blogging that I didn't already know! Thanks for posting! This is so helpful! I have always wondered what the Robots.txt was for, and now I know. I’m definitely going to make this a project of mine. I love reading blog posts that teach me something about blogging that I didn’t already know! Thanks for posting!

]]> By: Big Ben Patton http://www.realbloggingtips.com/you-need-a-custom-robotstxt-so-lets-make-one/comment-page-1/#comment-2236 Big Ben Patton Thu, 09 Oct 2008 19:11:15 +0000 http://www.realbloggingtips.com/?p=1420#comment-2236 No problem at all really, I noticed while searching myself for a clear and concise easy to read answer that none existed that told me everything that I needed to know for my blog. I also wonder how many blog owners are completely unaware of what this and what it can do for them. No problem at all really, I noticed while searching myself for a clear and concise easy to read answer that none existed that told me everything that I needed to know for my blog.

I also wonder how many blog owners are completely unaware of what this and what it can do for them.

]]> By: M. Pence http://www.realbloggingtips.com/you-need-a-custom-robotstxt-so-lets-make-one/comment-page-1/#comment-2235 M. Pence Thu, 09 Oct 2008 12:11:25 +0000 http://www.realbloggingtips.com/?p=1420#comment-2235 Ben, Thank you so much for this article. This is something that has been on my list of things to Google and get too when I can--learning about robots.txt, but you've pretty much saved me the basic trip--that, and you've explained in a manner that has completely made sense to me. As an easily confused beginner, that has to be one of the most important things for me; finding an article that explains it exactly and easily for me to pick up--so thanks for that! I wonder how many other blog owners aren't aware of this? Ben,

Thank you so much for this article. This is something that has been on my list of things to Google and get too when I can–learning about robots.txt, but you’ve pretty much saved me the basic trip–that, and you’ve explained in a manner that has completely made sense to me.

As an easily confused beginner, that has to be one of the most important things for me; finding an article that explains it exactly and easily for me to pick up–so thanks for that!

I wonder how many other blog owners aren’t aware of this?

]]>