Comments on: You Need A Custom Robots.txt, So Lets Make One.

By: Part Time Jobs

Part Time Jobs — Fri, 11 Nov 2011 21:18:48 +0000

ok fine this is a very important for any site who run
Part Time Jobs´s last blog ..Online jobs with google

By: domain name web hosting

domain name web hosting — Mon, 24 Oct 2011 17:16:27 +0000

great issues altogether, you just received a new reader. What might you suggest in regards to your submit that you simply made a few days in the past? Any positive?

By: article wizard

article wizard — Tue, 18 Oct 2011 19:51:56 +0000

Nice post. I used to be checking constantly this blog and I am inspired! Extremely helpful information particularly the final section I care for such info much. I used to be seeking this particular information for a very lengthy time. Thank you and good luck.

By: Islamic Website

Islamic Website — Mon, 08 Aug 2011 17:38:47 +0000

Robots.txt file is very important as per giving access to spiders, a big thanks for summarizing the whole topic.

By: Social Networking Website

Social Networking Website — Fri, 27 May 2011 05:02:21 +0000

i had really gained awesome information, i will use this for my website thanks for sharing

By: DJ ARIF@Islam Blog

DJ ARIF@Islam Blog — Mon, 18 Apr 2011 13:08:06 +0000

I’ve updated my robots.txt file by watching yours, would you like to see that and suggest me to what to do with it next? waiting for your reply….
DJ ARIF@Islam Blog´s last blog ..Importance Of Robotstxt File In Search Engine Optimization

By: Narzuty

Narzuty — Thu, 02 Apr 2009 09:03:40 +0000

robots.txt is an extremely important part of every blog. I’m surprised to see so many people in comments that were not aware of its functions.

By: Beijinger

Beijinger — Sat, 07 Feb 2009 15:55:22 +0000

Request-rate: 1/5
Visit-time: 0600-0845

I was having some issue with certain bots which is eating my bandwich , glad to know above commands.

By: Ajith Edassery

Ajith Edassery — Thu, 23 Oct 2008 19:22:04 +0000

Though I was reasonably well-versed with the robot.txt file, I wasn’t aware of those extended keyword set.

Thanks a lot,

Cheers,
Ajith

By: Ben Moreno

Ben Moreno — Mon, 13 Oct 2008 00:07:59 +0000

Hey this is cool. Thanks for the info man. I need to do this for my site. My name is Ben too! I like your blog. Nice design.

By: Sunday Hyper-Linkage & Updates | The Net Fool

Sunday Hyper-Linkage & Updates | The Net Fool — Sun, 12 Oct 2008 07:31:24 +0000

[...] Benjamin Patton – You Need a Custom Robots.txt, So Let’s Make One [...]

By: Armen Shirvanian

Armen Shirvanian — Sun, 12 Oct 2008 06:11:36 +0000

This is wonderful because it allows me to see the specifics of a file that is used ubiquitously. The internals of the file are quite easily altered, and you have presented them smoothly.

By: Donace

Donace — Sat, 11 Oct 2008 21:46:47 +0000

One thing I will chip in here is that following robot.txt and stuff like do-follow no-follow is all ‘recommendations’ for search engine guidelines.

Yahoo is notorious is disobeying robot.txt files and the such.

By: John @ Maryland Real Estate

John @ Maryland Real Estate — Sat, 11 Oct 2008 19:27:49 +0000

good post Ben,

it’s also a good idea under the User-agent to disallow a bunch of the site scraping scripts out there. You can really save you a bunch of bandwidth if you have a large site. Just make sure not to disallow any of the important search engine spiders of course.

By: Dinh Trung

Dinh Trung — Sat, 11 Oct 2008 14:02:13 +0000

Hi Benjamin,

Sorry to ask you this question, since I do not know to buil a robot.txt for mine !!

In my case, and my site (kienthucdulich.info), i faced problem with google sitemp, sometimes.

Pls help to clarify !

Thanks and best regards,
Dinh Trung
http://www.kienthucdulich.info

By: Big Ben Patton

Big Ben Patton — Fri, 10 Oct 2008 22:46:56 +0000

more or less yes one will send the comments one will notify you that their are some comments.

By: Kevin@Great Wall of China Facts

Kevin@Great Wall of China Facts — Fri, 10 Oct 2008 22:46:03 +0000

Thanks for info on this. I need a robots.txt. I will do more research on this!

By: Mark Sierra at MeAndMyDrum.com

Mark Sierra at MeAndMyDrum.com — Fri, 10 Oct 2008 12:10:58 +0000

Hi Ben, thanks for helping me. The link is:
http://www.accelerated-nursing.com/

By: Mark Sierra at MeAndMyDrum.com

Mark Sierra at MeAndMyDrum.com — Fri, 10 Oct 2008 12:10:27 +0000

Thanks for checking!

Surprisingly, that wasn’t the domain I had an error on. Even Google isn’t acknowledging anything wrong with the file, however, it is weird that it’s one line though. I’ll check into that.

The site is http://www.accelerated-nursing.com/

By: egk

egk — Fri, 10 Oct 2008 08:33:58 +0000

BTW Ben,
What is the difference between “Subscribe to comments via email” (above comment box) and “Notify me of followup comments via e-mail” (below comment box)? Does the first option send you an email for every comment on BPP, even if you didn’t comment on that post?