30 Jun '14, 1pm

The robots.txt Celebrates 20 Years Of Blocking Search Engines

The robots.txt Celebrates 20 Years Of Blocking Search Engines

Today is the 20th anniversary of the robots.txt directive being available for webmasters to block search engines from crawling their pages. The robots.txt was created by Martijn Koster in 1994 while he was working at Nexor after having issues with crawlers hitting his sites to hard. All major search engines back then, including WebCrawler, Lycos and AltaVista, quickly adopted it and even 20 years later, all major search engines continue to support it and obey it. Brian Ussery posted on his blog about the 20-year anniversary and documented the most common robots.txt mistakes he has seen over his SEO tenure. It is well worth scanning through, because the robots.txt, if implemented wrong, can be severely detrimental to your rankings and search marketing success. For more details on the robots.txt see robotstxt.org , Wikipedia and Google support . Also, make sure to check out ...

Full article: http://searchengineland.com/robots-txt-celebrates-20-year...

Tweets

Take an extra 20% off clearance items!

payless.com 02 Jul '14, 9pm

Your browser's Javascript functionality is turned off. Please turn it on so that you can experience the full capabilities ...

@MissAnnalieza start with the robots.txt file. ...

searchengineland.com 29 Jun '14, 9am

After you make your file, upload it to the root directory of your web site. If you don’t know what that is, find someone w...

What Local Business Leaders Must Know About Sea...

searchengineland.com 04 Jul '14, 5am

When I write about SEO and business success, I often realize that I've touched upon a point that merits a column of it's o...

The energy paypack for a 2-megawatt wind turbine that lasts over 20 years is... 5-8 months

The energy paypack for a 2-megawatt wind turbin...

treehugger.com 30 Jun '14, 6pm

Talk about a great return on investment! Some people who are against renewable energy - often when you follow the money yo...

Daily Search Forum Recap: July 1, 2014: Here is...

seroundtable.com 01 Jul '14, 8pm

AP's New Robot Writers; What Will Google Think? The Associated Press, AP, announced they will be using algorithms, machine...

What Local Business Leaders Must Know About Sea...

searchengineland.com 03 Jul '14, 1pm

When I write about SEO and business success, I often realize that I’ve touched upon a point that merits a column of it’s o...

Three Years of Bucket Living: Why I Travel via ...

runawayjuno.com 01 Jul '14, 6am

I left my engineering office for good . I wasn’t sure what was going to happen in my future, but I decided to stop being a...

Search Engine Marketing Helps Drive Back-to-Sch...

searchengineland.com 02 Jul '14, 1pm

We welcome constructive comments and allow any that meet our common sense criteria. This means being respectful and polite...

Google Killer Robots.txt File Blocks Terminator...

seroundtable.com 04 Jul '14, 1pm

Google posted a new robots.txt file at google.com/killer-robots.txt that is designed to help secure the Google co-founders...

Search & Apply for Careers online

sedo.co.uk 06 Jul '14, 10pm

Your search did not return any results. Get started by browsing Featured Domains or Browse By Category - we offer over 2,0...

Search Engine Adventures: Google is Misunderstood. via @seoinc

Search Engine Adventures: Google is Misundersto...

seoinc.com 07 Jul '14, 11am

Let’s have some fun with our favorite search engines! At SEO Inc., we like to think that Google, Bing, and Yahoo live pret...

'Google removing author pictures from search: Your input?' by @DevSEOChat

'Google removing author pictures from search: Y...

seochat.com 02 Jul '14, 6pm

Maybe it was negatively affecting AdWords CTR. Google wants more eyes on knowledge graph. Now that x number of people are ...