Proper SEO and the Robots.txt File

When it comes to SEO , most people understand that a website must have content, "search engine friendly" site architecture/HTML, and meta data (title tags and meta descriptions ). Another meta element, if implemented incorrectly, that can also trip up websites is robots.txt. I was recently reminded of this while reviewing the website of a large company that had spent considerable money on building a mobile version of their website, on a sub-directory. That’s fine, but having a disallow statement in their robots.txt file meant that the website wasn’t accessible to search engines (Disallow: /mobile/) Let’s review how to properly implement robots.txt to avoid search ranking problems and damaging your business, as well as how to correctly disallow search engine crawling. What is a Robots.txt File? Simply put, if you go to domain.com/robots.txt, you should see a list of directo...

Full article: http://searchenginewatch.com/article/2064412/Proper-SEO-a...

Tweets

@bphall » 10 Jul '12, 7pm
RT @haydens30: A good starting point - Proper #SEO and the Robots.txt File [link]
@seotweetz » 08 Jul '12, 6pm
RT @paulculshaw: A good starting point - Proper #SEO and the Robots.txt File [link] via @sewatch
@paulculshaw » 08 Jul '12, 6pm
A good starting point - Proper #SEO and the Robots.txt File [link] via @sewatch
@seoaus2010 » 26 Jun '12, 5am
Our clients barely understand the Robots, so we help | Proper SEO and the Robots.txt File by @MarkJackson [link] via @sewatch
@AlexMcCormick27 » 25 Jun '12, 3pm
RT @Ignition_SEO: Proper SEO and the Robots.txt File: When optimizing a website, don't disregard the power of robots.txt - [link]
@NickRink » 25 Jun '12, 1pm
Quick refresher on Robots.txt [link] #seobasics
@eLegacy_India » 24 Jun '12, 8pm
Proper #SEO and the #Robots.txt File. When it comes to SEO, most people understand that a website must have... [link]
@WebandSEOtips » 23 Jun '12, 5pm
Proper SEO and the Robots.txt File [link] ^NM
@DigTrafficSquad » 23 Jun '12, 9am
Great article on the proper use of Robo.txt files - [link]
@poiskbystro » 23 Jun '12, 5am
Proper SEO and the Robots.txt File by @MarkJackson [link] с помощью @sewatch
@sewatch » 22 Jun '12, 4pm
@neilatong here ya go [link]
@webakit » 21 Jun '12, 4am
RT @DigitalWebTrend: Proper SEO and the Robots.txt File [link]
@SEWseo » 21 Jun '12, 3am
RT @dirkdupont: Does your website have a proper robots.txt file for SEO? [link]
@iamSamuel__ » 20 Jun '12, 11pm
RT @ErikFoss8: Proper SEO and the Robots.txt File [link]
@seotweetz » 20 Jun '12, 5am
RT @worldsallinfo: Proper #SEO and the Robots.txt File by @MarkJackson [link] via @sewatch #World #Business
@worldsallinfo » 20 Jun '12, 5am
Proper #SEO and the Robots.txt File by @MarkJackson [link] via @sewatch #World #Business
@marketingAmity » 19 Jun '12, 6pm
Pay attention, don't set yours up the wrong way: Proper SEO and the Robots.txt File by @MarkJackson [link] via @sewatch #li
@seotweetz » 19 Jun '12, 9am
RT @an_in_di_ta: Proper #SEO and the #Robots.txt File by @MarkJackson [link] via @sewatch
@seotweetz » 19 Jun '12, 8am
RT @invinciblesaad: Proper #SEO and the Robots.txt File by @MarkJackson [link] via @sewatch
@BuzzOutsource » 19 Jun '12, 8am
RT @CPC_Andrew: Proper SEO and the Robots.txt File by @MarkJackson [link] via @sewatch #Ecommerce
@CPC_Andrew » 19 Jun '12, 7am
Proper SEO and the Robots.txt File by @MarkJackson [link] via @sewatch #Ecommerce
@crossfuse » 19 Jun '12, 12am
The Do's and Don'ts of SEO Robots.txt files - [link]
@miguelsilvar » 18 Jun '12, 9pm
RT @edstorini: Proper SEO and the Robots.txt File - [link] by @MarkJackson
@seotweetz » 18 Jun '12, 9pm
RT @sarah_musselman: What do you do with robots.txt? A helpful & necessary reminder: Proper #SEO and the Robots.txt File [link] (via @sewatch)
@RavenSarahM » 18 Jun '12, 9pm
What do you do with robots.txt? A helpful & necessary reminder: Proper #SEO and the Robots.txt File [link] (via @sewatch)
@vmavalankar » 18 Jun '12, 9pm
Proper SEO and the Robots.txt File [link] #seo #sem #marketing #startup #entrepreneur #strategy #analyst #search #searchengine
@seotweetz » 18 Jun '12, 8pm
RT @LoonieGuy: Get yourself a good robot! Proper SEO and the Robots.txt File by @MarkJackson [link] via @sewatch #SEO
@LoonieGuy » 18 Jun '12, 8pm
Get yourself a good robot! Proper SEO and the Robots.txt File by @MarkJackson [link] via @sewatch #SEO
@dreamwebservice » 18 Jun '12, 5pm
Proper SEO and the Robots.txt File: However, the website had virtually no presence in the search engine results ... [link]
@seotweetz » 18 Jun '12, 4pm
RT @aseohosting: Proper SEO and the Robots.txt File via @sewatch [link] #seo
@SoDigital2012 » 18 Jun '12, 4pm
RT @SeanClark: Proper #SEO & the Robots.txt File [link]
@seotweetz » 18 Jun '12, 4pm
RT @Ouwendorp: Reading: Proper #SEO and the Robots.txt File by @MarkJackson [link] via @sewatch #in
@seotweetz » 18 Jun '12, 4pm
RT @CoryGrassell: Proper #SEO and the Robots.txt File: [link] by @MarkJackson
@globallover2 » 18 Jun '12, 3pm
Proper SEO and the Robots.txt File [link] (via @summify)
@aragop » 18 Jun '12, 3pm
Great primer on robots > Proper SEO and the Robots.txt File by @MarkJackson [link] via @sewatch
@AMFowke » 18 Jun '12, 2pm
Good Review of what and what not to do with robots.txt: Proper SEO and the Robots.txt File by @MarkJackson [link] via @sewatch
@MartinWSI » 18 Jun '12, 2pm
RT @dcarty: Good summary on....Proper SEO and the Robots.txt File by @MarkJackson [link] via @sewatch
@RepEquity » 18 Jun '12, 2pm
Search Engine Watch (@sewatch) has a great article today about the usefulness and implementation of robot.txt files: [link]
@dcarty » 18 Jun '12, 2pm
Good summary on....Proper SEO and the Robots.txt File by @MarkJackson [link] via @sewatch
@seotweetz » 18 Jun '12, 2pm
RT @OnlineOutcomes: Advice for managing Robots.txt file. Larger web teams: have a documented directory usage convention! #seo @MarkJackson [link]
@OnlineOutcomes » 18 Jun '12, 2pm
Advice for managing Robots.txt file. Larger web teams: have a documented directory usage convention! #seo @MarkJackson [link]
@ElliotTelford » 18 Jun '12, 2pm
Crawler Control: Proper SEO and the Robots.txt File by @MarkJackson [link] via @sewatch #CWR12 #dotcom
@ferviddesigns » 18 Jun '12, 2pm
RT @MirandaM_EComm: Great overview, important info - Proper SEO and the Robots.txt File by @MarkJackson [link] via @sewatch
@MirandaM_EComm » 18 Jun '12, 2pm
Great overview, important info - Proper SEO and the Robots.txt File by @MarkJackson [link] via @sewatch
@seotweetz » 18 Jun '12, 2pm
RT @stonetemple: Proper #SEO and the Robots.txt File [link] by @MarkJackson RT @SEOdojo @DuaneForrester via @sewatch
@stonetemple » 18 Jun '12, 2pm
Proper #SEO and the Robots.txt File [link] by @MarkJackson RT @SEOdojo @DuaneForrester via @sewatch
@ThoughtShiftUK » 18 Jun '12, 1pm
How to use a robots.txt effectively for SEO [link] #seo
@seotweetz » 18 Jun '12, 1pm
RT @nlakeland: Useful advice about the robots.txt file for website owners & #seo specialists: [link] (via @sewatch)
@nlakeland » 18 Jun '12, 1pm
Useful advice about the robots.txt file for website owners & #seo specialists: [link] (via @sewatch)
@proformaexpress » 18 Jun '12, 1pm
Do's and Dont's of Robots.txt : [link]