28 Feb '13, 4pm

Have You Considered Privacy Issues When Using Robots.txt & The Robots Meta Tag?

Have You Considered Privacy Issues When Using Robots.txt & The Robots Meta Tag?

Understanding the difference between the robots.txt file and Robots <META> Tag is critical for search engine optimization and security. It can have a profound impact on the privacy of your website and customers as well. The first thing to know is what robots.txt files and Robots <META> Tags are. Robots.txt Robots.txt is a file you place in your website’s top level directory, the same folder in which a static homepage would go. Inside robots.txt, you can instruct search engines to not crawl content by disallowing file names or directories. There are two parts to a robots.txt directive, the user-agent and one or more disallow instructions. The user-agent specifies one or all Web crawlers or spiders. When we think of Web crawlers we tend to think Google and Bing, however, a spider can come from anywhere, not just search engines, and there are many of them crawling the Interne...

Full article: http://searchengineland.com/have-you-considered-privacy-i...

Tweets

So we shipped a page explaining what it's like to work with us @envylabs. It's responsive and has moving robots.

So we shipped a page explaining what it's like ...

envylabs.com 27 Feb '13, 3pm

Tell us your plans, your goals, your deadlines, and your budget; we want to know it all. In this step, we explore our comp...

Google Appending Brand In Front Of Search Resul...

seroundtable.com 28 Feb '13, 2pm

Typically, when Google shows a clickable headline in the search results, it pulls from your title tag meta data in the sou...

AJW shares 10 ways to avoid or deal with stomach problems during ultras.

AJW shares 10 ways to avoid or deal with stomac...

irunfar.com 02 Mar '13, 10am

4. I eat all solid food during the first 50 miles so that the gels taste new and different during the second 50. I usually...

Your government lies to your reflexively, thoug...

ace.mu.nu 07 Mar '13, 8pm

On Twitter, Ibrahim is quite blunt regarding her views. On July 18 of last year, after five Israeli tourists and a Bulgari...

Using Authority with a Single-Sign-On App: I ju...

rubyflow.com 28 Feb '13, 1pm

Using Authority with a Single-Sign-On App Posted by NathanLong on February 28, 2013 — 0 comments I just blogged a general ...

Disadvantages of Using a HDB Loan

thefinance.sg 28 Feb '13, 4pm

A way to circumvent having to use the CPF Ordinary Account balances to pay for the flat is to invest it before you make th...

Privacy Vs Censorship: Google, Spanish Governme...

searchengineland.com 27 Feb '13, 6pm

In a test case that could have significant implications for Google throughout Europe the company faced off against the Spa...

Using the Web to Motivate Customers to Save Energy - new article by me and @ESourceKim in @HomeEnergyMag

Using the Web to Motivate Customers to Save Ene...

homeenergy.org 05 Mar '13, 5pm

Can an online game that teaches you how to save energy ever be as popular and addicting as Angry Birds? Are people going t...

Facebook Shares How People Are Using Graph Sear...

searchengineland.com 28 Feb '13, 5pm

Getting people to use Facebook Graph Search other than finding people could help Facebook build a search business that cou...