31 Mar '14, 12pm

Spider Large Sites With Screaming Frog

Spider Large Sites With Screaming Frog

ScreamingFrog's a great tool and if you know how to use it you can make it incredibly powerful. Take for example spidering large sites. The site I'm using in this example, currently has over a million pages indexed within Google – now in relation to some of the sites on the internet that's a drop in the ocean but still you wouldn’t call it small by any means, yet time and time again I see/hear of people struggling to crawl a site of this size. I'm going to run you through how you might spider this site, first things first you'll need to identify how the site is set up and where all these URLs are coming from, in this case it's an ecommerce site so the usual suspects are faceted navigation, pagination and site search; among others but these are the main offenders. Once you have identified these, generally speaking they are a bulk-fix or at least a few bulk fixes for section...

Full article: http://www.davidnaylor.co.uk/spider-large-sites-screaming...

Tweets

A (Pretend) Open Letter From Google To Local Directory Sites

A (Pretend) Open Letter From Google To Local Di...

searchengineland.com 31 Mar '14, 3pm

As you know, it is our stated goal to organize the world’s information while simultaneously doing no evil. And, while you ...

Google Penalized Large Guest Blogging Network |...

submitshop.com 30 Mar '14, 4pm

Google today penalized “MyBlogGuest”, a large guest blogging network being used by publishers for guest blogging. Matt Cut...

Climbing The Ladder To Guest Posting On Tier 1 ...

searchengineland.com 31 Mar '14, 1pm

Content marketing is all the rage these days and, if you are pursuing guest posts as part of your link-building strategy, ...

How Large Cities Can Advance a Renewable-Powered #EV Infrastructure - @ELPCenter @SEIA @evobsession @E…

How Large Cities Can Advance a Renewable-Powere...

ecowatch.com 30 Mar '14, 1pm

The recent Chicago Auto Show was a great place to see new car models and automakers’ futuristic designs. Until recently, f...

Large LNG Explosion Displaces Hundreds in Washington | EcoWatch via @ecowatch

Large LNG Explosion Displaces Hundreds in Washi...

ecowatch.com 01 Apr '14, 1pm

Nearly 1,000 area residents and agricultural workers were displaced Monday following a liquefied natural gas (LNG) explosi...

Nike Air Force 1 High "Poison Dart Frog"

Nike Air Force 1 High "Poison Dart Frog"

sneakernews.com 30 Mar '14, 4pm

Kobe Bryant proved that PDFs aren’t just for tech sheets with a handful of ‘Poison Dart Frog’ colorways on the Zoom Kobe V...