Crawl Maps for SEO | Sitebulb

Sitebulb MCP Coming Soon! Easily query audit data.Sitebulb MCP Coming Soon! Join the waitlist now.

Crawl Maps for SEO

\ Patrick Hathaway](/content/resources/authors/patrick-hathaway/index.html)

Updated April 6, 2021

One of the features in Sitebulb that people seem most excited about is Crawl Maps, which, if you've not seen them yet, are interactive visualizations of your website architecture.

And they look awesome.

Example Crawl Maps

Basically, Sitebulb will take your crawl data and map it out using a force-directed crawl diagram, displaying URL 'nodes' as dots, with links represented by the connecting lines ('edges').

The result is an interactive graph that can be incredibly useful for technical SEO audits, often revealing patterns in the site architecture that you'd struggle to spot otherwise. Visual site mapping can be a great way to take a high-level view of your website's structure.

I'll stop talking about it, and show you some examples instead.

Flat site architecture

This is like a classic 'SEO friendly' flat website architecture, with almost every page no more than 2 clicks from the homepage. You're looking at the big green dot in the middle as the homepage, then the smaller green dots are at crawl depth/level 1 (i.e. they are linked to from the homepage).

The orange dots are at a crawl depth of 2. In this instance, they represent product URLs, and the depth 1 nodes are sub-category URLs.

If you haven't guessed already, this is an ecommerce site with a pretty extensive mega menu.

Duplicate content

Not an easy one to digest at first glance, but this is a significant case of duplicate content. In the upper left is the homepage (big green circle) and all the proper site content. But this also links off to two similar structures down at the bottom...duplicate homepages and in fact a duplicate website (twice over!).

Pagination chains

The long whip thing coming out the side is a string of paginated pages.

I've got another example of pagination actually, this one even weirder:

That one was actually caused by some legacy pagination markup that wasn't even being used anymore! (hence the 'bare branches' with nothing coming off them).

Content Explosion

I like this one a lot. Shared by one of our beta testers, Gareth Edwards from Wolfgang Digital. It shows a relatively small 'product' site, with a large and complex blog.

The homepage is the big green circle at the bottom, and everything coming down off that is the marketing site which lists their products and services. The little green dot in the centre is the blog homepage, with posts, sections, categories and pagination coming off that.

Content marketing FTW.

Interactivity

At this point you might be asking yourself how I'm so sure what I'm looking at. That's because of the bit that doesn't come across so well with static images - the Crawl Maps are also interactive.

If any particular 'node' piques your interest, you can hover over it to find out which URL it represents, along with data about its crawl depth and internal inlinks.

Attribution & Learning More

It would be remiss of me to not give credit where it's due, to the innovative marketers who inspired us to build this feature in the first place.

Finding a way to visualize website architecture has been pretty much at the top of our wish list when building the tool, and the method we were most keen to replicate was the one we first saw demonstrated by Ian Lurie, in his article ' SEO Using Force-Directed Diagrams'.

As you can see, the crawl maps produced by Sitebulb are very similar to Ian's, and I don't believe for a second that we'd have been able to come up with something this awesome on our own, so thank you Ian (and sorry for nicking your idea!).

To produce his visualizations, Ian used Gephi, and we were actually first introduced to Gephi many moons ago by Justin Briggs, when he set about visualizing external links in his blog post ' How to Visualize Open Site Explorer Data in Gephi.'

This was where we first learned about the concept of using graph theory to represent link data, and going back over his old posts helped us solidify our ideas for implementing Crawl Maps in Sitebulb.

As ever, we are extremely grateful to the wonderful members of the SEO community for consistently sharing such inspirational ideas.

The posts I linked to above can also serve as education pieces, if you are interested in learning more about this kind of data visualization. We have also published our own ' [Crawl Maps FAQ](/content/resources/guides/crawl-map-examples/# "Crawl Maps FAQs"/index.html)' which gives some more specific insight into how Sitebulb's Crawl Maps are built.

Crowd-sourced Examples

Below you will finda collection of Crawl Map examples from the community, which have been grouped together loosely by yours truly, with their own colourful commentary included as extra sauce.

Manipulative

We've only got one of these so far, but it's a doozy:

Twitter Embed

Not found

@sitebulb #sitemaps

This sitemap is from a black hat site where they were framing and hiding high ranking sites and injecting their own SEO services into. Bad boys.

Site now gone. pic.twitter.com/L8wyB6KZfl

— Simon Cox (@simoncox) February 15, 2018

Content Mess (Where to begin...?)

Crawl Maps are perhaps best for quickly showing you that you've got a real mess to untagle.

Twitter Embed

Crawl Maps for SEO

Table of contents:

Example Crawl Maps

Flat site architecture

Duplicate content

Pagination chains

Content Explosion

Interactivity

Attribution & Learning More

Crowd-sourced Examples

Manipulative

Content Mess (Where to begin...?)

Before and Afters

Content Explosion

Extreme Pagination

Videos

Trolling

Nice site structure (huh? what?!)

Calling all Crawl Map fanatics!

Create Your Own Beautiful (or ugly) Crawl Maps

Articles for every stage in your SEO journey. Jump on board.