How can I possibly relate spiders to chocolate cake?  When I hear a report about how information overload can affect our abilities to make good choices, they were the first thing that came to mind.

Website crawl and search engine crawling by search engine spiders

The study, conducted by Stanford University professor, Baba Shiv went like this:  Several dozen subjects were split into two groups.  The first group received a 2 digit number, while the second group received a 7 digit number.  They were then instructed to walk down the hall, enter another room and recite that number back.  That was all they thought they would need to do.  So imagine, the test subject is walking down the hallway, recalling the number over and over again so they can recite it back.  Then, out of nowhere, they encounter a person that asks if they would like a snack as gratitude for helping with the study.  The person is holding two different snack options that they can choose from.  One snack, considered not healthy (although some might argue) is chocolate cake, the other snack, a healthy bowl of fruit salad.

The results are where it gets interesting.  Since the subjects given the 7 digits numbers were using up more brain power to remember the number as they walked down the hallway, they were two times more likely to make what is considered a poor decision and pick the chocolate cake.  The people given the 2 digits almost always choose the fruit.  Professor Shiv referred to this as a “cognitive load”…where “the prefrontal cortex is so overtaxed, that all it takes is five extra bits of information before the brain starts to give in to temptation”.  So what does this have to do with search engine spiders?

When Spiders Website Crawl

Of course as online marketers, we always start to put everything into terms of how it relates to our field.  In this case, I was instantly correlating it to search engine crawling.  When the search engine spiders come into your site to crawl it, they do so in a very methodical way.  You can say they are a bit anal – they like structure and order because their crawling is automated and efficient.  For that reason, it is crucial to have order and structure on your site, i.e. give the spiders 2 numbers, not 7 numbers.

When we talk about SEO and how it relates to a website crawl, we often make fun of these spiders, calling them stupid and dumb.  Instead of blaming the search engine spiders for indexing issues, I would like to propose giving them benefit of the doubt and just consider them to be on cognitive overload.

Search Engine Crawling Made Easier

When we present goobledygook to these spiders, it is essentially like feeding them chocolate cake, which will only distract them from the job they have set out to do.  Sites that are strictly visual eye candy are great for humans, but terrible for the search engine spiders.  When the spiders become confused, i.e. they want the chocolate cake, they don’t want to try to index your site or figure out what it is about.  Flash images without no script, Javascript slide shows, and the lack of relevant content that is often missing from these types of websites, all contribute to spider cognitive overload.

Search engine spiders represented as Louise Bourgeois spider sculpture

Louise Bourgeois Spider Sculpture

I will not toot my horn again about the importance of content, you can read more about online content creation from my last post, but I will stress again that any opportunity you have on your website to make it easily crawled by the search engine spiders, and therefore indexed, is an opportunity to improve your sites ranking.  So, instead of confusing the spiders and serving them up an unhealthy heap of chocolate cake, follow these simple steps:

The search engine spiders will be happier and healthier and it will have been a pleasure for them to do the search engine crawling of your site, all while you have increase search engine visibility for your website.

The rational and emotional parts of the brain will always continue to be at war with each other.  While we often have little control over our own temptations and it becomes difficult to give our brains the room and space to be able to make good choices, we do have control over what the search engine spiders see when they crawl our websites.  Feed them well and let them do their jobs easier.  So back to chocolate…any other ways to relate chocolate cake to SEO?

Related Posts
Related Case Studies

Dragon 360 Logo