What Does Your Website Look Like to a Crawler?
You’ve read a lot about search engine optimization for your website (SEO), and you know that creating a website that’s easily indexed is paramount to being listed on the search engine results pages (SERPs). But:
Wouldn’t it be great to see your website from the crawlers’ point of view? To see your pages without any bells and whistles, and easily assess the changes necessary to make your site more friendly to the search engines?
It can be done, and it’s surprisingly simple! There are some great tools available that display a web page simply by its text content, which is what the crawlers see. Some of these free tools can be found on websites like:
Identifying Non-Text Content
Even though crawling technology has advanced, bots tend not to index non-text content. This means content like images and videos, or even Flash files and Java plug-ins, among others, aren’t indexed. These elements certainly do make the user experience more interesting, but a balance needs to be struck between style and appeal, thus ensuring your indexing potential. It's very common for websites to have problems with indexable content, so using these tools to check your pages is a great idea.
Check for These 5 Red Flags
Make sure the pages you’re building are visible to the engines by taking the URL of any website and plugging it into the tool. Once you’ve done that, you’ll notice how bare-bones it is, and how some of your content isn’t showing. That is what the crawlers see! Now that you can see what the crawlers see, here are some things to check:
- VISUAL FILE NAMES
We’ve talked a lot about remembering to name your image and video files, and if you can use a keyword to do so, even better! Instead of using the image with its numerical or nonsensical name, take the chance to give it a name that can be indexed and later retrieved for SERPs. - ONLINE FORMS
Do you have content that’s only accessible after users submit an online form? It’s probable that the content behind that login is invisible. It still might be worth it to keep the pages hidden and build your contact base, but weigh the pros and cons. - SITE NAVIGATION
The structure of the site needs to be crawlable. 61 Design Street explains it like this— “Most of the websites make a mistake of structuring their navigation in a way that search engines cannot access, obstructing their capability to get pages listed in the search engines’ indexes.”
Imagine, for instance, a page that’s been created for a project and the link is distributed for a promotion, but the page isn’t accessible through the site navigation. That page and the valuable content it holds will not be indexed. Pay attention to site architecture. - JAVASCRIPT LINKS
Are you using JavaScript for links? Consider replacing them with—or adding—HTML links. - LINK LISTING
Crawlers like content, but won’t necessarily see multiple links listed on a page as indexable content. Add helpful links to logical places on the site, and remember to look at your site architecture to make sure every page is able to be indexed. (See #3)
Take some time this week to view some of your web pages using these tools. Plan to work with your team to start making some of these changes, which will improve your user experience as well as your sites indexability.
What did you notice when you plugged your URL into the tool? Leave us a comment and tell us where you’re going to make site improvements.
Update your next website with a new .BIBLE domain! Register today.