Welink Agencies Welink Agencies

SEO: What does crawling mean exactly? An overview of on-page SEO and indexing

Welink Agencies

Welink Agencies

On and Off


SEO, or Search Engine Optimisation, is the process of boosting your website’s ranking on a search engine’s results page. There are steps you can take both on and off your website to improve your SEO, gain more visibility, and subsequently increase traffic and lead conversion on your website.


Related articles


What is SEO? What you should know about online visibility

White Hat vs. Black Hat SEO: The guidelines for ethical optimisation

SEO: More than a 2021 trend

How to improve your webite's SEO in 3 Easy Steps

On-page vs. Off-page SEO at a glance

There are two types of SEO: on-page and off-page.


On-page SEO is the optimisation that takes place on the web page and includes elements like content, UX, data, and code.


Off-page SEO is the optimisation that takes place outside of the website and whose purpose is to increase traffic and lead conversions on the website. It includes such techniques as social media advertising and link building.


 However, this article is about one on-page SEO element in particular, and that’s crawling.

What’s crawling? 

Crawling refers to a search engine’s ability to find and understand a web page. A crawlable website is a website that’s accessible to users and one that the search engine can find while it’s searching the Internet for content.


In fact, Google sends bots or spiders to crawl pages and in doing so checks if their content is available, whether they are original pages or duplicates, whether the links to different pages are dead or functional… In short, Google wants to 


  • See if the website is operating 
  • See if it’s adhering to the Google guidelines
  • Check its legitimacy


Having a crawlable page means that you’re successfully communicating with Google and letting it know what your site is about, that it’s useful, and ultimately that it deserves a higher ranking on its results page. 


The fewer dead links you have on your website, the better. Similarly, the more targeted and relevant your content is, the more crawlable your pages are. You want to increase your website’s crawlability to make it as accessible and visible as possible to your target audience.


One thing to keep in mind is that crawlers are very consistent, much more so than users. This means that if you have a lot of dead links on your website, they will find them. This goes hand in hand with user experience: dead links don’t only alert Google that the site is not fully optimised, it also communicates to Google that it has a less than perfect UX.


It’s important to know that your website should first and foremost be addressed to your human audience and not to your search engine. The good news is that once you take your audience and their user experience into consideration and build your website (in content and code) accordingly, Google is likely to boost your ranking on the results page.

Crawling and on-page SEO

As you can probably tell, the elements of on-page SEO are not really separate from one another. Good content and UX improve your website’s crawlability, and this in turn signals to Google that your website deserves a high ranking.


Think of this process like a chain of successive events set off by crawling to look for dead links, relevant content, and good overall coding. Crawling allows Google to 

  1. Discover your website, 

which then allows it to 

  1. Understand it and
  2. Regulate its position amongst the search results.

The discovery step happens as a result of your website being easy to navigate and well coded. This means that your website should be secure, fast to load, and in adherence to Google’s rules (no plagiarism or any other “black hat” SEO tactics). 


Importantly, good coding also includes making your website’s data (its different pages, its different sections within a single page) well-structured


The criterion for this is a standardised format that’s universally recognisable by Google, according to which a website should be structured. 


The most important thing to keep in mind in terms of structure is that your website should be coherent with specific and targeted content and pages that link well to one another. Then, Google will be able to understand what your website is about by putting it into a particular context.


Let’s say your website is a travel blog with a section dedicated to your favourite travel-related products in which you provide affiliate links to commercial websites. If you structure your pages and your sections correctly you will communicate to Google that your website serves two distinct purposes, informational and commercial. 


By structuring your website properly, you also let Google know you care about your visitors’ experience and provide them with content that’s relevant to their specific needs. Google will then be able to understand what your website’s purpose is, see that users are finding your content useful and your website easy to navigate, and classify it on its results page accordingly.

Crawling and Indexing

Once Google finds your website in its crawling process, it moves on to index your content, i.e. list it on its database. When your website is indexed on Google’s database, it can then appear amongst the search results.


An indexed page is a page that Google deemed good enough to serve as a search result. Once your website is on Google’s index, you can then start seeing how it moves up and down the results page and finetune your SEO tactics.

Crawling and Ranking

The final step in this process is what Google decides to do with your content in terms of classification. First it finds your page, then it indexes it on its database, and finally it ranks it according to several factors:


  • How relevant the content is for users
  • How good the navigation is
  • How it contributes to the existing content
  • How well users are responding to it…


The list goes on, and this is where all the other elements of SEO, both on-page and off-page, come into play. 


You should therefore consider crawling to be the first step in your quest to increase your visibility to Internet users and to your target audience more specifically. If you want to know more about SEO strategies in general and improve your page’s crawlability and rankings, check out our Ebook. 

Are you an agency?

List your agency on Welink Agencies.