Skip to content

The 2018 Webmaster Tools Guide – Google Index, Crawl, and more

In 2010, we wrote a comprehensive guide for beginners on Google Webmaster Tools. But since then, there have been significant changes in Google Webmaster Tools.

We've updated this guide to include new ways to set up your website with Webmaster Tools, new data included in Webmaster Tools, important data you've forgotten, and ongoing monitoring of Webmaster Tools. problems could affect your rankings in the search engines.

Setting Up Your Website with Webmaster Tools

If you have not already done so, the first thing to do is to set up your website with Webmaster Tools.

To do this, go to the Google Webmaster Tools website, sign in with your Google Account, preferably the one you already use for Google Analytics.

Click the red Add property button to begin.

Then you will have to check that this site is yours.

Previously, this involved embedding code in the header of your website or uploading an HTML file to your web server.

Now, if you already have Google Analytics, you can check your site by connecting Webmaster Tools to Google Analytics.

To do this, select the option Use your Google Analytics account.

Once your site is verified, you will need to submit a sitemap if you have one.

This is a simple XML file that will tell Google Webmaster Tools which pages you have on your site.

You will find the option to add a sitemap under the "Explore" tab of your toolbar.

If you already have one, you can usually find it by typing to see it in your browser.

To create a sitemap if you do not already have one, you can use online tools such as XML Sitemaps.

If you use a website on your own domain with WordPress, you can install the Google XML Sitemaps plugin.

Once you have activated the plugin, look under Settings in the WordPress dashboard and click XML-Sitemap.

The plugin should have already generated your sitemap, so there is nothing else to do.

You will find your url at the top of the page:

Copy the link address and return to the Webmaster Tools Page

Then paste the part of the URL after the your website into the box to submit your sitemap to the Google Webmaster Tools.

You will now be able to refresh the page and view the sitemap you sent.

Tools for webmasters may take a few days to start extracting information from your website if you are setting up your website for the first time on webmaster tools.

Make sure you wait a bit, then continue to see what you can learn from the tools for webmasters.

Valuable Information in Webmaster Tools

Once you have data in Webmaster Tools, you will be able to view the following information about your website.

These are just the highlights of the new data types in Google Webmaster Tools and the most important data you should always remember from time to time.


When you visit your website in Webmaster Tools, you first go to your dashboard.

This is an overview of important data in the tools for webmasters. You can visit specific areas such as your crawl errors, your search cues, and your sitemaps from this screen by clicking the applicable links.

You can also access these areas using the menu in the left sidebar.

Appearance of the research

In the left sidebar, the first option you will see is Search Appearance.

This section gives you an overview of the appearance of your site on the search engine results page.

See also  Side by Side: Comparison of Two Performance Marketing Tools / Agencies

Optimization is one way to check the SEO of your site.

Structured data

The Structured Data tab takes you to a page that looks like this:

This gives you a glimpse of how Google displays the content of your page, and if there are errors that stand out.

You can learn more about how Structured Data works for you with Google's handy guide.

Rich cards

Enriched cards are a form of structured data that you can add to your site's HTML code.

This helps the search engine to better understand what is on your webpage and display it in the results.

You will often see them posting when looking for a book, a movie or even a recipe:

Google has another guide to help you discover ways to create enriched maps for your own site.

When you do, you will come here to analyze and manage their performance.

Data highlighter

Google's Data Highlighter is a point-and-click tool that lets you tell Google manually what to point out in its search engine results.

This is a simpler method to create the own captions that you see on professional sites.

This video gives a quick overview of how it works and how you can start using it immediately.


HTML Enhancements

This page gives you a brief overview of the HTML issues detected by Google.

You will still see this message:

This means that your updates, changes, and overall structure work as expected.

If you get a hit, Google uses this tool to point you in the right direction for revisions and updates.

Accelerated Mobile Pages

Accelerated Mobile Pages is an open source initiative recently launched by Google to provide fast-loading mobile websites that work with slow connection speeds.

You can start here to create your first page if you do not already have one.

You will receive a piece of coding that you can customize on your site.

When you set it up, your webmaster tool will be able to break down your page and display an array of successfully indexed pages as well as AMP specific errors.

Traffic search

The Traffic Search section is the second option in the drop-down lists that you can select. Here you will find a breakdown of the performance of your site from different angles and possible improvements in the future.

Search Statistics

This section of your Google Webmaster Tools provides a global view of clicks, impressions, clickthrough rate, and location.

Why is it now more important than ever?

It gives an overview of keyword searches in which your website appears. It also shows the relationships between your impressions and clicks, average position and position changes.

You can also see how you perform on some devices, and which pages are lifting heavy loads.

Use this information to help you track your overall SEO and find high performing or underperforming pages.

Links to your website

Curious about your backlinks?

Google Webmaster Tools shows you the domains that bind you most, as well as the pages on your website that contain the most links.

This is probably the most complete list of your backlinks that you will find, at least for free.

This is a powerful tool for knowing where your content is being exploited on the Web, and what works best in Google's eyes.

Internal Links

These are the additional external links from your site, displayed below in the search results. If you use Google Kissmetrics, for example, you will see their list as well as six additional links from this site.

Unfortunately, you can not specify which pages you want to display in sitelinks.

See also  Crowd Machine prepares its blockchain-assisted network for application-free development

Manual actions

The Manual Actions tab lets you know if any of your pages do not follow Google's guidelines for the quality of websites.

This is one of the ways that Google has taken action against spam.


Mobile Usability

In the Mobile Usability tab, you can verify that all pages on your website are aligned with what Google considers best practice.

As you can see, you may have problems with text size, window settings, or even the proximity of your clickable elements.

All of these problems, as well as other errors, can negatively affect the ranking of your mobile site and push you further down the results page. Finding and correcting these errors will help your user experience and results.

Google index

Index Status

This report provides data on the URLs that Google has attempted to index on the selected property during the last calendar year from the date of the consultation.

When Googlebot searches the Internet, it processes every page it encounters to compile an index of every word it sees on every page.

It also looks at content tags and attributes like your alt titles or texts.

This chart shows a breakdown of your site's URLs that have been indexed by Google and may appear in search results.

When you add and delete pages, this graphic changes with you.

And do not worry too much if you have a smaller number of indexed pages than you think. Googlebot filters URLs that it considers to be duplicate, non-canonical, or those whose meta tag is not indexed.

You will also notice a number of URLs that have not been crawled by your robots.txt file.

And you can also check how many URLs you deleted with the removal tool. It will probably always be a low value.

Blocked Resources

If you want to know if your site is somehow outside of Google's crawling ability, you will be able to view your report on blocked resources.

However, this report does not show all the resources. Just those that Google thinks can control and fix.

Because Googlebot needs to access a lot of information on your page to index you correctly, a blocked resource can affect the performance of your page in Google's search results.

Finding and correcting these errors will help Google to rank you accurately.

Delete URLs

If, for one reason or another, you need to temporarily block a page from Google's search results, that's where you would go.

You can hide a page for about 90 days before it disappears.

If you want to permanently delete a page from Google's crawl, you will have to do it on your current website.


This tools section helps you to break down the performance of your site based on how Googlebot sees it.

Using the information here, you can find broken links, see how often Google looks at you, and set up technical parameters for Google to search your site.

Errors of exploration

It's never good to have broken links on your website.

But when you do it, or if you think you can, you can check what's really going on with this section of your Webmaster Tools.

You will get an overall graph of the performance of your site:

This lets you know how errors have increased or decreased over time. You can see here that we have some broken links to fix.

Under the graph, you will find a list of URLs, response codes and the date on which the error was detected.

See also  Marketing Day: news from Pinterest executives, Facebook CEO to testify before Congress and more

Google gives them a priority ranking, so you know which one should be treated first.

By clicking on the URL, you will also have a detailed overview of what exactly is going on.

If you have a lot of errors like these, focus on redirecting those that contain the most inbound links.

Crawl Statistics

For a more in-depth look at how often Googlebot visits your site, you can select the "Crawl Statistics" tab.

Here, Webmaster Tools shows you how often pages of your site are crawled, how many kilobytes are downloaded per day, and what are the download times for your site.

According to Google, there is no "good" number of exploration, but they have tips for sudden peaks or declines in your exploration rates.

Recover as Google

This tool is useful because it allows you to test how Google searches and displays a specific URL on your site.

This is a useful way to ensure that Googlebot can access a page that could be left to arbitrary.

If you succeed, the page will appear and you will see if any resources are blocked in Googlebot.

When you get to the debug point of web development, you can not beat this free tool.

Robots.txt Tester

If you use a robot.txt file to block Google's crawlers from a specific resource, this tool allows you to verify that everything is working properly.

So, if you have an image that you do not want to appear in Google Image Search, you can test your robot.txt here to make sure your image does not appear where you do not want it.

When you test, you receive an Accepted or Blocked message and you can edit accordingly.

Site map

I mentioned sitemaps earlier. I will return briefly to this topic.

Here you will see information about your sitemap.

If you notice that the last date your sitemap was uploaded is not new, you may want to submit your sitemap to refresh the number of URLs sent.

Otherwise, it helps you track how Google reads your sitemap and if all your pages are viewed as you wish.

URL parameters

Google itself recommends using this tool sparingly because an incorrect URL setting may have a negative impact on your site's scan.

You can read more about how to properly use Google's URL settings.

When you use them, this tool will help you keep tabs on their performance and make sure they do not point Googlebot in the wrong direction.


The tools for webmasters can give you an overview of your site's performance, as well as what you can do to keep Google's attention.

Do you use tools for webmasters? Which areas do you find most useful? Please share your thoughts in the comments below, and happy data analysis!

About the author: Neil Patel is the co-founder of Neil Patel Digital.