Ajit Gaddam: TechNews and Security

Google Sitemaps is an excellent tool used by a lot of webmasters out there including us at AskStudent.  As a webmaster, you would always want to exercise greater control over what is indexed by the search engines. For this purpose, most of use a robots.txt file or robots meta tag to tell the search engine spiders what we don’t want indexed. But sometimes, you like to remove content that was already indexed.

As an example, at AskStudent, couple of months ago we had .html then .php static pages serving content before we moved to WordPress to serve out content. What happened during this move was that a lot of static .html pages from AskStudent were already indexed. This resulted in errors such as the ones shown below showing up for the search engine crawlers.

So how do you remove such content that’s already been indexed? Well…. Google Sitemaps has added a brand new tool today which will allow webmasters to expedite the removal of outdated links instead of waiting for the next crawl.

In your webmaster tools account, under the Diagnostic tab, you will see a new option called URL Removals. To get started, click on the URL removals link, then New Removal Request and then choose the option that matches the type of removal you want to persue.

Individual URLs

You can choose to remove an individual URL or an image using this option. According to Google, in order for the URL to be eligible for removal, you must meet one of the following criteria.

• The URL most return a status code of either 404 or 410.
• The URL must be blocked by the site’s robots.txt file.
• The URL must be blocked by a robots meta tag.

Once you enter the URL, indicate if its an error that you see in the web search results or image results. According to Google, you can add upto 100 URLs in a single request.

A Directory

If you wish to remove all files and folders within a directory on your site then use this option. This could be a good option if you have password or subscription only pages. Even if you have a Wordpress blog, then enter the directory where you have images or any other content stored which you don’t wish to be indexed.

In order for a directory to be eligible for removal , you must block it using a robots.txt file. For instance, for the example above, http://www.askstudent.com/robots.txt could include the following:

User-agent: Googlebot
Disallow: /network

Entire Site and Google Cache 

Similarly, you can also remove your entire site from the Google Search Index. This option will remove all subdirectories and files too. Also, you can remove cached copies of your site from the Google index too. This could be beneficial for sites such as experts exchange. Experts exchange provides solutions to problems for paid subscribers. The workaround for this for free has been to use Google cache.

Requesting removal of content you don’t own

But what if you want to request removal of content that’s located on a site that you don’t own? It’s just gotten easier to do that as well. Google’s new Webpage removal request tool steps through the process for each type of removal request.

Since Google has no control on the content of webpages it doesn’t own just like you, they generally can’t remove results from their index. If you would like content removed, you can work with the site owner to do so, and then use this tool to expedite the removal from our search results. This could be particularly useful if you think your content has been plagiarized by someone else.

The exception comes when you have found search results that contain specific types of personal information, you can request removal even if you’ve been unable to work with the site owner. Examples of such content include

For this type of removal, provide your email address so that Google can work with you directly.