It was only a few months ago, that Kerry reported that the major search engines had been working together to create a new tag that allows webmasters to specify which version of a URL they want to be indexed.
It was decided that this was necessary, as there are lots of issues of duplicate content on many websites caused by a variety of problems – some of which can be easily fixed, with others that are quite difficult to overcome.
Duplicate content is a problem for lots of websites – certainly it’s something that we encounter on a daily basis working on a number of campaigns, particularly at the beginning of a campaign, where we have to unravel the mess that we are sometimes given to work with.
Sometimes this is caused by shopping carts – lots of shopping carts will have the same products across different categories, each with its own URL. Previously this was very difficult to fix, as short of only having a product in 1 category (which wouldn’t be sensible from a user of the sites perspective – they want to see products that are relevant to that category regardless of whether or not this means having more than page on a website for that product – ultimately that doesn’t matter to them) there wasn’t really a fix for it.
With the introduction of this tag, you can now add this to each of these duplicates, selecting just 1 URL that you want this page to be seen as, and only this page will then be seen as the “main” page for this content – as long as the content on all those groups of pages is substantially the same. Minor differences in content are allowed, but it does have to be very close from page to page to be effective.
Another useful situation where I have utlised this tag is in resolving site canonical problems – I’ve come across a number of hosting packages with clients that don’t allow for this to be resolved satisfactorily.
The site canonical problem is where you have a website that can be seen with and without the www part of the URL. Another instance where this can be an issue is with http and https versions of a website being indexed – and in some cases, all 3 versions of the site can have pages indexed. By using this tag, all 3 versions of the web page(s) can be displayed, but only 1 of them gets the authority that all 3 accrue over time.
It still remains to be seen if this is as effective as removing the duplicate content – the search engines are all making the right noises about how it should, but there are a lot of things in the SEO world that don’t always hold the same weight when it comes to the application than is being said to be the case. If it does prove effective at eliminating these issues, then it is something that we will be using far more frequently to resolve this type of issue.