Webmaster Central Blog
Official news on crawling and indexing sites for the Google index
Handling legitimate cross-domain content duplication
tirsdag, desember 15, 2009
Webmaster level: Intermediate
We've recently discussed several ways of
handling duplicate content on a single website
; today we'll look at ways of handling similar duplication across different websites, across different domains. For some sites, there are legitimate reasons to duplicate content across different websites — for instance, to migrate to a new domain name using a web server that cannot create server-side redirects. To help with issues that arise on such sites, we're announcing our support of the
cross-domain rel="canonical" link element
.
Ways of handling cross-domain content duplication:
Choose your preferred domain
When confronted with duplicate content, search engines will generally take one version and filter the others out. This can also happen when multiple domain names are involved, so while search engines are generally pretty good at choosing something reasonable, many webmasters prefer to make that decision themselves.
Reduce in-site duplication
Before starting on cross-site duplicate content questions, make sure to
handle duplication within your site
first.
Enable crawling and use 301 (permanent) redirects where possible
Where possible, the most important step is often to use appropriate
301 redirects
. These redirects send visitors and search engine crawlers to your preferred domain and make it very clear which URL should be indexed. This is generally the preferred method as it gives clear guidance to everyone who accesses the content. Keep in mind that in order for search engine crawlers to discover these redirects, none of the URLs in the redirect chain can be disallowed via a
robots.txt file
. Don't forget to handle your www / non-www preference with appropriate redirects and in
Webmaster Tools
.
Use the cross-domain rel="canonical" link element
There are situations where it's not easily possible to set up redirects. This could be the case when you need to move your website from a server that does not feature server-side redirects. In a situation like this,
you can use the
rel="canonical" link element
across domains
to specify the exact URL of whichever domain is preferred for indexing. While the rel="canonical" link element is seen as a hint and not an absolute directive, we do try to follow it where possible.
Still have questions?
Q: Do the pages have to be identical?
A: No, but they should be similar. Slight differences are fine.
Q: For technical reasons I can't include a 1:1 mapping for the URLs on my sites. Can I just point the rel="canonical" at the homepage of my preferred site?
A: No; this could result in problems. A mapping from old URL to new URL for each URL on the old site is the best way to use rel="canonical".
Q: I'm offering my content / product descriptions for syndication. Do my publishers need to use rel="canonical"?
A: We leave this up to you and your publishers. If the content is similar enough, it might make sense to use rel="canonical", if both parties agree.
Q: My server can't do a 301 (permanent) redirect. Can I use rel="canonical" to move my site?
A: If it's at all possible, you should work with your webhost or web server to do a 301 redirect. Keep in mind that we treat rel="canonical" as a hint, and other search engines may handle it differently. But if a 301 redirect is impossible for some reason, then a rel="canonical" may work for you. For more information, see our
guidelines on moving your site
.
Q: Should I use a noindex robots
meta tag
on pages with a rel="canonical" link element?
A: No, since those pages would not be equivalent with regards to indexing - one would be allowed while the other would be blocked. Additionally, it's important that these pages are not disallowed from crawling through a robots.txt file, otherwise search engine crawlers will not be able to discover the rel="canonical" link element.
We hope this makes it easier for you to handle duplicate content in a user-friendly way. Are there still places where you feel that duplicate content is causing your sites problems? Let us know in the
Webmaster Help Forum
!
Posted by
John Mueller
, Webmaster Trends Analyst, Google Zürich
Hey!
Check here if your site is mobile-friendly.
Etiketter
accessibility
10
advanced
195
AMP
13
Android
2
API
7
apps
7
autocomplete
2
beginner
173
CAPTCHA
1
Chrome
2
cms
1
crawling and indexing
158
encryption
3
events
51
feedback and communication
83
forums
5
general tips
90
geotargeting
1
Google Assistant
3
Google I/O
3
Google Images
3
Google News
2
hacked sites
12
hangout
2
hreflang
3
https
5
images
12
intermediate
205
interstitials
1
javascript
8
job search
2
localization
21
malware
6
mobile
63
mobile-friendly
14
nohacked
1
performance
17
product expert
1
product experts
2
products and services
63
questions
3
ranking
1
recipes
1
rendering
2
Responsive Web Design
3
rich cards
7
rich results
10
search console
35
search for beginners
1
search queries
7
search results
140
security
12
seo
3
sitemaps
46
speed
6
structured data
33
summit
1
TLDs
1
url removals
1
UX
3
verification
8
video
6
webmaster community
24
webmaster forum
1
webmaster guidelines
57
webmaster tools
177
webmasters
3
youtube channel
6
Archive
2020
nov.
okt.
sep.
aug.
juli
juni
mai
apr.
mars
feb.
jan.
2019
des.
nov.
okt.
sep.
aug.
juli
juni
mai
apr.
mars
feb.
jan.
2018
des.
nov.
okt.
sep.
aug.
juli
juni
mai
apr.
mars
feb.
jan.
2017
des.
nov.
okt.
sep.
aug.
juni
mai
apr.
mars
feb.
jan.
2016
des.
nov.
okt.
sep.
aug.
juni
mai
apr.
mars
jan.
2015
des.
nov.
okt.
sep.
aug.
juli
mai
apr.
mars
feb.
jan.
2014
des.
nov.
okt.
sep.
aug.
juli
juni
mai
apr.
mars
feb.
jan.
2013
des.
nov.
okt.
sep.
aug.
juli
juni
mai
apr.
mars
feb.
jan.
2012
des.
nov.
okt.
sep.
aug.
juli
juni
mai
apr.
mars
feb.
jan.
2011
des.
nov.
okt.
sep.
aug.
juli
juni
mai
apr.
mars
feb.
jan.
2010
des.
nov.
okt.
sep.
aug.
juli
juni
mai
apr.
mars
feb.
jan.
2009
des.
nov.
okt.
sep.
aug.
juli
juni
mai
apr.
mars
feb.
jan.
2008
des.
nov.
okt.
sep.
aug.
juli
juni
mai
apr.
mars
feb.
jan.
2007
des.
nov.
okt.
sep.
aug.
juli
juni
mai
apr.
mars
feb.
jan.
2006
des.
nov.
okt.
sep.
aug.
Feed
Follow @googlewmc
Give us feedback in our
Product Forums
.
Subscribe via email
Enter your email address:
Delivered by
FeedBurner
