Posted by Melanie Phung on Thursday, August 3, 2006 at 10:25 pm
As regular readers may know, I’m endlessly fascinated with how del.icio.us pages end up ranking well in search results, considering each page has robots noindex and noarchive instructions. About 2 weeks ago, I noticed that the snippet for the result (in Google) had changed. Whereas before it only displayed the URL, it now was also displaying text from within the page. (Compare this to what the same search result looked like earlier.)
Does this mean Google was not ranking the page based only on a “guess” regarding the page’s relevance, based on the combination of domain and URL? Consider, it had to actually crawl the del.icio.us page to display this snippet. It’s reasonable to assume that if it’s displaying the snippet text, it’s also reading and storing it somehow.
[So if "noindex,nocache,nofollow" together don't mean "don't crawl"... is there a robots tag (not including a robots.txt file) that would instruct a spider not to read the content at all?]
Social bookmarking sites like del.icio.us add the noindex robots instruction to discourage SEOs from gaming the site. The idea is that no one would bother posting not-bookmark-worthy links solely for the “link juice” — those links are not supposed to “count” (for link weight, not traffic, obviously). But I also thought noindex and nocache was supposed to prevent Google or other SEs from displaying snippets from the page, and that assumption was proved wrong.
If the del.icio.us page for a specific tag — your company’s name, for example — has PageRank value, ranks well for that keyword in a Google search, and lists your site at the top of the page, it becomes harder to believe, in light of this snippet being displayed, that there still is no IBL value in making sure your site is frequently del.icio.us’d.
Sorry, the comment form is closed at this time.