All About Content

More on and ‘noindex’

Posted by Melanie Phung on Thursday, August 3, 2006 at 10:25 pm

As regular readers may know, I’m endlessly fascinated with how pages end up ranking well in search results, considering each page has robots noindex and noarchive instructions. About 2 weeks ago, I noticed that the snippet for the result (in Google) had changed. Whereas before it only displayed the URL, it now was also displaying text from within the page. (Compare this to what the same search result looked like earlier.)

Does this mean Google was not ranking the page based only on a “guess” regarding the page’s relevance, based on the combination of domain and URL? Consider, it had to actually crawl the page to display this snippet. It’s reasonable to assume that if it’s displaying the snippet text, it’s also reading and storing it somehow.

[So if “noindex,nocache,nofollow” together don’t mean “don’t crawl”… is there a robots tag (not including a robots.txt file) that would instruct a spider not to read the content at all?]

Social bookmarking sites like add the noindex robots instruction to discourage SEOs from gaming the site. The idea is that no one would bother posting not-bookmark-worthy links solely for the “link juice” — those links are not supposed to “count” (for link weight, not traffic, obviously). But I also thought noindex and nocache was supposed to prevent Google or other SEs from displaying snippets from the page, and that assumption was proved wrong.

If the page for a specific tag — your company’s name, for example — has PageRank value, ranks well for that keyword in a Google search, and lists your site at the top of the page, it becomes harder to believe, in light of this snippet being displayed, that there still is no IBL value in making sure your site is frequently’d.

Comments (4)

Category: Googlebot,Social Media


Comment by Paul Drago

Made Wednesday, 30 of August , 2006 at 2:18 am

Question: Have you tried switching your User Agent to Googlebot (for example) and see if the no follow still exists? (I haven’t but I plan to do it today)

Comment by Paul Drago

Made Wednesday, 30 of August , 2006 at 2:33 am

Sorry to comment spam you. But, I changed my useragent and I am still pulling up a no follow (ie– dont bother trying)

Comment by Melanie Phung

Made Wednesday, 30 of August , 2006 at 12:48 pm

Thanks for checking into that. I must admit I hadn’t thought to do it myself… what a counter-intuitive way to implement IP cloaking if that’s indeed what had been doing.

Do you have any other thoughts on what might be going on with the snippets being displayed as they are?

Comment by Paul Drago

Made Wednesday, 30 of August , 2006 at 6:14 pm

1) if you look at Yahoo Directory to users it looks like a redirect…but try looking at it under a different user agent, search engines seem them as links

2) I’ve noticed a few links pointing at a site of mine that should have been nofollowed. In fact, I checked the links and they were nofollowed– yet Google is still counting them. I wonder if maybe the syntax isn’t being used correctly.

3)am I missing the noarchive no index functions somewhere? Email me the information (screen shot if it isn’t too much trouble)

Sorry, the comment form is closed at this time.