Colin Cochrane wrote some days ago about what happened to him when he tried to do a search inside his bookmarks at Del.icio.us but using Google, he couldn’t find any results. With some research done by him, he could find out that Del.icio.us was preventing Search Engines to index its pages by using robots.txt; what search engines see when trying to access it, is a 404 error page, what doesn’t let them index the pages.
This decision by Del.icio.us affects everyone in at least 3 different ways:
- Links: links on del.icio.us pages aren’t worth anything for Search Engines as they can’t even see the page. However they still got the value of needing them to be on Del.icio.us’ popular links page.
- Reputation Management: another use marketers gave to del.icio.us was for reputation management on search engines, but they won’t be able to use it any more for that purpose as Del.icio.us won’t even index the pages.
- Search via Google: it won’t be possible to search your bookmarks by using Google, I loved using Google Search for my bookmarks on del.icio.us as it always displayed the most relevant content first (yes, i have too much bookmarks).
Why I think del.icio.us took this decision?
As you might be aware, del.icio.us is owned by Yahoo, so the decisions behind del.icio.us were really taken by Yahoo. But they also blocked the content from Yahoo!’s Slurp bot, in my opinion, the real reason why they are doing this is because Yahoo doesn’t want other search engines to benefit from del.icio.us content, and they blocked themselves as they don’t need a crawler to know all the pages of a site they own.
