Search Engines Blocked From Your Content
By Lee Odden
Article Date: 2008-10-16
At the DMA08 conference in Las Vegas this week, the "Search Engine Experience" panel on Tuesday was presented with a number of site reviews which are great sessions because the audience gets free SEO consulting advice and speakers get to demonstrate their expertise.
One of the questions that came up was related to a site that has most of it's content behind a login. Because search engine spiders don't/can't fill out forms (with some exceptions) they cannot find the content behind the login to index and include in the search results.
Why would you want your "members access only" content available to search engines?
The more web pages a search engine can find, index and include in the search results, the more entry points there are for new customers to find your web site content. A broader, yet relevant, footprint on the web can make a significant difference in traffic to a web site. If most of the content is hidden behind a login process, that content is also hidden from the search engines.
Of course there are a number of good reasons for web sites to put their content behind a login. The site in question at DMA08 concerned an auction site that involved being a member to access the listings. Other examples include content that is sensitive, industry regulated or that is time sensitive.
In the case of the auction site, two issues were involved: most of the site was behind a login and the auction content was only relevant for a certain period of time.
A common solution for making content hidden behind a login available to search engines is to modify the templates that pull data from the database so that an excerpt is pulled into a crawlable web page. The excerpt content and links between them would be available to search engine spiders and get included in search results. Search visitors that find the excerpts would be presented with a link to read more of the article/content and then be presented with an option to become a member, register or whatever the criteria for access are.
In the case of the auction site, outdated content would be of no use to the customers so it makes little sense to allow it to be indexed by search engines. Sitting in the audience, I was doing my best to mind meld with the panel (Heather Lloyd-Martin, Detelv Johnson and Jeannette Kocsis) a suggestion but Detlev already had it: The auction site could create content summaries such as the top bids for a category or pictures and descriptions of the most popular items. A blog talking about the products being sold might also be of use.
When content is regulated by an industry, it's obviously important to work with the compliance "powers that be" within an organization to determine how much information can be made publicly available. Membership sites also need to gauge how much information to make public before asking readers to sign up for more.
A site with thousands of web pages can see a dramatic increase in pages indexed when using an excerpt strategy. However, it will take some convincing of IT/web development to make the necessary changes in the content management system and database, if that's even possible. Then there's the cost justification.
To build a business case for making these kinds of programming changes requires some forecasting of potential traffic increases which shouldn't be too difficult using existing historical web analytics. Some SEO consultants are fairly good at this and can help you out. :)
If your web site has undergone the programming changes in order to make some of your previously hidden content available to search engines, it would be great to hear about it. What were the biggest hurdles in getting budget, implementation and even how it panned out once the content was live?
About the Author:
Lee Odden is President and Founder of
TopRank Online Marketing, specializing in organic SEO, blog
marketing and online public relations. He's been cited as a search
marketing expert by publications including U.S. News & World Report and
The Economist and has implemented successful search marketing programs
with top BtoB companies of all sizes. Odden shares his marketing
expertise at Online Marketing Blog offering
daily news, interviews and best practices.