Improving Search Is there an inevitable innovator's dilemma with an advertising driven search business model?

What happens to advertising revenues when search gets better?

Last spring, I wrote a post about the conflict of interest with regard to algorithmic search results and paid search listings. This is a follow-up post, with a different spin on the same idea: that there is an "innovators dilemma" with regards to search innovation.  Better search results may in fact cannibalize ad clicks, and thus reduce expected revenues for a given query. image Here's an example case to consider - a query for: "Student Loan" In theory, a search engine dedicated to organizing the world's information should very much be able to provide valuable structured results for this term. So suppose a hotshot whiz-bang engineer comes up with an awesome way to amass and store up-to-the minute results on lenders, rates, reviews, etc. laid out in a clean, organized manner, and all based on what is most relevant to the searching party. What would be the response from the "business" side of the house that pays that engineer's salary? Here are some accumulated statistics from Google's AdWords tools for "Student Loan" and 50 recommended related terms:
CPC Weighted Average $ 14.31
U.S Monthly Search Volume 4,484,113
Percentage of queries that result in an Ad Click (my estimate!) 6%
Total Monthly Revenue $ 6,415,170
So - suppose this awesome new search innovation was so good that more queries were now satisfied by algorithmically generated data, and the percentage of clicks on ads falls in half to 3%. This search innovation just cost over $3 M in monthly losses. Take this example to its logical end, and the theoretically perfect search engine's ad-click rate will be 0, as the algorithm will always answer my query, and 0 x any $CPC will always be 0. There are two obvious rebuttals here: 1) Increase Price: the CPC that advertisers are willing to pay will go up. Because that price is determined by a second price auction bid stack, presumably destination sites are bidding based on the value of the arriving traffic.  It's debatable whether or not this is exogenous to organic results yielded by the search engine. But perhaps if users are still unsatisfied with the awesome improvements described above, their clicks will prove more valuable to the advertisers, and thus advertisers will be willing to pay more, and the $14.31 will go up. 2) Increase Volume: If more total people search on the site, then total number of clicks may be the same, even if a smaller percentage of people click on ads. Perhaps the search engine can grab more market share because of its (newly improved!) ability to answer queries such as "student loan." Or perhaps the entire search market will continue to grow. But, while this rationale may have been sensible over the past ten years, you can only replace margin with volume for so long, as you either reach market saturation (market won't grow) or market dominance (you own the market, and there's no more share to grab from competitors). There's more to say on this subject - in the meantime, I am hopeful, though pessimistic, that the technology press will start to think more critically about search business models -- as they like to say in mutual fund prospectuses: "past performance is not a good indicator of future success." Update, 2/13/2011: since I first wrote this post, there have been many articles written about how search all of a sudden feels broken. Most of the criticism has to do with "content farms" such as Demand Media, that produce low cost (and quality) content, do aggressive SEO, and then make money from clicks on ads when searchers arrive and find crappy content. This is arbitrage, essentially. IMHO, this is essentially the same "Innovator's Dilemma" dynamic as I describe above. Google makes money on clicks that occur at these content farm sites. By improving search relevance and not having those links listed in the organic results, they will cannibalize advertising revenue.