Search Results and Advertising Is there a conflict of interest between quality organic search results and pay-per-click sponsored links?

This starts with a simple story. My brother runs a small translation company in Berkeley called ION Translations. Like many small business owners, he has explored using Google's uber-popular AdWords service to help promote his business.

"Berkeley Translation" is one the search phrases for which he currently bids, and indeed, if you search on Google, as of 2/14/09, you will see his ad listed on the right-hand side sponsored links list. You do not, however, see a listing for "Ion Translations" in the "organic" search results. So perhaps my brother needs to do some work on optimizing his site for search engines?

Ad, but no organic's where it starts to get a little fishy...if you search for "Berkeley Translation Service", a phrase that my brother is not bidding for - lo' and behold, there is Ion Translations, listed in the "organic" search results local link section. Note as well that otherwise the organic list result is very similar.

No ad, organic search

Curious. Is it indeed a quirk in the Google algorithm that by including the term "service" in the query that all of a sudden my brother's company shows up in the organic search listings? Really? Or...(cue the organ music...) is there a more nefarious story about how Google is (not) rendering "organic" links based what sponsored links are associated with the query ...?

Indulge me the conspiracy for a few, if you will, and let's try and put aside the quirky lovable do-no-evil brand that Google has admirably built over the past 8 years. Let's think plain and simple about incentives.

First, recall how the AdWords model works: advertisers bid for keywords, Google lists them based on expected revenue, and only when a user clicks on an ad does Google get paid. (see appendix for more info on how the model works...)

If Google aims to maximize the revenue of this single instance query, then of course they would not want links to the same page ( in this instance) in the "paid" ads section to also appear in the "organic" list. After all, if the same link appears in both lists, Google would risk losing revenue if the searcher clicked on the "organic" link. And from Joe Searcher's point of view, he just wants to find what he's looking for, and probably doesn't really know or care how Google's business model works and will just click away on the most relevant seeming link. The only person who would prefer the organic click to the paid link click is the advertiser who would pay in that case (in the longer term the customer would also prefer that the advertiser not be unnecessarily charged, as higher costs would be passed along...)

...But of course, life ain't so simple...and in Google's (and all other search engines') defense, let's kick a little lay-man's game theory. Searching is not a "one shot game." Google has been able to build and maintain its formidable lead as the top search engine because users trust its results.. Trust is built over time, by delivering stellar service time and again. Any kind of public disclosure of the type of tinkering to drive revenue would really be damaging (I think, right?) But it would also be exceptionally difficult to prove, as Google's algorithm for ranking results is a proprietary black box.

Given that Google's search mechanics are, unlike many of the other offerings from the company, not "open", I can't help but be suspicious with the structure of the paid search listings business model. For these more fringey searches, like "Berkeley Translation", for which we know there is a very long tail, might a search engine be able to get away with it?

In fact, I think there is a darn uncanny parallel here to the conflict of interests in investment banking that have ebbed and flowed throughout the years.

In the banking world, there are analysts and traders. Analysts are supposed to be objective and report on the fundamental strength or weakness of securities. Traders buy and sell and make money off of commissions. But as we saw in the dot-com boom, analysts have the power to manipulate information flow, in such a way that unsuspecting sheep (the rest of us) buy or sell stocks and allow the traders (and analysts - see Blodget, Henry) to make money because of market swings that they can anticipate. Sarbanes-Oxley, in fact, was largely a response to this condition. And banks also had the same reputation interests to protect, as banking is not a one-shot game either.

So - let's transpose: in the search world, "organic" results are the analysts (an objective, scientific, heuristically driven list of links sorted by relevance), and paid listings are the "traders" (commissions are made based on trades/clicks.). And there they are - sitting on that sparse page together, with the tenuous white column serving as the firewall. If the analysts happen to have a moral slip-up and manipulate the objective information...they can drive business to the traders...

OK OK OK, I'll stop channeling Michael Moore. To be clear - I am not accusing Google of any wrong-doing. I do actually think this is a strange quirk of circumstance. But - and this is really my point: how can I be sure?

The bottom line is that this model deserves more scrutiny than it's received. And not just Google, but Yahoo, Microsoft, etc...who all have essentially the same approach. There is something quite troubling about these incentive structures, especially given search has become the way in which we gather and consume information. All we can do is trust that companies are doing the ethical thing...which we know all-too-well doesn't always work out...

The paid keywords business model, used by Google and others, is that advertisers pay if and only if a searcher clicks on the advertisement. The advertisers have stated a willingness to pay for a click, and the search engine calculates a likelihood of a click, and then ranks the list based on the product of those two numbers: (Probability of a Click) x (Willingness to Pay). The special sauce is really in the first number - the probability of a click, which is based on lotsa data from previous queries, which is why Google has such a strong advantage in this space.

(Yahoo used to charge for listing, not for click, which had its own issues...but I'll leave a comparison on the sidelines for now...).

Full disclosure: I work for Microsoft, not on anything related to search. I am writing this on a Mac. I use a bunch of Google products, which are for the most part very very very good. But I try not to click on their ads. When I can actually tell that they are ads.