You buy keywords to run paid search ads. Users type keywords into search engines to perform searches. It’s easy to forget that they’re not the same things.

I’ve taken to calling the ones you buy ‘keywords’ and the ones people type ‘queries’. And about 100 times a week in one conversation or another I have to pose the question: Do you mean keywords or queries?

Beyond the linguistic fascination this poses, there is a very practical issue for paid search marketers. These diabolically similarly named elements interact in strange and important ways within your campaigns.

Confusion or indifference isn’t too good for the bottom line.

The Impossibly Simple Report

To understand the performance of any keyword (the kind you buy) it is vital to look at the queries (the ones users type) that get counted as clicks.

Of course, you probably can’t.

At least not using the tools and configurations you have today. Neither Google Adwords nor Yahoo Search Marketing or MSN AdCenter offers a report showing each keyword you buy and all the queries that were matched to it. Somehow they seem to think it’s unimportant (or perhaps not in their self-interest) to show exactly how your money is being spent.

There is a work-around of sorts: you can manually tag each keyword in every one of your campaigns to pass back the name of the keyword you purchased (see this post for info on the way to do this in Google, there are similar methods available for Yahoo and MSN) and then figure out how to grab that tag and marry it to the query string which is often (but not always) returned in the URL when the click actually arrives.

The actual process depends upon the tools you’re using, and in my experience this isn’t really complete, accurate, or practical.

That’s Why We Have Analytics Software, Right?

Nope. In their default state, neither Google Analytics or the larger more generally well equipped web analytics packages can tell you which of your purchased keywords drove traffic from which search queries either.

I can never decide if this is more or less startling.

• In Omniture SiteCatalyst the ‘Paid Search Keywords’ report is a query report. There is no capture of your paid search keywords (the ones you bought) unless you code them into the inbound URL’s and grab them as part of the Campaign tracking codes. If you do this, then use Excel to extract the keywords into a SAINT classification you should be able to run a classification report broken down by Paid Search Keywords and see the desired report. I’ve found this can work for queries which generate orders, but haven’t been able to make it work for every click. If you add SearchCenter, Keywords are tracked automatically so all that tagging and SAINT extraction goes away, but still only a fraction of the queries appear.
• HBX / WebTrends ML2 – I’m far less experienced at either of these, but haven’t seen nor been able to find simple complete keyword by query reports here either. Anyone with more definitive information is encouraged to comment.
• Google Analytics 2 – In GA2 you can pass in data via the target URL much as described for SiteCatalyst if you’re willing and able do tag the target for every keyword you buy (instructions here). I don’t believe (but am not 100% sure) that when tagged you can then produce a keyword by query report. Anyone?

What If I Pay More?

They say money can’t buy happiness, but let’s admit that it can make the misery a lot easier to put up with. In the case of search analytics, Omniture SiteCatalyst/SearchCenter can produce the report we all need and desire – if you’re aware of and then purchase the Unified Sources DB Vista Rule.

Once this little gem is setup, you get a new eVar with the user query properly populated (the vast majority of the time) and you can therefore break down SearchCenter keyword reports with an accurate query list.

There is still some separation between the SearchCenter data and the SiteCatalyst data (which the evar is) so not everything is full allocated across the queries (keyword cost for example so you can’t fully calculate ROI) but none-the-less this is fantastically useful for understanding and tuning your paid search campaigns.

Demonstrating some of the useful information in these reports was the reason I started writing this post, but as it’s far too long already, I’ll save the examples and details for another day.

(BTW: This feature is only a bonus of the Unified Sources Vista Rule - it’s main utility is to classify organic search traffic and referring URLs into your Campaign tracking code so that Campaign Reports cover nearly every traffic source, and you can SAINT organic and referring traffic along with other campaigns. This capability had also been very high on my wish list, so I recommend this Vista rule highly.) 

The Developing Trend

If the core process of paid search marketing is buying keywords to connect with users when they type queries into search engines, then transparent reporting on the way our keywords are matched with user queries is essential.

Without knowing which queries are being matched to which keywords, it’s pretty hard to select or tune your use of match types. It’s a lot harder than it needs to be to see better ways to organize your ad-groups or rewrite text-ads to address user concerns or intent.

And across the performance metrics tied up in the CTRs, CPCs, and Conversion Rates of those keywords, match types, ad-groups, and creatives, it becomes virtually impossible to optimize your campaigns and your spend to maximize YOUR revenue.

This fact is the primary driver behind our development of ClickEquations – you don’t have a fair shot today due to both the complexity of the marketplace and the information you’re selectively not being provided.

There are undoubtedly many reasons why we don’t currently get clear complete information. The rapid development of the engines and their software and the complexity of what they’re trying to accomplish dictates that they haven’t had time to get to everything. It also seems reasonable to assume that since their economic interests and those of their advertisers don’t always align, some decisions and prioritizations are made without putting advertiser interests first.

But advertisers also own some of the responsibility. Without a loud and clear call for the kind of information and transparency we all need to manage our campaigns and budgets, we’re going to continue to get all kinds of bells and whistles from both the engines and the analytics vendors while some very basic features remain difficult or unavailable.