Google loves data. Google loves sharing data. But can we trust the data that Google shares with us?
Over the past few days I’ve found myself confronted with inconsistencies, suspicions, and downright inaccuracies in the data that Google chooses to share with us, its users.
First of all let me say that I love what Google is doing with its tools. The new Webmaster Tools features are all very welcome and extremely helpful, Google Adwords is improving with every iteration, and Google Analytics is made of pure awesomeness topped off with awesomesauce.
But can we trust these tools to give us the full picture? Let’s look at some of Google’s tools and the data it provides.
Google Adwords
The Google Adwords Keyword Tool has for a long time been viewed with some degree of suspicion, the data it yielded on search volumes not to be entirely relied on.
And now with the latest update to the tool any pretence of accurate, unbiased data is thrown entirely out the window. It’s a tool designed purely to help boost Google’s PPC revenue, and as such it has lost a lot of its relevance and value for SEOs.
But it’s not just the keyword tool that raises questions. Recently I was handed control over an existing Google Adwords account that hadn’t had any active campaigns for over 6 months. My first action was to restart one of the existing campaigns that had been dormant for half a year, with the intention of tweaking and improving it along the way.
The first week of this restarted campaign turned out to be a very good week, with a high amount of clicks, a very good CTR, and much lower CPC than the campaign had originally achieved.
But then things changed. In the second week, before I had made any alteration to the campaign, all metrics changed for the worse. Clicks and CTR dropped, CPC went up, and all for no discernible reason. The keywords the campaign was bidding on were not seasonal by any means, so I couldn’t identify any reason for this sudden and quite pronounced drop in campaign performance.
The drop continued in week 3, after which the campaign stabilised on a level of performance significantly lower than that of its first week.
Now a cynic (like me) could possibly suspect Google of having gamed the system a bit. It sees the re-activation of a dormant Adwords account and decides to ‘welcome’ the lapsed user back with suitably high metrics so as to instil a level of confidence with PPC advertising. It then quickly reverts to its normal performance, having done its duty, leaving the campaign’s manager to strive to achieve that first burst of phenomenal performance.
Google Analytics
I love Google Analytics. It’s a great web analytics tool that can provide its users with enormous amounts of highly useful data, which when interpreted correctly can yield incredibly valuable insights.
But I don’t trust Google Analytics very much. Too often I find discrepancies in its data, gaps in its reporting, and flaws in its figures.
Here’s one example I came across very recently. As part of my work for the Belfast Telegraph I do a weekly report on which articles performed the best, and there was one article that stood out in more ways than one. It was the most viewed article of the week – it was an article featuring a video of the Chile miners’ rescue – but Google Analytics gave me two entirely different pageview numbers, depending on how I approached the metric.
In one view (Top Content) the article was reported as having a bit over 9k pageviews, while in another view (Content Detail with a focus on Entrance Sources) it was shown to have over 33k pageviews.
While some discrepancies of numbers between different views are to be expected, I’m pretty sure a discrepancy of over a factor 3 is more than a bit excessive.
As it turns out the 33k number was the more accurate figure (as reported by our secondary analytics system HBX), but that number was only gained from Google Analytics after some additional digging. The original number as shown in the Top Content report was that much smaller (and highly inaccurate) 9k figure.
Needless to say this kind of highly inaccurate reporting can lead to a distorted view of a website’s performance, which in turn can lead to the wrong actions being taken.
Google Webmaster Tools
I don’t know about you but I find myself spending more and more time looking at the shiny reports conjured forth by Google Webmaster Tools. Over time it’s changed from a handy little toolset in to a full-fledged must-have toolkit, useful in a variety of ways.
The most recent improvements to GWT have been the most useful to SEOs – data on search query rankings & clicks and improved reporting on incoming links.
Yet once again we find ourselves wondering how accurate this data really is. Comparing the search query clicks as reported in GWT to the keyword data in Google Analytics there’s a discrepancy ranging from 45% to more than 100% on many keywords, as well as some keywords being omitted entirely from the GWT report.
We’ve already established that Google Analytics can’t be relied on entirely, and the numbers reported by Google Webmaster Tools are too generic (reminiscent of the data provided by the Google Keyword Tool), so we’re left wondering which figure to put our faith in.
The Struggle for Data
I realise that with data like this it’s not so much the actual figures we should be looking at as much as the trends they provide us, but in some cases those actual numbers really do matter. And it’s becoming painfully obvious that Google is unable, or unwilling, to provide us with truly accurate data.
These are just three minor examples of how Google presents us with a distorted view of what’s actually happening on the SERPs and on our websites. I’m sure there are many more such stories out there, and if you have one of your own to share by all means leave a comment.
The struggle to get accurate and reliable data on the performance of our sites continues, and while I applaud Google’s efforts to help webmasters and internet professionals with more data and insights, the fact remains that we can’t entirely trust what the big G is telling us.
“It then quickly reverts to its normal performance, having done its duty, leaving the campaign’s manager to strive to achieve that first burst of phenomenal performance.”
Agreed that is possible, however, I’d be more inclined to look to the blackbox in Mr. Schmidt’s office with “Quality Score” written across the side. It may be the first week it was going off an old QS until it built the new one which would be affected by the competition and new CTR. Google also seems to give more impressions to things they are trying to set a benchmark on. For instance that’s also why RealTime Search Started at the top of the SERP made it’s way down until it got moved to a vertical when they realized no one really cares about Real Time Search.
#JustAnotherPossibility
Hey Barry
The Google Adwords KWT is a bit of a mystery to me. When I compare the global search volumes they’re more or less the same as the API volumes. The local search data is a different story – and the free tool, on a local basis is fundamentally flawed – usually totally different to what I see coming out of the API. the external tool data is usually way, way higher than the API volumes.
The garbage about “commercial intent” IMO was exactly that – pure nonsense. Even the external tool is punching out a lot of non commercial terms in the data. Dictionary definitions, for example.
The data’s not so bad if you use the API’s average monthly search data, locally, for the past year. With a little due care and attention you can make a good decision provided you understand the occasional flaws.
Tap me up if you want to see some data,
Richard
Thanks for sharing real world numbers Barry.. We (the collective smart people we) have talked about this for a long time but it is always tough to get people to reveal real numbers on these things.. Now we have something to point to when this comes up..
Good piece, Barry. I can sure see why a discrepancy between 9k and 33k might shake someone’s confidence in the numbers (assuming they had any to begin with.
As for the GA numbers, when they seem to be getting better at everything else, why is it that they’re getting worse at something that is so simplistic, in comparison? GA ought to be a cake-walk! I can accept a time-lag for reporting, if the data they report is +/- a couple of percent. But 3:1 is ridiculous!
Matt Cutts video today… http://www.youtube.com/watch?v=iabVIPKFi84&feature=uploademail
At the end, he inadvertently uttered a Great Truth… “Consider the source.”
On Google Adwords, the keyword tool has never been that accurate. I would only ever use it for general trends and then test via PPC. As Richard says above, I don’t buy into the “commercial intent” either.
On Google Analytics, I am pretty sure the Google Analytics issue isn’t inaccurate data, it’s just a misleading report. The report “Content Detail with a focus on Entrance Sources” shows you the number of page views that originated from your page (Chilean Miners),this includes people who viewed the landing page and then went on to view a bunch of other pages. This is why it’s higher than the Top Content report. There are a lot of misleading reports in GA – I am going to do a post on that.
On GWT, I agree, the data doesn’t match GA, but I don’t feel these tools will ever match, due to the way they measure. Best used for general trends.
Thanks all for your comments – I’m glad to see that more people are struggling with data accuracy.
@Terry: yeah I did think about Quality Score as well, but the landing page hadn’t changed from the original campaign, nor had it undergone any change when we restarted the campaign. The site the landing page was on is also a fairly static site in a relatively small niche in Northern Ireland, so I can’t pinpoint any single aspect that might be causing such a huge discrepancy in Adwords performance.
@Richard: ah I’ve only ever looked at the keyword tool’s data, never dug in to the API – it’s time to change that I reckon. 🙂 Might hold you to that data-sharing offer.
@Steve: I was a bit hesitant to share real figures but I decided it was for the greater good and there wasn’t anything particularly sensitive being shared anyway. If we don’t address these flaws vocally and publicly, they’ll never get fixed in any meaningful way.
@Doc: thanks for that video – will take a close look at it when I’m home (buggering IT dept here at the BelTel offices have YouTube throttled).
@searchbrat: yeah I suspected the GA discrepancy might have a perfectly legitimate explanation, but as you said it’s a very misleading report that can result in all kinds of flawed decision making. Eager to read your views on that.
Agreeing with the general consensus here. I don’t trust google for anything, other than looking out for googles best interests.
Noticed so many sleazy practices they pull. As I’ve been trying to learn SEM/PCC. Which of course it’s not just google … All the megatech corps seem needlessly sleazy in the way they conduct themselves.
Like they want to get every last nickle they can, by screwing people over. Don’t know how that’s a formula for success or long term mutual benefit … Just the reality though.
Any data google provides, that stands to make google money. I automatically assume is skewed in their favor. Them or anyone else who stands to profit from playing with the numbers.
Since Im a newb, trying to get started on a shoestring budget … Means I have to work a lot harder to get quality metrics.
But Im willing to do the work to avoid the pitfalls. Will let the lazy/dull newbs walk into the traps, not be one of them. If I can help it.
Seasoned people, like the one’s who gather here aren’t fools though. You guys all know data is questionable. My take on it is turning to as many reasonably reputable sources of info and avging out the differences.
Think the only way someone is going to know what’s accurate ( works and doesn’t) Is their experience and in my case I won’t know until I acid test and actually give things a try in real life testing.
Anyway, thanks for the info.
I think the important question here is did Google know that extra data was in the content being captured? As a developer who has in the past for many many years reverse engineered file formats and represented them in an entirely different ‘common’ format I find it very difficult to believe that this data existed and was being produced undetected late in the development stage or was missed in the testing stage.
Given that Google was collecting enormous amounts of data with their street cars it would be intuitive to suggest that this data was looked at prior to application release and was tweaked to ensure that it was presented to the hard disk while capturing, with the most economical storage method available in mind. It would be foolish and incomprehensible to suggest that Google allowed the saved file format to contain ‘bloated’ raw data containing this extra ‘unwanted’ data that nobody knew about and was ignored. If I went about mapping an entire country, I would ensure that the data collection process was lean and mean for the purpose intended with no unwanted ‘side affects’ taking up valuable disk space. Anything other than lean and mean would be a gross error on my part.
f you ask me why pay more for cordless drills if on the market there’s a great range of cheaper plug-in drills my answer will be simple ““ it’s freedom. I’m sure that everyone was in a situation when you had to do some fixing in rather odd places. There’s always something to do in the backyard, everything can happen with your car somewhere on a lonely driveway. Of course, you don’t expect an outlet be available there. So cordless drill comes to relief. All modern cordless drill kits come with a drill itself, a charger and at least one battery.
Your Post is very useful, I am truly happy to post my note on this blog . It helped me with ocean of awareness so I really consider you will do much better in the future.
I think the important question here is did Google know that extra data was in the content being captured? As a developer who has in the past for many many years reverse engineered file formats and represented them in an entirely different ‘common’ format I find it very difficult to believe that this data existed and was being produced undetected late in the development stage or was missed in the testing stage.
Given that Google was collecting enormous amounts of data with their street cars it would be intuitive to suggest that this data was looked at prior to application release and was tweaked to ensure that it was presented to the hard disk while capturing, with the most economical storage method available in mind. It would be foolish and incomprehensible to suggest that Google allowed the saved file format to contain ‘bloated’ raw data containing this extra ‘unwanted’ data that nobody knew about and was ignored. If I went about mapping an entire country, I would ensure that the data collection process was lean and mean for the purpose intended with no unwanted ‘side affects’ taking up valuable disk space. Anything other than lean and mean would be a gross error on my part.
It’s time to make some big decisions, so the Google guys are slipping on their white lab coats. After eight years in the spotlight running a company that Wall Street values at more than $100 billion, Google co-founders Sergey Brin and Larry Page are still just in their early 30s and, with the stubbornness of youth, perhaps, and the aura of invincibility, keep doing things their way. So the white coats go on when it’s time to approve new products. For a few hours, teams of engineers will come forward with their best ideas, hoping to dazzle the most powerful men in Silicon Valley. Google paid crazy money to attract top talent–supercharging the nerd market in the process–and this is the recruits’ chance to show the investments were worth it.
conscience led me here. As many of you know, I was once a Calvary Chapel Pastor and missionary, but it slowly stopped making sense. I do have reasons for this, but that is not what I am writing about now. I was taught to believe that people like me became this way because of a gross sin, or because they loved the world more than they loved God. I now know this to be a lie. I am really not much different morally than I was then, nor do I wish to pursue things that would harm myself or others. Ironically, to be quite honest, I think I’m a bit more moral than I was then. I am most definately way more honest.