Why Google AdWords’ Keyword Volume Numbers Are Wildly Unreliable – Whiteboard Friday

Posted by randfish

Many of us rely on the search volume numbers Google AdWords provides, but those numbers ought to be consumed with a hearty helping of skepticism. Broad and unusable volume ranges, misalignment with other Google tools, and conflating similar yet intrinsically distinct keywords — these are just a few of the serious issues that make relying on AdWords search volume data alone so dangerous. In this edition of Whiteboard Friday, we discuss those issues in depth and offer a few alternatives for more accurate volume data.

why it's insane to rely on Google adwords' keyword volume numbers

Click on the whiteboard image above to open a high-resolution version in a new tab!

<span id=”selection-marker-1″ class=”redactor-selection-marker”></span>

Video Transcription

Howdy, Moz fans. Welcome to another edition of Whiteboard Friday. This week we’re going to chat about Google AdWords’ keyword data and why it is absolutely insane as an SEO or as a content marketer or a content creator to rely on this.

Look, as a paid search person, you don’t have a whole lot of choice, right? Google and Facebook combine to form the duopoly of advertising on the internet. But as an organic marketer, as a content marketer or as someone doing SEO, you need to do something fundamentally different than what paid search folks are doing. Paid search folks are basically trying to figure out when will Google show my ad for a keyword that might create the right kind of demand that will drive visitors to my site who will then convert?

But as an SEO, you’re often driving traffic so that you can do all sorts of other things. The same with content marketers. You’re driving traffic for multitudes of reasons that aren’t directly or necessarily directly connected to a conversion, at least certainly not right in that visit. So there are lots reasons why you might want to target different types of keywords and why AdWords data will steer you wrong.

1. AdWords’ “range” is so broad, it’s nearly useless

First up, AdWords shows you this volume range, and they show you this competition score. Many SEOs I know, even really smart folks just I think haven’t processed that AdWords could be misleading them in this facet.

So let’s talk about what happened here. I searched for types of lighting and lighting design, and Google AdWords came back with some suggestions. This is in the keyword planner section of the tool. So “types of lighting,” “lighting design”, and “lighting consultant,” we’ll stick with those three keywords for a little bit.

I can see here that, all right, average monthly searches, well, these volume ranges are really unhelpful. 10k to 100k, that’s just way too giant. Even 1k to 10k, way too big of a range. And competition, low, low, low. So this is only true for the quantity of advertisers. That’s really the only thing that you’re seeing here. If there are many, many people bidding on these keywords in AdWords, these will be high.

But as an example, for “types of light,” there’s virtually no one bidding, but for “lighting consultant,” there are quite a few people bidding. So I don’t understand why these are both low competition. There’s not enough granularity here, or Google is just not showing me accurate data. It’s very confusing.

By the way, “types of light,” though it has no PPC ads right now in Google’s results, this is incredibly difficult to rank for in the SEO results. I think I looked at the keyword difficulty score. It’s in the 60s, maybe even low 70s, because there’s a bunch of powerful sites. There’s a featured snippet up top. The domains that are ranking are doing really well. So it’s going to be very hard to rank for this, and yet competition low, it’s just not telling you the right thing. That’s not telling you the right story, and so you’re getting misled on both competition and monthly searches.

2. AdWords doesn’t line up to reality, or even Google Trends!

Worse, number two, AdWords doesn’t line up to reality with itself. I’ll show you what I mean.

So let’s go over to Google Trends. Great tool, by the way. I’m going to talk about that in a second. But I plugged in “lighting design,” “lighting consultant,” and “types of lighting.” I get the nice chart that shows me seasonality. But over on the left, it also shows average keyword volume compared to each other — 86 for “lighting design,” 2 for “lighting consultant,” and 12 for “types of lighting.” Now, you tell me how it is that this can be 43 times as big as this one and this can be 6 times as big as that one, and yet these are all correct.

The math only works in some very, very tiny amounts of circumstances, like, okay, maybe if this is 1,000 and this is 12,000, which technically puts it in the 10k, and this is 86,000 — well, no wait, that doesn’t quite work — 43,000, okay, now we made it work. But you change this to 2,000 or 3,000, the numbers don’t add up. Worse, it gets worse, of course it does. When AdWords gets more specific with the performance data, things just get so crazy weird that nothing lines up.

So what I did is I created ad groups, because in AdWords in order to get more granular monthly search data, you have to actually create ad groups and then go review those. This is in the review section of my ad group creation. I created ad groups with only a single keyword so that I could get the most accurate volume data I could, and then I maximized out my bid until I wasn’t getting any more impressions by bidding any higher.

Well, whether that truly accounts for all searches or not, hard to say. But here’s the impression count — 2,500 a day, 330 a day, 4 a day. So 4 a day times 30, gosh, that sounds like 120 to me. That doesn’t sound like it’s in the 1,000 to 10,000 range. I don’t think this could possibly be right. It just doesn’t make any sense.

What’s happening? Oh, actually, this is “types of lighting.” Google clearly knows that there are way more searches for this. There’s a ton more searches for this. Why is the impression so low? The impressions are so low because Google will rarely ever show an ad for that keyword, which is why when we were talking, above here, about competition, I didn’t see an ad for that keyword. So again, extremely misleading.

If you’re taking data from AdWords and you’re trying to apply it to your SEO campaigns, your organic campaigns, your content marketing campaigns, you are being misled and led astray. If you see numbers like this that are coming straight from AdWords, “Oh, we looked at the AdWords impression,” know that these can be dead f’ing wrong, totally misleading, and throw your campaigns off.

You might choose not to invest in content around types of lighting, when in fact that could be an incredibly wonderful lead source. It could be the exact right keyword for you. It is getting way more search volume. We can see it right here. We can see it in Google Trends, which is showing us some real data, and we can back that up with our own clickstream data that we get here at Moz.

3. AdWords conflates and combines keywords that don’t share search intent or volume

Number three, another problem, Google conflates keywords. So when I do searches and I start adding keywords to a list, unless I’m very careful and I type them in manually and I’m only using the exact ones, Google will take all three of these, “types of lights,” “types of light” (singular light), and “types of lighting” and conflate them all, which is insane. It is maddening.

Why is it maddening? Because “types of light,” in my opinion, is a physics-related search. You can see many of the results, they’ll be from Energy.gov or whatever, and they’ll show you the different types of wavelengths and light ranges on the visible spectrum. “Types of lights” will show you what? It will show you types of lights that you could put in your home or office. “Types of lighting” will show you lighting design stuff, the things that a lighting consultant might be interested in. So three different, very different, types of results with three different search intents all conflated in AdWords, killing me.

4. AdWords will hide relevant keyword suggestions if they don’t believe there’s a strong commercial intent

Number four, not only this, a lot of times when you do searches inside AdWords, they will hide the suggestions that you want the most. So when I performed my searches for “lighting design,” Google never showed me — I couldn’t find it anywhere in the search results, even with the export of a thousand keywords — “types of lights” or “types of lighting.”

Why? I think it’s the same reason down here, because Google doesn’t believe that those are commercial intent search queries. Well, AdWords doesn’t believe they’re commercial intent search queries. So they don’t want to show them to AdWords customers because then they might bid on them, and Google will (a) rarely show those, and (b) they’ll get a poor return on that spend. What happens to advertisers? They don’t blame themselves for choosing faulty keywords. They blame Google for giving them bad traffic, and so Google knocks these out.

So if you are doing SEO or you’re doing content marketing and you’re trying to find these targets, AdWords is a terrible suggestion engine as well. As a result, my advice is going to be rely on different tools.


There are a few that I’ve got here. I’m obviously a big fan of Moz’s Keyword Explorer, having been one of the designers of that product. Ahrefs came out with a near clone product that’s actually very, very good. SEMrush is also a quality product. I like their suggestions a little bit more, although they do use AdWords keyword data. So the volume data might be misleading again there. I’d be cautious about using that.

Google Trends, I actually really like Google Trends. I’m not sure why Google is choosing to give out such accurate data here, but from what we’ve seen, it looks really comparatively good. Challenge being if you do these searches in Google Trends, make sure you select the right type, the search term, not the list or the topic. Topics and lists inside Google Trends will aggregate, just like this will, a bunch of different keywords into one thing.

Then if you want to get truly, truly accurate, you can go ahead and run a sample AdWords campaign, the challenge with that being if Google chooses not to show your ad, you won’t know how many impressions you potentially missed out on, and that can be frustrating too.

So AdWords today, using PPC as an SEO tool, a content marketing tool is a little bit of a black box. I would really recommend against it. As long as you know what you’re doing and you want to find some inspiration there, fine. But otherwise, I’d rely on some of these other tools. Some of them are free, some of them are paid. All of them are better than AdWords.

All right, everyone. Look forward to your comments and we’ll see you again next week for another edition of Whiteboard Friday. Take care.

Video transcription by Speechpad.com

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

Moz Blog

Rankings Correlation Study: Domain Authority vs. Branded Search Volume

Posted by Tom.Capper

A little over two weeks ago I had the pleasure of speaking at SearchLove San Diego. My presentation, Does Google Still Need Links, looked at the available evidence on how and to what extent Google is using links as a ranking factor in 2017, including the piece of research that I’m sharing here today.

One of the main points of my presentation was to argue that while links still do represent a useful source of information for Google’s ranking algorithm, Google now has many other sources, most of which they would never have dreamed of back when PageRank was conceived as a proxy for the popularity and authority of websites nearly 20 years ago.

Branded search volume is one such source of information, and one of the sources that is most accessible for us mere mortals, so I decided to take a deeper look on how it compared with a link-based metric. It also gives us some interesting insight into the KPIs we should be pursuing in our off-site marketing efforts — because brand awareness and link building are often conflicting goals.

For clarity, by branded search volume, I mean the monthly regional search volume for the brand of a ranking site. For example, for the page https://www.walmart.com/cp/Gift-Cards/96894, this would be the US monthly search volume for the term “walmart” (as given by Google Keyword Planner). I’ve written more about how I put together this dataset and dealt with edge cases below.

When picking my link-based metric for comparison, Domain Authority seemed a natural choice — it’s domain-level, which ought to be fair given that generally that’s the level of precision with which we can measure branded search volume, and it came out top in Moz’s study of domain-level link-based factors.

A note on correlation studies

Before I go any further, here’s a word of warning on correlation studies, including this one: They can easily miss the forest for the trees.

For example, the fact that Domain Authority (or branded search volume, or anything else) is positively correlated with rankings could indicate that any or all of the following is likely:

  • Links cause sites to rank well
  • Ranking well causes sites to get links
  • Some third factor (e.g. reputation or age of site) causes sites to get both links and rankings

That’s not to say that correlation studies are useless — but we should use them to inform our understanding and prompt further investigation, not as the last word on what is and isn’t a ranking factor.


(Or skip straight to the results!)

The Moz study referenced above used the provided 800 sample keywords from all 22 top-level categories in Google Keyword Planner, then looked at the top 50 results for each of these. After de-duplication, this results in 16,521 queries. Moz looked at only web results (no images, answer boxes, etc.), ignored queries with fewer than 25 results in total, and, as far as I can tell, used desktop rankings.

I’ve taken a slightly different approach. I reached out to STAT to request a sample of ~5,000 non-branded keywords for the US market. Like Moz, I stripped out non-web results, but unlike Moz, I also stripped out anything with a baserank worse than 10 (baserank being STAT’s way of presenting the ranking of a search result when non-web results are excluded). You can see the STAT export here.

Moz used Mean Spearman correlations, which is a process that involves ranking variables for each keyword, then taking the average correlation across all keywords. I’ve also chosen this method, and I’ll explain why using the below example:


SERP Ranking Position

Ranking Site

Branded Search Volume of Ranking Site

Per Keyword Rank of Branded Search Volume

Keyword A





Keyword A





Keyword A





Keyword A





Keyword A





For Keyword A, we have wildly varying branded search volumes in the top 5 search results. This means that search volume and rankings could never be particularly well-correlated, even though the results are perfectly sorted in order of search volume.

Moz’s approach avoids this problem by comparing the ranking position (the 2nd column in the table) with the column on the far right of the table — how each site ranks for the given variable.

In this case, correlating ranking directly with search volume would yield a correlation of (-)0.75. Correlating with ranked search volume yields a perfect correlation of 1.

This process is then repeated for every keyword in the sample (I counted desktop and mobile versions of the same keyword as two keywords), then the average correlation is taken.

Defining branded search volume

Initially, I thought that pulling branded search volume for every site in the sample would be as simple as looking up the search volume for their domain minus its subdomain and TLD (e.g. “walmart” for https://www.walmart.com/cp/Gift-Cards/96894). However, this proved surprisingly deficient. Take these examples:

  • www.cruise.co.uk
  • ecotalker.wordpress.com
  • www.sf.k12.sd.us

Are the brands for these sites “cruise,” “wordpress,” and “sd,” respectively? Clearly not. To figure out what the branded search term was, I started by taking each potential candidate from the URL, e.g., for ecotalker.wordpress.com:

  • Ecotalker
  • Ecotalker wordpress
  • WordPress.com
  • WordPress

I then worked out what the highest search volume term was for which the subdomain in question ranked first — which in this case is a tie between “Ecotalker” and “Ecotalker wordpress,” both of which show up as having zero volume.

I’m leaning fairly heavily on Google’s synonym matching in search volume lookup here to catch any edge-edge-cases — for example, I’m confident that “ecotalker.wordpress” would show up with the same search volume as “ecotalker wordpress.”

You can see the resulting dataset of subdomains with their DA and branded search volume here.

(Once again, I’ve used STAT to pull the search volumes in bulk.)

The results: Brand awareness > links

Here’s the main story: branded search volume is better correlated with rankings than Domain Authority is.

However, there’s a few other points of interest here. Firstly, neither of these variables has a particularly strong correlation with rankings — a perfect correlation would be 1, and I’m finding a correlation between Domain Authority and rankings of 0.071, and a correlation between branded search volume and rankings of 0.1. This is very low by the standards of the Moz study, which found a correlation of 0.26 between Domain Authority and rankings using the same statistical methods.

I think the biggest difference that accounts for this is Moz’s use of 50 web results per query, compared to my use of 10. If true, this would imply that Domain Authority has much more to do with what it takes to get you onto the front page than it has to do with ranking in the top few results once you’re there.

Another potential difference is in the types of keyword in the two samples. Moz’s study has a fairly even breakdown of keywords between the 0–10k, 10k–20k, 20k–50k, and 50k+ search volume buckets:

On the other hand, my keywords were more skewed towards the low end:

However, this doesn’t seem to be the cause of my lower correlation numbers. Take a look at the correlations for rankings for high volume keywords (10k+) only in my dataset:

Although the matchup between the two metrics gets a lot closer here, the overall correlations are still nowhere near as high as Moz’s, leading me to attribute that difference more to their use of 50 ranking positions than to the keywords themselves.

It’s worth noting that my sample size of high volume queries is only 980.

Regression analysis

Another way of looking at the relationship between two variables is to ask how much of the variation in one is explained by the other. For example, the average rank of a page in our sample is 5.5. If we have a specific page that ranks at position 7, and a model that predicts it will rank at 6, we have explained 33% of its variation from the average rank (for that particular page).

Using the data above, I constructed a number of models to predict the rankings of pages in my sample, then charted the proportion of variance explained by those models below (you can read more about this metric, normally called the R-squared, here).

Some explanations:

  • Branded Search Volume of the ranking site – as discussed above
  • Log(Branded Search Volume) – Taking the log of the branded search volume for a fairer comparison with domain authority, where, for example, a DA 40 site is much more than twice as well linked to as a DA 20 site.
  • Ranked Branded Search Volume – How this site’s branded search volume compares to that of other sites ranking for the same keyword, as discussed above

Firstly, it’s worth noting that despite the very low R-squareds, all of the variables listed above were highly statistically significant — in the worst case scenario, within a one ten-millionth of a percent of being 100% significant. (In the best case scenario being a vigintillionth of a vigintillionth of a vigintillionth of a nonillionth of a percent away.)

However, the really interesting thing here is that including ranked Domain Authority and ranked branded search volume in the same model explains barely any more variation than just ranked branded search volume on its own.

To be clear: Nearly all of the variation in rankings that we can explain with reference to Domain Authority we could just as well explain with reference to branded search volume. On the other hand, the reverse is not true.

If you’d like to look into this data some more, the full set is here.

Nice data. Why should I care?

There are two main takeaways here:

  1. If you care about your Domain Authority because it’s correlated with rankings, then you should care at least as much about your branded search volume.
  2. The correlation between links and rankings might sometimes be a bit of a red-herring — it could be that links are themselves merely correlated with some third factor which better explains rankings.

There are also a bunch of softer takeaways to be had here, particularly around how weak (if highly statistically significant) both sets of correlations were. This places even more emphasis on relevancy and intent, which presumably make up the rest of the picture.

If you’re trying to produce content to build links, or if you find yourself reading a post or watching a presentation around this or any other link building techniques in the near future, there are some interesting questions here to add to those posed by Tomas Vaitulevicius back in November. In particular, if you’re producing content to gain links and brand awareness, it might not be very good at either, so you need to figure out what’s right for you and how to measure it.

I’m not saying in any of this that “links are dead,” or anything of the sort — more that we ought to be a bit more critical about how, why, and when they’re important. In particular, I think that they might be of increasingly little importance on the first page of results for competitive terms, but I’d be interested in your thoughts in the comments below.

I’d also love to see others conduct similar analysis. As with any research, cross-checking and replication studies are an important step in the process.

Either way, I’ll be writing more around this topic in the near future, so watch this space!

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

Moz Blog

Online Marketing News: Generational Video, Facebook Ups the Volume and Hiring an SEO

Infographic: How Gens X, Y and Z Consume Video Content A new infographic shows how content consumers across generations interact with video content — unsurprisingly, Gen Z consumes video at a higher frequency than their Gen Y and Gen X counterparts, but the platform and preferred format data may surprise you. AdWeek Facebook Videos Will Now Play With the Sound on by Default In contrast with a decision made last year to keep the sound from auto-playing on videos in the News Feed, Facebook has opted to turn the automatic sound back on for videos after rumored pressure from advertisers. User groups have reviewed this favorably, but there is also the option to disable auto-play with sound in your individual settings. AdAge How to Hire an SEO [Video] Hiring a good SEO is a daunting task for many — but Google’s Maile Ohye is here to help with some useful tips. First and foremost, SEO isn’t black magic. She also points out that an SEO is only as good as the site they have to work with in terms of credibility and content. Google Webmasters The Impact of Email List Segmentation on Engagement Personalizing content in your emails by segmentation works, according to a new study from MailChimp. The study saw open rates increase by 14% where segmentation was used, and a whopping 101% increase in click through rate. MarketingProfs Google Asked to Remove Over a Million Websites for Copyright Infringement Google’s Transparency Report showed that the search engine recently hit a major milestone – they’ve been asked to remove one million domains from their search results and two billion individual URLs. Some of these are due to copyright infringement or illegal content, but others are due to personal preference of the reporting user. For the latter, Google has the option to deny the request. Search Engine Journal Get Ready for Pinners to Search Outside the Box On February 8th, Pinterest announced Pinterest Lens (in Beta): “Pinterest Lens uses people’s mobile cameras to search for ideas using objects they see out in the real world. Just point Lens at a pair of shoes, and tap to see related styles.” Pinterest Facebook’s Rolling Out a New Job Posting Option for Pages Social Media Today reports: “This week, Facebook has confirmed that this new functionality is being rolled out to all business Pages, starting with North American-based organizations […] the workflow is fairly straightforward – when you want to advertise an open position, you click on the ‘Create Job’ option, which will be added to the new Page post options buttons.” Social Media Today Mobile makes up 21 pct. of online spending in Q4, as digital commerce reaches $ 109 billion ComScore released their 2016 eCommerce spending figures, and the results are in – consumers spent 21% ($ 22.7 Billion) of total online revenue (109.3 Billion) on mobile. Online revenue still didn’t overtake retail during the holiday season, but there’s always next year. Marketing Land What were your top online marketing news stories this week? We’ll be back next week with more online marketing news! In the meantime, keep the conversation going on Twitter @toprank or drop a note in the comments.

The post Online Marketing News: Generational Video, Facebook Ups the Volume and Hiring an SEO appeared first on Online Marketing Blog – TopRank®.

Online Marketing Blog – TopRank®

Looking Back: 2015 Holiday Email Volume

If you think you get a lot of email each day, it’s because you do. Research shows that in 2016, an estimated 215 billion emails are sent daily, an increase of 5 percent over the previous year. On any given day, the average email user receives approximately 123 emails in his or her inbox. 

At VerticalResponse, these are exciting numbers. We love email, of course — no surprise there. It’s easy, everyone uses it (including your customers), and it continues to thrive as a method of communication. As popular as social media and even texting are, nothing beats a carefully constructed email for getting your message out. 

In fact, VerticalResponse users contribute quite a bit to those emails flying around the cybersphere each day. We crunched some of our own numbers to see what our users have been doing, and how that activity changes around the holiday season. Here’s what we found:

VerticalResponse users sent over 650 million emails in 2015…

Our users generated more than 3 percent of total global email traffic last year! That includes more than 220 thousand separate email campaigns, with nearly 40 million opened emails. That’s an open rate that averages out to more than 6 percent.

…and nearly 750 million in the first nine months of 2016

In the first three quarters of 2016, VerticalResponse users have sent a whopping 744 million emails — that’s nearly 100 million more than all of last year, before taking October, November, or December into account. The number of email campaigns in that same timeframe was over 192,000, with an average open rate of nearly 7 percent.

The fourth quarter sees the heaviest use

In 2015, the fourth quarter dwarfed the others in number of emails sent: Almost 245 million compared to 177 million in the next highest quarter, the third. That’s an increase of 66 million (37.6 percent) over the third quarter, and a whopping 160 million (189 percent) over the first quarter, which saw the lowest number of emails sent. The number of email campaigns in the fourth quarter was also at its highest for 2015, with 66 thousand campaigns sent. That’s a lot of holiday emails! Open rates in the fourth quarter were more than 6 percent, which is the second highest rate of the year. 

Holidays and days of the week affect traffic

Certain email traffic patterns emerge from our 2015 data. In October 2015, VerticalResponse users’ email sends peaked on Thursday and Friday. In November, the highest-trafficked day was Monday, primarily because VerticalResponse usage skyrocketed on Nov. 30 — Cyber Monday itself. In December, Tuesdays and Thursdays saw the busiest email traffic, with, unsurprisingly, the two weeks before Christmas showing the highest numbers on those days. Throughout the entire fourth quarter, VerticalResponse customers sent the fewest emails on Saturday and Sunday.

Time of day matters

While the most popular days of the week varied based on calendar events happening in any given month, traffic according to time of day was more consistent regardless of the month. In the last quarter of 2015, the most popular time VerticalResponse users sent email was between 11 a.m. and 3 p.m. EST. The next most popular window was the three hours between 7 a.m. and 11 a.m. EST. Clearly, email marketing is most popular during the morning. But don’t discount afternoons too quickly: 1 p.m. to 5 p.m. EST was the third most trafficked window of the holiday months. Needless to say, sending emails during the night isn’t the most popular business practice. Email numbers drop to their lowest between 11 p.m. and 7 a.m. EST — but they don’t disappear completely. Some night owls or early risers are up during those hours, email marketing to their customers.

What this means for 2016 

This year’s fourth quarter will no doubt see even more email traffic than last year’s holiday season did, especially considering 2016 has already outperformed the entirety of 2015 in terms of email use. Retailers, restaurants, and other small businesses are making the most of their customers’ interest in holiday specials and year-end events, and they’re launching email campaigns accordingly. For a small business owner, determining which day and which time to send your emails may seem confusing. But take comfort in the fact that despite the astronomically high number of emails sent every single day, they still work: email marketing brings a 4,300 percent ROI on average, and email marketing is 40 percent more effective than social media. As our VerticalResponse users have discovered, small business owners cannot go wrong by using email. Happy sending this holiday season!


Don’t Have a VerticalResponse Account Yet?

It’s easy to use and free to get started. Sign up and send up to 4,000 emails per month for free.

Start for Free

© 2016, John Habib. All rights reserved.

The post Looking Back: 2015 Holiday Email Volume appeared first on Vertical Response Blog.

Vertical Response Blog

The Best of The Writer Files: Volume One


Before kicking off the next season of the show, we wanted to share with you some highlights from our previous seasons.

I don’t want to shortchange the most recent interviews with inspiring guests including Jay McInerney (’80s defining author of Bright Lights, Big City), Stephanie Danler (the bestselling author of Sweetbitter), the co-founder of Wired magazine Kevin Kelly, or How Neuroscientist Michael Grybko Defined Writer’s Block for us.

But I do want to dig into the archives with you and pull out a few of my favorites from a handful of the other 40 authors The Writer Files has cross-examined to learn how they keep the ink flowing, the cursor moving, and avoid writer’s block.

You’ll find links to these shows in the show notes on Rainmaker FM, and past episodes are easy to find in the archives of your favorite podcast app, in iTunes, or at WriterFiles.fm.

If you’re a fan of The Writer Files, click subscribe in iTunes to automatically see new interviews.

In this “Best of” Volume One, we’ll hear from a handful of past guests, including:

  • Advice columnist and critic Heather Havrilesky on social media and managed procrastination
  • NYTimes Bestselling Author of The Martian Andy Weir on productivity vs. laziness
  • Bestselling debut novelist Cynthia D’Aprix Sweeney on beating fear and procrastination
  • Bestselling thriller author Mark Dawson on how to publish more than a million words in a year
  • Bestselling author Ann Handley on the only reason to write a book

Listen to this Episode Now

The post The Best of The Writer Files: Volume One appeared first on Copyblogger.


Sweating the Details – Rethinking Google Keyword Tool Volume

Posted by rjonesx.

[Estimated read time: 13 minutes]

I joined Moz in August of 2015 and fell right into the middle of something great. Rand had brought his broad vision of a refined yet comprehensive SEO keyword tool to a talented team of developers, designers, data scientists and project managers… and now, me.

I was hoping to ease in with a project that was right up my wheelhouse, so when the “Volume” metric in Keyword Explorer was pitched as something I could work on, I jumped right on it. In my mind, I was done the second the work was offered to me. I already had a giant keyword volume database at my disposal and a crawling platform ready to fire up. All I had to do was tie some strings together and, voilà.

Peer pressure

It was subtle at first, and never direct, but I quickly began to see something different about the way Moz looked at problems. I’ve always been a bit of a lazy pragmatist — when I need a hammer, I look around for something hard. It’s a useful skill set for quick approximations, but when you have months set aside to do something right, it’s a bit of a liability instead.

Moz wasn’t looking for something to use instead of a hammer; they were looking for the perfect hammer. They were scrutinizing metrics, buttons, work flows… I remember one particularly surreal discussion around mapping keyboard shortcuts within the web app to mimic those in Excel. So, when on my first attempt I turned up in a planning meeting with what was, essentially, a clone of Google Keyword Planner volume, I should have seen it coming. They were polite, but I could feel it — this wasn’t better, and Moz demanded better in their tools. Sometimes peer pressure is a good thing.

If it ain’t broke, don’t fix it.

Rand was, unsurprisingly, the first to question whether or not volume data was accurate. My response had always been that of the lazy pragmatist: “It’s the best we got.” Others then chimed in with equally valid questions — how would users group by this data? How much do we have? Why give customers something they can already get for free?

Tail tucked between my knees, I decided it was time to sweat the details, starting with the question: “What’s broke?” This was the impetus behind the research which lead to this post on Keyword Planner’s dirty secrets, outlining the numerous problems with Google Keyword Planner data. I’ll spare you the details here, but if you want some context behind why Rand was right and why we did need to throw a wrench into the conventional thinking on keyword volume metrics, take a look at that post.

Here was just one of the concerns — that Google Adwords search volume puts keywords into volume buckets without telling you the ranges.

Image showing that Google Keyword Planner averages are heavily rounded.

Well, it’s broke. Time to sweat the details!

Once it became clear to me that I couldn’t just regurgitate Google’s numbers anymore and pretend they were the canonical truth of the matter, it was time to start asking the fundamental questions we want answered through a volume metric. In deliberation with the many folks working on Keyword Explorer, we uncovered four distinct characteristics of a good volume metric.

  1. Specificity: The core of a good volume metric is being specific to the actual average search volume. You want the volume number to be as close as possible to reality.
    We want to be as close to the average annual search volume as we possibly can.
  2. Coverage: Volume varies from month to month, so not only do you want it to be specific to the average across all months, you want it to be specific to each individual month. A good volume metric will give you reasonable expectations every month of the year — not just the whole year divided by 12.
    We want the range to capture as many months as possible. Highlighted range on graph with spike at the end.
  3. Fresh: A good volume metric will take into account trends and adjust to statistically significant variations which diverge from the previous 12 months.
    We want to detect trending keywords early on so we can predict volume and track it more closely. Graph with spike at the end.
  4. Relatable: A good volume metric should allow you to relate keywords to one another when they are similar in volume (i.e.: grouping).

We can actually apply these four points to Google Keyword Planner and see its weaknesses…

  1. Specificity: Google’s Keyword Volume is a yearly rounded, bucketed average of monthly rounded, bucketed averages
  2. Coverage: For most keywords, the average monthly search is accurate only 33% of the months of the year. Most months, the actual volume will land in a different volume bucket than the average monthly search.
  3. Fresh: Keyword Planner updates once a month, with averages not providing predictive value. A hot new keyword will look 1/12th its actual volume in the average monthly search, and it won’t show up for 30 days.
  4. Relatable: You can group keywords in 1 of the 84 different volume buckets, with no explanation as to how the groups were formed. (They appear to be associated with a simple logarithmic curve.)

You can see why we were concerned. The numbers aren’t that specific, have ranges that are literally wrong most of the time, are updated regularly but infrequently, and aren’t very group-able. Well, we had our work cut out for us, so we began in earnest attacking the problems…

Balancing specificity and coverage

As you can imagine, there’s a direct trade-off between specificity and coverage. The tighter the volume ranges, the higher the specificity and lower the coverage. The broader the ranges, the lower the specificity and higher coverage. If we only had one range that was from zero to a billion, we would have horrible specificity and perfect coverage. If we had millions of ranges, we would have perfect specificity but no coverage. Given our weightings and parameters, we identified the best possible arrangement. I’m pretty sure there’s a mathematical expression of this problem that would have done a quicker job here, but I am not a clever man, so I used my favorite tool of all: brute force. The idea was simple.

  1. We take the maximum and minimum boundaries of the search volume data provided by Google Keyword Planner, lets say… between 0 and 1 billion.
  2. We then randomly divide it into ranges — testing a reasonable number of ranges (somewhere between 10 and 25). Imagine randomly placing dividers between books on a shelf. We did that, except the books were keyword volume numbers.
  3. We assign a weighting to the importance of specificity (the distance between the average of the range min and max from the keyword’s actual average monthly search). For example, we might say that it’s 80% important that we’re close to the average for the year.
  4. We assign a weighting to the importance of coverage (the likelihood that any given month over the last year falls within the range). For example, we might say it’s 20% important that we’re close to the average each month.
  5. We test 100,000 randomly selected keywords and their Google Keyword Planner volume against the randomly selected ranges.
  6. We use the actual average of the last 12 months rather than the rounded average of the last 12 months.
  7. We do this for millions of randomly selected ranges.
  8. We select the winner from among the top performers.

It took a few days to run (the longer we ran it, the rarer new winners were discovered). Ultimately, we settled on 20 different ranges (a nice, whole number for grouping and displaying purposes) that more than doubled the coverage rate over the preexisting Google Keyword Planner data while minimizing damage to specificity as much as possible. Let me give an example of how this could be useful. Let’s take the keyword “baseball.” It’s fairly seasonal, although it has a long season.

Bar graph showing seasonality of keyword "baseball." The actual search volume falls within the Moz range 10 out of 12 months of the year. The Google search volume only matches 3 out of 12 months. Ranges give us predictability with best and worst case scenarios built in.

In the above example, the Google Average Monthly Search for Baseball is 368,000. The range this covers is between around 330K and 410K. As you can see, this range only covers 3 of the 12 months. The Moz range covers 10 of the 12 months.

Now, imagine that you’re a retailer that’s planning PPC and SEO marketing for the next year. You make your predictions based on the 368,000 number given to you by Google Keyword Planner. You’ll actually under-perform the average 8 months out of the year. That’s a hard pill to swallow. But, with the Moz range, you can use the lower boundary as a “worst-case scenario.” With the Moz range, your traffic under-performs only 2 months out of the year. Why pretend that we can get the exact average when we know the exact average is nearly always wrong?

Improving relatability

This followed naturally from our balancing specificity and coverage. We did end up choosing 20 groupings over some higher-performing groupings that were less clean numbers (like 21 groupings) for aesthetic and usability purposes. But what this means is that it’s easy to group keywords by volume and not in an arbitrary fashion. You could always group by ranges in Excel, if you wanted, but the ranges you came up with off the top of your head wouldn’t have been validated in any way regarding the underlying data.

Let me give an example why this matters. Intuitively, you’d imagine that the ranges would increase in broadness in a similar logarithmic fashion as they get larger. For example, you might think most keywords are 10% volatile, so if a keyword is searched 100 times a month, you might expect some months to be 90 and others 110. Similarly, you would expect a keyword searched 1,000 times a month to vary 10% up or down as well. Thus, you would create ranges like 0–10, 100–200, 1,000–2,000, etc. In fact, this appears to be exactly what Google does. It’s simple and elegant. But is it correct?

Nope. It turns out that keyword data is not congruent. It generally follows these patterns, but not always. For example, in our analysis, we found that while the volume range after 101–200 is 201–500 (a 3x increase in broadness), the very next optimal range is actually 501–850, only a 1/6th increase in broadness.

This is likely due to non-random human search patterns related to certain keywords. There are keywords which people probably search daily, weekly, monthly, quarterly, etc. Imagine keywords like “what is the first Monday of this month” and “what is the last Tuesday of this month.” All of these keywords would be searched a similar number of times by a similar population a similar number of times each month, creating a congruency that is non-random. These patterns create shifts in the volatility of terms that are not congruent with a natural logarithmic scale you would expect if the data was truly random. Our machine-learned volume ranges capture this non-random human behavior efficiently and effectively.

We can actually demonstrate this quite easily in a graph.

Upward-trend line graph of log of keyword planner ranges. Google's range sizes are nearly perfectly linear, meaning they are not optimized at all to accommodate the non-linear, non-random nature of search volume volatility and seasonality.

Notice in this graph that the log of Google’s Keyword Planner volume ranges are nearly linear, except at the tail ends. This would indicate that Google has done very little to try and address patterns in search behavior that make the data non-random. Instead, they apply a simple logarithmic curve to their volume buckets and leave it at that. The R2 value shows just how close to 1 (perfect linearity) this relationship is.

Upward-trend line graph of log of Moz ranges. Moz's Keyword Explorer volume ranges are far less linear, as they're trained to maximize specificity and coverage, exploiting the non-random variations in human search patterns.

The log of Moz’s keyword volume ranges are far less linear, which indicates that our range-optimization methodologies found anomalies in the search data which do not conform to a perfect logarithmic relationship with search volume volatility. These anomalies are most likely caused by real non-random patterns in human search behavior. Look at positions 11 and 12 in the Moz graph. Our ranges actually contract in breadth at position 12 and then jump back up at 13. There is a real, data-determined anomaly which shows the searches in that range actually have less volatility than the searches in the previous range, despite being searched more often.

Improving freshness

Finally, we improved freshness by using a completely new, thirrd-party anonymized clickstream data set. Yes, we analyze 1-hour delayed clickstream data to capture new keywords worth including both in our volume data and our corpus. Of course, this was a whole feat in and of itself; we have to parse and clean hundreds of millions of events daily into usable data. Furthermore, a lot of statistically significant shifts in search volume are actually ephemeral. Google Doodles are notorious for this, causing huge surges in traffic for obscure keywords just for a single day. We subsequently built models to look for keywords that trended upward over a series of days, beyond the expected value. We then used predictive models to map that clickstream search volume to a bottom quartile range (i.e.: we were intentionally conservative in our estimates until we could validate against next month’s Google Keyword Planner data).

Finally, we had to remove inherent biases from the clickstream dataset itself so that we were confident our fresh data was reliable. We accomplished this by…

  1. Creating a naive model that predicts Google Keyword Volume from the clickstream data
  2. Tokenizing the clickstream keywords and discovering words and phrases that correlate with outliers
  3. Building a depressing and enhancing map of these tokens to modify the predictive model based on their inclusion
  4. Applied the map to the naive model to give us better predictions.

This was a very successful endeavor in that we can take raw clickstream data and, given certain preconditions (4 weeks of steady data), we can predict with 95% accuracy the appropriate volume range.

A single metric

All of this above — the research into why Google Keyword Planner is inadequate, the machine-learned ranges, the daily freshness volume updating, etc. — all goes into a single, seemingly simple, metric: Volume Ranges. This is probably the least-scrutinized of our metrics because it’s the most straightforward. Keyword Difficulty, Keyword Opportunity, and Keyword Potential went through far more revisions and are far more sophisticated in their approach, analysis, and production.

But we aren’t done. We’re actively looking at improving the volume metric by adding more and better data sources, predicting future traffic, and potentially providing a mean along with the ranges. We appreciate any feedback you might offer, as well, on what the use cases might be for different styles of volume metrics

However, at the end of the day, I hope what you come away with is this: At Moz, we sweat the details so you don’t have to.

A personal note

This is my first big launch at Moz. While I dearly miss my friends and colleagues at Angular (the consulting firm for whom I worked for the past 10 years), I can’t say enough about the amazing people I work with here. Most of them will never blog here, won’t tweet, and won’t speak at conferences. But they deserve all the credit. So, here’s a picture of my view from Google Hangouts from a Keyword Explorer meeting. Most of the team was able to make it, but those who didn’t, you know who you are. Thanks for sweating the details.

Google Hangout screenshot of the Moz Keyword Explorer team during a meeting. Russ is connected remotely in the corner.

Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!

Moz Blog