The Ultimate Guide to Keyword Research (and Why It's Important)

There are several factors involved in determining how a website performs in search engine results pages, or SERPS. A site’s history is one element taken into consideration, alongside estimated page loading times or the quantity of inbound links.
Then there’s the content. Search engines process queries for specific words and phrases, so including these keywords within website text has a huge impact on how that site performs in search results. Keywords can be divided into three distinct categories:

1.     One-word phrases like “bread”, known as head keywords. These generic terms identify a product or industry, but are best avoided in marketing campaigns because they often have little relevance to potential customers.
2.   Two- or three-word phrases like “Manhattan bakery”. Typically attracting thousands of monthly searches, these narrow down an industry to a specific location or niche.
3.     Multi-word phrases like “gluten free bakery Manhattan”, known as long tails. These less common searches collectively comprise 70 per cent of web searches. Because they’re specific, they’re also most likely to drive traffic when managed correctly.

Nobody knows the precise composition of the algorithms Google and Bing/Yahoo use to rank page results, but pages containing an exact long tail match will always perform strongly. Searches where long tail words appear in a different order will also rank relatively well. This even applies to searches including stop words, which are effectively ignored by the search engines to speed up results. Searching for “a gluten free bakery in Manhattan” will disregard “a” and “in” from the search string, so this enquiry matches the long tail in point 3 above.
In this article, we consider the importance of optimizing your site by researching relevant keywords and long tails. We look at various tools of the trade, and explain why it isn’t necessary to employ a professional agency to undertake keyword research. But first, let’s consider a pressing question…

Why does keyword research matter?

There are over a billion websites currently in existence, typically comprising several pages, each of which is populated with a significant amount of content. To ensure your site performs strongly in search engine results pages, or SERPS, it needs to contain relevant data that elevates it above competitors. If it doesn’t, the only traffic your site will receive will be from people who deliberately type its address into their browser bar. Everyone else will remain unaware of its existence.
Since search engines are effectively the internet’s librarians, they decide where to direct people performing web searches. They scan and log text on every website and subpage in an endless process known as crawling. This article contains the term “keyword research tool” five times, so crawlers will identify the URL currently displaying in your web browser as a useful resource for future searches involving that particular phrase.
Keywords don’t just drive traffic to your website – they drive relevant traffic, as opposed to people with no interest in your products and services. Companies can target specific demographics, promote key products and services, and outperform their competitors. Achieving this is a process known as search engine optimization, or SEO, and it can only happen on pages with written content. If a page is full of photos without any accompanying text or captions, it simply won’t perform in search results.
The Holy Grail of SEO is to appear on page one of search results, where over 90 per cent of searches end. This means failing to appear on the first page of Google or Bing causes you to miss over 90 per cent of your potential target audience. Even worse, if you’re not on the first two pages, over 96 per cent of audiences will be missed. Indeed, some might argue ranking 21st or worse is relatively pointless.
Of course, organic SEO rankings are only part of the puzzle. Many companies undertake targeted advertising campaigns through platforms like Google AdWords, where ads are displayed beside results for specific web searches. Ensuring you’re targeting the right audiences is critically important, since every clickthrough from an advert to your site incurs a fee. Common terms command higher prices, so it’s vital to strike a balance between popular phrases and those most likely to generate new custom or inquiries. Without adequate keyword research, you’ll probably end up wasting your monthly advertising budget on irrelevant searches.

Okay, I’m convinced. What will keyword research tell me?

Keyword research is effectively a form of online market research into your customers. It reveals what they’re looking for, when, and how. It highlights where competitors are generating their organic traffic from, as well as any keywords and long tails they’re underperforming on. That might enable you to steal a march on them.
Keyword research often identifies specific audience demographics. It could, for example, reveal that most searches are carried out by women in their twenties and thirties, and in particular neighborhoods. You can then choose keywords with specific appeal to these market segments, instead of targeting people of all ages and genders throughout your home city. This is known as identifying buyer personas, though it’s important to recognize a business may appeal to numerous personas at once.

Mistaken identity: who is your audience?

A common mistake made during the creation of online content is to second-guess your audience. You might assume that the public usually search for one specific product or service, when they’re actually looking for something entirely different. Equally, a product designed for a specific demographic may appeal more to other groups. It’s easy to create content for a non-existent audience, or waste money on misplaced advertising, all because of a lack of research.
Another common mistake involves taking a shortcut with a generic keyword research tool like Google Keyword Planner. Let’s assume you’re setting up an artisan bakery business. The cornerstone of any bakery is bread, so you type the word “bread” into GKP. A list of related terms is instantly generated – “bread recipe”, “bread retailer”, and so forth.
It’s easy to therefore assume that these are the terms to build your keyword strategy around, but that’s not always the case:

1.               Everyone else will see these terms, potentially targeting the same audience.
2.               GKP takes no account of your particular involvement with (or interest in) bread.
3.               Every term will include the word “bread”, which limits the relevance of results.

The last point is particularly important. Running a GKP for “bread” won’t identify searches for “best loaf”, “artisan bakery” or “where can I buy nice bagels”. It won’t call up geographic identifiers which may attract local audiences. It won’t deliver topical results, since Google averages search volumes out over a twelve-month period. In short, it provides a very narrow insight into your audience.
Deciding which keywords to search for and expand upon requires a degree of industry knowledge. It also involves patience – some of the terms you might expect to be popular will generate very little web traffic, and aren’t worth bothering with. Other terms may be dominated by competitors who’ve already undertaken this process, making it impractical to outperform them. And the process of brainstorming keywords to include in your written content can be frustrating if you’re not good at similes or word association. Fortunately, there are software tools dedicated to keyword research – and websites providing it almost by accident…

Identifying value

Before we consider which keyword research tools should be used to undertake research, it’s worth considering a few mainstream platforms that might be of assistance:

1.    Wikipedia. This is great for identifying subtopics or categories. For instance, the Wiki page for bread includes a subsection on sourdough. Sourdough is a good keyword to target when setting up an artisan bakery.
2.    Reddit. Trawling Reddit often reveals pertinent terms like “gluten free bread” or “favorite bagel flavor”. If these are prevalent or trending topics, create a blog or news story on your site and then link to it from the subreddit.
3.     Twitter. One of Twitter’s enduring strengths is its hashtag functionality, which has since been copied by Facebook. Trawl for keywords and trending topics, see the terms other people use, and consider whether these related words and phrases are of value.
4.    Google. If you search for “artisan bakery Manhattan” in Google, a variety of related search terms will appear below page one’s results. These indicate what people might also be looking for, and can be used to compile a list of related search terms.

Keyword Research Tools of the Trade

When it comes to dedicated keyword research tools, these are some of the best platforms to use. Each will compile a detailed report in seconds, enabling users to drill down into their statistics for comprehensive information:

1.     Soovle. This is effectively an aggregator for search engines, social media platforms and reference sites like Answers.com or Wikipedia. As you start entering a term, predictive keywords and long tails associated with these sites will be overlaid across their logos. Some results will be surprising or irrelevant, but there’s also great scope for identifying popular terms that should be published liberally across your own site.
2.  Ahrefs Keywords Explorer. With a database of four billion keywords, Ahrefs provides a wealth of keyword data. Metrics include search volumes (with and without clicks), required backlink targets and an extensive list of keyword variations. Ahrefs also measures the percentage of search results that are clicked on, which gives a far more accurate indicator of traffic potential than just a basic search volume.
3.     UberSuggest. As easy to use as the search engines it’s inspired by, UberSuggest enables you to search for words or phrases across specific components of the internet like news sites or YouTube. A comprehensive list of alphabetically ordered results will appear within seconds, so you can identify new keywords or test the popularity of existing ones.
4.     Google Analytics. Google’s keyword tool has a considerable advantage, since it’s been designed by the world’s largest search engine. GA knows which keywords your site is currently being found with, where traffic comes from and the demographics of each visitor. Its sheer volume and complexity of information is overwhelming at first, but time spent understanding this uniquely informative platform is worthwhile.
5.     SEMrush. SEMrush is particularly useful for identifying the keywords your competitors are already performing well for. By entering a domain name, it’s possible to view the organic keywords that rival webpages rank for. Competitor analysis can identify areas of weakness or steal a march on them. It’s even possible to view a rival’s own competitor sites, drilling down into their rankings in turn.
6.   KWFinder. The sheer amount of information contained within this long tail research platform may feel daunting. Its main interface displays trends, volumes, the likely cost per click for advertising purposes and so forth. Each keyword is given a difficulty level from one to 100, breaking results down by backlinks, social shares and traffic volumes. Keywords can also be trawled by city, state or country.

Do I need professional assistance?

The simple answer is no. The internet is awash with legitimate marketing companies offering to optimize SEO, and boost web traffic. Your email inbox is probably full of dubious overseas agencies, guaranteeing a position on the first page of Google search results. Since they send the same emails to thousands of companies at once, that’s clearly a fraudulent promise. It may also involve disreputable black hat marketing techniques, potentially downgrading the site’s ranking or even getting it blacklisted from future results.
The true path to keyword enlightenment comes from within one of the packages outlined above. Having determined which terms you want to perform well for, start editing and revising written web content accordingly. Target topical keywords with a news page or blog, complementing core pages without replacing or detracting from them. Updating your site on a weekly basis reassures web crawlers that it’s active and relevant, providing another boost to overall ranking performance.
Finally, there are a few general rules to be aware of.  Stuffing a website with keywords will cause search engines to downgrade it, so incorporate each keyword three or four times per page. Prioritize a handful of terms rather than squeezing in too many – content has to be readable by people, as well as appealing to web crawlers. And never consider your work to be finished, since search engine algorithms evolve over time and websites are always being launched or deleted. Keyword optimization is an ongoing process, not a one-off challenge.