Search improvement

How We Improved Search Accuracy by Indexing Keywords Extracted From Descriptions

We show that instead of using long text like description for scoring, using keywords extracted from description leads to significant improvement in accuracy.

Alexander Kingson September 11, 2026 4 min read Solr OpenSearch Elasticsearch

We were getting too many irrelevant documents ranked highly in a movie and TV search index.

Users searching for "romantic comedy" were seeing dark dramas that merely mentioned a failed romance. Searches for "space documentary" were returning sci-fi movies with one sentence about "the mysteries of space." At first glance, nothing looked obviously broken. The mappings looked reasonable, and the usual relevance controls were already in place.

After reviewing a sample of low-precision queries, the culprit was almost always the same field: movie and TV descriptions.

The Problem With Movie Descriptions

Movie and TV descriptions are usually written to summarize, market, and create intrigue. They are not written as structured search relevance data.

A description might say:

"Unlike traditional space documentaries, this fictional thriller explores isolation aboard a failing orbital station."

To a viewer, those descriptions are useful. To a search engine, they can be dangerous.

The engine sees terms like comedy, romance, space, documentary, child, adventure, family, and thriller. But many of those terms are not actually describing the content category. They may be plot references, comparisons, negations, mood-setting language, or promotional copy.

That distinction matters. A dark drama that mentions a comedian should not rank highly for comedy. A fictional thriller described as "unlike traditional space documentaries" should not rank highly for space documentary.

Why Boosting Did Not Fully Solve It

The first instinct in relevance tuning is often to adjust boosts: boost the title, lower the description, add more weight to genre, add cast matching, tune BM25, or add phrase matching.

Those changes helped, but they did not solve the problem. The reason was simple: the description field still contained too much noisy evidence.

If a field contains misleading terms, lowering its boost only reduces the damage. It does not remove the bad signal.

Removing descriptions from search entirely was not a good option either. They still helped users find titles with specific natural-language searches, such as "movie about a detective in London" or "show about time travel." Those details often live only in the synopsis, not in fields like title, genre, cast, or director.

The description field had value. The problem was that it mixed useful searchable concepts with noisy promotional language.

The Fix: Extract Searchable Description Keywords

Instead of using raw descriptions as a primary search field, we added a preprocessing step before indexing.

The goal was to extract only the terms that actually conveyed the meaning of the description. In this case, the extraction layer used a lightweight classifier to identify useful searchable terms. A MaxEnt-style classifier is one way to do this, but the exact model is less important than the principle:

Separate descriptive signal from synopsis noise before indexing.

For example, given this description:

"Unlike traditional space documentaries, this fictional thriller follows a stranded engineer aboard a failing orbital station as she fights to survive."

The extracted searchable terms were:

fictional thriller
stranded engineer
orbital station

The ignored terms were:

traditional space documentaries

The topic of this document was not a documentary. It is a fictional thriller set in space. Those are very different relevance signals.

The Indexing Change

The original description stayed in the index for display, but primary matching moved away from the raw description field and toward a curated field:

description_keywords

A simplified document looked like this:

{
  "title": "Orbital Silence",
  "type": "movie",
  "genres": ["thriller", "science fiction"],
  "description": "Unlike traditional space documentaries, this fictional thriller follows a stranded engineer aboard a failing orbital station as she fights to survive.",
  "description_keywords": [
    "fictional thriller",
    "stranded engineer",
    "orbital station",
    "survival",
    "space setting"
  ]
}

The search query changed from treating raw description as a major relevance field to using the extracted keyword field instead:

{
  "multi_match": {
    "query": "space thriller",
    "fields": [
      "title^5",
      "genres^4",
      "cast^3",
      "director^2",
      "description_keywords^2"
    ]
  }
}

The Results

The improvement was immediate. False positive matches from synopsis noise dropped sharply, and relevance tuning became more predictable.

That is an underrated part of relevance engineering:

Cleaner index signals make every downstream ranking decision easier.

The General Lesson

Movie and TV descriptions are editorial copy. They are not structured relevance data. Treating them the same as titles, genres, cast, directors, ratings, and content attributes invites false positives.

That does not mean descriptions should be ignored. It means they should be handled carefully.

For media catalogs, the better approach is often:

store raw descriptions for display
extract useful searchable concepts
search extracted concepts at a controlled boost
keep structured metadata separate
measure the impact on real queries

Sometimes the best fix is cleaning the signal before it ever reaches the index.


Alexander Kingson is the founder of IndexNgin, a managed search and search consulting platform. He previously built search infrastructure at Adobe and DirecTV.