Natural Language Processing (NLP) Solutions

Which language task does the process actually need?

“Natural language processing” covers several different jobs, and the wrong job is a common reason text projects disappoint. A team that wants fewer manual reviews of incoming messages needs classification. A team that wants the amounts and dates out of supplier invoices needs extraction. A team that wants a long case history reduced to what the next person needs to know needs summarization. Those tasks have different inputs, different failure modes and different ways of being checked, so the first decision is which one the process requires, written in terms of what the output will be used for.

This page is for the operations manager, service lead or digital owner who is looking at text that is currently read, sorted or retyped by people, and who has to decide whether language technology should take on some of that work. It explains how to match task to output, how to judge whether the text is suitable, and how to evaluate a result before a person relies on it.

Matching the language task to the output the process uses

The right column matters more than the left. An output that looks fluent can still be wrong in a way that is hard to notice, so the check should be designed for the failure the table names rather than for overall impressions.

Why language coverage and domain terms decide accuracy

Language models and classic text pipelines both learn from examples, which means their accuracy follows the examples they were built on. Text that uses the vocabulary of a particular trade, a region, a product line or an internal abbreviation may be handled far less reliably than general prose. A message that reads “the RMA for the unit is stuck in the portal” depends on knowing what an RMA is, what the portal does and what stuck means in that team. Before any evaluation, collect the terms the business uses, the abbreviations that appear only internally and the phrasing customers actually choose, because these are where a generic system most often drifts.

Language coverage needs the same attention. Mixed-language messages, transliterated names, and text that switches between languages within a sentence can all produce confident errors. Where the process serves several languages, measure performance in each one separately rather than trusting a single blended figure. A process that works well in the dominant language and poorly in a minority one has a fairness problem that an average will hide.

Evaluating on a labeled sample from your own traffic

General benchmarks say little about your text. The evaluation that matters uses messages, documents or records drawn from the same channel and period the service will run on, with a label applied by people who do the work. Several practical details make the difference between a useful evaluation and a misleading one.

• Sample across the whole period, including the busy weeks and the unusual ones, so that rare but costly cases are present.
• Have two people label a subset independently and measure how often they agree. If they disagree, the task definition is not clear enough yet.
• Keep the evaluation set separate from any examples used to tune the system, and do not reuse it after changing the design without accepting that its results are now optimistic.
• Report errors by category, so that a misrouted complaint is counted differently from a misrouted routine query.
• Record examples the system was least sure about, since those will shape the review queue in production.

Setting thresholds for human review

Most text services that work in practice combine automation with a review step. The decision is where the line falls. Text that customers, patients or staff will read directly needs a different threshold from text that is only used to route a message. A reasonable design sends low-confidence results, results touching sensitive categories and results in a language with weaker performance to a person, and it lets high-confidence results in routine categories flow through with sampling. The sampling rate should be set in advance and reviewed when the error pattern changes.

A reviewer’s time is part of the design. If review queues grow faster than the people available, the threshold is effectively set at whatever point the team gives up, and the service will quietly stop being reviewed. Measure the review queue as a service metric from the first day.

Classic pipelines or large language models: how to choose

Older text pipelines are narrow, predictable and cheap to run. They do one labeling or extraction job well when the inputs are consistent and the categories are fixed. Large language models handle variety and unfamiliar phrasing more gracefully, and they can summarize or answer questions that a fixed pipeline cannot. They are also harder to constrain, more expensive per request, and more prone to producing text that sounds right without being supported by the source. For a stable routing task with a clear category list, the simpler approach often remains the better choice. For open-ended reading of varied documents, the more flexible approach may be necessary, provided its outputs are checked against sources.

The choice should be revisited when the task changes. A routing service that later needs to draft replies has crossed into a different risk category, and its evaluation, review and data handling need to be rebuilt for the new output.

Handling personal details in text

Customer messages, case notes and supplier correspondence contain names, contact details, account references and sometimes health or financial information. Decide before any processing what the service may keep, where processing happens, who may see the output and how long traces of the text remain in logs. Redaction before processing can reduce exposure for some tasks, though it can also remove the context a classifier needs, so it should be tested rather than assumed. Data-protection obligations apply to the text whether or not the service is automated, and they can determine whether a given provider can be used at all.

Questions teams ask about language processing

Can a chatbot simply replace the contact-center queue?

Only for a narrow set of requests with clear answers and a clean handover to a person. A conversational interface that cannot recognize when it is out of depth creates more work than it saves, so the handover rules are part of the design.

How do we know a summary is faithful?

Test it against the source. Check whether each statement in the summary can be traced to a passage in the record, and whether anything material has been dropped, especially caveats, dates and negatives.

Is translation good enough for customer-facing text?

For routine content with reviewed terminology, often yes. For legal, medical or contractual wording, a fluent reviewer should check the terms that carry the meaning, because a fluent sentence can still change a commitment.

Next step for a text-processing assessment

Bring a description of the text, the output the process needs, the volume and the languages involved, and a few real examples with their correct handling. From those it becomes possible to judge which task fits, what an evaluation sample would need to contain and where review must sit. The document and knowledge workflows page covers classification, extraction, drafting, retrieval and routing in document workflows, and the knowledge retrieval readiness page covers whether a system can find the right source, respect access and expose uncertainty. Where the text is handled by a ServiceNow Virtual Agent, the NLU model concept explains how Virtual Agent classifies and routes requests, and the virtual agent topic concept explains how a topic guides a conversation toward a task. Discuss your text-processing requirement with us when a sample of real messages is ready to review.