What is Natural Language Processing (NLP)?

First of all, let’s say a few words about the specialized data used in NLP. When textual data is generated from conversations, tweets, declarations, posts and comments on Facebook, it is arbitrary and ambiguous. These pieces of machine-readable texts are collected in a dataset or a text corpus (pl. corpora), a name usually used in linguistics and NLP. The content of the corpus is defined by specification of the NLP task. In topic segmentation, corpora contain a balanced number of texts on different topics: religion, science, science fiction, detective fiction, etc. The Brown corpus is an example of such a corpus, consisting of 500 English text samples (1 million words), which are tagged with 80 different tags. In machine translation tasks, corpora contain texts in multiple languages. Another example of a corpus is the Switchboard corpus, which was collected for speech recognition in phone operation. It contains conversational speech and corresponding text of the different sexes and all different dialects in the United States.

1 What is Natural Language 
Processing (NLP)_

The first step in the pipeline is to split our working document into units. In text analysis, the unit is called “token”, a string of symbols that represent text. Usually, one token corresponds to one word. Splitting into tokens starts with sentence segmentation. A machine can easily separate sentences using a punctuation mark (modern NLP pipelines use more complex techniques in the case of unformatted documents). Then, we split apart words whenever there is a space or punctuation mark between them - this is called tokenization.

The next step is to predict a part of speech for each token and tag it accordingly. Parts of speech (e.g., verbs, nouns, adjectives) are important, because they indicate how a word is functioning within the context of a sentence. NLTK is one of the more widely-used tools for part-of-speech tagging.

Due to the fact that, in most languages, words appear in different forms (e.g. be, is, was, are, or apple, apples) it is very helpful to find a basic form or lemma of each word in the sentence. This is called lemmatization. The lemmatization process is often used to decrease the number of overall word classes. We can provide lemmatization by using a table with lemma forms of words (for example, WordNet for English) based on their parts of speech.

There are also words that you might want to filter before statistical analysis, like “the”, “and”, “is”. These words are called stop words and they produce a high amount of noise due to their frequency in texts.

These steps are basic for extracting deeper levels of understanding from your data, like morphological, syntactic and semantic levels. Then, depending on the NLP task, one or several models are used for further analysis. We’ll consider some examples in this article.

Classification is a very popular text classification technique used for sentiment analysis of reviews, message polarity, reactions, etc. For example, this technique computes relative frequencies of positive words (“good”, “awesome”) and negative words (“bland”, “horrible”) in order to find negative and positive reviews.

The most commercially successful text classification model is one used for spam filtering: the Naïve Bayes classification model uses corpus of spam and not spam emails to predict the probabilities of a word’s presence in spam emails based on their frequencies.

Effective language models should take into account the localization words in their context. Such an approach will be able to detect sentiment much better than classification models. The “bag-of-words” technique evaluates the frequency with which words co-occur with themselves and other words in a specific context. Thus, the model explores relationships between documents that contain the same mixture of individual words. This can be very useful when we want to do topic segmentation, because texts of a specific discipline usually contain specific vocabulary.

The context in which the words appear plays a huge role in conveying meaning. Extensions of the “bag-of-words” model consider co-occurrences of phrases in texts. This is called n-gram analysis (n simply means number of words in a phrase to scan on) and it is very effective in extracting meaning from texts.

Currently, neural networks play an increasingly significant role in text processing. Almost all big challenges of NLP are addressed now to neural networks, such as machine translation, summarization, paraphrasing, question-and-answer, dialog, and other challenges; Alexa’s speech recognition andGoogle Translate are just two examples of such applications. There are still many limitations in existing NLP models, e.g. it is very difficult for a machine to recognize sarcasm in sentiment analysis, and, furthermore, natural language changes with time (new words appear, and the meaning of existing words evolves).