1. Overview
Natural Language Processing (NLP) is a branch of Artificial Intelligence (AI) that enables computers to understand written or spoken language similarly to humans. It has diverse applications in this era of the AI revolution.
In this tutorial, we’ll explore different NLP libraries in Java and see how to implement some NLP tasks using Apache OpenNLP and Stanford CoreNLP.
2. What Is NLP?
NLP empowers computers to process texts and words in a similar way to humans. It combines computational linguistics with statistics, deep learning, and machine learning.
Humans interact with each other online via various media every day. In doing so, they share different types of data, such as text, speech, images, etc. These data are essential for understanding human behavior and habits. Hence, they are used to train computers to mimic human intelligence.
NLP uses data to train machines to mimic human linguistic behavior. To do this, it follows a process that consists of several steps:
- It segments the text into smaller units, such as sentences or words.
- It tokenizes the text, which means it assigns a unique identifier to each word.
- It removes stops words, which are common words that don’t add much meaning to the text, such as “the”, “a”, “and”, etc.
- It stems or lemmatizes the text, which means it reduces each word to its root form or dictionary form.
- It tags each word with its part of speech.
- It tags each word with its named entity, such as a person, location, organization, etc.
3. Use Cases of NLP
NLP is the driving power behind machine intelligence in numerous modern real-world applications.
Machine translation is an example use case. We have systems that can translate from one specific language to another language. One such example is Google Translate. The technology that drives machine translation is based on NLP algorithms.
Furthermore, another popular use case is spam detection. Most popular email service providers use spam detectors to determine if an incoming message is a spam. Spam detection applies NLP text classification techniques to identify spam emails based on their language patterns.
Moreover, AI chatbots are now very common. Popular examples include Siri, Google Assistant, Alexa, etc. These applications use speech recognition and natural language to recognize patterns in voice and respond with appropriate, helpful comments.
NLP is the core logic of these applications because it enables them to process natural language inputs and outputs, such as text and speech, and to understand the meaning and intent behind them.
4. OpenNLP
The Apache OpenNLP is a toolkit that utilizes machine learning to process natural language text. It provides support for common NLP tasks like tokenization, segmentation, speech tagging, etc.
The major goal of Apache OpenNLP is to provide support for NLP tasks and provide a large number of pre-built models for diverse languages. Furthermore, it provides a command line interface (CLI) for easy experiments and training.
There are various pre-built models available for download from Apache OpenNLP. Let’s implement a simple language detector using a pre-built model. First, let’s add the OpenNLP dependency to the pom.xml:
<dependency>
<groupId>org.apache.opennlp</groupId>
<artifactId>opennlp-tools</artifactId>
<version>2.1.1</version>
</dependency>
Next, let’s implement the language detector by using the langdetect-183.bin pre-built model:
@Test
void givenTextInEnglish_whenDetectLanguage_thenReturnsEnglishLanguageCode() {
String text = "the dream my father told me";
LanguageDetectorModel model;
try (InputStream modelIn = new FileInputStream("langdetect-183.bin")) {
model = new LanguageDetectorModel(modelIn);
} catch (IOException e) {
return;
}
LanguageDetectorME detector = new LanguageDetectorME(model);
Language language = detector.predictLanguage(text);
assertEquals("eng", language.getLang());
}
In the example above, we grab the pre-built model to detect language from OpenNLP and place it in the root directory. Then, we define input data. Next, we load the language detector model. Finally, we create a new instance of LanguageDetectorME and try to detect the language. We test an expected language with the return language.
5. Stanford NLP
The Stanford NLP group helps with algorithms that allow machines to process, generate, and understand human text and languages.
CoreNLP is a suite of programs written in Java by the Stanford NLP group that can perform various NLP tasks like tokenization, part-of-speech tagging, lemmatization, etc. It can be used via the command line, in Java code, or with calls to a server.
Let’s see an example of performing tokenization with Stanford CoreNLP. We’ll need to add its dependency to the pom.xml:
<dependency>
<groupId>edu.stanford.nlp</groupId>
<artifactId>stanford-corenlp</artifactId>
<version>4.5.3</version>
</dependency>
Next, let’s perform tokenization:
@Test
void givenSampleText_whenTokenize_thenExpectedTokensReturned() {
Properties props = new Properties();
props.setProperty("annotators", "tokenize");
StanfordCoreNLP pipeline = new StanfordCoreNLP(props);
String text = "The german shepard display an act of kindness";
Annotation document = new Annotation(text);
pipeline.annotate(document);
List<CoreMap> sentences = document.get(CoreAnnotations.SentencesAnnotation.class);
StringBuilder tokens = new StringBuilder();
for (CoreMap sentence : sentences) {
for (CoreLabel token : sentence.get(CoreAnnotations.TokensAnnotation.class)) {
String word = token.get(CoreAnnotations.TextAnnotation.class);
tokens.append(word).append(" ");
}
}
assertEquals("The german shepard display an act of kindness", tokens.toString().trim());
}
In the example above, we set up the StanfordCoreNLP object with tokenized annotator. Next, we create a new instance of Annotation. Finally, we implement the logic to generate tokens from the sample sentence.
6. CogComp NLP
CogComp NLP is a collection of Natural Language Processing (NLP) libraries developed by the Cognitive Computation Group. It provides various tools and modules for NLP tasks, such as tokenization, lemmatization, part of speech tagging, etc.
CogComp NLP can be used either as a command line tool or as a Java API. One of the popular modules in CogComp NLP is cogcomp-nlp-pipeline, which performs basic NLP tasks on a given text. However, the cogcomp-nlp-pipeline works only for plain text in English.
Another module is similarity, which measures the similarity between texts or other objects and returns a score.
7. GATE
General Architecture for Text Engineering (GATE) is a toolkit that is capable of solving problems with text analysis and language processing. It’s a great infrastructure for developing software components to process human language. Additionally, it’s a great toolkit for NLP.
This toolkit has a large community of developers and researchers who use it for information extraction, sentiment analysis, social media mining, and biomedical text processing.
GATE helps developers and researchers by providing an architecture for language processing software. Furthermore, it provides a class library that implements the architecture.
8. Apache UIMA
Unstructured Information Management Applications (UIMA) are software systems that can process and analyze huge amounts of unstructured data, including text, audio, and video. They help to create components that can detect sentiments, entities, and other kinds of information from content. The components are written in Java or C++.
Furthermore, Apache UIMA is a framework that enables us to build applications using UIMA components and handle large volumes of unstructured data. It helps us to extract relevant information from the data and use it for various purposes.
9. MALLET
MAchine Learning for LangaugE Toolkit (MALLET) is a Java package that provides various tools and algorithms for NLP tasks, such as document classification, topic modeling, and sequence tagging. One of the algorithms included in MALLET is the Naive Bayes algorithm, which is widely used in NLP for text classification and sentiment analysis.
MALLET is an open-source Java package that offers various tools for text analysis. One of these tools is topic modeling, which can discover the main themes in a large collection of unlabeled text documents.
Furthermore, MALLET can also convert text documents into numerical vectors that can be used for machine learning. Additionally, it can be used either as a command line tool or as a direct Java API.
10. Conclusion
In this article, we learned key things about NLP and the use cases of NLP. Furthermore, we saw different Java NLP libraries and toolkits. Additionally, we looked at examples of tokenization and sentence detection using CoreNLP and OpenNLP, respectively.
As always, the complete source code for the examples is available over on GitHub.