N-Gram Language Modelling with NLTK Using Python

Discover the essentials of N-Gram Language Modelling with NLTK in Python: Learn how to build and analyze models for effective text processing.

Language modelling using N-gram models in NLTK is a technique that involves predicting the likelihood of a sequence of words. It fundamentally operates by analyzing a large corpus of text to understand the frequency and patterns of word sequences. In N-gram language modelling, the context is defined by the preceding words, with the number of these words being determined by the 'n' in N-gram.

NLTK provides efficient tools for creating and manipulating these N-gram models. The process begins with tokenizing text into words or characters, followed by the construction of N-grams. For instance, bigrams and trigrams, which are two and three-word sequences respectively, are commonly used in this modelling approach.

Probability distributions are then computed for each N-gram, forming the basis of the language model. These probabilities enable the model to predict the next word in a sequence or to generate new text based on learned patterns. N-gram models, despite their simplicity, are powerful tools in NLTK for various NLP tasks, including text generation, speech recognition, and other machine learning algorithms for translation.

Methods Of Language Modelling

The two types of Language Modelling that are used are as follows.

Statistical Language Modelling

Statistical Language Modeling is a technique in natural language processing that involves the computation and utilization of probabilities to predict word sequences. It is grounded in the principle that the likelihood of a word in a text can be inferred from its previous words. This method leverages large corpora to determine the frequency and patterns of word sequences, often using N-gram models as a foundation. The core idea is to quantify the probability of a sequence of words, enabling tasks like speech recognition, text prediction, and machine translation. Statistical language models are fundamental in understanding the structure and dynamics of languages, forming the basis for more advanced models in computational linguistics and AI-driven language processing.

Neural Langauge Modelling

Neural Language Modeling represents a significant advancement in natural language processing, employing deep learning techniques to understand and generate human language. Unlike traditional statistical models, neural models use layers of the artificial intelligence and neural networks to process and predict linguistic patterns. These networks, often in the form of recurrent neural networks (RNNs) or transformers, are adept at capturing long-range dependencies and nuances in language by learning from vast datasets. They effectively encode contextual information, allowing for more accurate predictions of word sequences and the generation of coherent and contextually relevant text. Neural language models like GPT (Generative Pre-trained Transformer) and BERT (Bidirectional Encoder Representations from Transformers) have revolutionized fields like machine translation, conversational AI, and content creation, showcasing the power of deep learning in understanding and emulating human language.


An N-Gram in the context of language modeling with NLTK is a sequence of 'n' items (typically words) extracted from a larger text corpus. These items are contiguous and used to predict the likelihood of sequences in text analysis. In NLTK, N-grams are fundamental building blocks for creating language models, with common types being bigrams (sequences of two words) and trigrams (sequences of three words).

The construction of N-grams involves parsing text data into these sequential units, which are then analyzed to understand language patterns and probabilities. NLTK provides efficient tools for generating and manipulating N-grams, allowing for the creation of models that can predict subsequent words in a sentence or assist in tasks like text completion. The effectiveness of an N-gram model in NLTK largely depends on the size of the N-gram (the value of 'n') and the quality of the text corpus used for training.

N-Gram Language Model

An N-gram Language Model in NLTK is a probabilistic model that predicts the occurrence of a word based on the occurrences of its preceding 'n-1' words. This model is essential in natural language processing for tasks such as text prediction and sentence completion. In NLTK, constructing an N-gram model typically involves first tokenizing text data into words or characters, and then forming N-grams, which are sequences of 'n' contiguous items from the text.

Mathematically, an N-gram Language Model in NLTK is expressed through the conditional probability of a word given its preceding words. For a sequence of words 1,2,...,w1​,w2​,...,wn​, the model computes the probability of the word wn​ occurring, given the sequence of its n−1 preceding words. This is represented as P(wn​∣wn−1​,wn−2​,...,wn−(n−1)​).

In a bigram model (2-gram), for example, the probability of a word is calculated based on its immediate predecessor: P(wn​∣wn−1​). The overall probability of a sentence is the product of the probabilities of each word given its previous word(s). Mathematically, for a sentence w1​,w2​,...,wn​, the probability is P(w1​)×P(w2​∣w1​)×...×P(wn​∣wn−1​).

NLTK's functionality allows for the calculation and manipulation of these probabilities, using the frequencies of N-grams derived from a corpus of text data. This mathematical framework is crucial for understanding and implementing language models that are capable of predicting word sequences and generating coherent text

The Python Implementation.

import nltk
from nltk import bigrams
from nltk.probability import FreqDist
from nltk.tokenize import word_tokenize

# Sample text
text = "This is a sample sentence for N-gram language modeling."
tokens = word_tokenize(text)

# Create bigrams
bigram_list = list(bigrams(tokens))

# Calculate frequency distribution
freq_dist = FreqDist(bigram_list)

# Output the frequency of each bigram
for bigram, frequency in freq_dist.items():
    print(f"{bigram}: {frequency}")

When you run this code, it outputs the frequency of each bigram in the given string of text. For instance, if the text contains the bigram ("sample", "sentence") once, the output will be.


('sample', 'sentence'): 1

This output indicates the count of each bigram's occurrence in the text, which is fundamental in calculating the probabilities needed for the language model. The NLTK library simplifies the process of tokenizing text, creating N-grams, and computing their frequency distribution, making it an invaluable tool for N-gram language modelling in Python.

The N-Gram Language Modelling with NLTK in Python is a powerful and accessible tool for natural language processing tasks. This method, utilizing the NLTK library, allows for the efficient creation and analysis of N-gram models, which are essential in understanding and predicting language patterns. Through the use of bigrams, trigrams, and higher-order N-grams, NLTK facilitates the exploration of text data, providing insights into linguistic structures and probabilities.

The practical applications of N-gram language models are vast, ranging from text prediction and auto-completion of sentences to more complex tasks like machine translation and speech recognition. The ease of implementation in Python, combined with NLTK's comprehensive toolkit, makes N-gram modelling an invaluable skill for anyone venturing into the field of computational linguistics and natural language processing.

You can also check these blogs:

  1. Namespaces In Python
  2. Scope In Python
  3. Constructor Overloading In Python
  4. Indentation In Python
  5. Removing Non-Alphanumeric Characters in Python