NLP Tokenization Practicals

- Overview 

Tokenization is the foundation of Natural Language Processing (NLP). It breaks down a raw string of text into smaller, meaningful units called tokens (such as words, punctuation, or contractions). 

While a simple Python command like text.split() can separate words by whitespace, it fails to handle punctuation or special edge cases properly. For example, split() treats "don't" as one token, whereas dedicated NLP libraries recognize that "do" and "n't" carry distinct semantic meanings. 

Below are simple, practical examples of how to tokenize text using NLTK and spaCy, the two most popular libraries in Python.

- Option 1: Using NLTK (Natural Language Toolkit)

NLTK is highly modular and widely used for teaching and research. It offers distinct functions for splitting text into sentences or words. Use code with caution.

import nltk
from nltk.tokenize import word_tokenize, sent_tokenize

# Download the required pre-trained tokenization models
nltk.download('punkt')

text = "NLP is amazing! Let's tokenize this sentence, okay?"

# 1. Sentence Tokenization (Splits a paragraph into sentences)
sentences = sent_tokenize(text)
print("Sentences:", sentences)
# Output: ['NLP is amazing!', "Let's tokenize this sentence, okay?"]

# 2. Word Tokenization (Splits text into individual words & punctuation)
words = word_tokenize(text)
print("Words:", words)
# Output: ['NLP', 'is', 'amazing', '!', 'Let', "'s", 'tokenize', 'this', 'sentence', ',', 'okay', '?']

- Option 2: Using spaCy 

spaCy is designed to be fast and production-ready. Instead of calling separate functions, you pass your text through a pipeline (nlp), which automatically builds a structured Doc object containing the tokens. 

(Note: Before running this, you need to download the English model in your terminal using: 

1. Code: Use code with caution.

python -m spacy download en_core_web_sm)

import spacy

# Load the small English language pipeline
nlp = spacy.load("en_core_web_sm")

text = "NLP is amazing! Let's tokenize this sentence, okay?"

# Process the text
doc = nlp(text)

# Extract token text using a list comprehension
tokens = [token.text for token in doc]
print("Tokens:", tokens)
# Output: ['NLP', 'is', 'amazing', '!', 'Let', "'s", 'tokenize', 'this', 'sentence', ',', 'okay', '?']


2. Expected Output:

text

--- spaCy Sentence Tokens ---
['Hello!', "I'm learning NLP.", 'Is tokenization easy?', 'Yes, it is.']

--- spaCy Word Tokens ---
['I', "'m", 'learning', 'NLP', '.']

- NLTK vs. spaCy 

•  Design Philosophy: NLTK is essentially a massive academic sandbox. It gives you multiple different algorithms for the exact same task (e.g., Porter Stemmer vs. Lancaster Stemmer) so you can experiment. spaCy follows a "opinionated" philosophy - it chooses the single best, state-of-the-art algorithm for each task so you don't have to guess.
• Deep Learning & Model Support: spaCy features seamless, built-in support for pretrained transformers like BERT, making it much easier to integrate into modern deep learning pipelines. NLTK relies on older, more traditional statistical models.
• Text Modification: Because spaCy uses strict object-oriented pipelines, its Doc objects are immutable (you cannot change the text inside them directly). NLTK just gives you standard Python lists, making it much easier to modify, delete, or scramble strings on the fly.