- LPU MCA Semester 1
- CAB107
- Unit 6
CAB107 · Unit 6: Natural language processing and vision
Text preprocessing, tokenization, stemming and lemmatization
10 practice questions · in the ETE syllabus
Key points
- Tokenization splits text into words or sentences
- Stemming chops suffixes; lemmatization returns dictionary forms
- Over-stemming merges unrelated words
- Removing stop words like not can reverse sentiment
Raw text must be cleaned before modelling. Typical steps: lower-casing, removing punctuation, tokenization, stop-word removal and stemming or lemmatization.
Tokenization splits text into smaller units such as words or sentences. Lower-casing makes Data and data a single token, which reduces vocabulary size. Stop words are very common words such as the, is and of that carry little meaning for many tasks.
| Technique | How it works | Output |
|---|---|---|
| Stemming | Chops suffixes by rules | Possibly a non-word, for example studies becomes studi |
| Lemmatization | Uses vocabulary and grammar | A valid dictionary base form, for example better becomes good |
A Porter stemmer reduces both university and universe to univers, an over-stemming error where different meanings are merged. Under-stemming leaves related words with different stems.
from nltk.stem import PorterStemmer, WordNetLemmatizer
print(PorterStemmer().stem('running'))
print(WordNetLemmatizer().lemmatize('running', pos='v'))This prints run for both.
Stop-word removal can harm sentiment analysis, because words such as not may be on the stop list and removing them can flip the meaning of a phrase.
Exam trap: stemming is fast but crude and can output non-words; lemmatization is slower but returns real words.
Try two questions
What list results from lowercasing, tokenizing on spaces and removing the stop words the, is, on from 'The cat is on the mat'?
- ['cat', 'mat']
- ['The', 'cat', 'mat']
- ['cat', 'is', 'mat']
- ['the', 'cat', 'on', 'mat']
Show the answer
A. After lowercasing the tokens are the, cat, is, on, the, mat; removing the three stop words leaves cat and mat.
Why not B: Lowercasing makes 'The' become 'the', which is then removed as a stop word.
Why not C: The stop word 'is' is removed as well.
Why not D: Both 'the' and 'on' are stop words and are removed.
Why is text usually converted to lower case before building a vocabulary?
- So that every word in the text is replaced by a unique number in the vocabulary
- So that the order of the words in each sentence is reversed before the counting starts
- So that extra stop words are added to the text to make it longer for the model to learn from
- So that words such as Data and data are treated as one token, which reduces vocabulary size
Show the answer
D. Case folding merges variants and reduces sparsity.
Why not A: Conversion to numbers happens in a different step.
Why not B: Case folding does not reorder the words.
Why not C: Lowercasing does not add words.