- LPU MCA Semester 1
- CAB107
- Unit 6
CAB107 · Unit 6: Natural language processing and vision
Bag of Words and TF-IDF
10 practice questions · in the ETE syllabus
Key points
- BoW counts words and ignores order
- TF-IDF = term frequency times inverse document frequency
- Common terms get low IDF
- Bigrams add some word-order context
Bag of Words (BoW) represents a document by the counts of the words it contains, ignoring word order. It is simple and gives sparse vectors, mostly zeros. Its weakness: it loses order and context, so dog bites man and man bites dog get identical vectors. Adding bigram features recovers some local context, for example making not good its own feature.
TF-IDF weights words by how informative they are:
TF-IDF = TF x IDF, where TF is how often the term occurs in the document and IDF (inverse document frequency) is the log of total documents divided by the number of documents containing the term.
A word scores high when it is frequent in the document but rare across the collection. A term in every document gets a low IDF and therefore a low weight, but it is not deleted from the vocabulary.
| Word | In document | In collection | TF-IDF |
|---|---|---|---|
| the | Frequent | Every document | Low |
| tensor | Frequent | Few documents | High |
from sklearn.feature_extraction.text import TfidfVectorizer
X = TfidfVectorizer().fit_transform(['good film', 'bad film'])
print(X.shape)This gives 2 documents by 3 vocabulary words (bad, film, good): shape (2, 3). Film appears in both documents, so it receives a lower weight than bad or good.
Exam trap: IDF lowers the weight of common terms but never changes the term frequency, and BoW vectors are sparse, not dense.
Try two questions
What is a main weakness of a bag-of-words representation?
- It loses word order and context, so it cannot tell dog bites man from man bites dog
- It cannot be computed for English text, only for languages that have fewer word forms
- It needs a labelled target value for every single word in the vocabulary before counting
- It produces dense vectors that contain no zero entries for any word in the vocabulary
Show the answer
A. Order and syntax are discarded; n-grams partly recover local order.
Why not B: BoW is easy to compute for English.
Why not C: BoW is not supervised.
Why not D: BoW vectors are usually sparse.
A collection has 50 documents, and the word the appears in all of them. Using idf = log(N / df), what is the idf of the?
- 0
- 1
- 50
- Undefined
Show the answer
A. log(50 / 50) = log(1) = 0, so its TF-IDF is zero in every document.
Why not B: log(1) is zero, not one.
Why not C: 50 is the number of documents.
Why not D: The ratio is 1, so the logarithm is defined.