Natural Language Processing Basics
How computers turn messy human language into something they can process.
Inspired by learning material from Microsoft
Natural Language Processing (NLP) is how computers understand and generate human language.
Core steps
1. Tokenization: split text into words or sub-words. 2. Normalization: lowercase, remove noise, handle punctuation. 3. Representation: convert tokens into numbers the model can use.
Common NLP tasks
- Sentiment analysis: is this review positive or negative?
- Named entity recognition: find names, places, dates.
- Translation: convert between languages.
- Summarization: shorten while keeping meaning.
From bag-of-words to embeddings
Early NLP counted words. Modern NLP uses embeddings — dense vectors that capture meaning, so 'king' and 'queen' sit close together.
Key takeaways
- NLP turns language into numbers, processes it, and turns it back.
- Tokenization and representation are foundational.
- Embeddings capture meaning, not just word counts.