1. Next-Token Prediction (NLP)
Welcome to Lesson 5! Today, we explore how AI understands and generates human language, a field called Natural Language Processing (NLP).
Many people believe Large Language Models (like ChatGPT) have conscious thoughts or access facts inside a database. In reality, they are just **large-scale autocompletes**.
AI reads text by breaking words into small chunks called Tokens, turns those tokens into numbers, and calculates the mathematical probability of what word is most likely to come next based on previous ones.
2. Interactive Word Predictor
Let's see how probability autocompletes a sentence! Below is a text prompt. Click "Calculate Next Word" to see the top 5 candidate words and their statistical chances.
Click on any candidate word to append it to the sentence, and watch the probabilities instantly update for the subsequent step. You are directing the generative path!
3. Notebook: Bigram Text Generator
Let's build a text-generator in code! We train a simple **Bigram Model** (which looks at adjacent word pairs) on the book *Alice in Wonderland*. Run cells 1-3, type a starting word in cell 4, and generate speech!
with open("alice_in_wonderland.txt", "r") as f:
corpus = f.read().lower().split()
print(f"Corpus loaded: {len(corpus)} words")bigram_pairs = {}
for w1, w2 in zip(corpus[:-1], corpus[1:]):
bigram_pairs.setdefault(w1, []).append(w2)
print("Bigram probability dictionary built successfully!")4. Core Concept Summary
Congratulations on completing the AI Lessons series! Let's review the final terms in our cheat sheet companion page.
• **Next-Token Prediction:** Generative AI guess rules. The model simply computes what word fits best based on all prior text context.