3 · Tokens, embeddings, and representations

Chapter status: outline

This is the planned scope of Chapter 3. The complete lesson, executable examples, and solutions will be developed later. Chapter 1 is available now.

Question. How does text become something a numerical model can use?

• Learn: tokenization, vocabulary, one-hot representations, embedding lookup, positional information, cosine similarity, and contextual representations.

• Statistical lens: design matrices, learned feature maps, matrix factorization, and identifiability.

• Python practice: tokenize a small public text collection and compare bag-of-words with embedding-based similarity. Inspect examples where similarity is misleading.

• Self-check: explain why a token is not necessarily a word and why a two-dimensional embedding plot is not evidence of causal structure.