AI Ecosystem CourseBy Queensly Austin
Module 14

Foundation Models: Tokens, Attention, And Generation

Introduction Available · Lessons In Development

Introduction

A foundation model is trained on broad data so it can support many later uses, directly or with adaptation. The name tells us about breadth and reuse. Evaluate capability, safety and generality separately for each system. Large language models are one family; other models work with images, audio, video, or combinations of them.

We will follow the language-model process step by step. Text becomes tokens, tokens become numerical representations, layers transform those representations, and the model scores possible next tokens. A decoding procedure selects output while the application supplies context, tools, and controls. Following the chain makes model claims easier to question.

Revisit vectors, matrix multiplication, parameter updates, and training versus inference in Module 1, plus memory and throughput in Module 2. The module explains the main transformer components and how they fit together. The small calculations explain what each operation contributes. Work through representation and attention, training and generation, then context and deployment. Your project combines a model explanation with a controlled comparison experiment.

Planned lessons

  1. Broad Learning And Downstream Adaptation
  2. Tokens: The Model's Sequence Units
  3. Embeddings And Hidden Representations
  4. Learned Parameters Versus Temporary Activations
  5. Attention: Selecting Relationships Within The Sequence
  6. A Numerical Attention Example
  7. Heads, Feedforward Layers, Residuals, And Normalization
  8. Encoder, Decoder, And Encoder–Decoder Families
  9. Next-Token Training And The Loss Signal
  10. Autoregressive Inference: One Selected Token At A Time
  11. Decoding, Temperature, Top-K, And Top-P
  12. Context Capacity And Usable Context
  13. Dense Models And Mixture Of Experts
  14. Reasoning Behavior And Inference-Time Computation
  15. Scaling: Model Size, Data, And Compute
  16. Prompts, Instructions, And Examples
  17. Knowledge, Hallucination, And Uncertainty
  18. Model Access And Controlled Comparison