AI Ecosystem CourseLearn with Q
AI Ecosystem · In Development

Foundation Models: Tokens, Attention, And Generation

Understand how language models turn words into numbers and generate an answer.

About This Course

Introduction

A foundation model is trained on broad data so it can support many later uses, directly or with adaptation. The name tells us about breadth and reuse. Evaluate capability, safety and generality separately for each system. Large language models are one family; other models work with images, audio, video, or combinations of them.

We will follow the language-model process step by step. Text becomes tokens, tokens become numerical representations, layers transform those representations, and the model scores possible next tokens. A decoding procedure selects output while the application supplies context, tools, and controls. Following the chain makes model claims easier to question.

Revisit vectors, matrix multiplication, parameter updates, and training versus inference in How Machine Learning Learns, plus memory and throughput in Compute Fundamentals. the course explains the main transformer components and how they fit together. The small calculations explain what each operation contributes. Work through representation and attention, training and generation, then context and deployment. Your project combines a model explanation with a controlled comparison experiment.

Planned Lessons

  1. Broad Learning And Downstream AdaptationSee how a model trained on many examples can later be used for a specific task.
  2. Tokens: The Model's Sequence UnitsUnderstand how a language model breaks text into smaller pieces it can process.
  3. Embeddings And Hidden RepresentationsSee how words and other inputs become numbers that a model can work with.
  4. Learned Parameters Versus Temporary ActivationsSeparate the numbers a model learns from the temporary calculations it makes for one question.
  5. Attention: Selecting Relationships Within The SequenceExplore how a model gives different parts of a sentence different amounts of attention.
  6. A Numerical Attention ExampleUse a small example to calculate how attention combines information from several words.
  7. Heads, Feedforward Layers, Residuals, And NormalizationFollow the main calculations inside one layer of a language model.
  8. Encoder, Decoder, And Encoder–Decoder FamiliesCompare model designs by how they read information and produce an answer.
  9. Next-Token Training And The Loss SignalSee how guessing the next piece of text gives a language model a way to learn.
  10. Autoregressive Inference: One Selected Token At A TimeFollow an AI answer as the model adds one piece of text at a time.
  11. Decoding, Temperature, Top-K, And Top-PSee how answer settings change which words a model chooses and how varied its replies are.
  12. Context Capacity And Usable ContextUnderstand why fitting a long document into a model does not guarantee a useful answer.
  13. Dense Models And Mixture Of ExpertsCompare models that use all their learned parts with models that select specialised parts.
  14. Reasoning Behavior And Inference-Time ComputationExplore how doing more calculations while answering can affect the result.
  15. Scaling: Model Size, Data, And ComputeConnect a model’s size with the examples and computing work needed to train it.
  16. Prompts, Instructions, And ExamplesSee how a clear request and useful examples can guide an AI response.
  17. Knowledge, Hallucination, And UncertaintyUnderstand why a model can give an answer that sounds plausible but lacks support.
  18. Model Access And Controlled ComparisonCompare models using the same questions and clearly stated settings.