Understand the chips and memory that run AI, and why some systems are faster than others.
About This Course
What you will be able to diagnose
'We need more compute' can mean several things. A system might be short of processing power, memory space, or the ability to move information quickly. It might also be held back by communication, queues, software, electricity, or cooling. Identify the limiting resource before choosing additional equipment.
We will connect the parameters, inference, vectors, matrices, and learning loop from Modules 0–1 to the physical resources that make them possible. Your project is a request-to-compute diagram and a performance dashboard. Work through the units as you go. Unless a dated source is identified, the numerical scenarios are hypothetical examples for learning.
Planned Lessons
What A Processor Actually DoesSee how a processor follows instructions to do the calculations behind an AI tool.
GPUs And Parallel WorkFind out why GPUs can do many AI calculations at once and when that helps.
Matrix Operations Connect Learning To HardwareConnect the rows and tables of numbers in a model to the chips doing the work.
Bits, Bytes, And Numerical PrecisionUnderstand how computers store numbers and why the number of digits matters for AI.
Quantization And Its Actual Trade-OffsSee how using fewer digits can make a model smaller and faster, and what may be lost.
FLOPS And The Meaning Of A PeakUnderstand what a chip’s advertised speed tells you about running an AI task.
Memory Capacity: What Must FitWork out what needs to fit in memory for a model to run.
Memory Bandwidth: What Must MoveSee why moving information to a chip can slow an AI system down.
Arithmetic Intensity And The RooflineFind out whether an AI task is waiting for calculations or for information to arrive.
Training Has More Work Than Forward PredictionSee the extra calculations needed to teach a model, beyond using it to make a prediction.
Prefill And Decode Create Different Inference WorkCompare the work of reading a question with the work of writing an AI answer.
KV Cache And Long-Context MemorySee how remembering earlier words uses memory as an AI conversation grows.
Batching, Concurrency, And QueueingFind out how serving several people at once affects waiting time and computing needs.
Speculative Decoding And Why Verification MattersSee how a smaller model can suggest words that a larger model checks to speed up answers.
Communication: Scale-Up, Scale-Out, And Rack SystemsExplore how AI computers exchange information within a machine and across a building.
Utilization, Goodput, And Useful EconomicsCompare the computing capacity paid for with the useful AI work actually completed.
Compare Accelerator Strategies As Complete SystemsCompare AI chips using the whole system needed to run a particular task.