Introduction
Image interpretation and image generation require different capabilities. Speech recognition converts audio into text; speech generation produces audio. 'Multimodal' means working with more than one kind of representation, but we still need to ask which inputs and outputs a particular system supports.
We will look at how media becomes numerical input and how those representations connect. The route covers pixels, patches, visual encoders, contrastive learning, OCR, speech recognition, audio samples, video frames, diffusion, latent representations, provenance, and evaluation. Each term comes with a problem it helps explain. Extracting, understanding, and generating content remain different tasks.
Review tokens, embeddings, attention, and generation in Module 14, and source meaning and lineage in Module 13. Begin with images and documents, then audio and video, then generation and product design. Your project is a multimodal workflow with a test packet that checks whether the outputs actually do the job.