AI Ecosystem CourseLearn with Q
AI Ecosystem · In Development

Edge And On-Device AI

Explore when it makes sense to run AI on a phone or other device instead of in the cloud.

About This Course

Introduction

Should a task run in a distant data center, on a nearby gateway, or on the user's device? The answer affects delay, connectivity, privacy, hardware limits, updates, and costs. Edge AI is a choice about where the work happens, with consequences for the whole product.

We will connect compute in Compute Fundamentals, inference memory in Foundation Models: Tokens, Attention, And Generation, optimization in Post-Training, Adaptation, And Evaluations, and application architecture in AI Application Architecture. You will decide what can run locally, what needs the cloud, and how to test a combination of both. The project is an offline study assistant that explains selected course material and preserves progress.

Start with placement and hardware, then optimization and operations, and finish with the local-versus-cloud comparison. Keep returning to the task: will the proposed setup work on the actual device, network, and user conditions?

Planned Lessons

  1. What Counts As The Edge?Identify AI work done on or near the device collecting the information.
  2. Why Place Computation Locally?Compare keeping AI work on a device with sending it to a remote service.
  3. CPUs, GPUs, And NPUsExplore the different chips a phone or other device may use for AI tasks.
  4. Memory Is More Than The Model FileCount the working memory an AI app needs beyond the model file itself.
  5. Battery, Power, Energy, And HeatSee how running AI affects battery life, power use and device temperature.
  6. Compression And Smaller ModelsCompare ways to make a model fit within a device’s limits.
  7. Conversion, Runtime Compatibility, And PackagingCheck whether a model works with a device’s hardware and software before release.
  8. Hybrid Local-Cloud ArchitectureDecide which parts of an AI task should run on the device and which should run remotely.
  9. Local Retrieval And PersonalizationUse information kept on a device to help an AI app give more relevant answers.
  10. Privacy And Security Across Device BoundariesCheck what private information stays on a device and what leaves it.
  11. Deployment To A Fragmented Device FleetPlan updates for users with different devices, chips and software versions.
  12. Measuring The Whole Local ExperienceTest useful results, response time and resource use on the device people will use.
  13. Hardware-Feature Economics And Cloud-Service EconomicsCompare earning money from AI features in hardware with selling an online service.
  14. Streaming Tasks And Deadline BudgetsKeep up with a continuous flow of sound, pictures or measurements within the available time.
  15. Federated Learning And Local Adaptation BoundariesExplore how models can learn from many devices and what information still needs to be shared.