AI Ecosystem CourseBy Queensly Austin
Module 13

Data: From Business Records To Model Inputs

Introduction Available · Lessons In Development

Introduction

An AI system receives specific inputs from a business workflow. It receives selected rows, documents, images, events, labels, vectors, and instructions. People and software choose how to collect, organize, transform, check, and share them. An apparent model problem may begin with one of those earlier choices.

We will start with records and databases, then move through pipelines, data quality, labels, synthetic data, retrieval, memory, rights, and governance. Your project is a data architecture for a business use case. Explain what belongs in training, what should be retrieved for a request, what the application retains as state, and why.

Use Module 0 for data versus instructions and Module 1 for training and evaluation splits. Module 7 can help with versioning and pipelines. Lessons 1–6 cover storage and movement, 7–12 cover quality and learning, and the rest cover application use and control. Keep an orders-and-returns assistant in mind as a running example.

Planned lessons

  1. A Record Represents Something, Under A Convention
  2. Tables, Keys, And Relationships
  3. Queries, Joins, And Missing Values
  4. Operational Databases, Warehouses, Lakes, And Lakehouses
  5. Ingestion, ETL, ELT, And Streaming
  6. Pipeline Layers And Lineage
  7. Data Quality Is Fitness For A Particular Use
  8. Sampling, Representation, And Selection Bias
  9. Labels, Annotation, And Ground Truth
  10. Pretraining, Post-Training, And Evaluation Data
  11. Synthetic Data: Useful Examples And Inherited Errors
  12. Proprietary Data And The Conditions For An Advantage
  13. Retrieval Data Is Available Evidence At Request Time
  14. Embeddings, Vector Search, And Exact Matching
  15. Memory And Application State
  16. Rights, Privacy, Access, And Deletion
  17. Data Governance And Documentation That Can Be Used