AI Ecosystem CourseLearn with Q
AI Ecosystem · In Development

AI Security, Safety, And Governance

Identify ways an AI system can cause harm or be misused, and how organisations manage those risks.

About This Course

Introduction

Who can access the data? What could cause harm? Who is responsible for the decision? Security, safety, and governance overlap, but each brings a different question. Security addresses unauthorized access, manipulation, and disruption. Safety considers harmful outcomes. Governance establishes ownership, rules, evidence, and accountability.

The risks extend beyond the model to data, retrieval, prompts, tools, credentials, interfaces, vendors, people, and processes. We will identify those risks and design controls we can test. Your project is a threat model and risk register for the research agent from Agents, MCP, And Tool Use.

Begin with boundaries and threats, then evaluation and governance, then an incident example. The aim is to support useful, authorized work with a clear understanding of who can act and what the consequences may be.

Planned Lessons

  1. Model Risk And System RiskSeparate mistakes made by a model from problems elsewhere in an AI application.
  2. Assets, Actors, And Trust BoundariesIdentify what needs protecting, who can reach it and where permissions change.
  3. Prompt InjectionSee how instructions hidden in outside information can try to redirect an AI tool.
  4. Exfiltration And Sensitive Information DisclosureFind ways private information could leave an AI app without permission.
  5. Identity, Permissions, And Least PrivilegeGive each person or system only the access needed for its job.
  6. Output Handling And Executable ConsequencesCheck AI-generated content before it can run code or change real records.
  7. Data Poisoning And The Supply ChainExplore how misleading training examples or compromised software can affect AI behaviour.
  8. Availability And Unbounded ConsumptionSet limits that prevent an AI service from being overwhelmed or using uncontrolled resources.
  9. Hallucinations, Uncertainty, And MisuseConsider the harm caused by unsupported answers and deliberate misuse of AI tools.
  10. Fairness And Harmful Outcome PatternsCheck whether an AI system makes different kinds of mistakes for different groups of people.
  11. Red Teaming And Assurance EvidenceLook for weaknesses through targeted tests and keep evidence of what was found.
  12. Governance And AccountabilityMake responsibility for AI decisions and safeguards clear within an organisation.
  13. Auditability And Privacy-Conscious RecordsKeep records that help investigate a problem without collecting unnecessary personal information.
  14. Incident Response And RecoveryPlan how to contain an incident, restore the service and prevent the same problem happening again.