AI Security Framework
Security controls spanning the AI development lifecycle, prompt engineering, API security, identity management, and third-party supply chain protection.
5.1 Secure Development Lifecycle
AI systems shall be developed, deployed, and operated using a Secure Development Lifecycle (SDL) that integrates security controls into every phase. The SDL for AI extends traditional software security to address model-specific risks including prompt injection, data poisoning, model extraction, and adversarial evasion.
SDL Phases & Security Controls
| Phase | Security Activity | Deliverable | Owner |
|---|---|---|---|
| Requirements | Threat modelling, risk assessment, security requirements definition | Security Requirements Document, Threat Model | Security Architect |
| Design | Secure architecture review, privacy by design, access model design | Architecture Security Review, Data Flow Diagram | Solution Architect |
| Development | Secure coding standards, dependency scanning, secrets management | Code review sign-off, SCA report | Technical Lead |
| Training Data | Provenance verification, bias audit, poisoning detection, sanitisation | Data Security Assessment, Bias Report | Data Engineer |
| Model Training | Environment hardening, training log integrity, checkpoint security | Training Environment Hardening Checklist | ML Engineer |
| Testing | Adversarial testing, red teaming, penetration testing, fuzzing | Security Test Report, Red Team Findings | Security Tester |
| Deployment | Container security, API security, runtime protection, monitoring | Deployment Security Checklist | DevOps Engineer |
| Operations | Vulnerability management, incident response, model drift detection | Operational Security Runbook | SOC Analyst |
5.2 Secure Prompt Engineering
Prompt engineering for large language models and generative AI introduces unique security risks. This section establishes standards for secure prompt design, prompt injection mitigation, and prompt governance.
Secure Prompt Design Principles
- Least Privilege Prompting: Prompts shall request only the minimum information and capabilities necessary to fulfil the task. Avoid over-prompting that exposes system context or internal data.
- Input Sanitisation: All user inputs destined for LLM prompts shall be sanitised to remove or escape control characters, delimiters, and escape sequences that could alter prompt structure.
- Context Isolation: System prompts, user prompts, and retrieved context shall be clearly separated using structured formats that resist injection attacks.
- Output Validation: All LLM outputs shall be validated against expected schemas, length limits, and content policies before downstream processing or user display.
- Prompt Versioning: All production prompts shall be version-controlled, reviewed, and approved before deployment. Ad-hoc prompt changes are prohibited.
Prompt Injection Mitigation
Direct prompt injection (jailbreaking) and indirect prompt injection (via external data sources) are critical attack vectors. Implement defence-in-depth: input filtering, output encoding, privilege separation, human-in-the-loop for sensitive actions, and continuous monitoring for anomalous prompt patterns.
5.3 API & Identity Security
AI systems expose and consume APIs that require robust security controls. This section covers API security, identity management, access control, encryption, and secrets management for AI infrastructure.
API Security Requirements
| Requirement | Control | Verification |
|---|---|---|
| Authentication | OAuth 2.0 / OIDC with PKCE for all API access; no API keys in client-side code | Penetration test, code review |
| Authorisation | RBAC with principle of least privilege; ABAC for sensitive operations | Access review quarterly |
| Rate Limiting | Tiered rate limits per client; anomaly detection for abuse | Load testing, DDoS simulation |
| Input Validation | Schema validation, type checking, length limits, reject unexpected fields | Fuzzing, negative testing |
| Output Encoding | Context-appropriate encoding; no sensitive data in error messages | Code review, error message audit |
| Logging & Monitoring | Comprehensive audit logs with tamper protection; real-time alerting | Log review, SIEM coverage audit |
| Encryption in Transit | TLS 1.3 minimum; certificate pinning for mobile clients | SSL Labs scan, certificate inventory |
5.4 Monitoring, Logging & Threat Detection
Continuous monitoring and threat detection are essential for AI security. AI-specific threats including model drift, adversarial inputs, data poisoning, and anomalous inference patterns require specialised detection capabilities.
Security Monitoring Requirements
- All AI model inference requests and responses shall be logged with timestamp, client identity, model version, and input/output hashes.
- Model prediction distributions shall be monitored for drift, outliers, and unexpected class imbalances.
- Adversarial input detection shall be deployed using input perturbation analysis, anomaly detection, and rule-based filters.
- Privilege escalation attempts, unauthorised access to model artefacts, and abnormal API usage patterns shall trigger automated alerts.
- Third-party model and data supply chain components shall be monitored for disclosed vulnerabilities, license changes, and integrity compromises.
5.5 Third-Party & Supply Chain Security
Third-party AI components — including foundation models, APIs, training data, cloud infrastructure, and MLOps tools — introduce supply chain risks that must be systematically managed.
Vendor Security Assessment
| Assessment Area | Minimum Requirements | Evidence |
|---|---|---|
| Security Certifications | ISO 27001, SOC 2 Type II, or equivalent; AI-specific attestations preferred | Certificate, audit report |
| Data Handling | Clear data residency, processing, and retention commitments; no unauthorised training on client data | DPA, data flow diagram |
| Model Provenance | Documented training data sources, fine-tuning methodology, known limitations | Model card, data sheet |
| Incident Response | Defined SLAs for breach notification; cooperative incident investigation | Contract clause, IR plan review |
| Business Continuity | Redundancy, failover, and exit strategy for critical AI services | BCP documentation, test results |
| Subcontractor Transparency | Disclosure of all subcontractors with access to organisational data or models | Subprocessor list, assessment reports |
Supply Chain Integrity
Verify the integrity of all third-party model weights, container images, and libraries using cryptographic signatures, checksums, and software bills of materials (SBOMs). Maintain an AI Software Bill of Materials (AI-SBOM) for every production AI system.