AI Aviation Evaluation Working Group
The AI Aviation Evaluation Working Group, convened by Airside Labs, develops open, aviation-specific standards for evaluating AI systems — the safety-critical questions that generic AI testing misses. It maintains Pre-Flight, the first open-source aviation-specific LLM benchmark, now included in the UK AI Security Institute's Inspect AI, and brings aviation operators, regulators, and AI specialists together to shape how aviation AI is tested.
Shaping the Future of Aviation AI Safety
The AI Aviation Evaluation Working Group is a collaborative initiative bringing together aviation industry experts, regulatory bodies, AI specialists, and safety professionals to develop comprehensive evaluation frameworks for AI systems in aviation.
As AI adoption accelerates across the aviation sector—from operations to maintenance, from customer service to flight planning—the need for rigorous, aviation-specific evaluation standards has never been more critical. Our working group addresses this challenge head-on.
Our Mission
Develop aviation-specific AI evaluation methodologies that ensure safety, reliability, and regulatory compliance while enabling innovation. We create practical testing frameworks that address the unique challenges of aviation AI deployment.
Why Aviation Needs Specialized Standards
Aviation's safety-critical environment, complex regulatory landscape, and specialized terminology require evaluation frameworks that go beyond generic AI testing. We develop standards that understand aviation's unique requirements.
Key Focus Areas
Safety-Critical Decision Making
Evaluation methodologies for AI systems that influence safety-critical aviation decisions, including emergency procedures, operational limitations, and risk assessment capabilities.
Regulatory Compliance Testing
Frameworks for validating AI alignment with CAA, EASA, FAA, and ICAO requirements, including operational approvals, certification standards, and continuing airworthiness obligations.
Domain Knowledge Validation
Comprehensive testing of AI understanding of aviation terminology, procedures, regulations, and operational context across ground operations, flight operations, maintenance, and air traffic management.
Human Factors Integration
Evaluation of AI systems' interaction with human operators, including situational awareness support, workload management, automation bias mitigation, and crew resource management considerations.
Adversarial Robustness
Testing frameworks for identifying vulnerabilities to adversarial attacks, including prompt injection, data poisoning, and model manipulation that could compromise aviation safety or security.
Performance Under Degraded Conditions
Evaluation of AI system behavior under non-normal conditions, including partial data availability, system failures, emergency scenarios, and edge cases that aviation operations frequently encounter.
Aviation-Specific Evaluation vs Generic AI Testing
Generic AI benchmarks and general LLM-evaluation platforms were not built for a safety-critical, heavily regulated domain. This is the gap the working group's frameworks — and the open-source Pre-Flight benchmark — are designed to close.
| What is being evaluated | Generic AI evaluation (MMLU, general LLM-eval platforms) |
Aviation-specific evaluation (this working group + Pre-Flight) |
|---|---|---|
| Aviation domain knowledge (ICAO, ground ops, dispatch) |
Not covered | Built in and practitioner-authored |
| Regulatory alignment (CAA, EASA, FAA, ICAO, EU AI Act) |
Generic or none | Core focus area |
| Safety-critical failure behaviour | Rarely tested | Central — systems must fail safe |
| Adversarial robustness (prompt injection, data poisoning) |
Generic only | Aviation threat scenarios |
| Human factors & automation bias | No | Evaluated in operational context |
| Authored by aviation practitioners | No | Yes — ATM, ground ops & commercial-flying experience |
Who Should Join
Aviation Organizations
- • Airlines and operators deploying AI systems
- • Ground handling service providers
- • Maintenance, repair, and overhaul organizations
- • Air navigation service providers
- • Airport operators
Regulatory & Standards Bodies
- • Civil aviation authorities
- • International standards organizations
- • Safety investigation agencies
- • Industry associations
Technology Providers
- • AI/ML technology companies
- • Aviation software developers
- • Safety management system providers
- • Aviation data analytics platforms
Academic & Research
- • University aviation programs
- • AI safety research groups
- • Human factors specialists
- • Aviation safety researchers
Collaborative Approach
The working group operates through regular virtual meetings, collaborative document development, public consultation periods, and industry workshops. We believe that effective aviation AI standards can only be developed through genuine collaboration between all stakeholders.
Our outputs include evaluation frameworks, test datasets, compliance checklists, best practice guidance, and case studies. All materials are developed openly with input from the aviation community and published for industry adoption.
Join the Working Group
Help shape the future of aviation AI safety. Whether you represent an airline, regulator, technology provider, or academic institution, your expertise is valuable to developing effective evaluation standards.
Express Interest in Joining