Pre-Flight vs. ALUE: Two Aviation AI Benchmarks, Two Very Different Jobs

The Short Version

2025 gave the aviation industry not one but two purpose-built benchmarks for evaluating large language models. We launched Pre-Flight in January. Nine months later, MITRE and the FAA released the Aerospace Language Understanding Evaluation (ALUE). The natural instinct is to compare them head-to-head, but that misses the point. These benchmarks tackle different problems at different layers of the aviation stack, and understanding where each one fits matters more than picking a winner.

Pre-Flight was built for the commercial side of the house — testing whether an LLM actually understands ground operations, ICAO procedures, and dispatch. It is a domain-knowledge benchmark: a fast, like-for-like baseline of aviation competence for when you are choosing or comparing models. The wider job of security and regulatory audit readiness — against frameworks like the EU AI Act, OWASP LLM Top 10, NIST RMF, and MITRE ATLAS — is a separate piece of work, handled by Airside Labs' use-case-specific adversarial red-teaming, which runs alongside the benchmark rather than inside it.

ALUE, backed by the FAA through MITRE's federally funded research centre, is aimed squarely at the National Airspace System. Its focus is on aviation-specific language, nomenclature, and the kind of system-level safety assurance you need before letting an AI anywhere near air traffic management decisions. The FAA wants a "definitive library" of aviation terminology that LLMs must demonstrate they understand before they can be considered for use in NAS operations.

The two benchmarks complement each other. Pre-Flight answers an early question cheaply — does this model understand commercial aviation well enough to shortlist it? ALUE then validates whether the model has the deep aerospace domain understanding required for integration into federal safety-critical systems. It is a layered approach, not a competition.

Which Aviation AI Benchmark Should You Use?

The quick answer, by use case:

• Building AI for commercial airline operations, ground handling, dispatch, or customer service? Start with Pre-Flight to baseline a model's aviation competence. For audit-ready evidence against the EU AI Act, OWASP LLM Top 10, NIST AI RMF, and MITRE ATLAS, pair it with Airside Labs' separate, use-case-specific adversarial red-teaming.
• Integrating AI into the National Airspace System or air traffic management? You will need to meet ALUE, the FAA/MITRE federal standard for deep aerospace language understanding.
• Need comprehensive assurance for a safety-critical deployment? Use both benchmarks — Pre-Flight to baseline commercial aviation competence and ALUE for systemic aerospace domain understanding — together with dedicated adversarial red-teaming for the security and compliance evidence neither benchmark produces on its own.
• Just want to know whether a general model understands aviation at all? Run Pre-Flight first; it is open source, available on GitHub and Hugging Face, and integrated into the UK AI Security Institute's Inspect AI framework, so you can benchmark any model in an afternoon.

Why Aviation AI Needs Purpose-Built Evaluation

There is nothing controversial about saying that AI errors in aviation could be catastrophic. What is more interesting is how the industry is responding. Both benchmarks emerged from the same recognition: generic AI evaluation methods are not fit for purpose in safety-critical aerospace applications. You cannot rely on general-purpose leaderboards to tell you whether an LLM will hallucinate a non-existent NOTAM, confuse ICAO wake turbulence categories, or leak sensitive passenger data.

The Regulators Are Watching

EASA's September 2025 survey results confirmed what most of us already knew. Aviation professionals are most concerned about the limits of AI performance, data protection, accountability, and safety implications. A strong majority called for active regulation and supervision by EASA and national authorities. The message is clear: assurance is not optional, it is a prerequisite.

These concerns are exactly what both benchmarks aim to address, albeit from different angles.

ALUE: The Federal Standard

MITRE has been shaping aerospace capabilities for over sixty years through its work with the FAA's Center for Advanced Aviation System Development (CAASD). That institutional weight matters. When MITRE publishes a benchmark, it carries the authority of the federal aviation safety establishment.

ALUE is designed to set common, verifiable standards for AI tools operating within the NAS. The emphasis is on whether LLMs genuinely understand aviation language and context — not just surface-level pattern matching, but the kind of precise, technically accurate comprehension that air traffic management demands. Their research found that general-purpose models tend to produce verbose, unstructured outputs when confronted with aviation tasks, and that structured prompts and in-context examples significantly improved performance. This finding is baked into the benchmark's design.

Pre-Flight: A Fast Commercial Baseline

We built Pre-Flight to bridge the gap between cutting-edge AI capabilities and safe, reliable deployment for commercial aviation. The design reflects a market-driven need: airlines, ground handlers, and aviation technology providers need a rapid, demonstrable way to check whether a model actually understands aviation before they integrate it. Pre-Flight gives them that baseline. The next layer — producing audit-ready evidence against the EU AI Act, GDPR, OWASP, NIST RMF, and MITRE ATLAS — is a separate piece of work: Airside Labs' use-case-specific adversarial datasets and red-teaming, which build on the model selection that Pre-Flight informs.

Where the government-backed ALUE focuses on NAS safety and airspace integrity, Airside Labs' commercial offering addresses the immediate, tangible risks that commercial partners face: data privacy vulnerabilities, regulatory non-compliance, and the business consequences of deploying an AI system that cannot reliably follow standard operating procedures — with Pre-Flight providing the first-pass domain-competence check.

A Closer Look at ALUE

The ALUE benchmark, developed by Eugene Mangortey, Kunal Sarkhel, and their team, is built to be flexible. It supports open-source and domain-specific LLMs, custom datasets, user-defined prompts, and various quantitative metrics. This versatility is intentional — the FAA and broader aerospace community need a framework that can adapt to different evaluation needs across the sector.

Where ALUE Is Heading

The more interesting story is ALUE's roadmap. Future versions are expected to require LLMs to extract data from charts, consult aircraft operational manuals, and determine technical parameters like thrust settings and flap configurations under specific conditions. This is a fundamentally different challenge from answering knowledge questions. The model would need to synthesise information from multiple sources — text, charts, technical manuals — and arrive at a specific, safety-critical conclusion.

In practical terms, ALUE is moving toward validating Retrieval-Augmented Generation (RAG) and multi-modal reasoning. That positions it at the high end of the aviation operational hierarchy, where system-critical performance and real-time decision support become the bar to clear.

A Closer Look at Pre-Flight

Pre-Flight focuses on the operational layer of aviation — the "last mile" of commercial operations where things happen under the wing. The benchmark tests an LLM's understanding of ICAO annex documentation, flight dispatch rules, and critically, airport ground operations safety procedures: ground equipment operations, fuelling safety, aircraft towing, emergency response protocols, and scenarios like managing snow removal during winter operations. The dataset of roughly 300 questions is derived from standard international airline and airport ground operations safety manuals.

V2: Harder Questions as Models Saturate

We ran into the same problem that plagues most AI benchmarks — the leading models started saturating the top scores. The response was V2, which keeps the benchmark discriminating by adding harder, multi-step reasoning questions and questions that require structured answers rather than free prose.

The point is to test something closer to real operational use. In regulated environments, a model that will feed a downstream system has to follow a specific format reliably, not just sound fluent, so V2 rewards models that can reason through a multi-step aviation problem and return a clean, structured answer. That makes Pre-Flight a sharper signal of whether a model is a plausible candidate for an operational role — still as a benchmark and baseline, not a substitute for testing the deployed system itself.

The Separate Security and Compliance Layer

Pre-Flight is one tool in a broader risk-assessment process — not the whole of it. Domain knowledge is what the benchmark measures; security and compliance are a distinct piece of work that Airside Labs runs separately, using domain- and use-case-specific adversarial datasets to test against the EU AI Act, GDPR (data leakage risks), OWASP LLM Top 10 (common vulnerabilities), and MITRE ATLAS (adversarial threats against AI systems). The inclusion of MITRE ATLAS is worth highlighting — it means testing not just whether a model knows the right answers, but whether it can withstand deliberate attempts to make it produce wrong ones. This adversarial red-teaming is independent of the Pre-Flight benchmark, runs alongside it, and is where audit-ready compliance evidence actually comes from.

How They Compare

The honest assessment is that Pre-Flight and ALUE are largely complementary. They address different layers of the aviation ecosystem and different phases of the LLM deployment lifecycle.

ALUE targets the systemic layer — NAS, air traffic management, and real-time decision support. Pre-Flight targets the operational layer — ICAO procedures, dispatch, and ground operations. The overlap is limited to foundational regulatory knowledge and general safety concepts drawn from ICAO Annexes.

The methodological differences are equally significant. ALUE is structurally broad, supporting diverse data formats and focused on ensuring models produce concise, technically accurate outputs rather than verbose noise. Pre-Flight V2 pushes on harder, multi-step questions and structured answers, keeping the benchmark discriminating as models improve and giving a sharper read on a model's operational reasoning.

Dimension	Pre-Flight (Jan 2025)	ALUE (Sept 2025)	Assessment
Sponsor	Commercial R&D, Security Testing	Government/FFRDC, NAS Safety	Different mandates: commercial agility vs. federal standard
Domain	ICAO, Ground Ops, Dispatch (Under the Wing)	Aviation Nomenclature, NAS, ATM (In the Air)	Pre-Flight holds a specific operational niche
Method	MCQ (v1), structured answers and complex logic (v2)	Diverse tasks, custom prompts, quantitative metrics	ALUE is broader; Pre-Flight V2 adds harder multi-step reasoning questions
Complexity	Multi-step operational reasoning questions	Chart extraction, manual consultation, thrust/flap parameters	ALUE targets multi-modal/RAG; Pre-Flight targets harder operational reasoning
Compliance & audit evidence	Via Airside Labs' separate red-teaming, alongside Pre-Flight	Addresses hallucinations, biases, privacy generally	Compliance and audit readiness come from Airside Labs' use-case-specific red-teaming, not the benchmark

What This Means

For LLM developers targeting aviation, the takeaway is straightforward. If you are building AI tools for commercial airline operations, ground handling, or compliance-sensitive applications, Pre-Flight is where you start. If your ambition is integration into federal airspace management or safety-critical ATM systems, ALUE sets the standard you need to meet. For comprehensive assurance, you will likely need both.

The fact that 2025 produced two serious, purpose-built aviation AI benchmarks from very different corners of the industry is itself a signal. The era of evaluating aviation AI with general-purpose methods is ending. The question is no longer whether domain-specific assurance is necessary, but how quickly the industry can adopt it.

Works cited

1. MITRE and FAA Introduce Novel Aerospace Large Language Model Evaluation Benchmark
2. LLM Benchmarks January 2025 - Rinat Abdullin
3. Pre-Flight Aviation AI Benchmark | Open-Source Testing Suite - Airside Labs
4. Airside Labs: AI Security Testing & Compliance
5. EASA publishes survey results on ethics of Artificial Intelligence in Aviation at AI Days event
6. Aviation | MITRE
7. FAA System Security Testing and Evaluation - MITRE Corporation
8. AirsideLabs (Airside Labs) - Hugging Face
9. Aerospace Language Understanding Evaluation (ALUE): Large Language Benchmark with Aerospace Datasets | AIAA
10. MITRE, FAA Launch Aerospace LLM Evaluation Benchmark - ExecutiveGov
11. Benchmark Problem for Autonomous Urban Air Mobility - NASA
12. Defining Terminal Airspace Air Traffic Complexity Indicators - MDPI
13. AirsideLabs/pre-flight-06 - Hugging Face Datasets

Which aviation AI benchmark should I use — Pre-Flight or ALUE?

It depends on where your AI system sits in the aviation stack. For commercial aviation AI — airline operations, ground handling, dispatch, and customer service — start with Airside Labs' Pre-Flight to baseline a model's aviation domain knowledge (ICAO procedures, ground operations, dispatch, safety) during model selection. Compliance and adversarial assurance, mapped to the EU AI Act, OWASP LLM Top 10, NIST AI RMF, and MITRE ATLAS, come from Airside Labs' separate, use-case-specific adversarial datasets and red-teaming that run alongside Pre-Flight. For integration into the federal National Airspace System or air traffic management, the MITRE/FAA ALUE benchmark sets the standard you need to meet. Pre-Flight and ALUE are complementary layers, not competitors.

What is the MITRE/FAA ALUE benchmark?

ALUE (Aerospace Language Understanding Evaluation) is an open-source benchmark released in September 2025 by MITRE and the FAA to evaluate large language models on aerospace-specific tasks. It supports custom datasets, open-source and domain-specific models, user-defined prompts, and quantitative metrics, with tasks including hazard classification, extractive question answering for details such as tail numbers and runways, and analysis of air traffic control communications. It is available on GitHub to airlines, academia, and aerospace stakeholders.

Is the Pre-Flight aviation AI benchmark free and open source?

Yes. Pre-Flight is open source and published on GitHub and Hugging Face (huggingface.co/AirsideLabs), and it is included in the UK AI Security Institute's Inspect AI community evaluations package. You can download the question set, run it against your own models, and compare results against published baselines on the Airside Labs live results dashboard.