Pre-Flight Aviation AI Benchmark

Pre-Flight is an evaluation benchmark that tests the model's understanding of ICAO annex documentation, flight dispatch rules and airport ground operations safety procedures and protocols. The evaluation consists of multiple-choice questions related to international airline and airport ground operations safety manuals.

The benchmark was developed by Airside Labs' founder along with a small community of aviation experts with experience in Air Traffic Management, ground operations and commercial flying.

Pre-Flight is now described in full in our arXiv preprint, "Pre-Flight: A Benchmark for Evaluating Large Language Models on Aviation Operational Knowledge" by Alex Brooker (Airside Labs) and Tim Hughes (Mahino Research).

Also available as a direct PDF · DOI 10.48550/arXiv.2607.01829 · Licensed CC BY 4.0

Cite us

If you use the Pre-Flight benchmark in your work, please cite our arXiv preprint:

@misc{brooker2026preflight,
  title={Pre-Flight: A Benchmark for Evaluating Large Language Models on Aviation Operational Knowledge},
  author={Brooker, Alex and Hughes, Tim},
  year={2026},
  eprint={2607.01829},
  archivePrefix={arXiv},
  primaryClass={cs.AI},
  doi={10.48550/arXiv.2607.01829}
}

Why Airside Labs for Aviation AI Evaluation

Airside vs General Benchmarks

Pre-Flight is an aviation knowledge benchmark — ideal for shortlisting models. Adversarial robustness and regulatory audit evidence come from Airside Labs' separate, use-case-specific red-teaming, which runs alongside it.

Pre-Flight vs MITRE/FAA ALUE

The two purpose-built aviation benchmarks answer different questions. They are complementary, not competing — Pre-Flight helps you choose a model; MITRE and the FAA's ALUE is oriented to US federal airspace programmes.

For the full breakdown, see our Pre-Flight vs MITRE/FAA ALUE comparison, or the wider aviation AI evaluation & benchmarking landscape (Pre-Flight, ALUE and PilotBench).

The company behind Pre-Flight

Pre-Flight is built and maintained by Airside Labs, an independent aviation-AI company. We build and validate AI for aviation and travel — from prototypes, fine-tuned embedding and language models and RAG systems to domain-specific evaluation, red-teaming and assurance. Pre-Flight is one of our open tools; evaluation is one part of the picture. Airside Labs was founded by Alex Brooker (CEng, FIET, Chair of the IET Aerospace Technical Network), who spent 25+ years in safety-critical aviation and defence at BAE Systems and as VP of R&D at Cirium (RELX PLC), delivering certified flight-plan validation and arrival-management systems used by NATS, the FAA and IATA. The benchmark was authored with Tim Hughes of Mahino Research.

Dataset Overview

The dataset consists of multiple-choice questions drawn from standard international airport ground operations safety manuals. Each question has 4-5 possible answer choices, with one correct answer.

Topics Covered

Dataset Sections

Note: Gaps in ID sequences enable future additions

Example Questions

Note: Actual evaluation uses the multiple_choice solver. These examples have been lightly edited for readability.

Scoring Methodology

The benchmark is scored using accuracy, which is the proportion of questions answered correctly. Each question has one correct answer from the multiple choices provided. The implementation uses the multiple_choice solver and the choice scorer.

Launch Evaluation Report (March 2025)

These are the original results from running the full dataset (300 samples) on March 21, 2025. The models are the ones available at launch and are kept here as a record.

See the current leaderboard for newer models →

These results demonstrate the capability of modern language models to comprehend aviation safety protocols and procedures from international standards documentation. The strongest models achieve approximately 75% accuracy on the dataset.

Test Your AI on Aviation Safety

Contact us to evaluate your AI systems against the Pre-Flight benchmark or develop custom aviation-specific testing frameworks.