Latest Evaluation Results
Real-time performance dashboard for AI models tested on the Pre-Flight Aviation AI Benchmark — frontier hosted models vs. open-weight models you can run locally.
This live dashboard is the rolling source of truth. Our arXiv preprint reports a fixed leaderboard snapshot from 29 June 2026 (dataset revision 439d2d1); the numbers below update as new models are evaluated. Read the paper on arXiv.
Pre-flight Model Rankings
OpenAI
Alibaba
Anthropic
Meta AI
Google
SDAIA
NVIDIA
MiniMax
Models
45
Top Score
80.0%
Average
68.9%
Gap to Human
15.0%
Model Rankings
| Rank | Model | Organization | Type | Score | Release Date |
|---|---|---|---|---|---|
|
1
|
GPT-5.5 |
OpenAI
|
Frontier |
80.0%± 2.3%
|
Apr 2026 |
|
2
|
GPT-5 |
OpenAI
|
Frontier |
78.0%± 2.4%
|
Aug 2025 |
|
3
|
Claude Opus 4.8 |
Anthropic
|
Frontier |
76.7%± 2.5%
|
May 2026 |
|
4
|
Gemini 2.5 Pro |
Google
|
Frontier |
76.3%± 2.5%
|
Jun 2025 |
|
5
|
NVIDIA Llama 3.1 Nemotron |
NVIDIA
|
Open-weight |
75.0%± 2.5%
|
Apr 2025 |
|
6
|
Claude 3.7 Sonnet |
Anthropic
|
Frontier |
74.7%± 2.5%
|
Feb 2025 |
|
7
|
Qwen3.5 122B-A10B int4 (local 2x Spark) |
Alibaba
|
Open-weight |
74.7%± 2.5%
|
Jan 2026 |
|
8
|
GPT-5.1 |
OpenAI
|
Frontier |
74.3%± 2.5%
|
Nov 2025 |
|
9
|
GPT-4o Mini |
OpenAI
|
Frontier |
73.3%± 2.6%
|
Jul 2024 |
|
10
|
GPT-4o (Nov 2024) |
OpenAI
|
Frontier |
73.3%± 2.6%
|
Nov 2024 |
|
11
|
Gemini 3.5 Flash |
Google
|
Frontier |
73.3%± 2.5%
|
Apr 2026 |
|
12
|
GPT-5.4 |
OpenAI
|
Frontier |
73.0%± 2.6%
|
Mar 2026 |
|
13
|
Llama 3.3 70B |
Meta AI
|
Open-weight |
72.3%± 2.6%
|
Dec 2024 |
|
14
|
Llama 4 Scout 17B |
Meta AI
|
Open-weight |
72.3%± 2.6%
|
Apr 2025 |
|
15
|
Gemini 2.5 Flash |
Google
|
Frontier |
72.3%± 2.6%
|
Jun 2025 |
|
16
|
Qwen3.6 35B-A3B FP8 (local Spark) |
Alibaba
|
Open-weight |
72.3%± 2.6%
|
Jan 2026 |
|
17
|
Claude 4.5 Sonnet |
Anthropic
|
Frontier |
72.0%± 2.6%
|
Sep 2025 |
|
18
|
Llama 4 Scout |
Meta AI
|
Open-weight |
71.7%± 2.6%
|
Apr 2025 |
|
19
|
Claude 3.5 Sonnet (Oct 2024) |
Anthropic
|
Frontier |
71.3%± 2.6%
|
Oct 2024 |
|
20
|
Qwen3-Coder-Next FP8 (local Spark) |
Alibaba
|
Open-weight |
71.0%± 2.6%
|
Jan 2026 |
|
21
|
Llama 3 70B (Groq) |
Meta AI
|
Open-weight |
70.7%± 2.6%
|
Apr 2024 |
|
22
|
Gemma 4 26B-A4B (local Spark) |
Google
|
Open-weight |
70.7%± 2.6%
|
Mar 2026 |
|
23
|
Qwen3.5 35B-A3B FP8 (local Spark) |
Alibaba
|
Open-weight |
69.7%± 2.6%
|
Sep 2025 |
|
24
|
Claude Haiku 4.5 |
Anthropic
|
Frontier |
69.7%± 2.7%
|
Oct 2025 |
|
25
|
MiniMax M2.7 AWQ (local 2x Spark) |
MiniMax
|
Frontier |
69.7%± 2.6%
|
Jan 2026 |
|
26
|
Claude Sonnet 4.6 |
Anthropic
|
Frontier |
69.3%± 2.7%
|
Nov 2025 |
|
27
|
GPT-OSS 120B |
OpenAI
|
Open-weight |
69.0%± 2.7%
|
May 2025 |
|
28
|
GPT-OSS 120B MXFP4 (local Spark) |
OpenAI
|
Open-weight |
69.0%± 2.7%
|
Aug 2025 |
|
29
|
Nemotron-3 Super 120B-A12B NVFP4 (local Spark) |
NVIDIA
|
Open-weight |
69.0%± 2.7%
|
Jan 2026 |
|
30
|
GPT-OSS 120B |
OpenAI
|
Open-weight |
68.7%± 2.7%
|
Aug 2025 |
|
31
|
Qwen3-Coder-Next int4 (local Spark) |
Alibaba
|
Open-weight |
68.7%± 2.7%
|
Jan 2026 |
|
32
|
Claude 3 Haiku |
Anthropic
|
Frontier |
68.3%± 2.7%
|
Mar 2024 |
|
33
|
Qwen3 32B |
Alibaba
|
Open-weight |
68.0%± 2.7%
|
Apr 2025 |
|
34
|
DiffusionGemma 26B-A4B NVFP4 (local Spark) |
Google
|
Open-weight |
67.3%± 2.7%
|
Jan 2026 |
|
35
|
Claude 3.5 Haiku |
Anthropic
|
Frontier |
66.7%± 2.7%
|
Oct 2024 |
|
36
|
GPT-4 Preview (Jan 2024) |
OpenAI
|
Frontier |
66.0%± 2.7%
|
Jan 2024 |
|
37
|
Llama 3 8B (Groq) |
Meta AI
|
Open-weight |
66.0%± 2.7%
|
Apr 2024 |
|
38
|
Nemotron-3 Nano 30B-A3B (local Spark) |
NVIDIA
|
Open-weight |
65.7%± 2.7%
|
Jan 2026 |
|
39
|
GPT-3.5 Turbo |
OpenAI
|
Frontier |
64.0%± 2.8%
|
Jan 2024 |
|
40
|
GPT-OSS Safeguard 20B |
OpenAI
|
Open-weight |
63.3%± 2.8%
|
Oct 2025 |
|
41
|
Gemma 2 9B |
Google
|
Open-weight |
62.3%± 2.8%
|
Jul 2024 |
|
42
|
GPT-OSS 20B |
OpenAI
|
Open-weight |
61.7%± 2.8%
|
Aug 2025 |
|
43
|
Qwen QWQ 32B |
Alibaba
|
Open-weight |
58.7%± 2.8%
|
Mar 2024 |
|
44
|
Llama 3.1 8B |
Meta AI
|
Open-weight |
55.7%± 2.9%
|
Sep 2024 |
|
45
|
ALLaM 2 7B |
SDAIA
|
Open-weight |
24.3%± 2.5%
|
Jan 2025 |
Cite us
These results accompany our arXiv preprint, "Pre-Flight: A Benchmark for Evaluating Large Language Models on Aviation Operational Knowledge" by Alex Brooker (Airside Labs) and Tim Hughes (Mahino Research). If you use the benchmark, please cite:
@misc{brooker2026preflight,
title={Pre-Flight: A Benchmark for Evaluating Large Language Models on Aviation Operational Knowledge},
author={Brooker, Alex and Hughes, Tim},
year={2026},
eprint={2607.01829},
archivePrefix={arXiv},
primaryClass={cs.AI},
doi={10.48550/arXiv.2607.01829}
}
arXiv abstract · PDF · DOI 10.48550/arXiv.2607.01829 · Licensed CC BY 4.0