Latest Evaluation Results
Real-time performance dashboard for AI models tested on the Pre-Flight Aviation AI Benchmark — frontier hosted models vs. open-weight models you can run locally.
This live dashboard is the rolling source of truth. Our arXiv preprint reports a fixed leaderboard snapshot from 29 June 2026 (dataset revision 439d2d1); the numbers below update as new models are evaluated. Read the paper on arXiv.
Pre-flight Model Rankings
OpenAI
Meta AI
Alibaba
Anthropic
Google
SSDAIA
NVIDIA
MiMistral
MMMiniMax
xAI
MoMoonshot
DDeepSeek
UUpstage
ZZ.ai
Models
75
Top Score
84.0%
Average
73.9%
Gap to Human
11.0%
Model Rankings
| Rank | Model | Organization | Type | Score | Release Date |
|---|---|---|---|---|---|
|
1
|
GPT-5.6 Sol |
OpenAI
|
Frontier |
84.0%± 2.1%
|
Jun 2026 |
|
2
|
GPT-5.6 Terra |
OpenAI
|
Frontier |
83.7%± 2.1%
|
Jun 2026 |
|
3
|
Kimi K3 |
MoMoonshot
|
Frontier |
83.6%± 2.1%
|
Jul 2026 |
|
4
|
GPT-6 Astra |
OpenAI
|
Frontier |
83.6%± 2.1%
|
Aug 2026 |
|
5
|
Claude Opus 5 5 |
Anthropic
|
Frontier |
82.9%± 2.2%
|
Sep 2026 |
|
6
|
GPT-5.5 |
OpenAI
|
Frontier |
82.7%± 2.2%
|
Apr 2026 |
|
7
|
Gemini 3.8 Flash |
Google
|
Frontier |
82.5%± 2.2%
|
Aug 2026 |
|
8
|
Gemini 3.7 Flash |
Google
|
Frontier |
82.2%± 2.2%
|
Aug 2026 |
|
9
|
Claude Fable 5 |
Anthropic
|
Frontier |
82.0%± 2.2%
|
Jun 2026 |
|
10
|
Claude Fable 5.1 |
Anthropic
|
Frontier |
81.9%± 2.2%
|
Aug 2026 |
|
11
|
GPT-5.6 Luna |
OpenAI
|
Frontier |
81.3%± 2.3%
|
Jun 2026 |
|
12
|
Qwen3.8 Max |
Alibaba
|
Open-weight |
81.2%± 2.3%
|
Sep 2026 |
|
13
|
Claude Opus 5 |
Anthropic
|
Frontier |
80.5%± 2.3%
|
Jul 2026 |
|
14
|
GPT-5 |
OpenAI
|
Frontier |
80.3%± 2.3%
|
Jul 2025 |
|
15
|
Claude Opus 4.6 |
Anthropic
|
Frontier |
80.3%± 2.3%
|
Feb 2026 |
|
16
|
Claude Opus 4.7 |
Anthropic
|
Frontier |
79.7%± 2.3%
|
Apr 2026 |
|
17
|
DeepSeek V4 Pro |
DDeepSeek
|
Open-weight |
79.5%± 2.3%
|
Aug 2026 |
|
18
|
Grok 4.5 |
xAI
|
Frontier |
79.2%± 2.4%
|
Jul 2026 |
|
19
|
Gemini 2.5 Pro |
Google
|
Frontier |
79.0%± 2.4%
|
Jun 2025 |
|
20
|
Claude Opus 4.8 |
Anthropic
|
Frontier |
79.0%± 2.4%
|
May 2026 |
|
21
|
Claude Sonnet 5 5 |
Anthropic
|
Frontier |
78.5%± 2.4%
|
Sep 2026 |
|
22
|
Claude Sonnet 5 |
Anthropic
|
Frontier |
78.3%± 2.4%
|
Jun 2026 |
|
23
|
Claude Opus 4.1 |
Anthropic
|
Frontier |
78.0%± 2.4%
|
Aug 2025 |
|
24
|
GLM-5.3 |
ZZ.ai
|
Frontier |
77.8%± 2.4%
|
Aug 2026 |
|
25
|
Qwen3.5 122B-A10B int4 (local 2x Spark) |
Alibaba
|
Open-weight |
77.3%± 2.4%
|
Jan 2026 |
|
26
|
GPT-5.1 |
OpenAI
|
Frontier |
77.0%± 2.4%
|
Nov 2025 |
|
27
|
Gemini 3.6 Flash |
Google
|
Frontier |
76.8%± 2.4%
|
Jul 2026 |
|
28
|
NVIDIA Llama 3.1 Nemotron |
NVIDIA
|
Open-weight |
76.8%± 2.4%
|
Jun 2025 |
|
29
|
Mistral Large 2512 |
MiMistral
|
Frontier |
76.2%± 2.5%
|
Dec 2025 |
|
30
|
Grok 4.6 |
xAI
|
Frontier |
76.2%± 2.5%
|
Aug 2026 |
|
31
|
MiniMax M3 |
MMMiniMax
|
Frontier |
75.8%± 2.5%
|
May 2026 |
|
32
|
Qwen3.6 35B-A3B FP8 (local Spark) |
Alibaba
|
Open-weight |
75.7%± 2.5%
|
Jan 2026 |
|
33
|
Gemini 3.5 Flash |
Google
|
Frontier |
75.7%± 2.5%
|
Jan 2026 |
|
34
|
GPT-5.4 |
OpenAI
|
Frontier |
75.7%± 2.5%
|
May 2026 |
|
35
|
Claude Opus 4.5 |
Anthropic
|
Frontier |
75.3%± 2.5%
|
Nov 2025 |
|
36
|
DeepSeek V4 Flash |
DDeepSeek
|
Open-weight |
75.2%± 2.5%
|
Jul 2026 |
|
37
|
Gemini 2.5 Flash |
Google
|
Frontier |
75.0%± 2.5%
|
Jun 2025 |
|
38
|
GPT-5.4 Mini |
OpenAI
|
Frontier |
75.0%± 2.5%
|
Mar 2026 |
|
39
|
Llama 3.3 70B |
Meta AI
|
Open-weight |
74.7%± 2.5%
|
Jun 2024 |
|
40
|
Claude 3.7 Sonnet † |
Anthropic
|
Frontier |
74.7%± 2.5%
|
Feb 2025 |
|
41
|
Claude 4.5 Sonnet |
Anthropic
|
Frontier |
74.7%± 2.5%
|
Sep 2025 |
|
42
|
Llama 4 Scout 17B |
Meta AI
|
Open-weight |
74.3%± 2.5%
|
May 2025 |
|
43
|
GPT-5.2 |
OpenAI
|
Frontier |
74.3%± 2.5%
|
Dec 2025 |
|
44
|
Qwen3-Coder-Next FP8 (local Spark) |
Alibaba
|
Open-weight |
73.7%± 2.5%
|
Jan 2026 |
|
45
|
Solar Pro 4 |
UUpstage
|
Frontier |
73.5%± 2.6%
|
Aug 2026 |
|
46
|
GPT-4o Mini † |
OpenAI
|
Frontier |
73.3%± 2.6%
|
Jul 2024 |
|
47
|
GPT-4o (Nov 2024) † |
OpenAI
|
Frontier |
73.3%± 2.6%
|
Nov 2024 |
|
48
|
Gemma 4 26B-A4B (local Spark) |
Google
|
Open-weight |
73.3%± 2.6%
|
Jan 2026 |
|
49
|
Claude Sonnet 4.6 |
Anthropic
|
Frontier |
72.7%± 2.6%
|
Nov 2025 |
|
50
|
Claude Haiku 4.5 |
Anthropic
|
Frontier |
72.3%± 2.6%
|
Jan 2025 |
|
51
|
MiniMax M2.7 AWQ (local 2x Spark) |
MMMiniMax
|
Frontier |
72.3%± 2.6%
|
Jan 2026 |
|
52
|
Qwen3.5 35B-A3B FP8 (local Spark) |
Alibaba
|
Open-weight |
72.0%± 2.6%
|
Jan 2025 |
|
53
|
Llama 4 Scout † |
Meta AI
|
Open-weight |
71.7%± 2.6%
|
May 2025 |
|
54
|
Claude 3.5 Sonnet (Oct 2024) † |
Anthropic
|
Frontier |
71.3%± 2.6%
|
Oct 2024 |
|
55
|
GPT-OSS 120B MXFP4 (local Spark) |
OpenAI
|
Open-weight |
71.3%± 2.6%
|
May 2025 |
|
56
|
Nemotron-3 Super 120B-A12B NVFP4 (local Spark) |
NVIDIA
|
Open-weight |
71.3%± 2.6%
|
Jan 2026 |
|
57
|
Qwen3-Coder-Next int4 (local Spark) |
Alibaba
|
Open-weight |
71.3%± 2.6%
|
Jan 2026 |
|
58
|
Qwen3.6 27B |
Alibaba
|
Open-weight |
71.2%± 2.6%
|
Jan 2026 |
|
59
|
Llama 3 70B (Groq) † |
Meta AI
|
Open-weight |
70.7%± 2.6%
|
Apr 2024 |
|
60
|
Qwen3 32B |
Alibaba
|
Open-weight |
70.7%± 2.6%
|
Apr 2025 |
|
61
|
GPT-OSS 120B |
OpenAI
|
Open-weight |
70.7%± 2.6%
|
May 2025 |
|
62
|
DiffusionGemma 26B-A4B NVFP4 (local Spark) |
Google
|
Open-weight |
69.3%± 2.7%
|
Jan 2026 |
|
63
|
Claude 3 Haiku † |
Anthropic
|
Frontier |
68.3%± 2.7%
|
Jul 2024 |
|
64
|
Nemotron-3 Nano 30B-A3B (local Spark) |
NVIDIA
|
Open-weight |
68.0%± 2.7%
|
Jan 2026 |
|
65
|
Qwen3.8 27B |
Alibaba
|
Open-weight |
67.1%± 2.7%
|
Aug 2026 |
|
66
|
Claude 3.5 Haiku † |
Anthropic
|
Frontier |
66.7%± 2.7%
|
Oct 2024 |
|
67
|
GPT-4 Preview (Jan 2024) † |
OpenAI
|
Frontier |
66.0%± 2.7%
|
Jan 2024 |
|
68
|
Llama 3 8B (Groq) † |
Meta AI
|
Open-weight |
66.0%± 2.7%
|
Apr 2024 |
|
69
|
GPT-OSS Safeguard 20B |
OpenAI
|
Open-weight |
65.3%± 2.7%
|
Oct 2025 |
|
70
|
GPT-3.5 Turbo † |
OpenAI
|
Frontier |
64.0%± 2.8%
|
Jan 2024 |
|
71
|
GPT-OSS 20B |
OpenAI
|
Open-weight |
63.7%± 2.8%
|
May 2025 |
|
72
|
Gemma 2 9B † |
Google
|
Open-weight |
62.3%± 2.8%
|
Jul 2024 |
|
73
|
Qwen QWQ 32B † |
Alibaba
|
Open-weight |
58.7%± 2.8%
|
May 2024 |
|
74
|
Llama 3.1 8B |
Meta AI
|
Open-weight |
58.0%± 2.8%
|
Sep 2024 |
|
75
|
ALLaM 2 7B |
SSDAIA
|
Open-weight |
25.3%± 2.5%
|
Jan 2025 |
Cite us
These results accompany our arXiv preprint, "Pre-Flight: A Benchmark for Evaluating Large Language Models on Aviation Operational Knowledge" by Alex Brooker (Airside Labs) and Tim Hughes (Mahino Research). If you use the benchmark, please cite:
@misc{brooker2026preflight,
title={Pre-Flight: A Benchmark for Evaluating Large Language Models on Aviation Operational Knowledge},
author={Brooker, Alex and Hughes, Tim},
year={2026},
eprint={2607.01829},
archivePrefix={arXiv},
primaryClass={cs.AI},
doi={10.48550/arXiv.2607.01829}
}
arXiv abstract · PDF · DOI 10.48550/arXiv.2607.01829 · Licensed CC BY 4.0