Preflight - The Aviation Intelligence Benchmark: Guidelines for Contributors

Introduction

Objective

We are collecting the most challenging and comprehensive set of aviation-related questions ever assembled. Can you think of aviation knowledge that would challenge current artificial intelligence (AI) systems? Your contributions will help us evaluate the capabilities of AI systems in understanding and reasoning about aviation concepts. Contributors of accepted questions will be credited, have access to the entire benchmark and be invited to co-author the paper corresponding to this dataset.

Process

1. Write Question: Create a valid, highly challenging aviation-related question in English that both AIs and average humans would struggle to answer. Aim for questions that only exceptional aviation professionals could answer correctly.
2. AI Difficulty Assessment: State-of-the-art AI models will attempt to answer your question to help determine its difficulty level.
3. Write Answer Explanation: If your question proves sufficiently challenging for AIs, you'll be asked to provide a thorough yet concise solution.
4. Peer Review: Your submission (question, answer, and rationale) will undergo manual review to ensure benchmark quality. The answer rationale will aid experts and AIs in verifying the correctness of your provided answer.
5. Publication: Selected questions will be included in the dataset, with contributors' names highlighted. Top contributors may receive special recognition in the paper appendix.

Guidelines

1. Original: Questions must be your own original work, not copied from other sources. You may include questions from aviation exams you've created, provided the answers aren't publicly available and the questions are extremely challenging.
2. Challenging: Questions should be difficult for non-experts and not easily answerable by the general public or casual aviation enthusiasts. Avoid simple calculations or straightforward facts. Questions can be so challenging that they might be impractical for real-world aviation exams.
   
3. Objective and Self-Contained: Questions should have answers accepted by other aviation experts. Avoid highly disputed or subjective topics. Include all necessary context and definitions within the question. You may use technical aviation jargon or notation without definition if it's standard and unambiguous within the field.
   
4. Aviation-Focused: All questions must relate directly to aviation topics, which may include but are not limited to:

Aerodynamics and flight physics

Aircraft systems and avionics

Navigation and meteorology

Air traffic control and airspace management

Aviation regulations and safety procedures

Aircraft design and engineering

Aviation history and technological advancements

Airline and Airport commercial strategies

Airline Network Planning and Airport Slots

Remember, the goal is to create a benchmark that truly challenges AI systems in their understanding of aviation concepts and practices. Your expertise and unique insights are crucial to this effort.

5. Safety and Security: Do not include questions that could compromise aviation safety or security if the answers were widely known. Avoid questions about potential vulnerabilities in aviation systems or security procedures.

6. Diversity: Try to cover a wide range of aviation topics and difficulty levels to ensure a comprehensive benchmark.