In December, Harvard joined many other universities to eliminate ACT and SAT requirements – the standardized tests that these institutions have recognized will not tell the full story of applicants. Is it time for the aviation industry to follow?

The trend of colleges moving away from SAT admissions protocols — albeit primarily for social reasons to encourage fairness among applicants — illustrates that sometimes when we look at things with fresh eyes, we realize there there are several ways to solve a problem.

The FAA knowledge exam – usually a requirement to pass before applying for a control lap – is in my mind a long-lasting and flawed system that all stakeholders, from pilots to examiners and even down to decision-makers, recognized that they needed to be corrected.

Or, in my opinion, eliminate.

Tradition says it weeds out some candidates, but I would say this system provides very little evidence of actual knowledge. So why do we cling to it?

A faulty system

Much like university test scores, proponents of maintaining knowledge exams might argue that examiners need a yardstick to compare applicants against some standard. The examiners are responsible for questioning the candidates on the weak points revealed by the questions missed on the knowledge test. With the recent transition to airman certification standards, this link is made even more explicit by the alignment of task codes.

Getting rid of tests would mean reviewing the whole verification mechanism and rethinking a whole methodology. Surely that would be an expensive undertaking, but I counter that maintaining the tests puts a strain on the whole system.

The knowledge testing industry

Consider what students now have to go through to pass a knowledge test. In 2018, the FAA has awarded a 10-year contract to PSI Services LLC (PSI), a provider of testing services in North America.

PSI (formerly LaserGrade) offers proctored computerized test administration for professional and vocational licensing and certification exams, as well as FAA certification using its own secure network and software. PSI is one of only two computerized testing companies authorized by the federal government to administer all FAA Airman Knowledge Tests. PSI offers more than 60 mandatory exams for commercial and private pilots, as well as airframe and powertrain technicians.

As more and more universities find ways to consider applicants without SAT and ACT scores, it proves that options exist.

At first glance, it would seem that this decision to standardize things meant more efficiency. All you had to do was register on the FAA website for the specific knowledge test you want to take and visit your local PSI test center.

Except pilots know that the entire process of taking a test – from preparation to verification – is anything but simple.

Because knowledge testing plays such an important role as a precursor to most checks, and then as a component of the check itself, candidates know they need to make the most of it. Like the SAT and ACT, these knowledge tests have led to their own market, where an entire industry of test prep companies now walk students through the tests, charging high fees. In fact, a March 2019 report by IBISWorld valued the tutoring and test prep industry at $1.1 billion, with test prep services accounting for 25% of the industry.

To be clear, this is for the entire US test prep market for all levels of education, including college admissions and other industries outside of aviation. The aviation industry has its own commercial players such as King School, ASA, Sporty’s Jeppesen, Boldmethod and many more.

Although it is difficult to measure how many pilots use these commercial services to prepare, a Webinar 2020 by the FAA provides an overview of the industry’s reach, explaining that it administers more than 150,000 exams per year. In fact, the presentation calls it one of the “world’s largest and most complex testing programs in any industry.”

A slide from a January 2020 FAA webinar explaining changes to the Airman Knowledge Testing System.

Pilots might recognize this process: it’s almost time to do your lap and you must pass a knowledge exam before applying to do the lap with a score of at least 70%. Your well-meaning instructor might encourage you to pay a subscription fee to a popular test prep company that advertises high pass rates and provides advice on how to study the question bank. Also, when you get to the actual check and the examiner has to review your knowledge test to question you on any shortcomings, the driver is incentivized to get the highest score possible on the knowledge test. This makes the whole process even more rigorous for the pilots, who spend a lot of time memorizing the answers rather than fundamentally understanding the question issues and paying more to retake the test if they don’t pass the first time. .

Moreover, as an example to prove that the process has become a substitute for the actual result – the driver’s knowledge – the FAA Airman Knowledge Test Matrix, which the administration released in May 2021 in an attempt to clarify parts of the process, adds to the complexity. In other words, in an effort to make the process as efficient as possible, the system misses the fact that pilots still view the process as a pass.

The cost to already cash-strapped pilots

What else? Candidate testing costs are onerous. The fee for all FAA Airman Knowledge Examinations is now $175. Considering that the pilot’s journey from private pilot to flight instructor includes taking at least six knowledge tests, in the end, they spent a lot of capital.

It can be argued that this model discourages potential pilots. Of course, compared to the direct cost of flight training, it is much less, but for many people it is something.

Do the numbers prove anything?

Is there a correlation between knowledge tests and initial audit pass rates? Do the pilots who do well in the knowledge tests pass their check flights on the first attempts?

Here is the 2020 data on initial check pass/fail rates in 2020. Overall, at the private, commercial, and even flight instructor level, the average initial pass rate is 77.5%. Can we claim that this is enough?

In the meantime, here the 2020 Airman Knowledge Test Statistics. The interesting thing to note is the higher pass rate and average score of the knowledge tests compared to the initial pass/fail stats of the control ride. Is it enough to continue to argue that knowledge test scores are a good measure of whether or not someone will pass their checkride?

Test questions lack nuance for global aviation cohort

The FAA pushed for more comprehensive testing, with more emphasis on pilot skill level. What does a limited test question bank really prove about a candidate that a check pilot shouldn’t be able to understand?

Another thing to consider is that more and more foreign students, with English as a second language, are enrolling in training around the world. Although English is the industry standard language, the rigid multiple-choice structure of these leaves no room for maneuver for students who might want to clarify something. Invariably, this reinforces the test preparation cycle where more memorization takes place. Part of the system overhaul is creating room for a broader base of multicultural contributors around the table to encourage greater inclusion.

It’s time to rethink our approach to standardizing knowledge testing.

Is it time for the aviation industry to take inspiration from what is happening in the broader college testing industry? As more and more universities find ways to consider applicants without SAT and ACT scores, it proves that options exist. Likewise, if our industry were to admit the limitations of knowledge testing and the lack of perceived value, we might find that options exist. Although the FAA has made efforts – to its credit – more can be done, in my opinion.