Training Tips for Ships - Tip #43: What’s the Point of Testing (Part II)?

In the previous edition of Training Tips for Ships, we made the point that assessment can never tell us whether a candidate has all the knowledge required for a role or is able to perform the skills necessary under all conditions. Instead, assessment is essentially an audit or sampling process whereby we test a subset of skills or knowledge in the hope that what we learn about that subset can be extrapolated to all the required knowledge and skills. Understanding this core truth about assessment creates a number of implications about how we need to train and test for optimal performance. Let’s discuss some of them here.

One critical implication is that since we are not testing everything, we had better ensure that learners do not use knowledge of that fact to study only what they expect to be tested. Said another way, one core value of testing is the incentive it can create for the trainee to learn as well as they can and as much as they can. If they want to pass the test and they have no idea what is going to be on that test (i.e. they don’t know what items will be “sampled”), then they have no choice but to study everything equally. This is the goal we are trying to achieve - motivation to study as well and broadly as possible. If this is done, then learning is improved, and assessors can generally extrapolate the test results to those parts of the curriculum that were not explicitly tested - which is good.

Understanding this, we have to be very careful about our assessment practices because it is easy to “break” the audit effect, removing the motivation to study broadly and rendering useless any conclusions about those items we did not test. There are many common testing practices that break the audit effect entirely.

One such example is giving the same test to multiple learners. Doing so will create an incentive in them to share the test details, removing their incentive to study broadly and our ability to extrapolate. The end result is they know less, and we know less about what they know.

Another great way to break the audit effect is to allow for exams to be “open book”. An open book exam will provide you with information about whether the candidate was able to find the answers to the questions on the exam (and may even teach him or her the answers in the process), but will provide no information about knowledge and skills not present on the exam. The only valid reason to give open book exams is to test the candidate’s ability to successfully find information - so if that is your goal, they can be a good choice.

A third example of how to break the audit process is to allow repeated taking of an exam, especially with little or no required time interval in between. Efficient trainees will quickly determine that the fastest way to pass the exam is to take it over and over until they manage to achieve a passing score. Many can and will do this without ever studying the materials. As in the cases above, they will learn very little in the process, and our test results will tell us almost nothing of their readiness to perform.

Testing is a critical part of the training process. Sadly, it is easy to get it wrong. However, it is also easy to get it right - so long as there is a good understanding of what testing actually is. Knowing that testing is necessarily the process of sampling knowledge and skills makes it easy to make intelligent choices when designing and delivering your testing program.

There is more to be said on assessment and we will do so in future Training Tips for Ships. Until then, keep well and sail safely.