By Bob Baron, Ph.D
The Aviation Consulting Group
As your company’s Human Factors (HF) instructor, you have just finished conducting a two-day HF course for 20 aircraft mechanics. So, here’s a question for you: Was the training impactful enough to make a difference in on-the-job behaviors, and if so, is it possible to quantify the effects of the HF training?
Let’s start with the first part of the question. Unquestionably, the experience of the HF course itself can make a big difference in whether or not the mechanics absorb and transfer the classroom skills to the real world. A poorly developed, mundane course will likely bore the attendees and negatively impact the intended outcome of the training. This, combined with a disinterested instructor, who tells war stories for the entire course, or just reads the slides, pretty much guarantees that the training will be a waste of time for all concerned!
Now let’s shift to the second part of the question. With the assumption that the training course is well-developed and delivered, we want to know if we can observe and quantify the behavioral changes that have taken place as a result of the training. In other words, is there a positive transfer of knowledge, skills, and attitudes to the real world? To answer this question, we need to measure the performance objectives. Those objectives include, but are not limited to, building an awareness of errors and error-provoking conditions, and how the mechanic uses that awareness to minimize, or prevent, the commission of maintenance errors. To know if the HF skills, knowledge, and attitudes, as a result of the HF training, are having an effect, we can use the following methods:
Safety Performance Indicators (SPIs)
You already have, among others, the following SPIs-
- Accidents
- Incidents
- Occurrences
- On-the-job injuries
Use these indicators to look for trends on a year-over-year and/or month-over-month basis. If HF training is newly deployed, then theoretically there should be a downward trend in these indicators. If HF training has been an ongoing process, we hope to also see an ongoing downward trend. But trend is the key word. There will be safety events, even during downward trends. When safety events do occur, were they due to Human Factors? If so, you might need to further reinforce the transfer of training to the job. What went wrong? Why? How can we prevent it from happening again? Do we need to add/modify/expand on a particular HF topic in the training course (i.e., focus on particular problem areas such as fatigue, procedural deviations, assertiveness, communication, etc.)?
It should be noted that these SPIs can have both upward and downward trends, and that we are using correlations with the HF training. But, correlation does not mean causation, as there may be other intervening variables. However, with a strong correlation, we can be pretty confident in the results.
Observations
One of the most effective ways of determining whether there is behavioral change is by direct observation. Take a look around the hangar. Are the mechanics doing things the way they are supposed to be done? Or, are you noticing deviations from procedures (i.e., skipping functional checks, signing off inspections that were not completed, working on tasks from memory, etc.)?
It is important to point out that these types of observations are not meant to be punitive; but rather to identify systemic, human factors issues within the company. This can be done by surveying key people who observe the mechanics. Or, as the safety manager, you can just take a walk through the hangar at random times. You’d be surprised at how many things you will see walking from one end of the hangar to the other if you really pay attention.
Speaking of non-punitive observations, you might want to consider implementing a Line Operations Safety Audit (LOSA) for Maintenance Operations. A LOSA observation is conducted by a non-threatening company employee (typically another mechanic) who takes notes on what is happening during the course of normal maintenance activities. Once all the data are collected, then a report is written and recommendations are made to management to both reinforce the positives and improve the negatives.
Surveys
Surveys can be deployed online and be anonymous and confidential. Surveys can provide a “peek” into the effectiveness of the HF training directly from the mechanics. A carefully constructed, short survey can measure attitudes, opinions, and beliefs about various aspects of the HF training.
While surveys are an excellent way of collecting a wealth of information, it should be kept in mind that surveys are subjective and any changes (if needed) to the HF training program should be based on an appropriate sample size. If your company employs 500 mechanics, and only 15 of the mechanics respond to a survey, the results will be questionable. The typical response rate for surveys is 20%-30%. Although this seems rather low, it should still be enough for statistical significance.
Interviews
When it comes to interviews, trust is a must! With trust and openness, you will find that one-on-one, confidential interviews can provide an extremely useful source of information. Interviews should be done casually (avoid checklists and forms). Just listen and take notes. mechanics are generally willing to discuss things in a casual, personal interview that they may not want to bring up in other mediums. For instance, you might find a trend in mechanics not feeling comfortable submitting hazard reports in the voluntary reporting system because they feel it’s a waste of time and nothing will get done to fix the issues. You may also detect this trend in the survey responses discussed above. That’s why using a variety of methods to collect data is so important. There’s power in numbers!
Interviews can also be done in a focus group format. Just as with the one-on-one format, a focus group provides direct feedback from the mechanics. Focus groups can be more comfortable for the group attendees as they are discussing issues with other colleagues, rather than feeling singled out in a personal interview.
In either format, be sure to let the interviewees know that they have the right to refuse an interview and that there will be no repercussions if they choose not to be interviewed. Remember, this is all part of your non-punitive Just Culture.
Summary
For many companies, HF training is simply a “check the box” activity. These companies are not as concerned about the training’s efficacy (transfer to the job) as much as satisfying a regulatory requirement. That’s not what HF training is all about. You need to move away from the “check the box” approach and give your mechanics a reason to take what they’ve learned in the classroom and move it to the hangar. It starts with a well-developed HF course, with an effective and respected instructor/facilitator, and the appropriate, ongoing performance measures to ensure that the training is meeting its objectives.
On a final note, just because an HF class attendee scores 100% on the classroom written test, it does not guarantee the AMT will have the desired behavioral change on the job. Conversely, a mechanic that barely passes the written test may still have the desired behavioral change. This is one of the reasons why, in HF training, I put less emphasis on the results of written tests and more emphasis on the real test; the demonstrable transfer of knowledge to the AMTs on-the-job activities. This might include double checking work, not skipping steps, being assertive when needed, and much, much more.
Hopefully, this article has helped to answer the question of how do you know if your HF training is working. It’s a sad fact that many companies aren’t interested in reaching this level of detail regarding HF training effectiveness. But those companies that do, will undoubtedly be the ones that have a safer workforce, which in turn will show that HF training is an investment rather than an expense. The ROI will be quantifiable, and management will like that!
Dr. Bob Baron conducts aviation safety training, consulting, and program implementation for aviation operators on a global basis.
Sensitive and knowledgeable about various cultures, Dr. Baron uses his 32+ years of academic and practical experience to assist aviation organizations in their pursuit of safety and quality excellence. He has extensive experience working with developing nations and island countries. He also provides training and consulting to some of the largest airlines and aircraft manufacturers in the world, as well as civil aviation authorities and accident investigation bureaus.
If your aviation organization is interested in improving its culture, implementing programs such as Human Factors, SMS, or LOSA, or have an external, unbiased safety audit/Gap analysis, please get in touch.
Dr. Baron’s company, TACG, provides numerous training and consulting services. For more information, please go to www.tacgworldwide.com