Introduction
At last year’s FSEMC conference in Utrecht Mark Dransfield from our Sim Ops team gave a presentation entitled “Recurrent Evaluations - A Requiem”. At the conference, and subsequently, this presentation generated a lot of questions. Our editorial team caught up with Mark and posed some of these questions.
What are the issues that need to be considered?
Q - Sim Ops
In your presentation you started off asking the audience what they believed the real purpose of doing recurrent evaluations is. Did you get the answer you expected?
A - Mark Dransfield
It was interesting as I got some different answers, judging by the show of hands, when I asked if the purpose of NAA recurrent evaluations was one of the following:
for the NAA to determine for itself if the FSTD was still compliant with its original qualification status, or
for the NAA to check if the FSTD Operator (FSTDO) was doing its job in maintaining compliance in accordance with its Compliance Monitoring Manual (CMM), or
because EASA regulation PART-ARA mandates the NAA to conduct a yearly recurrent evaluation
Although all are correct to some extent the most relevant answer, from my regulatory experiences, is (b), that the NAA is checking the operator is, or should be, ensuring the FSTD is maintaining compliance properly with its original initial qualification criteria. That is certainly the sentiment I have heard from several EASA regulatory personnel over the course of the last year, and it makes sense to me. The million dollar question is what's the best way to achieve that, conducting traditional recurrent evaluations, or moving towards a more Performance-Based Oversight approach? The recurrent evaluation process we have today works but we haven't evolved it in over 30 years, yet the FSTD product and FSTDO operational capabilities have evolved tremendously.
Q - Sim Ops
You elaborated on what actually happens during a recurrent evaluation and emphasised the need for operators to be focussed on compliance monitoring; this message has been promulgated for some years now, is the message hitting home with operators yet?
A - Mark Dransfield
To an extent but I still believe there is some way to go, I have seen the good, the bad and the ugly in my experiences of assessing compliance management and monitoring systems. Let me explain.
A typical recurrent still consists of a sampling of the objective and subjective testing capabilities of the FSTD in addition to how the device has performed over the last year from operational metrics to quality and discrepancy reviews. However, I still think some operators regard NAA simulator qualification (recurrent) and NAA compliance monitoring (CMS audits) as separate activities…they are not and are inextricably linked. A recurrent evaluation is really a mini compliance audit for the FSTDO compliance management (as per EASA ORA.FSTD.100 General) that requires the FSTDO to develop some, or all, of the following:
Compliance Management System
Safety Management System
Organisation Management Manual
FSTD Operations Manual
This is to demonstrate that the FSTDO has the capability to maintain the performance, functions and other characteristics specified for the FSTD’s qualification level as well as controlling the installation, acceptance and initial/recurrent qualification process.
Q - Sim Ops
You mentioned the current Extended Evaluations Program[me] (EEP), has this been widely adopted, is it now common practice?
A - Mark Dransfield
I wouldn't say it's common practice today but it is on the increase and becoming more frequent as FSTDOs develop more mature compliance management systems and have a track record of hiring trained experienced personnel capable of conducting successful recurrent evaluations on specific FSTDs on behalf of the authority. EEP is something that has to be earned by the FSTDO through demonstrating its ability to develop and maintain robust compliance monitoring processes and reporting with the relevantly trained personnel dedicated to the task.
Q - Sim Ops
The Extended Evaluations program[me] requires an operator to have an assigned person(s) with adequate experience to review the QTG re-runs, conduct subjective flyouts, etc. but adequate experience is not defined and is quite loose, what is your take on that?
A - Mark Dransfield
I agree that we don't have a good definition of the experience or training that someone at an FSTDO must have to fulfil these roles,…. but that's not just relevant to EEPP,… it is expected that the operator should have the capability and experience to review the initial MQTG from the TDM in advance of the initial qualification. A lot of people still don't do this as they regard critiquing and approval of the MQTG as the NAA’s job at the initial qualification…it's not…the NAAs don't want to (and shouldn't have to) do the FSTDOs’ job for them anymore like they did in the past. I recently heard stories of some FSTDO’s initial evaluations being delayed or even cancelled because the FSTDO (as applicant for the initial qualification) could not provide evidence that they had appropriately or adequately reviewed with suitably experienced resources, and approved the TDMs QTG submission in advance and in accordance with their compliance procedures.
Q - Sim Ops
You introduced the concept of Performance Based Oversight (PBO) with a greater emphasis on the operator rather than the devices, how would this work in practice?
A - Mark Dransfield
To put it simply… Assess the operators’ compliance capability rather than the devices themselves.
FSTDs today tend to be produced to strictly defined configurations both in terms of hardware and software (see our blog about TDM software product loads, as that's another discussion) which means that they are becoming more production-line oriented. Once the product is established and initially qualified by an NAA then the real challenge is how to prove it remains serviceable and compliant.
The point I was trying to make is that the NAA really adds very little value to the compliance process when it conducts a recurrent evaluation of such an FSTD at an operator that has established an effective compliance monitoring process that has been recognised by the NAA as such. Typically the NAA performs a very brief review of the FSTD matrics for the last 12 months, runs a few QTG tests and conducts a flyout. Part of the operator’s compliance monitoring is to ensure that these relevant activities have already been successfully carried out with regard to running the quarterly QTG tests (and reviewing and approving them!) as well as conducting the quarterly functional and subjective flyouts by the operator’s assigned standards pilots.
The idea was to conduct a more detailed CMS audit of the operator’s compliance and capabilities annually rather than a series of recurrent evaluations on each FSTD stand alone.
Q - Sim Ops
In your presentation you concentrated on an unnamed large training centre with multiple devices running the same software load, how would PBO extend to centres with multiple devices that are all different?
A - Mark Dransfield
It's still possible but probably more difficult as the FSTDO’s CMS and maintenance procedures regarding operational configuration control will naturally be TDM-specific depending on the variety and types of devices they might have….but the principle remains the same. Having said that not every operator will warrant the PBO-based approach in my opinion…the smaller single FNPT device organisation, with only a few individuals wearing multiple hats, would actually get more value from the NAA performing a traditional recurrent evaluation annually.
Q - Sim Ops
You summed up the presentation with a call for change, and judging by the audience reaction (including some of the regulators) there appears to be agreement that change is needed, how would you see this coming about?
A - Mark Dransfield
Let's look at what happened during the Covid lockdown times… the NAAs couldn't travel to many FSTDO’s sites to conduct recurrent evaluations due to travel restrictions…. so what did we do instead… the NAAs conducted remote evaluations by reviewing all the information collected during the year as part of the operators’ independent compliance monitoring process (QTG reruns, flyout reports, metrics, internal audit findings, discrepancies etc). This is effectively Performance-Based Oversight of the FSTDO, albeit done that way at the time because there was no alternative. My point is that we should be moving more towards this as the norm and only doing traditional recurrent evaluations as the exception to the rule when the FSTDO or the FSTD performance deems it necessary to have a separate NAA recurrent evaluation.
I would expect we may see a more detailed FSTDO Compliance Management System audit conducted on a yearly basis by an NAA rather than a series of recurrent evaluations for each FSTD. As more FSTDs fall under EEP-type arrangements then we will see a move in this direction…but it is not to be viewed lightly…. if an NAA finds major findings during such a PBO-type audit then the operator could be looking at restrictions or limitations affecting some or all of its current FSTD qualification certificates.
It would be great perhaps if the industry operators, through a vehicle such as the FSEMC, were to develop some more guidance and share best practices on this subject, perhaps as an update to the existing document ARINC Report 433 Standard Measurements for Flight Simulation Quality”.
How can Sim Ops Help?
At Sim Ops our partners have helped both airlines and start-up training centres go through the process of writing their CMS processes and implementing them. We can even provide experienced and industry-recognised resources to help train your people in QTG assessment, quality flyout profiles etc Contact us for more information on how we can be of help.