In the past we have published blogs dealing with accepting your Flight Simulation Training Device (FSTD) from the Training Device Manufacturer (TDM). But acceptance with the TDM is just the precursor to having your device qualified and Ready for Training (RFT). So this month we thought we would take a look at the issues around the qualification of devices.
Qualification Vs Certification
Before going any further it is important to ensure we use the correct terminology (some inspectors get a bit sniffy about this). Whereas for an aircraft the type is certified and each subsequent production example is verified against that standard, in the FSTD world each FSTD has its own qualification. That said FSTD type certification is something the TDMs would dearly love to achieve, and is on the agenda in some EASA rulemaking activities, but that is another story.
Initial Qualification
A lot of the preparatory work for the initial qualification of a new device will, by necessity, be carried out by the TDM. It is TDM who will need to produce the initial Qualification Test Guide (QTG) as required by the regulation. What most operators completely fail to grasp is that although the TDM prepares the QTG it is their responsibility to ensure they fully review, accept and approve the QTG as a fundamental part of the FSTD acceptance process. In fact we know of recent examples where the initial QTG has been submitted to the authority (usually 30 days prior to the date proposed for the evaluation) with little or no evidence of FSTD operator review and acceptance by appropriately experienced or QTG competent personnel…the QTGs were swiftly returned and initial evaluations delayed as a result!!
It is thus important that as an operator you realise it is your qualification. A qualification that you will have applied to your authority for and that they are not just evaluating the FSTD but also that you, as the operator applying for qualification, have done a good job evaluating and accepting the device with the TDM and furthermore that you have the knowledge and resources to operate and maintain it….you are, after all, declaring to the authority that you believe the device meets the required standards and is suitable for the training credits you desire!; When the authority does arrive they will therefore expect the operator to take the lead, there is often a tendency for the TDM’s engineers to take over this activity but the authority will expect the operator to do so, indeed some of us at Sim Ops have been asked to leave meetings during qualifications whilst working for TDMs.
The in-brief
You should have a comprehensive presentation ready for when the regulatory authority arrives that covers the device, the status of the programme and your organisation (In the EASA regulatory world there is useful guidance in Part ORA, GM3 to ORA.FSTD.100). Importantly this in-brief should identify all/any areas where you don’t fully meet the standard you are asking for and/or any unserviceabilities/outstanding discrepancies affecting the device and, most importantly, your plan to correct them. This will normally be presented by your quality manager with the accountable manager and head of training present.
The evaluation team
The authorities will send at least two inspectors to your site, a technical inspector and an assessment pilot or training inspector who should be, but isn’t always, type rated or at least familiar with the class or type of aircraft that the FSTD represents and the training that it will be used for. If it is the first FSTD at your facility you will probably also require a separate regulatory visit to assess your quality system with regard to the simulator maintenance and operation. Have a private room available for them to use throughout the evaluation and remember that the authority usually only needs to deal with the key representatives of the operator and perhaps the TDM, everyone else should be behind the scenes at least until the final debrief. The days of walking into a inbrief and finding 20 plus people in there from the operator, user and TDM should be long gone by now.
The technical inspector is focused on the objective evaluation and should have already had a chance to review the draft QTG and submitted queries that they would normally expect to be resolved during the first day of the initial evaluation. After a quick review of the physical device and its installation the technical inspector will probably install themselves in their room to go through the final QTG. After this you will probably have a number of questions to answer on the document, some of which may need the TDM to answer. It is likely that you will be asked to run a number of tests during evaluation both automatically and manually, some requiring the use of special tools and test equipment for things like visual and sound system testing. Make sure your staff are fully able to do this rather than rely on the TDM (in a previous blog we emphasized the importance of ensuring the QTG was produced using the FSTD delivered tools and your staff). All that said, particularly in Europe, there is a change of emphasis. We have been in meetings where EASA in particular have stressed that it is not their job to evaluate a QTG to ensure it complies with the regulations, that is the operators job. So if the TDM is presenting QTG results to you that you as the operator cannot justify you will have problems, the authority is not there to adjudicate. They are just checking for compliance.
The end of the evaluation should result in the authority approving the QTG as the Master document or MQTG that essentially establishes the objective testing baseline for the future recurrent evaluations.
The assessment pilot will be primarily interested in the subjective evaluation; they may arrive with a pre-prepared flight profile/set of tests they want to fly, but not always. Be prepared yourselves with a suggested profile, the one you intend to use for your regular fly-outs is ideal. The inspector will normally occupy the left hand seat and the operator should have either the head of training or the SME pilot who took the lead in the acceptance for the right hand seat armed with all the usual documentation such as FCOMs, QRH, charts etc. It is essential that someone very familiar with the IOS occupies the instructor’s seat if not the technical inspector. The authority will also usually allow the TDM’s representative to occupy the observers seat and the technical inspector will no doubt come in for some sessions.
During the evaluation, as points are raised and during the daily debrief, it is perfectly acceptable to “clarify” and discuss the findings from each day to ensure there is no ambiguity. But beware…this is not the time to get involved in a heated debate with the authority on any specific issue even if you initially disagree….better to take time out and research or review the finding and discuss again with the inspectors at the next day’s inbrief.
.
At the end of the evaluation, typically three days, the inspectors will retire to their room to prepare a draft report based on the agreed findings which will then be presented to you. Make sure you ask questions to fully understand what is being asked of you on any outstanding items (particularly any unserviceability, limitations, reservations or unacceptable items).
Recurrent Qualification
A recurrent evaluation follows pretty much how an initial is conducted except it's usually completed in a single day. Of course the tone of recurrent inspections can be heavily influenced by your follow-up on issues arising from previous inspections, if you had actions that were not followed up on don’t be surprised if you get a thorough examination.
If you have added new capabilities or features during the year (which of course you will have notified the authority of at the time) they will naturally concentrate on those.
Red flags! Discrepancies that have been outstanding for an extended period and repeated/frequent re-mastering of QTGs being obvious ones. We have seen some sound QTGs that have been re-mastered on an annual basis!
General Considerations
At the end of the day the authority will be looking to ensure that the FSTD is fit to carry out the training for which it is intended; if the staff presenting the device are negative toward the device don’t expect a positive response from the authority. If you are not sure the FSTD is ready for qualification it probably isn’t.
In our experience the authorities are not trying to catch you out, their motives are to help you, so don’t hide things from them. Any evaluation is a team effort between the authority and the operator and must always be completely open and transparent. The chances are any issues you fail to declare or hide will be seen and it will erode trust.
One slightly dubious practice we have seen on some initial evaluations (and a recurrent during warranty) is for one of the operator’s instructors to “sign post” an issue to get an authority write up with the aim of getting the TDM to work on it, even if previously rejected. All we can say is this will end in tears for someone.
One final piece of advice. Don’t do last minute loads. We have seen on numerous occasions a new load implemented just prior to an evaluation to fix some minor bugs without proper, thorough, testing. Don’t. Far better to explain the issue to the authority and that you have an untested fix in development. If time allows, install the development load at the end of the evaluation and just test the particular issue in front of the inspectors. Authorities expect the operator to have strict configuration control….remember what is being evaluated during the initial qualification is the load to be used for training.
Summary
Remember that the authority is really in fact auditing that, as the operator, you have met your regulatory obligations to ensure the device has been properly tested and accepted by you as meeting the standard and is fit for training.
How can Sim Ops help?
Amongst our partners at Sim Ops we have extensive experience of qualifications as a regulatory authority approved FSTD technical and training inspectors, operator and TDM’s representative; we are well placed to guide you through the process, provide expert QTG reviews and analysis to EASA/FAA standards and act as an independent advisor on all matters related to FSTD qualification processes.