The FDA has released a January 2019 update to its working model of the precertification program (Pre-Cert) for software that is a medical device (SaMD). As we have previously addressed (here, here, and here) the idea of the Pre-Cert approach is that a reasonable level of safety can be assured by looking at the developer’s quality system for product development rather than by looking primarily at the product itself.  This “look” involves determining a level of excellence for the developer and matching this level to the risk level of the software. Where appropriate, a streamlined review process would then be triggered.

The update notes that for 2019, excellence level will be determined directly by the FDA by conducting excellence appraisals of companies (or subunits) participating in the Pre-Cert pilot program. FDA says it will determine what information about the organization, and the organization’s software products, should be part of a company’s application for an appraisal.  Fifteen elements and 5 excellence principles of such an appraisal are given in an Appendix to the update.

The FDA says it also intends to collect real-world information on the effectiveness and ease of use of its appraisal methodology. Real world product information is also an important part of determining the effectiveness of Pre-Cert (see below). One might note that real-world information is generally preferred over make-believe information, or at least it used to be. Two levels of excellence, cleverly named Level 1 and 2, will continue to be a key element of the model. Level 2 is the better of these, which may or may not be obvious. The FDA has never been consistent in rank ordering things noting that a Class III device requires the most pre-market scrutiny while a Class III recall is the least dangerous. An interesting contest would be to name the excellence levels, remembering that even Level 2 extends only to products that have no more than moderate risk. It remains an intriguing question whether companies will want to be known as only Level 1 excellent, and whether this information will be publicly available.

Since the FDA will be doing the appraisals, they have also proposed a test methodology with the following questions:

• Does it provide assurance that the organization's processes and activities are persistent, and does the evaluation work across varying organizational structures.

• Does it determine the clinical responsibility of varying organizational structures, and across varying track records of software products.

• Does it appropriately demonstrate organizational performance against excellence principles to assure safe and effective SaMD.

• Does it identify the metrics or indicators used by the organization to monitor and sustain their processes and organizational effectiveness.

• What are the time and resources needed to complete an appraisal for varying organizational structures.

Of the above list I think the safety and effectiveness question is the most open ended. No matter what kind of appraisal is done the ultimate safety and efficacy question can only be answered, if it is to be answered, after the product is in the field. Reliance on such real-world evaluation leaves the inherent possibility that the product will be found to be dangerous after it is on the market. When this is the case it means that the FDA failed in its core mission of keeping dangerous products out of the market place. It also means that people who were victimized by the product were actually subjects of a clinical trial, but unaware that this was the case.

The ability to collect real-world performance data will be part of the FDA’s appraisal.  In 2019 Pre-Cert participants would provide FDA with access to its real-world data on a periodic basis (e.g., quarterly). The FDA will use this information to refine the data framework and provide additional clarity to industry on attributes that have high concordance with ongoing excellence. Also tested will be sensitivity in detecting postmarket signals, and these align with signals and traditional postmarket reporting. Remember here that Pre-Cert extends only as far as moderate risk so that post-market signals are likely to be weak. FDA expects that this testing will enable expanded uses of real-world data in the future.

As the Pre-Cert pilot program continues to evolve key questions remain, which may or may not ever be actually answered. One is whether Pre-Cert provides an equivalent level (or perhaps a better level) of assurance to the public that SaMD that reaches them has a reasonable expectation of being safe and effective, compared to traditional regulatory oversight. Another is whether a one product start-up will be better served by becoming certified as to its level of excellence, or should they take the otherwise applicable route to market.