Valuing Private Certification

There are currently several private entities that seek to certify medical apps, connectivity solutions, EHR record exchange, and other products, services and people in our sphere of interest. Given the ongoing proliferation of private certifications, there is a growing need to evaluate them, judge their relative costs and benefits, and determine which – if any –  are worth adopting as either the one certified or as the consumer of certified products or services.

These private activities are usually distinct from governmental requirements (e.g. FDA or FTC  compliance, or state licensing), although in the case of EHR Meaningful Use (MU) certification, private entities function on behalf of the federal government to certify compliant EHRs.  Note here that compliant EHRs are those that are capable of achieving MU. Purchasing a product that is thus certified is a prerequisite for a provider then achieving MU.

Government/private interaction is also relevant to some Class II FDA device regulation which are eligible under third party review. Internationally, private  notified bodies are part of the CE process necessary to meet EU requirements. Private efforts can also become public when an authority having jurisdiction (AHJ) adopts or cites compliance with a non-governmental standard or certification as required by law or regulation. Private certifications can also have secondary private impacts such as when a third party requires compliance with a certification or standard. For example a seller of a product might require that it be certified, or an insurance company might be interested in whether or not their client is producing or using certified systems. Similarly, a hospital accreditation organization might cite standards of the National Fire Protection Association (NFPA), although the NFPA does not itself provide certification of compliance.

Private certification efforts share a number of common questions. The first is who is the certifying entity, and what is their stated purpose?  This might include some general public value, e.g. assured capability, safety, or some enhancement of commerce resulting from interoperability. However it might be necessary to look beyond the stated purpose to other purposes such as deriving fee income or limiting access to a market or profession. There may also be questions about related products or services associated with the certification process. For example, are consulting services or training available from the same entities that created the requirements? Or does a group of  certification supporters make products or have other interests that are met by their self-created requirements, while creating a barrier to others?

The second broad question is what are the criteria for certification, i.e what measurable and subjective criteria must be met in order for an applicant's product or a person to become certified? Typically there will be specified technical attributes and associated test methods. The certifying entity might conduct such tests, or have the applicant submit or attest to test results. In addition to more-or-less easily measurable features, there may also be softer criteria such as human factors (usability) considerations, or oral interviews for individual certification, for which truly objective tests might not be available. It is also good if there is feedback from testing to criteria development. If everyone passes the test or answers a question correctly, the assessment may be too easy. If few  pass, the test may be too difficult, and/or irrelevant. In this regard an interesting process in careful multiple choice written testing is the post-test identification of questions for which there were a large number of wrong answers. This allows the wrongness to be re-assessed, or ambiguity of the question to be resolved.

A related question is how broad is the scope of the requirements, i.e. do they broadly or narrowly define the necessary attributes of the product/person? Overly narrow requirements can result in a system/person passing a test when that test is not by itself sufficiently indicative of actual real world  performance requirements. For example a connectivity solution may be too limited in its performance scope to assure easy and reliable  usefulness. It might for example measure A to B connectivity but not A to C, or it might include data string definition but not patient identification. A related issue is how open was the requirements development process, and were the requirements written to benefit only one segment of the industry. Of course public disclosure of the requirements is also necessary if compliance with them is to have any external meaning. Here clarity is also a factor. When the developers of a standard offer consulting services that are necessary to explain it, that might be cause for suspicion.

Once the requirements are known the next question is how  rigorous, independent and fair is the testing? The answer here may be clearer for objective tests than for subjective ones. More to the point is the issue of whether a product passing the test is the result of a fair and unbiased assessment, or do the people doing the assessment have a self interest in the outcome. Fully public testing is attractive in this regard. In other cases testing results and other assessments have, at a minimum, to be fully available to the applicant. There would be heightened transparency if these results were also made publicly available, at least for systems that were deemed to have passed.

There are two ultimate questions related to whether the value of the certification is itself valuable. One is whether or not customers and other interested parties care whether or not certification has been obtained. This is the case for both product and personal skill certification. For the latter, other than self-satisfaction, there is the question of whether anyone else (e.g. employers) cares if one is certified. Thus it could easily be the case that there is a product or skill certification that is fair and meaningful but for which there is no consumer demand that the products being bought, or people being hired, have that certification. Lack of demand can in turn diminish the desire to be certified, and reduce compliance even when such compliance would actually be desirable.

The second value issue is that it could be the case that customers would seek out a certification that is not in  fact meaningful, presumably based on some level of promotion that has convinced them that it is meaningful. Sometimes this promotion comes from the same entities that have created the certification in the first place. Or, even if not actually meaningful, the consumer might value a certification if they can gain some advantage from being able to claim that they use certified products or people. This would be a secondary marketing consideration aimed at the customers/clients of the original certified entity.

As private certifications multiply and seek their foothold in the doors of e-health commerce, there is reason to remember the old droll adage: Standards are wonderful, that is why there are so many of them. This could be amended here to state: Certification is wonderful, that is why so many organizations want to provide it.