Assessing the magnitude and significance of cyber threats has at least two important purposes. One is to determine the extent of measures that have been or should be taken to respond to or counter the threat. This is part of the rational deployment of resources across the multiple risks that we face, whether cyber or otherwise. In this regard it is simply not possible or necessary to respond to all risks with equal vigor. A second purpose can be to communicate threat significance to or among interested parties. For such communication there is a tendency to reduce complex, multifaceted issues to a simple broad summary word, e.g., the threat level is "Guarded". Such simplicity is possibly attractive but not necessarily meaningful with regard to what to do with the information communicated.

Of interest here is how those issuing threat assessments are making their determinations. One approach is quasi mathematical in which components of a threat are "scored", and then those scores combined in some way, with the net score then indicating something of supposed significance. Those of us who have used methods such as those in ISO14971 are familiar with scoring severity and probability, and then presenting these in a two dimensional grid which is then divided into some number of risk zones (often 3). A "risk score" is also often determined by multiplying the severity score by the probability score to get the risk score, although there is no theoretical basis for such a multiplication. None-the-less, with a multiplication scheme additional factors can also be considered, scored and also multiplied. The best known of these additional factors is detectability, especially when risk assessment is applied to manufacturing and inspection is meant to ideally find and eliminate bad product from the production stream, thereby reducing the risk of a manufacturing defect reaching the end user. My colleague and I have discussed the limitations of these kinds of fake mathematical calculations here and here. Limitations do not mean that a method should not be used. Instead it means that it should be used knowingly and with caution.

In the cybersecurity space a different kind of math has emerged, as used for example by the National Health Information Sharing and Analysis Center (NH-ISAC). Here four factors are combined by addition and subtraction using the relationship Severity= (Criticality + Lethality) – (System Countermeasures + Network Countermeasures). Each of these four factors has a 1-5 scale with word descriptions of each, e.g., a criticality of 3 is "Less critical application servers" while a lethality of 3 is "No known exploit exists; Attacker could gain root or administrator privileges; Attacker could commit degradation of service." As far as I have been able to determine there is no theoretical basis for or validation of linearly adding and subtracting the four individual scores, nor is there a basis for each factor having the same 1-5 scale. In addition the use of the same scale for each factor introduces a false symmetry, e.g., a Criticality of 4 with a Lethality of 2 has the same effect as a Criticality of 2 with a Lethality of 4. There is also a false relativity with respect to, for example, 4 being logically perceived as twice as bad as 2.

The conversion of the calculated number into a particular threat level also appears to be more-or-less arbitrary. In NH-ISAC there are 5 threat levels: Low, Guarded, Elevated, High, and Severe (with corresponding color codes Green, Blue, Yellow, Orange, Red). The score range for these 5 threat levels is distributed with the intervals being 3, 2, 3, 2 and 2, which also seems arbitrary. This scheme also results in many different combinations of the four factors leading to the same overall rating. In addition the end result has high sensitivity to the inherent uncertainty in the individual scores. Furthermore there is an inherent limitation in knowing just the end result, e.g., Guarded, which is that it is not possible to work backwards to find out what contributed to that level. For example in a particular instance was it high criticality or low protection? Also, it cannot be determined how much taking an action that moves an individual score up or down would help unless the raw score is reported along with the level. How individual Severity scores are combined to produce a global Severity score is not described. In fact as used by NH-ISAC, it is not even possible to publicly determine what actual threat or threats they have considered in arriving at their overall threat level. If you don't even know the particular threat it is not possible to respond, even if a response was appropriate. Given that this methodology has no available documentation we also don't know anything about inter-rater variability, i.e., to what degree would two independent raters arrive at the same scoring.

A different group with an online presence in threat rating at first offered me the following explanation of their methodology (from an email): "The determination is somewhat more subjective, how the rubric is calculated has not been published to my knowledge. We essentially get together online and have a discussion about the current issue." This methodology clearly is not transparent. Subsequent to my inquiry they added to their website a linear combination scoring system that has at least some of the same limitations as the one discussed above.

Another characteristic of third party threat assessment is that those doing the assessing are not those who have to respond to the threat. Nor do they have to rank the cybersecurity threat in comparison to other threats to patient and system safety that have nothing to do cybersecurity. Yes, such threats do still exist. This can lead to a Chicken Little mentality in which those enmeshed in their own arena of doom come to believe that their perceived doom is the only doom, or at least a much worse doom than other people's doom. Moreover it leads to the self aggrandizement of their own domain, and attempted self propagation of their importance. In this regard the announcement of the  recent collaboration between two cybersecurity threat rankers included the assertion that, "This landmark partnership will enhance health sector cyber security by leveraging the strengths of..." the participants and "...will facilitate improved  situational awareness information sharing." all before they had done anything. How grand!

Health related cybersecurity is no doubt an issue that has to be treated seriously, but this must be done in the context of all the other issues that challenge the healthcare enterprise and the safety of patients. In this context we should not confuse potential vulnerabilities with actionable risk, nor confuse any risk with a critical risk. In this regard it is important to remember that safety is the absence of unreasonable risk, not the absence of risk.

William Hyman is Professor Emeritus of the Department of Biomedical Engineering at Texas A&M University. He now lives New York City where he continues his professional activities.