Over the past few years, medical device networking has become increasingly problematic. There is a growing perception of declining service levels and network reliability. (I've not seen or heard about any quantitative data to back this up - let me know if you've got any - but these are substantive issues.) As medical devices, by way of connectivity and workflow automation, have been pulled into the sphere of the IT department risks to patient safety have increased.
These growing problems are a key factor in the creation of IEC 8001 - which will have a major impact on how hospital's deploy and manage networked medical devices. The current situation provides a lot of the impetus behind a meeting this week at Villanova on wireless technology in health care. So whttp://medicalconnectivity.com/2008/06/16/iec-80001-to-impact-providers/hat's the big deal?
Private Medical Device Networks
In the good old days, all medical device systems operated on private networks that were designed, installed and supported by the medical device vendor. Vendors decided exactly how the would handle IP addressing (static or DHCP) and every other configuration option. This minimized vendor's product development costs, shortened time to market, and made service and support easy (because the network was their sandbox). When they got the inevitable calls from customers that, "your system just brought down our network," vendors could quickly and easily determine if that was the case - and just as importantly, quickly prove to hospital network administrators why their answer was correct.
Providers benefited from this approach in a number of ways. First, vendor system support was quick, efficient and reasonably priced. This approach also relieved hospitals of the need to assume responsibility for supporting life critical applications themselves.
Of course nothing's perfect, and relying on private networks created a few problems of its own. As more medical devices evolved to become components in medical device systems connected by networks, these private networks proliferated. The IT department of a thousand bed hospital system on the east coast did a survey a couple years ago of their networked medical device systems, and "discovered" over 200 private networks. And unlike enterprise networks that must be kept up to date to ensure cross vendor interoperability and coexistence, private medical device networks are like time capsules. With the exception of occasional failures of a switch or router, medical device networks remain unchanged from the day they're installed; the hospital mentioned earlier found numerous ThickNet and ThinNet Ethernet networks in their survey of medical device systems.
Over time, medical device vendors have given some ground by moving from physically separate private networks, to creating logically separate private networks through the use of network switches and routers.
The Transition to Enterprise Networks
Changing requirements in hospitals have reduced the viability of private medical device networks over the past several years. Due to changes in technology and care delivery processes, various medical devices have broken out of their traditional use models and are now deployed house-wide. Smart infusion pumps, patient monitors, point of care diagnostic testing (POCT) devices, and ventilators can be found almost anywhere in the hospital. (Admittedly, the POCT market lags in networking technology, and ventilator vendors have successfully held connectivity at bay - so far.) The reality here is that it is impractical to deploy multiple private networks house-wide to support each of these and future medical device systems.
Hospital EMR adoption is also driving the transition to enterprise networks. Automating medical device data acquisition for charting has two key requirements. First, one must have access to the data. Data access is frequently done through gateway devices that provide the EMR with HL7 data while retaining much of the walled garden of the medical device vendor's private network. The second requirement is that all data going into the EMR sport a consistent time stamp. Communicating with hospital's network time servers has further widened the crack in device vendor's private networks.
For many good reasons, the wide spread duplication and proliferation of private networks just rubs IT folks the wrong way. All that variability and complexity, not to mention duplication of network gear (all at vintages reaching back 10 years or more), is increasingly looked at as unnecessary and undesirable. Private networks are fading, and are not expected to make a come back.
What About Wireless?
In many ways, wireless networking has closely followed the evolution of wired networks. The earliest wireless medical devices (telemetry systems) used proprietary RF communications. Just like private data networks, medical device vendors designed, installed and supported these systems. Even early Wi-Fi based systems fit this model. The majority of these early Wi-Fi systems used 802.11FH rather than the 802.11b technology used by hospital IT departments. This difference in underlying technology effectively rendered them private networks.
Recently, wireless has moved in two direction. The old analog VHF has transitioned to the new WMTS bands, and 802.11FH has been replaced by 802.11b/g and 802.11a/b/g. While WMTS is still a private network, Wi-Fi has been forced on the the enterprise Wi-Fi infrastructure in most hospitals.
The transitions from private to enterprise networks has not been without its problems.
Culture Shift for Medical Device Vendors
Medical devices are what's known as embedded systems, purpose built black boxes that are completely controlled by the manufacturer. Medical device vendors like to keep complete control of what goes into their systems, and rightfully so. A key design principle for medical devices is to keep them as simple as possible in order to eliminate unnecessary variables. The more complex a system is, the longer it takes and the more it costs to develop. Variability and complexity become major factors in the verification test of medical device systems. And verification test impacts both initial product development and sustaining engineering (where bugs are fixed, product components replaced, and software updated). Consequently, it is ingrained in medical device vendors to minimize variability and only implement those features that are absolutely required by the product.
The purpose built mentality of vendors works great for embedded systems, the actual medical device. But when this mind set is extended to application software, personal computers, servers and networks, problems arise. The general purpose computing world in which hospital IT exists is full of options and is highly configurable; they revel in variability.
Private networks enabled medical device vendors to select and implement only those options and configurations they wanted to use in their medical device system. As already noted, this reduced development costs, shortened time to market, and facilitated support - mainly due to keeping a lid on variables. Obviously, this approach doesn't work well in an enterprise IT environment. It is the enterprise IT environment in which medical device vendors (many thinking fondly of the days of private networks) are struggling. Device vendors are not alone in their feeling the pain of medical device systems moving to enterprise networks.
Culture Shift for Hospital IT
Hospital IT departments have their own set of proclivities. The IT world is heterogeneous, with a large variety of vendors offering different applications, computing platforms, and networking technologies. The IT industry depends on standards developed by SDOs (standards development organizations) and test and certification provided by consortia (like the Wi-Fi Alliance) to make this heterogeneous environment possible. Hospital IT folks tend to take all this heterogeneity for granted, demanding that medical device vendors conform to whatever set of standards and configuration options they've chosen to adopt. Hospitals can be downright vociferous in their insistence that vendors - medical device and IT - conform to their requirements, rather than the other way around.
This intransigence can cause hospitals to reject superior clinical solutions because a medical device system does not support the specific standards and configurations adopted by the hospital. Hospitals also suffer reduced levels of service from medical device vendors as easy to diagnose and support private networks are replaced by enterprise networks. This results in higher costs for individual hospitals, as medical device vendors have to cover the increased costs of developing and supporting a full range of IT options.
Regulatory Factors
Medical device systems, made up of the embedded system, application software, computers and network, are regulated by the FDA. Vendors must follow the Quality System regulation when designing those products, follow Good Manufacturing Practices when building them, and conform to regulations for seemingly everything else they do. This means that the software, computers and network are part of the regulated medical device.
Regulatory approval is granted based on the labeling or intended use of the system as stated by the manufacturer. Only the holder of that regulatory approval can make any promotional claims about the medical device system, in whole or part. The intended use, claims, product design and verification test are all interrelated - change one and you likely impact the others. This becomes an issue when medical device vendors claim network capabilities that are by necessity supported by specifications and related verification testing. When hospitals adopt a medical device system and deploy it on a network that differs from the device's network specifications, the hospital is using the medical device "off label."Off label use by providers is not precluded by FDA regulations, or any other regulations that I'm currently aware of.
The FDA can only regulate manufacturers. Providers who use regulated devices off label don't need to worry about the FDA. That's not to say they shouldn't worry at all - plaintiff's' attorney's come to mind, along with possibly state regulators, accreditation bodies, and other parties interested in patient safety. The FDA has set the standard for quality systems intended to ensure the safety of medical devices. Should providers wish to use devices off label, the prudent organizations provide appropriate test and management oversight to bear so that their off label use does not impair patient safety.
Make no mistake, enterprise networking issues have killed patients by impacting the operation of medical device systems. Of course biomeds and clinical engineers are used to working within rigorous environments designed to ensure patient safety. This is sometimes a sobering realization for IT folks who may not even realize how their actions impact medical device systems already installed in their hospitals.
The bottom line here is that we can't all look to the FDA to solve these issues that are the consequence of putting medical device systems on enterprise networks - when you do this, your enterprise network becomes part of a medical device.
Let's look below the surface of the key issues arising from this transition from private to enterprise networks.
Provider Network Variability
By design, the general purpose network environment is very flexible and highly configurable. The hospital network design and configuration choices are highly variable - there's probably a hospital somewhere that's exercised every configuration choice available. This variability imposes significant costs on the industry.
As noted above, building in a full breadth of general purpose network configurability is a considerable expense for device vendors. This same flexibilty also adds considerably to verification test and sustaining engineering costs. Rather than creating a private network that remains mostly unchanged for 5, 10 years or longer while installed in the field, enterprise networking requires that certain changes be made to keep up with the general IT networking market. This sustaining engineering cost can easily dwarf the initial cost of developing the medical device system itself.
Proprietary network features are another big issue. In general buyers, hospitals and medical device vendors both, prefer using industry standards. Sticking with standards provides additional flexibility and can greatly reduce the cost of developing and supporting networked medical devices. When providers select an infrastructure vendor (for Ethernet, wireless or both), they typically go with one vendor. By "standardizing" on one vendor, the provider frequently looks more favorably at proprietary features from their vendor. This makes sense because they've only got that one vendor's products to content with, and thus don't see any duplication of effort or additional costs between a feature based on a standard and one that proprietary to their "standard" vendor. This is not the case for medical device vendors. Implementing a proprietary feature for authentication, encryption, quality of service, roaming or other capability means significantly more development cost and a big hit to sustaining engineering costs over the life cycle of their product. Not surprisingly network vendors are constantly looking to add value and differentiate their solution in a relatively mature and standards driven market (rather like medical device vendors themselves, come to think of it). Providers who want medical device vendors to adopt proprietary infrastructure features, even for a vendor who might have say 70 to 80 percent market share, is a losing proposition. They won't do it, and rightly so.
The industry also incurs costs in the area of service and support. Providers pay more for a lower level of service because of the new variables the device vendor must grapple with in diagnosing and fixing problems. Overall patient safety can suffer as well.
The days of private medical device networks as we know them are over. Yet a full blown general networking approach to medical device networks is less than satisfying. Both the medical device and IT realms must reach some accommodations with each other if we want to realize the patient safety levels of the recent past, at the same or lower costs.
Medical Device System Coexistence
Most medical device systems struggle with coexistence on an enterprise network with other medical device systems. There are two main reasons for this, certain applications impose stiff performance requirements on the network and many medical device systems aren't designed to be sufficiently well behaved in an enterprise network environment.
Because medical device life cycles are so long (typically 7 years minimum), most current networked medical device systems include software that was designed to operate on a private network. These systems lack the built in intelligence to manage network utilization, lack network management and performance monitoring capabilities, provide sometimes severely limited configurability, and can exhibit antisocial network behavior - like spewing gobs of multicast traffic, or over consuming other network resources. Some systems have considerable technical network performance requirements due to a combination of being demanding life critical applications and designs that may be inadequate for enterprise network environments. A common symptom of network coexistence limitations are vendor requirements for dedicated SSIDs and/or VLANs. Many of the problems will fade over time, as new products and versions of existing products come to market.
Wireless Coexistence
Coexistence for wireless networks is similar to that for Ethernet, although there are differences. Besides the actual network configuration and protocols (like you have with Ethernet), Wi-Fi is further complicated by things like RF interference. The ISM band, used by most wireless medical devices, is supported by standards that are intended to ensure coexistence. Mixing ISM based technologies (like Wi-Fi, Bluetooth and ZigBee), does complicate things, but is possible.
The alternative to the ISM band for medical devices is WTMS. (There is MICS and a proposed medical body area network, but are so specialized are rarely used that they will not be discussed here.) There are no standards specified for WMTS whatsoever. The FCC simply designates specific frequency bands and rules for registering your use of the band. Consequently there are no standards or formal mechanism to ensure coexistence between vendor's solutions. In fact, early in the existence of WMTS, the two predominate vendors using the band, GE and Philips, had considerable coexistence problems precluding, for a time, the installation of systems from both vendors being used in the same hospital. While a standard to support wireless medical devices in WMTS might be worthwhile, the relatively narrow bandwidth and economy of scale issues argue against such an effort.
Infrastructure Vendor Support
Because of the maturity of Ethernet standards, and the fact that networking engineers wrote the standards, most medical device systems can run on any Ethernet vendor's infrastructure.
When medical device vendors develop a new wireless product, most go through a similar process. First, the vendor selects a wireless LAN infrastructure to use when designing and testing their new product. The choice of wireless LAN or infrastructure vendor can be based on market requirements, like infrastructure vendor market share, or performance, especially when high performance life critical applications are involved.
Regardless of why an infrastructure vendor was chosen, when that new product is released it may specify that infrastructure vendor's network. The degree of specification can range from the general (a specific manufacturer and certain standards) to detailing down to product models and firmware software releases.
The greater specificity in medical device vendor's network specifications, the more narrow the choices for both device vendor and provider. Say vendor X build their new patient monitoring system on Cisco infrastructure. When their sales rep calls on Barnes Jewish in St Louis, they have a problem: Barnes has installed the Wi-Fi network from Meru. This leaves the vendor with 3 options: 1) walk away from the business, 2) try to convince the provider to install a second Wi-Fi network from Cisco (good luck!), and 3) spend $250,000 to $500,000 to repeat the verification test they did with Cisco using Meru equipment. And the quarter to half a million dollar verification cost estimate assumes they don't need to change their product to support the new network. If the vendor needs to say tweak their embedded radio device driver, the cost can easily go north of $1 million. Multiply that cost by 4 or 5 infrastructure vendors and the network support cost can easily exceed the cost to develop the actual medical device system. Of course all these costs are passed on to providers and again to payors and patients.
Sustaining Engineering
Assume we have a working wireless medical device system and it's widely installed in hospitals. Remember the product life cycle for a medical device is several years at least. Compare that to the life cycle of general purpose networking products, which can be as short as 18 months. Now imagine each medical device vendor having to assess their network vendor's product changes (down to firmware updates) and you can easily imagine a near full time verification test and sustaining engineering effort. This sustaining engineering burden falls on each wireless product individually or by product line, because most medical device vendors are organized in silos by market segments and tend to reinvent connectivity features across separate R&D teams.
This sustaining engineering effort is further complicated when the medical device system is deployed on an enterprise network managed by the provider. How are they supposed to know whether it's okay to install that new firmware upgrade on their access point controllers or not? Sadly, it's not unheard of for an enterprise to update their infrastructure and have the entire network fail when they turn everything back on. If that network is supporting life critical applications, patient lives are at risk.
What About Wireless, Again?
There are lots of networked medical devices in hospitals, especially in diagnostic imaging and the clinical lab. What makes the networking issues described above of urgent importance is networking at the point of care, where there's a great deal of mobility and wireless networking best meets user requirements. Because of this, much of the focus on this topic has been on wireless networking. In tackling these problems, we must remember that Wi-Fi represents the last several feet between the Ethernet network and the wireless client device. Any solutions for wireless medical devices must also consider the wired network that underlies the wireless connection.
Perfect Storm
The problems around wireless medical devices and networking have been growing for years. It was 2 years ago that the FDA assembled a workgroup to look at this problem (and that effort resulted in IEC80001). This year's FDA proposed draft of the Medical Device Data Systems rule tangentially addresses networking issues. Just last week the Joint Commission issued a Sentinel Event Alert on, "Safely implementing health information and converged technologies" (where they note, "Multiple networks can result in poor interoperability and increased costs.").
Tomorrow I'll be attending the "Villanova Workshop on Wireless Technology in Healthcare: What is needed for safe, secure, and reliable deployment?" I'll be blogging the meeting, as I know many folks involved in this area either weren't able to come, or could not be accommodated due to the limited space available.
Pictured up top is a VeriWave 90 test system chassis - a piece of diagnostic equipment that almost every hospital will one day have to keep their wireless medical devices healthy.
NOTE: This post was whipped out even more quickly than usual. You can expect minor revisions over the next few days (adding links to reference additional information, fixing typos, improving clarity). Revisions will be noted per usual as "UPDATE".
That’s an awesome piece of work, well done!
Great article, Tim, as usual. I do have one comment regarding the Hospital IT culture. From strategic viewpoint, if one has a better device or system clinically, I don’t believe IT should have a veto vote. The fact is the healthcare business is just that, not an information technology business. It should be a team approach and the clinical requirements should have a higher weight or priority in the decision to adopt or implement a clinical system.
Optimally, there are three aspects to a decision: clinical, technical and financial. A smart organization will drive the choices available to equality in meeting the clinical and technical requirements so that then any negotiations are based on price. If there is a conflict between the technical and clinical requirements, then I believe the clinical requirements should be of higher priority. Part of being engineers is learning to optimize performance based on a constrained set of variables and I believe our job is to provide those optimal solutions with the edge towards safe clinician use. I would frankly like my clinician to have the best set of tools possible before they use them on me.
Thanks for your comprehensive review of the medical device networking and I look forward to your blogging about it.
Tim, another excellant and valuable summary. I offer the following comments.
1. ECRI provides an good companion article “Coping with Convergence” October 2008 HealthDevices. It offers a sound 13 point plan.
2. My experience with hospital IT networks and departments suggests that IT networks are getting better and becomming more robust. I have reviewed a number of IT infrastructures that are designed to 99.99% availability or better, well beyond what any proprietary system can provide. These networks have plenty of capacity to segment medical applications if the networks are properly configured and verified at the time of installation. I have also noticed the level of technical expertise within hospital IT departments, large and small, to have dramatically improved over the past few years. AND, as Bridget suggests, these technical professionals are willing to engage medical device vendors in problem solving to the benefit of their clinical customers. An assessment (as the ECRI article suggests) of the network prior to implementation of a medical device is straightforward and a reasonable requirement to impose on the hospital IT and biomedical departments. Furthermore, most of the IT professionals I have encountered understand the importance of network configuration management and the consequences that network changes can have on system behavior. I know this is a broad generalization, but the trend is in the right direction.
3. There is a path through FDA clearence that addresses the use of medical devices on hospital IT networks. I do not share your assessment of the effort and cost necessary to qualify all variants of a network with the caveate that if a medical device system is designed from the start to share an IT infrastrucuture and based on IEEE standards, then appropriate design risks and mitigations considered at the time of design inputs can address anticipated limitations within the shared network including vendor and configuration variants as well as load considerations. Infrastructure variants are also addressed within the intent of ISO 80001 draft standard during the deployment stage. Verifications at the time of installations can ensure that the “system” functions as intended within the clearence secured by the FDA. This is not necessarily the case with systems migrating from proprietary networks to shared / open networks, since embedded design constraints may not allow some mitigation other than restrict the network itself (ex. a dedicated SSID). As all things in engineering, the devil is in the detail.
4. We are in the infant stages of a new generation of connected medical device systems. Just as token ring has dissappeared, this will likely be the case for proprietary networks, simply because the cost vs benefit of proprietary networks is not sustainable when new technologies prove otherwise, as Bob has suggested. Each generation of connectable medical devices will improve on it’s predecessor thus reducing or eliminating restrictions on connectivity, encryption, security,and QOS. As an example, WiFi modules are available today that are compliant with 802.11 a, b/g, e and i with all the common variants for security, encryption and QOS. These modules have only become available in the last few years. These subsystems may not conform to low power requirements or cost constraints today, but they are getting smaller, lower power, and lower cost with each new generation. Finally, there will always be the case for network segmentation to ensure QOS and latency management. These can be addressed with good vendor-neutral network designs.
5. I am not aware of any published data or reports of patient safety issues relating specifically to death or serious adverse events associated with the failure of an integrated medical device / IT network. I have heard the rumors, just like everybody else. BUT, as technologists, we should stick to the facts, not heresay. Perhaps the FDA or ECRI could elaborate on specific incidents relating to this topic.
Tim is always on the money.
Key to the wireless aspect of things is patient-data association in real-time, especially with little or no technology intervention by clinical staff (a real plus in terms of clean workflow). Associating medical devices with patients and their data is, perhaps, the most daunting challenge faced. The technology for pulling data is rather mature. But, when we’re talking about many devices with no common data models, standards or formats, and in which no common mechanism for associating these data with patients is available, then therein lies the essence of the problem.
Creative means (barcoding?, rfid?, zigbee? bluetooth?) for solving the association problem and then meeting the transport needs with a quality of service necessary to meet diagnostic, therapeutic and interventional standards can mandate that all aspects of the system must be managed and regulated. I think this is the key: we stop thinking about individual devices and think more about system integration of multiple components into a seamless web that lives and breathes according to standard communication, networking, and hardware protocols. A mighty challenge.
I see this as an operational problem for IT rather than a technical problem for the network infrastructure.
It reminds me of when VOIP was first deployed in large enterprises the IT departments were faced with similar challenges - voice is the most critical application - voice
needs to be on a dedicated network supported directly by the vendor, general IT network is not reliable enough, voice has networking needs a shared IT network can’t meet…..
Voice engineers had a long history of disciplined operational procedures and they brought those skills to IT for successful VOIP deployments, believe Biomed can do the same for medical device networks.
I feel it would be a failure of our industry if we ever got to the point of requiring regulation of the network.
Jim W, to your third point: I agree that many medical device manufacturers should have little trouble specifying and designing a network capability that minimizes the need for verification testing of their system with specific network infrastructures. A key goal of a good regulatory strategy is to balance verification demands against design specifications, marketing claims against risk analysis.
However, some applications — continuous patient monitoring for example — sometimes requires designs with a degree of technical specifications that necessitate detailed validation testing. This is especially true of systems that are fully or partially based on software that was written for private networks (a number of existing patient monitoring systems).
Jerry, you are spot on. Expecting network infrastructure manufacturers to “fix” a medical device manufacturer’s networking limitations is unrealistic — not to mention of questionable legality.
Health care providers who want to run medical device systems on enterprise networks must look to the medical device manufacturer for solutions. Until buyers make network support a key purchase criteria, manufacturers have little incentive to improve in this area.