Medical Device Networks Trouble Industry
Over the past few years, medical device networking has become increasingly problematic. There is a growing perception of declining service levels and network reliability. (I’ve not seen or heard about any quantitative data to back this up – let me know if you’ve got any – but these are substantive issues.) As medical devices, by way of connectivity and workflow automation, have been pulled into the sphere of the IT department risks to patient safety have increased.
These growing problems are a key factor in the creation of IEC 8001 – which will have a major impact on how hospital’s deploy and manage networked medical devices. The current situation provides a lot of the impetus behind a meeting this week at Villanova on wireless technology in health care. So what’s the big deal?
Private Medical Device Networks
In the good old days, all medical device systems operated on private networks that were designed, installed and supported by the medical device vendor. Vendors decided exactly how the would handle IP addressing (static or DHCP) and every other configuration option. This minimized vendor’s product development costs, shortened time to market, and made service and support easy (because the network was their sandbox). When they got the inevitable calls from customers that, “your system just brought down our network,” vendors could quickly and easily determine if that was the case – and just as importantly, quickly prove to hospital network administrators why their answer was correct.
Providers benefited from this approach in a number of ways. First, vendor system support was quick, efficient and reasonably priced. This approach also relieved hospitals of the need to assume responsibility for supporting life critical applications themselves.
Of course nothing’s perfect, and relying on private networks created a few problems of its own. As more medical devices evolved to become components in medical device systems connected by networks, these private networks proliferated. The IT department of a thousand bed hospital system on the east coast did a survey a couple years ago of their networked medical device systems, and “discovered” over 200 private networks. And unlike enterprise networks that must be kept up to date to ensure cross vendor interoperability and coexistence, private medical device networks are like time capsules. With the exception of occasional failures of a switch or router, medical device networks remain unchanged from the day they’re installed; the hospital mentioned earlier found numerous ThickNet and ThinNet Ethernet networks in their survey of medical device systems.
Over time, medical device vendors have given some ground by moving from physically separate private networks, to creating logically separate private networks through the use of network switches and routers.
The Transition to Enterprise Networks
Changing requirements in hospitals have reduced the viability of private medical device networks over the past several years. Due to changes in technology and care delivery processes, various medical devices have broken out of their traditional use models and are now deployed house-wide. Smart infusion pumps, patient monitors, point of care diagnostic testing (POCT) devices, and ventilators can be found almost anywhere in the hospital. (Admittedly, the POCT market lags in networking technology, and ventilator vendors have successfully held connectivity at bay – so far.) The reality here is that it is impractical to deploy multiple private networks house-wide to support each of these and future medical device systems.
Hospital EMR adoption is also driving the transition to enterprise networks. Automating medical device data acquisition for charting has two key requirements. First, one must have access to the data. Data access is frequently done through gateway devices that provide the EMR with HL7 data while retaining much of the walled garden of the medical device vendor’s private network. The second requirement is that all data going into the EMR sport a consistent time stamp. Communicating with hospital’s network time servers has further widened the crack in device vendor’s private networks.
For many good reasons, the wide spread duplication and proliferation of private networks just rubs IT folks the wrong way. All that variability and complexity, not to mention duplication of network gear (all at vintages reaching back 10 years or more), is increasingly looked at as unnecessary and undesirable. Private networks are fading, and are not expected to make a come back.
What About Wireless?
In many ways, wireless networking has closely followed the evolution of wired networks. The earliest wireless medical devices (telemetry systems) used proprietary RF communications. Just like private data networks, medical device vendors designed, installed and supported these systems. Even early Wi-Fi based systems fit this model. The majority of these early Wi-Fi systems used 802.11FH rather than the 802.11b technology used by hospital IT departments. This difference in underlying technology effectively rendered them private networks.
Recently, wireless has moved in two direction. The old analog VHF has transitioned to the new WMTS bands, and 802.11FH has been replaced by 802.11b/g and 802.11a/b/g. While WMTS is still a private network, Wi-Fi has been forced on the the enterprise Wi-Fi infrastructure in most hospitals.
The transitions from private to enterprise networks has not been without its problems.
Culture Shift for Medical Device Vendors
Medical devices are what’s known as embedded systems, purpose built black boxes that are completely controlled by the manufacturer. Medical device vendors like to keep complete control of what goes into their systems, and rightfully so. A key design principle for medical devices is to keep them as simple as possible in order to eliminate unnecessary variables. The more complex a system is, the longer it takes and the more it costs to develop. Variability and complexity become major factors in the verification test of medical device systems. And verification test impacts both initial product development and sustaining engineering (where bugs are fixed, product components replaced, and software updated). Consequently, it is ingrained in medical device vendors to minimize variability and only implement those features that are absolutely required by the product.
The purpose built mentality of vendors works great for embedded systems, the actual medical device. But when this mind set is extended to application software, personal computers, servers and networks, problems arise. The general purpose computing world in which hospital IT exists is full of options and is highly configurable; they revel in variability.
Private networks enabled medical device vendors to select and implement only those options and configurations they wanted to use in their medical device system. As already noted, this reduced development costs, shortened time to market, and facilitated support – mainly due to keeping a lid on variables. Obviously, this approach doesn’t work well in an enterprise IT environment. It is the enterprise IT environment in which medical device vendors (many thinking fondly of the days of private networks) are struggling. Device vendors are not alone in their feeling the pain of medical device systems moving to enterprise networks.
Culture Shift for Hospital IT
Hospital IT departments have their own set of proclivities. The IT world is heterogeneous, with a large variety of vendors offering different applications, computing platforms, and networking technologies. The IT industry depends on standards developed by SDOs (standards development organizations) and test and certification provided by consortia (like the Wi-Fi Alliance) to make this heterogeneous environment possible. Hospital IT folks tend to take all this heterogeneity for granted, demanding that medical device vendors conform to whatever set of standards and configuration options they’ve chosen to adopt. Hospitals can be downright vociferous in their insistence that vendors – medical device and IT – conform to their requirements, rather than the other way around.
This intransigence can cause hospitals to reject superior clinical solutions because a medical device system does not support the specific standards and configurations adopted by the hospital. Hospitals also suffer reduced levels of service from medical device vendors as easy to diagnose and support private networks are replaced by enterprise networks. This results in higher costs for individual hospitals, as medical device vendors have to cover the increased costs of developing and supporting a full range of IT options.
Medical device systems, made up of the embedded system, application software, computers and network, are regulated by the FDA. Vendors must follow the Quality System regulation when designing those products, follow Good Manufacturing Practices when building them, and conform to regulations for seemingly everything else they do. This means that the software, computers and network are part of the regulated medical device.
Regulatory approval is granted based on the labeling or intended use of the system as stated by the manufacturer. Only the holder of that regulatory approval can make any promotional claims about the medical device system, in whole or part. The intended use, claims, product design and verification test are all interrelated – change one and you likely impact the others. This becomes an issue when medical device vendors claim network capabilities that are by necessity supported by specifications and related verification testing. When hospitals adopt a medical device system and deploy it on a network that differs from the device’s network specifications, the hospital is using the medical device “off label.”Off label use by providers is not precluded by FDA regulations, or any other regulations that I’m currently aware of.
The FDA can only regulate manufacturers. Providers who use regulated devices off label don’t need to worry about the FDA. That’s not to say they shouldn’t worry at all – plaintiff’s’ attorney’s come to mind, along with possibly state regulators, accreditation bodies, and other parties interested in patient safety. The FDA has set the standard for quality systems intended to ensure the safety of medical devices. Should providers wish to use devices off label, the prudent organizations provide appropriate test and management oversight to bear so that their off label use does not impair patient safety.
Make no mistake, enterprise networking issues have killed patients by impacting the operation of medical device systems. Of course biomeds and clinical engineers are used to working within rigorous environments designed to ensure patient safety. This is sometimes a sobering realization for IT folks who may not even realize how their actions impact medical device systems already installed in their hospitals.
The bottom line here is that we can’t all look to the FDA to solve these issues that are the consequence of putting medical device systems on enterprise networks – when you do this, your enterprise network becomes part of a medical device.
Let’s look below the surface of the key issues arising from this transition from private to enterprise networks.
Provider Network Variability
By design, the general purpose network environment is very flexible and highly configurable. The hospital network design and configuration choices are highly variable – there’s probably a hospital somewhere that’s exercised every configuration choice available. This variability imposes significant costs on the industry.
As noted above, building in a full breadth of general purpose network configurability is a considerable expense for device vendors. This same flexibilty also adds considerably to verification test and sustaining engineering costs. Rather than creating a private network that remains mostly unchanged for 5, 10 years or longer while installed in the field, enterprise networking requires that certain changes be made to keep up with the general IT networking market. This sustaining engineering cost can easily dwarf the initial cost of developing the medical device system itself.
Proprietary network features are another big issue. In general buyers, hospitals and medical device vendors both, prefer using industry standards. Sticking with standards provides additional flexibility and can greatly reduce the cost of developing and supporting networked medical devices. When providers select an infrastructure vendor (for Ethernet, wireless or both), they typically go with one vendor. By “standardizing” on one vendor, the provider frequently looks more favorably at proprietary features from their vendor. This makes sense because they’ve only got that one vendor’s products to content with, and thus don’t see any duplication of effort or additional costs between a feature based on a standard and one that proprietary to their “standard” vendor. This is not the case for medical device vendors. Implementing a proprietary feature for authentication, encryption, quality of service, roaming or other capability means significantly more development cost and a big hit to sustaining engineering costs over the life cycle of their product. Not surprisingly network vendors are constantly looking to add value and differentiate their solution in a relatively mature and standards driven market (rather like medical device vendors themselves, come to think of it). Providers who want medical device vendors to adopt proprietary infrastructure features, even for a vendor who might have say 70 to 80 percent market share, is a losing proposition. They won’t do it, and rightly so.
The industry also incurs costs in the area of service and support. Providers pay more for a lower level of service because of the new variables the device vendor must grapple with in diagnosing and fixing problems. Overall patient safety can suffer as well.
The days of private medical device networks as we know them are over. Yet a full blown general networking approach to medical device networks is less than satisfying. Both the medical device and IT realms must reach some accommodations with each other if we want to realize the patient safety levels of the recent past, at the same or lower costs.
Medical Device System Coexistence
Most medical device systems struggle with coexistence on an enterprise network with other medical device systems. There are two main reasons for this, certain applications impose stiff performance requirements on the network and many medical device systems aren’t designed to be sufficiently well behaved in an enterprise network environment.
Because medical device life cycles are so long (typically 7 years minimum), most current networked medical device systems include software that was designed to operate on a private network. These systems lack the built in intelligence to manage network utilization, lack network management and performance monitoring capabilities, provide sometimes severely limited configurability, and can exhibit antisocial network behavior – like spewing gobs of multicast traffic, or over consuming other network resources. Some systems have considerable technical network performance requirements due to a combination of being demanding life critical applications and designs that may be inadequate for enterprise network environments. A common symptom of network coexistence limitations are vendor requirements for dedicated SSIDs and/or VLANs. Many of the problems will fade over time, as new products and versions of existing products come to market.
Coexistence for wireless networks is similar to that for Ethernet, although there are differences. Besides the actual network configuration and protocols (like you have with Ethernet), Wi-Fi is further complicated by things like RF interference. The ISM band, used by most wireless medical devices, is supported by standards that are intended to ensure coexistence. Mixing ISM based technologies (like Wi-Fi, Bluetooth and ZigBee), does complicate things, but is possible.
The alternative to the ISM band for medical devices is WTMS. (There is MICS and a proposed medical body area network, but are so specialized are rarely used that they will not be discussed here.) There are no standards specified for WMTS whatsoever. The FCC simply designates specific frequency bands and rules for registering your use of the band. Consequently there are no standards or formal mechanism to ensure coexistence between vendor’s solutions. In fact, early in the existence of WMTS, the two predominate vendors using the band, GE and Philips, had considerable coexistence problems precluding, for a time, the installation of systems from both vendors being used in the same hospital. While a standard to support wireless medical devices in WMTS might be worthwhile, the relatively narrow bandwidth and economy of scale issues argue against such an effort.
Infrastructure Vendor Support
Because of the maturity of Ethernet standards, and the fact that networking engineers wrote the standards, most medical device systems can run on any Ethernet vendor’s infrastructure.
When medical device vendors develop a new wireless product, most go through a similar process. First, the vendor selects a wireless LAN infrastructure to use when designing and testing their new product. The choice of wireless LAN or infrastructure vendor can be based on market requirements, like infrastructure vendor market share, or performance, especially when high performance life critical applications are involved.
Regardless of why an infrastructure vendor was chosen, when that new product is released it may specify that infrastructure vendor’s network. The degree of specification can range from the general (a specific manufacturer and certain standards) to detailing down to product models and firmware software releases.
The greater specificity in medical device vendor’s network specifications, the more narrow the choices for both device vendor and provider. Say vendor X build their new patient monitoring system on Cisco infrastructure. When their sales rep calls on Barnes Jewish in St Louis, they have a problem: Barnes has installed the Wi-Fi network from Meru. This leaves the vendor with 3 options: 1) walk away from the business, 2) try to convince the provider to install a second Wi-Fi network from Cisco (good luck!), and 3) spend $250,000 to $500,000 to repeat the verification test they did with Cisco using Meru equipment. And the quarter to half a million dollar verification cost estimate assumes they don’t need to change their product to support the new network. If the vendor needs to say tweak their embedded radio device driver, the cost can easily go north of $1 million. Multiply that cost by 4 or 5 infrastructure vendors and the network support cost can easily exceed the cost to develop the actual medical device system. Of course all these costs are passed on to providers and again to payors and patients.
Assume we have a working wireless medical device system and it’s widely installed in hospitals. Remember the product life cycle for a medical device is several years at least. Compare that to the life cycle of general purpose networking products, which can be as short as 18 months. Now imagine each medical device vendor having to assess their network vendor’s product changes (down to firmware updates) and you can easily imagine a near full time verification test and sustaining engineering effort. This sustaining engineering burden falls on each wireless product individually or by product line, because most medical device vendors are organized in silos by market segments and tend to reinvent connectivity features across separate R&D teams.
This sustaining engineering effort is further complicated when the medical device system is deployed on an enterprise network managed by the provider. How are they supposed to know whether it’s okay to install that new firmware upgrade on their access point controllers or not? Sadly, it’s not unheard of for an enterprise to update their infrastructure and have the entire network fail when they turn everything back on. If that network is supporting life critical applications, patient lives are at risk.
What About Wireless, Again?
There are lots of networked medical devices in hospitals, especially in diagnostic imaging and the clinical lab. What makes the networking issues described above of urgent importance is networking at the point of care, where there’s a great deal of mobility and wireless networking best meets user requirements. Because of this, much of the focus on this topic has been on wireless networking. In tackling these problems, we must remember that Wi-Fi represents the last several feet between the Ethernet network and the wireless client device. Any solutions for wireless medical devices must also consider the wired network that underlies the wireless connection.
The problems around wireless medical devices and networking have been growing for years. It was 2 years ago that the FDA assembled a workgroup to look at this problem (and that effort resulted in IEC80001). This year’s FDA proposed draft of the Medical Device Data Systems rule tangentially addresses networking issues. Just last week the Joint Commission issued a Sentinel Event Alert on, “Safely implementing health information and converged technologies” (where they note, “Multiple networks can result in poor interoperability and increased costs.”).
Tomorrow I’ll be attending the “Villanova Workshop on Wireless Technology in Healthcare: What is needed for safe, secure, and reliable deployment?” I’ll be blogging the meeting, as I know many folks involved in this area either weren’t able to come, or could not be accommodated due to the limited space available.
Pictured up top is a VeriWave 90 test system chassis – a piece of diagnostic equipment that almost every hospital will one day have to keep their wireless medical devices healthy.
NOTE: This post was whipped out even more quickly than usual. You can expect minor revisions over the next few days (adding links to reference additional information, fixing typos, improving clarity). Revisions will be noted per usual as “UPDATE”.