The HIStalk guy has been pretty hot lately with lots of interesting content. The following is an excerpt from his latest post in response to an email asking questions about the IT ranking firm KLAS.
Anonymous Reader II asks about KLAS: "I
am trying to understand the logic of the KLAS company and I am
confused. Given the appearance of importance my upper management is
placing on this, they skim through and say 'here are the top vendors'"
according to KLAS. I know what their site says, but can YOUR readers
provide direct insight BOTH from the vendor side and from the Facility
IT side. How are their test scores computed? Are products tested? Do
vendors pay or advertise? How does this ranking come to be? I am also
seeing companies ranked top one year and not the next. Products in my
previous industry took years to change... I assume this is not the case
in HIS?" KLAS is a survey company, basically, paid mostly by
vendors who buy reports at full price and pay to use the results in
advertising (or so I assume, anyway.) A few hospitals subscribe, but
only because they get heavy discounts for answering KLAS surveys.
KLAS does not perform product testing. They do not release information
on which hospitals they survey, how many surveys were completed, who
they talked to, or what statistical manipulation they used to leap to a
particular conclusion. They talk to the users of software, but you
can't find out what role those folks hold, how big their facility is,
how long they've been live, what version of software they're on, how
long ago they were surveyed, etc. KLAS tries to convince everyone that
movement up or down the rankings is due to their keenly tuned detection
of trends, but it's more a function of sloppy methodology and
statistical stretching. On the other hand, they often arrive at
conclusions that reflect the market in general, though I certainly
wouldn't buy a product on that basis alone. Use KLAS to confirm what
you suspect, to focus your evaluation process, to get contractual
protection against apparent weaknesseses, and to read what other
customers say about your vendors. Personally, I find little
disagreement with the 1-2 products at the top and bottom of a category,
but any attempt to compare the middle of the pack to one another is
generally unsatisfying.
am trying to understand the logic of the KLAS company and I am
confused. Given the appearance of importance my upper management is
placing on this, they skim through and say 'here are the top vendors'"
according to KLAS. I know what their site says, but can YOUR readers
provide direct insight BOTH from the vendor side and from the Facility
IT side. How are their test scores computed? Are products tested? Do
vendors pay or advertise? How does this ranking come to be? I am also
seeing companies ranked top one year and not the next. Products in my
previous industry took years to change... I assume this is not the case
in HIS?" KLAS is a survey company, basically, paid mostly by
vendors who buy reports at full price and pay to use the results in
advertising (or so I assume, anyway.) A few hospitals subscribe, but
only because they get heavy discounts for answering KLAS surveys.
KLAS does not perform product testing. They do not release information
on which hospitals they survey, how many surveys were completed, who
they talked to, or what statistical manipulation they used to leap to a
particular conclusion. They talk to the users of software, but you
can't find out what role those folks hold, how big their facility is,
how long they've been live, what version of software they're on, how
long ago they were surveyed, etc. KLAS tries to convince everyone that
movement up or down the rankings is due to their keenly tuned detection
of trends, but it's more a function of sloppy methodology and
statistical stretching. On the other hand, they often arrive at
conclusions that reflect the market in general, though I certainly
wouldn't buy a product on that basis alone. Use KLAS to confirm what
you suspect, to focus your evaluation process, to get contractual
protection against apparent weaknesseses, and to read what other
customers say about your vendors. Personally, I find little
disagreement with the 1-2 products at the top and bottom of a category,
but any attempt to compare the middle of the pack to one another is
generally unsatisfying.
Recent Comments