Research Methodology FAQs

What is the evaluation instrument?

• KLAS uses two unique evaluation instruments, one for Software & Medical Equipment vendors/products (40 questions) and another for professional service firms (19 questions).
• The same KLAS vendor evaluation instrument is used for all Software & Medical Equipment vendors/products.
• The same KLAS vendor evaluation instrument is used for all Professional Services Firms.
• The exact questions for the vendor/product and professional service evaluations are available on the KLAS web site here

How are evaluations received by KLAS?

1. Evaluations are received online, via fax, email or directly from one-on-one interviews with professionals from provider organizations.
2. All completed evaluations are followed up with an in-depth confidential interview by a KLAS research analyst.
3. The results are posted to the KLAS database and subsequently made available for public use.
   • KLAS reserves the right to quarantine any incomplete or suspect evaluation.
   • A dedicated KLAS researcher is assigned to resolve data issues from quarantined evaluations.
   • KLAS reserves the right to discard evaluations it cannot confirm.

How does KLAS verify the data is valid and from verifiable end user organizations?

• Every evaluation is verified face-to-face or via telephone, by a KLAS research analyst. Verification timeframes vary based upon the availability of both KLAS researchers and the participant.
• KLAS confirms the role and responsibility of the participant and assesses conflict of interest. If an evaluation cannot be confirmed or corroborated as valid, it is discarded.

Does KLAS keep the participant’s identity confidential?

• KLAS keeps the identity of evaluation participants confidential.
• A list of participants is never shared with any outside organization.
• KLAS does not share or sell information about a participant's needs or future plans to any outside organization.
• Evaluations and commentary submitted are not linked to a participant or organization in any public use of the data (online query, published reports, articles, presentations, etc.).

What level of healthcare professional can submit data to KLAS?

• The data collected represents input from individuals who have direct product and/or service responsibility and/or evaluate and manage vendor decisions and relationships. Participants are at the manager/supervisor level or above.
• When an evaluation is submitted from someone below a manager/supervisor level, KLAS must receive validation from a manager (or other key decision maker) before the data is accepted.
• Evaluations that cannot be verified, do not meet submission requirements, or are for products or services not in productive use by the participant (unless they are rating applicable indicators such as selling and contracting) are discarded.

Is there a limit to how much and how often data can be submitted?

• There is no limit to the number of evaluations that can be submitted on a given product or service— except for the obvious limit governed by the number of live organizations.
• There are limits as to the number of evaluations accepted from the same organization and frequency from the same person. KLAS allows up to three evaluations for a particular product/service from the same organization. In most cases, the IT, operational and C-level executives rate the product. When there are more than three submissions for a product or service from the same organization, KLAS uses the three evaluations from the highest level persons in each area and discards excess evaluations. New evaluations from higher level submitters can cause an existing evaluation from a subordinate to become inactive.
• KLAS maintains one active evaluation per participant for each product the participant rates. Subsequent to verification, acceptance of a newer evaluation causes the first or older evaluation to be archived for trending purposes. A replacement of this nature within a 12-month period will cause a slight trickle-down effect on any calculations made from the archived evaluation.
• Every evaluation participant is contacted annually for an update of their previous evaluation. KLAS provides the participant with their previous responses and participants are asked to rescore the vendor, product, or service. Every updated evaluation includes a personal interview before entry into the KLAS database.

Can a vendor "stack the deck"?

• KLAS expects vendors to provide complete client lists and works with and encourages vendors to seek participation from their entire client base. Download official KLAS text to invite all clients to participate—please ensure that ALL clients are invited to participate.
• It is our objective to insure that no vendor "stacks the deck" resulting in scores that misrepresent product and/or service performance. KLAS will use whatever lawful and commercially available means at our disposal to validate performance data received. In the event that suspect data is received, KLAS will work closely with the submitting individuals, organizations or vendors to resolve questions about the validity of the ratings received. KLAS may perform random evaluations of the entire vendor client base or specifically gather additional scores from the participating organization. In the event that a random sampling of the entire vendor client base and/or a reasonable validation of the ratings cannot be achieved, the suspect data is discarded.
• When there is a question as to whether the data represents a random sample for the product or vendor even if it exceeds the minimum threshold of verified evaluations in the database, KLAS reserves the right to asterisk the reported results until we can confidently remove the asterisk. Only products or vendors without an asterisk will be considered for "Best in KLAS" awards and considered for top rated products or vendors in reports.

How does KLAS avoid bias and remain impartial?

• KLAS’ mission is "to improve the delivery of Healthcare Information Technology by honestly, accurately and impartially measuring vendor performance for our provider partners." Everything KLAS does is in support of this mission.
• KLAS utilizes a controlled group of industry professionals to collect performance data. Audit control measures and specific checks and balances are in place to both monitor and gauge productivity and data quality. Evaluation data is entered into the database, as individual entries and data are obtained. In addition, with more than 20 research analysts gathering data across all vendors and products, it is difficult for any single individual to consciously influence outcome.
• Virtually every vendor, in addition to many consulting firms and investment banking firms, are KLAS clients. KLAS bases the credibility of its services on remaining independent from any vendor, consultant or investment firm. KLAS is open to an audit by an independent party given the appropriateness of such a request and subject to cost and availability.
• Providers who have equity ownership in a vendor may not evaluate that vendor except for rare circumstances that will be noted. They may rate other vendors' products.
• Please note that Scott Holbrook owns equity in KLAS and serves as a KLAS board member. Scott is employed by a vendor rated by KLAS. Mr. Holbrook is one of the original founders of KLAS. KLAS stands for the first initials of the company founders (Kent Gale, Leonard Black, Adam Gale and Scott Holbrook.) In order to mitigate any potential conflicts of interest and to avoid potential bias in any ratings, Mr. Holbrook is not given access to confidential information supplied to KLAS by providers or vendors (other than information from his own employer), nor does he collect or review research. Mr. Holbrook does not influence the nature of the research, the products for which research is pursued, or the method of reporting the data.

What are the requirements for inclusion in KLAS vendor/product performance reporting?

Before vendors/products are included in the KLAS performance ratings:

1. A vendor/product must meet the minimum reporting threshold (see requirements below).
2. KLAS must ensure the sample of reporting provider organizations is random.
3. KLAS must verify that an appropriate balance of provider respondents is represented (ie: job title, department, etc…)
4. KLAS must determine which market segment the vendor/product fits into.

Inclusion thresholds in the KLAS ratings apply both to Specialty Reporting and Online Ratings.

• KLAS reporting applies to both (1) traditional KLAS Specialty Reports (drilling down into an industry segment, i.e. PACS) and (2) KLAS data available for on-line query and self-generated reports.
• While the minimum thresholds are the same for both industry reports and online query, the data is presented differently depending on the medium being used. The traditional KLAS industry reports identify and report on minimum threshold data through the reports' Addendum while KLAS data available for online query and self-generated reports identifies minimum threshold data with an asterisk and uses a system of check marks to indicate data confidence level.

Software & Medical Equipment Minimum Inclusion Threshold:

• To be included in the KLAS ratings at Minimum Confidence, a vendor/product must:
  • Be live and verified by KLAS in at least six (6) unique provider organizations
  • Have product evaluations from greater than fourteen (14) unique provider organizations in a twelve month period.
• To be included in the KLAS ratings for trending purposes only (no ranking assigned), a vendor/product must:
  • Be live & verified by KLAS in at least six (6) unique provider organizations.
  • Have product evaluations from a minimum of six (6) unique provider organizations in a twelve month period.
• The following table outlines the various data thresholds and confidence levels used by KLAS, specific to Software and Medical Equipment.

Organizations Reporting	Industry Reports	On-line Query & Self-generated Reports
1-5	Not reported	Not reported
6-14	Results identified with an asterisk Results appear only in the Addendum Results are not compared to the vendor/products appearing in the main body of the report	Product marked with an asterisk indicating that it does not meet the minimum KLAS confidence level Confidence level of zero check marks indicating that the confidence level on this product rating is low, the results are for early trend detection, and as more performance data is received, the result is likely to have a wide variance
>15	Results reported in the main body of the report Results compared with other main body reporting vendors, the average of the industry segment(i.e. PACS) and the average HIT vendor (all vendors, all products in the KLAS database)	Product not marked with an asterisk Data confidence reported using a check mark system, ranging from one check mark, the lowest confidence level (for products with a minimum of 15 organizations), to three check marks, the highest confidence level (for products with a minimum of 30 or more organizations)

Professional Services Firm (PSF) Minimum Inclusion Threshold:

• Performance monitoring is based on 19 criteria versus the 40 criteria used for its vendor/product counterparts
• To qualify as valid, an evaluated service must be either 1) a one-time service completed in the last 24 months or 2) an ongoing service of which a measurable portion was completed in the last 24 months.
• The Professional Services segment represents a small volume of activity, compared to software vendor products. As a result, most professional service categories have few clients per Professional Service Firm. To accommodate providers’ needs, the threshold for PSF services ranking assignments vary depending on different engagements and sizing issues. Data at this threshold is not asterisked in Professional Services categories. In such cases, KLAS notifies the reader of the low confidence level—even when KLAS has 100% of organizations reporting. The low number of reporting sites increases the possibility of significant variation when new data is added.
• To be included in the KLAS ratings at Minimum Confidence, a PSF must:
  • Be live and verified by KLAS in at least three(3) unique provider organizations.
  • Have product evaluations from greater than five (5) unique provider organizations in an eighteen month period.
• To be included in the KLAS ratings for trending purposes only (no ranking assigned), a PSF must:
  • Be live and verified by KLAS in at least three(3) unique provider organizations.
  • Have product evaluations from greater than five (5) unique provider organizations in an eighteen month period.
• Be aware that some Professional Service market segments have different sample size requirements (depending on different engagements and sizing issues) to meet confidence level standards.
• The following table outlines the various data thresholds and confidence levels used by KLAS, specific to Professional Services Firms.

Organizations Reporting	Industry Reports	On-line Query & Self-generated Reports
1-2	Not reported	Not reported
3-5	Results identified with an asterisk Results appear only in the Addendum Results are not compared to the vendor/products appearing in the main body of the report	Product marked with an asterisk indicating that it does not meet the minimum KLAS confidence level Confidence level of zero check marks indicating that the confidence level on this product rating is low, the results are for early trend detection, and as more performance data is received, the result is likely to have a wide variance
>6	Results reported in the main body of the report Results compared with other main body reporting vendors, the average of the industry segment(i.e. PACS) and the average HIT vendor (all vendors, all services in the KLAS database)	Product not marked with an asterisk Data confidence reported using a check mark system, ranging from one check mark, the lowest confidence level (for services with a minimum of 6 organizations), to three check marks, the highest confidence level (for services with a minimum of 15 or more organizations)

How does performance monitoring for professional services differ?

• Unlike its IT vendor or product counterparts, whose data has been collected since 1996, professional services monitoring began early 2004.
• Today, performance monitoring is based on 19 criteria versus the 40 criteria used for its vendor/product counterparts. As data collection grows, KLAS does anticipate growth in this area.
• Professional Services performance monitoring is broken down into eight market segments: Implementation-Principal; Implementation-Supportive; Planning & Assessment; Vendor Selection; Technical Consulting; Full or CIO IT Outsourcing; Departmental Outsourcing; and Staff Augmentation.
• To qualify as valid, an evaluated service must be either 1) a one-time service completed in the last 24 months or 2) an ongoing service of which a measurable portion was completed in the last 24 months.

What are the Top 20 Reports and how does a vendor get into a Top 20 Report?

• Top 20 Reports show the top 20 scoring products from the 14 most-rated healthcare segments and the market leader for each market segment KLAS rates. Top 20 Reports also show vendor ratings, by category, for every vendor or product that has been evaluated by more than six organizations.
• There are two Top 20 Reports: the Mid-year Report, issued in June, summarizing data collected and verified between April 15 (previous year) to May 15 (current year); and the Year-end Report, issued in December, summarizing data collected and verified between October 15 (previous year) to November 15 (current year).
• The year-end report carries with it a "Best in KLAS" award, which is the primary difference between the two reports.

What is a "Best in KLAS" award and how does a vendor receive one?

• A "Best in KLAS" Award is an annual award given to a vendor/product that scores the highest in its industry segment (i.e. PACS) in the Top 20 Year-End Report and meets all the other "Best in KLAS" requirements.
• Other "Best in KLAS" requirements:
  • Vendor/product data meets the minimum confidence level.
  • The product fits properly into a defined Best in KLAS category.
  • Represents vendor/product scoring over the past 13 months.
  • Data validation able to be completed by November 15th.

NOTE: It is not appropriate to assume that if enough clients have submitted data to KLAS by November 15th this will qualify one for award consideration since all data still needs KLAS validation. KLAS performs four validation steps before data is reported without an asterisk:

1. Ensure the survey mix is within a normal range of respondents (by title) and client sizes and represents a random sample (not a select set of clients funneled by the vendor to KLAS).
2. Ensure that a single organization has not submitted too many evaluations.
3. Ensure the product includes the core modules required by the category.
4. Ensure the vendor has not coached clients to bias the survey.

How old is the data I am looking at?

• Data and commentary represent performance reporting from the most recent 12 months unless otherwise stated.
• Data that is more than 12 months old is archived and used for trending purposes only; 1-3 year online trending is available in Platinum KLAS.
• In KLAS Online, commentary from the past 12 months is available. In Platinum KLAS, you may choose to display commentary from a 6, 6-12, 12, or 12-24 month window.

Sometimes the Top 20 scores do not match those seen online in Platinum KLAS or in individual reports, why is this?

• Due to production requirements and in an effort to be fair and provide for "expiring" 12 month data scores, the Top 20 incorporates 13 months of data while Platinum KLAS (PK) and other reports include a rolling 12 months of performance data.
• As daily performance scores come in and others roll off the active 12 month window, the actual score in a report can be different than a PK score a few days later and a Top 20 score covering 13 months.

How does the KLAS stoplight system work?

• Stoplights represent the variance from the average of all products in the KLAS database.

  Well Above Average - Rating of 1 or more points (10% for Business Indicators) above industry average for that criteria

  Above Average - Rating is between .5 and .9 points (5%-9% for Business Indicators) above industry average

  Average - Rating is within .5 points( 5% for Business Indicators) of industry average

  Below Average - Rating is between .5 and .9 points (5%-9% for Business Indicators) lower than industry average

  Well Below Average - Rating of 1 or more points (10% for Business Indicators) below industry average for that criteria

What is the difference between KLAS Online & Platinum KLAS?

• KLAS Online offers online access to KLAS ratings, exclusively for healthcare providers. Providers who evaluate their vendors receive immediate access to KLAS Online.

Learn more about KLAS Online
Evaluate a vendor and get access now

• Platinum KLAS offers robust, subscription-based access to KLAS ratings. Platinum KLAS delivers advanced functionality, and includes free access to specialty-report summaries, as well as KLAS’ semi-annual Top 20 Report. Providers who evaluate their vendors receive a 50% discount on Platinum KLAS.

Learn more about Platinum KLAS.
Rate your vendor to receive a 50% discount on Platinum KLAS.