Highlights of the APCO NENA Quality Assurance and Improvement (QA/QI) ANSI Standard

​​​ ​apho2.pngTwo years ago, APCO and N​​ENA introduced the standard for Quality Assurance and Improvement (QA/QI) for Public Safety Answering Points (PSAPs). The obje​ctive was to “establish a long overdue quality assurance and improvement process for all Public Safety Answering Points and their telecommunicators to ensure call taking and radio dispatch actions are delivered at the highest possible standard.” 

In the past two years, we’ve seen several public safety agencies embrace and adopt the standard to various degrees, and encourage others who haven’t yet, to do so. We recently hosted an APCO Webinar with the Denise Amber Lee Foundation and I was encouraged to see that more than 800 PSAPs registered to attend the Webinar to learn more about the standard, best practices for implementing it, and helpful tools to improve the objectivity and consistency of their QA programs.

The voluntary standard offers a highly recommended roadmap that can be adopted and implemented by any PSAP. We hope you find these recap highlights from the standard helpful.

Scope and Purpose

The NENA APCO 911 Quality Assurance standard defines the recommended minimum components and best practices of a Quality Assurance/Quality Improvement (QA/QI) program within a public safety communications center. Recommended minimum requirements for the evaluation and QA of call processing are intended to ensure a consistent, effective, and efficient level of service.

The document organizes information into the following sections:

 1. General Agency Responsibilities
 2. Evaluator Duties and Responsibilities
 3. Components of a QA/QI Program
 4. Review Process Requirements
 5. Performance Review and Evaluation Procedures
 ​
 6. Documentation Criteria
 7. Reporting Criteria
 8. Recommendations for Improvement
 9. Feedback Criteria
 ​​10. Action​ Plans
 ​​ ​

The following is a quick summary of each section.

General Agency Responsibilities

This section advises that the QA/QI program should be formally defined and standardized by each Agency – for part time, full time, temporary and volunteer telecommunicators – as the process of evaluation of telecommunicators’ strengths and identifying areas for improvement.  The QA/QI process must be regularly calibrated, monitored and revised as needed to assure the highest possible level of reliability and integrity. The Quality Assurance Evaluator (QAE) shall be periodically assessed to ensure that they are being fair and equitable in their evaluation techniques. The Agency shall be thoroughly familiar with quality assurance training and systems available to enhance the performance of its program and in the performance of case evaluations. The Agency should provide resources and/or technologies necessary to conduct the quality assurance process.

It is recommended that Agencies define their internal QA standards based on Industry Best Practices as well as by soliciting internal inputs from employees. Upon evaluation of recorded calls, employees should receive feedback as soon as possible, in order to provide prompt training and remediation assistance where needed, or to promptly recognize employees for a job well done. The QA/QI program, processes, and requests for release of information shall comply with all applicable federal, tribal, state/provincial and local laws, rules and regulations.

Evaluator Duties and Responsibilities

The Quality Assurance Evaluator (QAE) is expected to be thorough, consistent and objective. Playback of recorded calls/screens and rating them on QA forms should be combined with reviewing CAD records and call notes, and a confirmation that all information was requested, collected, and entered accurately.  The QAE is to provide fast feedback to telecommunicators. He or she is also responsible for generating and disseminating QA reports defined by the Agency.

Components of a QA/QI Program​​

Agencies should define and communicate their standard evaluation guidelines to include minimum performance expectations for each rated category.  The following is a list of recommended components:

  • Case Review Criteria
  • Evaluation Guidelines
  • Oversight Committee as per protocol
    and/or agency guidelines
     
  • Program Monitoring
  • Record Keeping
  • Reporting and Feedback
  • Written Directives​
     

All parts of incident communications are to be reviewed, to include call taking, dispatch, data entry (collecting Callback telephone number; Caller’s name; Chief complaint and/or incident type selection; Location/Address Verification; Post Dispatch Instructions; Pre-Arrival Instructions; and Protocol/Systemized interrogation questions, status keeping, and others. Since the objective of a QA program is performance improvement, short-term and long term reporting and in-person coaching should be incorporated into the QA process.  

Agencies should establish discipline-specific (i.e. law enforcement, fire/rescue, EMS) oversight committees who formally meet to insure compliance with Agency policies and procedures.

Review Process Requirements

As a guideline, PSAP agencies shall review:

  • at least 2% of all calls
  • all cases involving catastrophic loss and/or high-acuity events as soon as possible after the receipt of the call and/or following the radio dispatch or at least within five (5) days.
  • any other call or event types as defined by an agency
     

The Agency shall maintain appropriate documentation when, in the normal course of business, case reviews are not conducted pursuant to established agency and/or protocol criteria.

Performance Review​ and Evaluation Procedures

The process of measuring the quality and performance of service should include at least the following criteria:

  • Analysis of performance trends
  • Compliance to protocols and
    standard operating procedures
  • Customer Service
  • Optimizing the use of agency resources
  • Overall performance of each employee
  • Reviewing the operation as a whole

Documentation Criteria

Review results shall be recorded and submitted to the appropriate personnel for cumulative reporting in accordance with the Agency’s written directives. The evaluation findings and any actions resulting from the quality assurance process shall be documented and filed in accordance with the Agency’s written directives.

Reporting Criteria

The Agency should develop reporting systems to inform management and employees about performance levels and help identify areas of superior performance as well as areas for improvement and remedial action. The level of detail in each report is to correspond to the needs of the recipient.

The Agency should develop procedures for the use of each report type, such as whether the reports should be intended for written records or face-to-face session, for private or public release, or those intended for formal presentations to internal vs. external users.

Recommendations for Improvement

All recommendations for improvement shall be consistent, derived from a structured process. They shall be clearly articulated during the review process and retained on file according to agency policy. They should, at a minimum:

  • Provide a means to identify gaps in performance
  • Evaluate strengths and weaknesses
  • Suggest steps to improve individual performance as well as Agency operations, and
  • Commendations for superior performance

Feedback Criteria

The Agency shall develop a communication plan that ensures timely, accurate and objective feedback to each telecommunicator. It shall allow for and solicit input into, and review from all employees involved in the QA/QI process.

Action Plans​​

The Agency shall establish action plans to address performance issues in response to QA/QI review findings.
The Agency shall develop short-term goals and objectives for final recommendations as well as the program as a whole. Responsibility for timelines should be assigned to each action plan.

Addendum 1

CUSTOMIZING AND UTILIZING THE CASE EVALUATION TEMPLATES

1.  Setting Up Your QA Forms

It is highly recommended to thoroughly explain the QA process to your staff members prior to implementing the program.  The program will be better accepted when you start out selecting calls or dispatches that have positive results so that telecommunicators do not become fearful of the QA process.  Include an “Exceeds Standards” category in your forms, so that excellent performance is promptly noted and recognized.

Determine the desired point value for each question - the more important the question, the larger the point value should be. Each category is scored separately based on the percentage of total points for that group. A target for the total score for each evaluation should be at least 90%, though you may start lower such as at 80% and work up to this target over time, such as in steps of 2% or 5%.  For telecommunicators with inadequate performance (i.e., poor customer service), you may use more extensive evaluation to determine patterns of improvement or further assistance needed.

The Standard provides sample forms for each type of communication (police, fire, EMS) in Addendum 2 and Guidelines for Call Taking and Dispatch, to be used for developing and training on QA form questions in Addendum 3

2.   Scoring Call Taking or Dispatching

Scoring should include compliance to protocols, Agency policies and procedures, and providing excellent customer service. Review the entire process of call processing, from accuracy and speed of CAD data entry to appropriate dissemination of information and dispatch if called for. The purpose of scoring is ultimately improving in areas where each telecommunicator needs a hand. Answer choices can be Yes, No, Refused (when the caller refuses to provide information) and N/A. Use of questions that have no point value, but improve follow-up is recommended, such as notes (e.g. “See Supervisor”) or a checkbox for “Exceeds requirements.”  The comments section adds greatly to the effectiveness and objectives of the QA process.

3.  Categories of Evaluation Results

Each evaluation will fall into one of the following categories:

  • Exceeds requirements
  • Meets requirements
  • Needs improvement – assure prompt employee review, call playback and coaching when score is awarded
     

Any unacceptable QA score (below the bar) should be signed by both the supervisor and the recipient. It is helpful to have the​ staff member review the audio of the calltaking or dispatching and grade the incident themselves. This methodology assists in making the conversation with the supervisor more objective and therefore more productive.

4.  Dissemination

The staff member should always receive a copy of their completed QA evaluations as soon as possible.  Time is of the essence in making the QA process a valuable learning tool. The standard recommends that the original evaluation should be placed in the employee’s personnel file to be incorporated as part of their periodic (annual, quarterly) performance evaluation.

5.  Calibration

QA evaluators need to be periodically assessed to ensure that they are being fair and equitable in their evaluation techniques. It is therefore very important to periodically, (weekly to begin with, then monthly) review their evaluations to ensure fairness and consistency. Evaluations by senior management should be used as benchmarks for assessing evaluation calibratio​n sessions.

 

Source: APCO/NENA ANS Standard for the Establishment of a Quality Assurance and Quality Improvement Program for Public Safety Answering Points. Various trademarks held by their respective owners. All information is believed to be accurate at the time of publication.​

Share this:
Twitter LinkedIn Facebook Email