Nursing-Sensitive Outcome Implementation and Reliability Testing in Two Cardiac Surgery Intensive Care Units

Kathy A. Demmer MS, RN, Cheryl McKane MS, RN, Linda Griebenow MS, RN, Julia Behrenbeck  MS, RN and Jane Timm  MS, RN

Demmer, K.A., McKane, C., Griebenow, L., Behrenbeck, J., and Timm, J. (October, 2004) Nursing-Sensitive Outcome Implementation and Reliability Testing in Two Cardiac Surgery Intensive Care Units, (OJNI). Vol. 8, No. 3 [Online]. Available at


The profession of nursing has increasingly found itself in the position of having to demonstrate that it makes a difference in patient outcomes. The language used in nursing care documentation plays an important role in validating the effect nursing interventions have on patient outcomes. Standardized nursing language has become essential to ensure nursing care representation in today’s health care environment and clinical information systems. In recent years, several standardized nursing language classifications have been developed and are being used to measure and compare outcomes sensitive to nursing care. Clinical testing is necessary to evaluate the usefulness, accuracy, and reliability of these language classification systems.

A pioneer in the development of standardized nursing language; the University of Iowa, sponsored a research team to develop Nursing Outcomes Classification (NOC). NOC consists of 260 outcome labels, each with a definition, list of indicators, and corresponding measurement scale. The outcomes and indicators are variable concepts that allow measurement of outcome states at any point on a continuum from most negative to most positive over time 1.

There were five phases of the development research. This clinical test site participated in Phase IV of the research focusing on the Evaluation of Measurement Scales. The following were the study aims given to the test sites:

This manuscript will focus on the experience of being one of the clinical test sites of the research study and will describe the impact of a computerized charting system, patient acuity and staff education on the study processes. Preliminary study results discovered by this clinical test site will also be presented.

Study Environment

Two Cardiac Surgery Intensive Care Units at a large mid-western tertiary hospital had an opportunity to serve as clinical test sites in this study. Both patient care units specialize in adult and pediatric care, while one unit subspecializes in transplantation and the other in congenital heart disease. The focus of this study was on the adult patient as cared for by the 200 registered nurses who served as study participants. An average of over 2000 cases of cardiac surgery occur in a year. Both patient care units average together about twelve cases per day.

Electronic Environment

Both intensive care units utilized an electronic documentation system that supports the use of standardized nursing language. The nursing staff was familiarized with the terminology and use of the care plan prior to the beginning of the research study through utilizing the North American Nursing Diagnosis Association (NANDA) and NOC. Both patient care units document utilizing an electronic medical record documentation system. The system is highly configurable and was developed to meet the needs of the ICU for critical care multi-system assessments, interventions, patient education and care planning.

Originally, the system was developed with three documentation flowsheets—a flowsheet for assessments and interventions, a patient education flowsheet, and a care plan flowsheet. The care plan was comprised of a drop-down list of NANDA nursing diagnoses and NOC outcome labels with selected indicators. “Family” rows were used as well, which allowed a group of nursing diagnoses and outcomes to be picked for a particular type of patient. The NOC indicators were difficult to incorporate due to system limitations related to the size of the flowsheet sections and the number of characters allotted to each row.

The documentation involved indicating the nursing diagnosis as initiated, ongoing, or discontinued. The Nursing Outcomes (NOC) were charted using a five-point scale to establish the patient at a particular outcome level. The level chosen would appear in the flowsheet charting cell, which allowed for the ability to track the patients’ progress over time. Approximately one year after instituting the electronic care plan flowsheet, the nurses requested that the nursing diagnosis rows and outcomes be moved from the care plan flowsheet to the assessment/intervention flowsheet. This would eliminate the need to navigate between flowsheets, thus making it more convenient and less time-consuming. The nurses reported that it helped them to see the “big picture” as related to the patient and the effectiveness of their nursing care. The combined flowsheet has proven to be an excellent model of integration of the care planning process with the patient data.

The combined flowsheet reached its character limitations at the introduction of nursing diagnoses and outcome labels. Consequently, NOC indicators could not be included on the flowsheet, which proved to be an issue with the introduction of the NOC study. Additional labels were added to accommodate the study, but the indicators could not be added. It was necessary to use paper for this portion of the study in order to rate the NOC label and indicators.

Staff Preparation

Staff education and preparation was crucial to meaningful study participation and success. The Registered Nurse Research Assistant (RNRA) held meetings with ICU unit leadership members (Nurse Manager, Nursing Education Specialist and Clinical Nurse Specialist) to familiarize them with the study expectations. Each unit’s patient care environments, routines, and learning preferences were assessed in relation to the study.

The educational in-services developed were entitled “Do You Have a Knack for NOC?”. The educational in-service reviewed and discussed Standardized Nursing Language and how it is incorporated into nursing practice guidelines and care planning. After familiarizing staff with the purpose and design of the study, staff had the opportunity to practice completing NOC label instruments according to the assessment information presented in a case study. Each staff member practiced comparing their rating results with a colleague and calculating the percent of their results in agreement ( figure 1). This Interrater Reliability Testing exercise served to assess the reliability of the NOC instrument. Participants were instructed to rate the labels upon admission to the unit and again at the time of transfer in order to measure instrument sensitivity. All of these measurements were to be done in conjunction with regular patient care.

The RNRA and unit leadership members implemented two educational strategies to prepare staff for the study. In the Cardiac Transplant ICU, several one-hour optional in-services were offered. The Cardiac Surgery ICU staff received the in-service during the patient care unit’s team meeting. Those unable to attend the in-services received individual education on a one-to-one basis. If staff attended the education, their willingness to participate in the study was evident.

Study Implementation

On the first study patient care unit, Cardiac Transplant ICU, three-ring binders with tabbed NOC study instruments were placed in the patient’s anteroom. Abbreviated directions and examples of the completed NOC label instruments were placed in the binders, on unit bulletin boards and in patient rooms. Reminder cards were positioned on the computers, in patient care plans, and in the medication/supply rooms ( Figure 2). A large NOC study poster displaying the status of the study was situated in an area where it would be highly visible and updated weekly to serve as a reminder and to foster team spirit ( Figure 3). Periodic communication with staff occurred via e-mail and included information regarding the progress of the study and suggestions for problem solving. The RNRA remained as accessible to the participants as possible and solicited volunteers from the staff to serve as resources during off-hours.

Each patient admitted to the ICU from surgery received a research number from the unit secretary. The nurse selected two to three outcomes after the initial assessment of the patient. Then a colleague was enlisted to rate the same outcomes within a half-hour of admission, or as soon as possible. The nurses compared the ratings and calculated the percentage of agreement which is the inter-rater reliability (IRR).

The RNRA also performed a predetermined measurement, as close as possible to the staff label rating, to establish instrument validity. Each NOC label has its own unique criterion measurement (predetermined by the University of Iowa NOC research team) that required either a patient assessment or gathering information from the patient's chart. The selected and rated NOC instruments were kept on the chart or in the care plan until the second rating at the time of transfer from the ICU.

We learned from the first patient care unit’s participation that NOC label completion was significantly hampered by the amount of time needed to select, look for, take out, and file the completed study instruments. As a result, a list of commonly used NOC labels were selected by a representation of unit staff and leadership for the second study patient care unit, Cardiac Surgery. The labels that were not applicable for the patient population were removed. Several NOC labels were pre-selected and placed on the patient's chart for nurse completion.

Study Implementation Limitations

The staff struggled with not having a standard reference in which to compare their patient. Many participants felt that the rating of the NOC labels and indicators was too subjective, leading them to question instrument reliability. Rating the NOC labels with a colleague was time consuming. When patient acuity increased, study participation decreased, the process was forgotten or paperwork was misplaced. Lastly, the staff was not familiar with the wording for some of the indicators.

Study Implementation Successes

The staff had an opportunity to closely reflect upon their nursing practice and visualize how nursing interventions directly impacted patient outcomes. The staff had an opportunity to participate in a large multi-site nursing research project and gained more exposure to the concept of Standarized Nursing Language, especially as related to Nursing Outcomes Classification. And lastly, staff was able to participate in label development. As a result of this research process two new labels were identified: "Adult Mechanical Ventilation" and "Adult Ventilatory Weaning Responses".


The most vital role that staff played in this study was providing input and testing the reliability of the outcome instrument. The staff also provided input as to the appropriateness of some outcome indicators. Though an outcome label seemed to apply to their patient population, some of the associated indicators did not. The IRR results are reported in two modes: first, the average IRR, which is how much agreement there is between raters as they rate the label and its qualifying indicators (within the 1-5 scale). The second mode is the absolute IRR, which is how much exact agreement there is between raters on the outcome label only. Finally, the average number of indicators for rating outcome labels will be shared.

Average IRR Cardiac Transplant ICU

Data was collected on 42 outcome labels. Thirty of the 42 outcomes had an average IRR coefficient of 85% or above. The top five outcome labels were "Blood Transfusion Reaction Control", "Nutritional Status: Nutritional Intake", "Safety Behavior: Fall Prevention", "Self Care: ADLs, and Thermoregulation"

Average IRR Cardiac Surgery ICU

Data was collected on 30 outcome labels. Twenty-five of the 30 outcomes had an average IRR coefficient of 85% or above. The top five outcomes were "Communication Ability", "Self Care: ADLs", "Tissue Integrity: Skin and Mucous Membranes", "Tissue Perfusion: Pulmonary, and Wound Healing: Primary Intention".

Percent with Absolute Agreement on Label Rating Cardiac Transplant ICU

Data was collected on 42 outcome labels. Sixteen of the 42 outcomes had an absolute IRR rating of 85% or above. The top five outcome labels were "Ambulation: Walking, Mobility Level", "Neurological Status: Cranial", "Tissue Perfusion: Cerebral, and Self Care: ADLs".

Percent with Absolute Agreement on Label Rating Cardiac Surgery ICU

Data was collected on 30 outcome labels. Six of the 30 outcomes had an absolute IRR rating of 85% or above. The top four outcome labels were "Fluid Balance", "Mobility Level", "Self-Care: ADLs", and "Tissue Perfusion: Abdominal".

The staff was asked to examine the qualifying indicators of each NOC label as they rated them. If they felt an indicator did not apply to this patient population or if an indicator did not make sense to them, they were asked to rate the indicator as not applicable and, when possible, provide rationale as to why they chose not to rate the indicator.

Average Number of Indicators used for Rating Outcome Label Cardia Transplant ICU

The top four outcomes with 80% or more of its indicators used were "Communication Ability", "Neurological Status", "Tissue Integrity: Skin and Mucous Membranes", and "Vital Sign Status". The lowest indicator use outcomes (with 51% or less of its indicators used) were "Nutritional Status: Food and Fluid", "Respiratory Status: Ventilation", "Tissue Perfusion: Cardiac", and "Safety Behavior: Fall Prevention".

Average Number of Indicators used for Rating Outcome Label Cardiac Surgery ICU

The top five outcomes with 87% or more of its indicators used were "Blood Transfusion Reaction Control", "Knowledge: Medications", "Pain Level", "Vital Sign Status", and "Nutritional Status: Nutritional Intake". The lowest indicator use outcomes (with 51% or less of its indicators used) were "Comfort Level", "Infection Status", "Tissue Perfusion: Cardiac", and "Safety Behavior: Fall Prevention".


The experience of being part of a clinical test site for outcome classification nursing research enabled nursing staff to re-examine and discover unique elements of critical care nursing practice. By re-examining the process of patient assessment step-by-step, it was discovered that nurses’ skillful assessments, interventions and resulting outcomes could be described with a standardized vocabulary rating patient outcomes.

Although patient assessment and resulting nursing interventions seemed like a highly subjective process, through this research process, the participants discovered they were using similar criteria to rate their patients.

Standardized nursing language plays a role in validating the effects of nursing interventions. Standardized nursing language is essential to ensure representation of nursing care effectiveness in today’s health care environments. Much more research is needed in the future to validate the use of standardized language in patient care.


Johnson M, Maas M, editors. Nursing outcomes classification (NOC). St. Louis: Mosby-Year Book, Inc., 1997.

Authors’ Bios

Kathy A. Demmer MS, RN

Ms. Demmer has been an obstetrical care nurse for 15 years at the Mayo Clinic-Rochester. She also has 10 additional years of experience in the areas of public health, nursing research (standardized nursing language testing), quality improvement and Parish Nursing. Kathy has served as Adjunct Faculty for Winona State University and in addition to her bedside care, instructs her colleagues in electronic nursing care documentation.

Cheryl L. McKane, MS, RN, CCRN

Ms. McKane has been a critical care nurse for 24 years at the Mayo Clinic-Rochester. Her clinical area of expertise is cardiovascular surgery and transplantation. For the past 15 years, Cheryl has been working in her specialty as a Nursing Education Specialist in the area of Education and Professional Development. In 2000, Cheryl was given the academic status of Assistant Professor in the Mayo Medical School. She has expertise in the area of critical care orientation, teaches in Mayo’s Essentials of Progressive and Intensive Care Program (EPIC), and has presented internationally, nationally and locally.

Linda Griebenow MS, RN

Ms. Griebenow is an Informatics Nurse Specialist at the Mayo Clinic-Rochester. She is known for her work in the development and implementation of electronic documentation tools for nursing and has lectured on the topic of: NOC Implementation and Testing, Electronic Plan of Care/Kardex Development and Paving the Way for Electronic Nursing Documentation at a Large Multidisciplinary Practice. Linda is also recognized for her development of the electronic Plan of Care for nursing and current work on the Computerized Physician Order Entry. She is also a recipient of the Excellence Through Teamwork Award at Mayo Clinic-Rochester.

Julia G. Behrenbeck RN, MS, MPH

Ms. Behrenbeck received her BSN from the University of Wisconsin-Oshkosh and her MS in Nursing and MPH in Information Management for Human Services from the University of Minnesota-Twin Cities. She has worked at the Mayo Clinic-Rochester for 17 years: 11 years as an Informatics Nurse Specialist and 5 years as Director of Nursing Information Operations. Her expertise is in the development and implementation of clinical information systems and information technologies to support nursing.

Jane A. Timm, MS, RN

Ms. Timm has been an Informatics Nurse Specialist at Mayo Clinic-Rochester since 1994. Her background in nursing includes general care and OB staff nursing, as well as positions in Infection Control and nursing management. Ms. Timm has worked with the development and implementation of clinical systems in critical care, general care, OB, and outpatient clinic settings. She has been a co-investigator in research on outcomes classification terminology and has held leadership positions on the North American Nursing Diagnosis Association and the CareCast User Exchange (formerly LINX). Ms. Timm is currently the clinical lead on the IDX CareCast Flowsheet team and is one of the nursing leaders in implementation of multiple CareCast modules as part of the Hospital Charting Package at Mayo Clinic-Rochester. She is also the facilitator for the Outpatient Nursing Documentation initiative at Mayo Clinic-Rochester.