Introduction

            Complex cognitive tasks, such as learning to use a computer simulation, can be difficult to analyze because of the number of factors that can affect an individual's learning and performance outcomes (e.g., previous education, prior computer experience, motivation, simulation usability and learnability). Prietula, Feltovich and Marchak (2000) recently proposed a new conceptual model to facilitate the analysis of this class of complex cognitive tasks.  Their model is adapted from one originally developed by Jenkins (1979) to characterize the kinds of variables that influence laboratory studies of human memory and guide researchers in planning research and interpreting the results.   Prietula et al. suggest that four categories of task factors may influence the analysis of complex cognitive tasks:  task goals, knowledge characteristics, task materials, and strategies.  Task goals are defined as what the user is trying to accomplish in the task.  Knowledge characteristics are defined as the particular kind of knowledge demanded by the task and depend on the user's background and experience.  Task materials are defined as characteristics of the materials the user has to work with, including the characteristics of the problem (e.g., well-structured or ill-structured) and its structure, environmental constraints, and available tools (e.g., the interface design).  Finally, strategies are defined as the goal-directed activities that the user applies to solve the problem (e.g., allocation of attention, pattern recognition, knowledge access, and interaction with task materials). The model has been used to explain a variety of aspects of cognition (Hoffman, Feltovich & Ford, 1997; Honek & Temple, 1992) and serves as the conceptual framework for the research reported here.  In this paper, we describe the results of research evaluating the interaction of two of the model's components, task materials (computer displays) and task goals.

         In previous research, Effken and colleagues developed several complex computer simulations of hemodynamics using design principles derived from ecological psychology’s theory of perception and action (Effken, 1993; Effken, Kim, Kadar & Shaw, 1992; Effken, Kim & Shaw, 1994, 1997).  Subsequent empirical tests showed that the etiological potentials display (EPD) consistently improved the speed and accuracy of novice and expert clinicians’ problem identification and treatment when compared with their performance using an integrated balloon display (IBD) or a traditional strip-chart display (TSD).  In a recent study, Effken & Doyle (in press) investigated the effects of cognitive style on nursing students' performance outcomes with each display.  As in the previous studies, variation was observed in both learning and subsequent performance with the displays across participants and experiments that could not be explained by gender, age, prior education, experience or cognitive style. For example, qualitative differences in performance were seen when the experimenter-defined goal varied across different experiments.

         As any teacher or researcher will attest, not all learners readily adopt the instructor's goal, nor do all experiment participants follow the specific directions of the experimenter.  Even when they adopt the same overall goal, individual learners may intentionally or unintentionally adopt very different sub-goals, or ways of fulfilling their intentions (Biggs, 1987; Howie, 1996).  In previous studies, we sometimes asked participants to solve the problems as quickly as they could, while using as little drug as possible.  Novices tended to try to optimize speed, at the expense of drug usage.  Only experts were able to respond to both goals simultaneously. In those experiments, the differences were simply treated as "noise" or variability in the data.  The purpose of the present research was to investigate systematically how task goal interacts with interface design to affect student nurses' abilities to learn to use the a computer simulation and their subsequent problem-solving performance using the simulation.

         Research question and hypotheses.  The general question to be answered in this study was: To what extent are differences in students’ abilities to learn to use an instructional computer simulation presented via two different interface and their subsequent problem-solving performance with the simulation moderated by treatment goals?  To assess performance, we measured diagnostic accuracy and treatment speed, operationalized as time to initiate treatment, time to reach target range, percentage of time in target range, number of scenarios corrected, and number of drugs used.  Based on our previous results, we generated the following hypotheses: 

1.      Diagnostic accuracy and treatment speed of participants using EPD will exceed that of participants using IBD.  

2.      Experimenter-defined goals will affect performance with the displays, particularly in terms of time to initiate treatment and number of drugs used, but performance differences will be mediated by display type. 

Methods

            Participants.  Sixteen undergraduate students (15 females and 1 male) enrolled at the University of Arizona College of Nursing were recruited to participate in the experiment.  Participants were required to have normal or corrected-to normal visual acuity (self reported) and normal color vision (tested by asking participants to point to red and green areas of the display).  Participants were randomly assigned to one of two display groups, IBD or EPD.  This assignment resulted in the IBD group including 1 pre-nursing student, 5 first year and 2 second year nursing students and the EPD group including 2 pre-nursing students, 3 first year and 3 second year nursing students.  

         Displays.  We presented the simulations on a WinNT workstation with a 17” color monitor. Guyton’s (1980) computer model of hemodynamics, which shows the intrinsic constraints on pressure and flow, serves as the basis for the simulations.  The model is simple, but includes the components thought to be most critical by clinicians (preload, afterload, and contractility).  In addition, the physiological relationships in the model have been well validated clinically by Guyton and his colleagues.  To create a scenario (“illness”), the experimenter changes the values of the control parameters in the Guyton model (e.g., contractility, resistance and fluid volume).  The simulations depict four output variables of the Guyton model (cardiac output, and arterial, venous and right atrial pressure) in two different visual formats, an integrated balloon display (IBD) and an etiological potentials display (EPD). 

         The computer interface uses two windows.  The simulated patient parameters are shown in the main window and ‘drug’ options in another window.  Drugs are the same for both displays.  Drugs act directly on the etiological factors (resistance, heart strength and volume).  No drugs affect more than one factor. Participants select a desired drug and dose (high, medium low), then press a mouse button to give discrete ‘drug’ doses.

In IBD, mean arterial pressure, right atrial pressure and central venous pressure are represented by three balloons that expand and contract horizontally (Figure 1) as pressures increase or decrease.  Increased or decreased cardiac output is shown by changes in the vertical dimension of a "bellows-like" object.  Small bar graphs superimposed over the balloons and the bellows show normal and high and low danger areas for each. The student's objective is to administer drugs until all pressures and cardiac output are within the normal range.  The lower part of the main window contains a reflected bar graph showing the overall status of the system (computed as the absolute value of the distances of the three pressures and cardiac output from normal). When participants select a drug and a dose, the name of the drug and the dose are shown on the "bottle" at the upper left of the screen.  For a more detailed description, see Effken et al, 1997.

   Refer to Figure 1

In EPD (Figure 2), the three pressures and cardiac output are indicated by the corners of a four-sided object (box).  The box moves through a space defined by two large red bars representing resistance (the horizontal dimension) and contractility (the vertical dimension).  The central crossing point for the two bars represents the optimal value for each.  A shrinking or expanding "box" shows fluid volume changes.  Very small bar graphs over the corners of the box indicate target ranges for each pressure and cardiac output. Green areas represent the target area; high and low danger areas are black.  When all pressure and cardiac output values are normal, the box approximates a square of a particular size positioned at the intersection of the red bars.

   Refer to Figure 2

            The computer records values for the three pressures and cardiac output, heart strength, resistance and volume, as well as the amount and strength of each drug used and the number of key presses made, at 1-second intervals.

         Design and procedure.  Display type was controlled between subjects and task goal, scenario and trial within subjects in the 2 (display type) x 3 (goal) x 3 (scenario) x 2 (trial) mixed experimental design. Two displays were compared, IBD and EPD.  Three goals were assigned by the experimenter: 1) correct the problems as quickly as possible (speed), 2) correct the problems with as little drug therapy as possible (drug), and 3) correct the problems (correct). Students were presented with three test scenarios developed previously with the help of clinical experts to simulate common hemodynamic problems (hypertension, heart failure and hypovolemia).   Each scenario was presented twice in each goal condition, in fully randomized order. There were three sets of experimental trials, one for each goal.  The first set of trials was always with the instruction simply to correct the problems.  The order of the second and third sets of trials (speed and drug goals) was counterbalanced.

            After reading and signing a disclaimer, participants were given written instructions that explained the purpose of the experiment, gave a brief description of hemodynamics, and described the display.  The experimenter then initiated the display and demonstrated the kinds of changes the model can undergo. After reading a description of the drugs, participants practiced using them (with the display in a normal state) until they stated they felt comfortable using these “tools.”  Participants then were given two practice problems that were repeated until each had been solved twice.  Participants were tested individually, in the presence of the experimenter.

         Data analysis.  Means and standard deviations were computed to summarize group scores for each of the dependent variables.  Separate mixed design ANOVAs were used to compare the effect of goal, display type, scenario, and trial on: 1) time to initiate treatment, 2) percentage of problems solved, 3) number of drugs used, and 4) percentage of time the patient system was maintained within target range. 

Results

Refer to Table 1

            Percentage of problems solved. The results are summarized in Table 1. Overall, participants solved 97% of the problems they viewed with EPD, but only 73% of those viewed with IBD.  In addition, there was much more variability among the students using IBD than those using EPD (Mean SDs = .36 and .05, respectively). Students solved 83% of the problems in the baseline "correct problems" condition, 86% in the "as fast as possible" condition, and 88% in the "minimize drug" condition. A 2 (display) x 3 (goal) x 3 (scenario) x 2 (trial) ANOVA found only a significant main effect of display, F₁ = 14.32, p < .01.

         Time to initiate treatment.  Mean treatment initiation time differed little across the two display groups (M =  7.9 s for IBD and M = 7.3 s for EPD). In the "as fast as possible" goal condition (M = 6.81 s), participants started treatment 1 s (IBD) to 2 s (EPD) earlier than in the baseline "correct" condition (M = 8.26 s).  In the "as little drug as possible" condition, students initiated treatment about the same time or even later (M = 8.38 s) than in the baseline condition. A 2 (display) x 3 (goal) x 3 (scenario) x 2 (trial) ANOVA revealed only a main effect of goal, F₂ = 10.67, p = .00.  When the "correct" condition was eliminated from the ANOVA, an interaction of display type and goal emerged, F₁ = 5.05, p < .05.  A post hoc Tukey test showed that in the "speed" condition, students using EPD were significantly faster at initiating treatment than students using IBD.  In the "drug" condition, the two display groups did not differ. 

            Percentage of time the system was maintained in target range. Students using IBD maintained the patient in target range considerably less time (M = 46%) than students using EPD (M = 80%).   In the baseline "correct" goal condition, students maintained the patient in target range 57% of the time, while in the "as fast as possible condition" and "minimize drugs" conditions they maintained the patient's parameters in target range 68% and 64% of the time, respectively. Because the baseline condition was always presented first, the difference between that condition and the test conditions, which were presented in counterbalanced order, is most likely a learning effect.  Students were faster, on average, on the second trial with each scenario, further evidence of a learning effect.  A 2 (display) x 3 (goal) x 3 (scenario) x 2 (trial) ANOVA revealed significant main effects of display, F₁ = 30.31, p = .00; goal, F₂ = 7.5, p< .01; and trial, F₂ = 6.47, p < .05.        

            Number of drugs used.  Students using IBD used more drugs to correct problems (M = 3.31) than students using EPD (M = 1.77).  In the baseline "correct problem" condition, students used more drugs (M = 3.01) than in either the "as fast as possible" (M = 2.39) or "minimize drugs" (M = 2.23) conditions. The difference in the baseline and test conditions is again most likely a learning effect.  In the "drug" condition, IBD users decreased their drug use from 3.88 in the baseline condition to 2.92 while EPD users went from 2.13 (baseline) to 1.54.  Both groups also reduced their drug use in the "speed" condition, but not by as much.  A 2 (display) x 3 (goal) x 3 (scenario) x 2 (trial) ANOVA revealed significant main effects of display type, F₁ = 12.34, p < .01; goal, F₂ = 9.83, p = .001; and scenario, F₂ = 5.30, p < .05.  The latter is expected because different amounts of drugs are required for solving each problem (scenario).

Discussion

         The major effect of experimenter-imposed goals to maximize speed or minimize drug use was on how quickly treatment was begun.  In the "speed" condition, participants started treatment 1-2 seconds earlier, whereas in the "drug" condition, they initiated treatment about the same time or even later than in the baseline "correct" condition. When only the "speed" and " drug" conditions were considered, the ANOVA revealed a significant interaction of display type with goal.

         The lack of difference in drug use in the two experimenter-assigned goal conditions is somewhat surprising. It may be that the students in this experiment had already internalized a goal of using few drugs, either on the basis of their previous education or as a constraint imposed by both displays.  Another possible reason for the lack of effect of goal on other performance measures is the fact that learning to use IBD requires considerable practice.  Students using IBD solved only 73% of the problems and varied widely in their performance levels (SD = .38).  Although students had to meet a performance criterion (solving two problems with the display) before beginning the experimental trials, this may not have ensured that they had enough skill to vary their performance to meet specific goals.  In contrast, EPD requires very little practice to learn; as a result, students using EPD solved over 97% of the problems with little variability (SD = .05).   Because they had already developed a degree of comfort with the display, students using EPD had sufficient skill to be able to optimize different goals.  A second potential reason for the lack of effect on number of drugs used is that the two goal conditions are not truly independent.  That is, minimizing drug use in these problems can also improve the speed of solution.  Each problem can be solved most efficiently by using only one drug.  As a result, by adopting a goal of using fewer drugs, students could also improve their speed.  

Conclusion and Implications

            Our previous studies showed effects of a number of user characteristics (gender, level of expertise, and cognitive style) on students' learning to use a computer situation.  However, that research also showed variability in performance that was unexplained by gender, level of expertise, age or cognitive style.   The goal of this study was to investigate explicitly one possible source of that variability: task goal.

         We investigated the effects of task goal on students' abilities to learn to use a computer simulation. The simulation presented fundamental principles of hemodynamics in two different visual formats: IBD and EPD.  Students used a mouse to select and administer simulated "drugs" to treat simulated clinical hemodynamic problems.

         Our manipulation of goal was at the level of sub-goal.  The overall goal was always to solve each problem.  The experimenter-defined sub-goals simply constrained how the overall goal was to be attained, that is, as quickly as possible, or with as little drug therapy as possible.

         Goal did not affect the number of problems solved, but did affect the strategies students used to solve them.  When asked to optimize speed, students initiated treatment earlier; when asked to limit drugs, they tended to start treatment later and used fewer drugs (although not significantly fewer than in the "correct the problem" condition).  The results also showed that display made a difference on how well students could meet the experimenter's goals.  Because students learned to use EPD so easily, their performance in the "correct" condition already was very high.  In the first set of 2-minute trials, they quickly (M = 42 s) corrected 96% of the problems and used an average of 2.13 drugs (an optimal solution requires one drug).  Because of this, these students had little margin for improvement (a ceiling effect).  Still, the students took only about half as long (M = 24 s) to solve the problems in the speed condition.  In contrast, some of the students using IBD were still struggling to master the display throughout the experiment, as shown by their solving only 73% of the "speed" and 77% of the "drug" trials.  Because the students were still learning, they lacked the flexibility to respond to additional experimenter goal constraints.  Even so, IBD users cut their time for solving problems from 87 s in the control condition to 63 s in the speed condition and to 73 s in the drug condition.  The difference between the speed and drug conditions represents an 11% change for IBD users (based on their solution time in the control condition) in contrast to a 33% variation shown by the EPD users.

         Using the Cognitive Task Model as a framework, we investigated the interaction of task materials (interface design) and task goals, while controlling knowledge characteristics and found an effect on the fourth factor in the model (strategies).  Task materials (specifically, the computer interface design) affected all performance variables.  This is consistent with our previous research, as well as with findings in studies involving a variety of complex cognitive tasks (Hoffrage & Gigerenzer, 1996; Larkin & Simon, 1987; Payne, Bettman & Johnson, 1988; Roy & Lerch, 1996; Zhang, 1997; Zhang & Norman, 1994).  Task goal affected how quickly treatment was initiated. This is consistent with research reported by others (e.g., Baron & Galizio, 1983; Pilgrim & Johnston, 1988) showing that even subtle orienting instructions can cause substantial performance differences.  Speed was also moderated by interface design.

         Limitations.   The two goal conditions used were not completely independent.  To the extent that the drug used is the drug needed to solve the problem (a minimizing drug use solution), the problem is likely to be solved faster because additional unneeded drugs simply create additional disturbances that then have to be corrected. 

         We looked at the effect of goal in one computerized simulation task with two interfaces only.  In addition, our sample size was relatively small and was restricted to novices.  Thus we cannot generalize the results beyond the conditions of this experiment.
         Implications.  From a research standpoint, our results support previous data showing that even subtle differences in a researcher's instructions can have a significant effect on the outcome of the research (Baron & Galizion, 1983; Pilgrim & Johnson, 1988).  Researchers have two options to deal with this issue.  The first is to make every effort to ensure that each participant in the research is given precisely the same instructions and is working to achieve precisely the same task goal.  It may even be necessary to remind participants of the goal throughout the experiment.  The second option is to make task goal an independent variable in the study and manipulate it systematically.  The latter may ensure a more detailed view of problem solving under a variety of goal conditions (Preitula et al., 2000).  This approach may be particularly useful in evaluating the impact of technology.  However, we should emphasize that it is difficult for novices to optimize different task goals until they have considerable competence using the tools and in the task itself.  In the experiment reported here, we found a significant main effect of trial on treatment initiation time; and performance improved under both speed and drug goal conditions over the initial baseline condition.  Both these results suggest that learning was still taking place throughout the experiment, particularly for those students using IBD.  Finally, we found a display-goal interaction for treatment initiation time.  Students were able to learn to use EPD more quickly and were therefore able to vary their performance in response to goal.  Some students in the IBD condition were struggling with the display (as evidenced by the number of failed trials) throughout the experiment.  For them, varying performance to meet the experimenter's demands was simply not possible. 

         Many human-computer interaction models have been proposed (e.g., Card, Moran, & Newell, 1983; Clarke, 1986; Jacgodzinski & Clarke, 1988).  We found the Model of Cognitive Tasks (Preitula et al., 2000) provided a useful, generic framework for organizing the analysis of data in a complex cognitive task, learning to use a computer simulation.  The model has some similarity to the Nurse-Computer Interaction Framework (Staggers & Parks, 1995), but does not include context and is not a developmental model.  Because it includes only four categories, it has intuitive appeal for the researcher looking for a simple model.  However, each of those categories requires subcategories to operationalize it, so the model is not as simple as it initially seems.  It is likely that the model can provide a framework for a variety of informatics-related research.  In any applied research study there are many degrees of freedom to be controlled if analysis is to be possible.  It seems likely that any research dealing with complex computer activities will have to take into account the relevant degrees of freedom pertaining to each factor in the model: task materials, strategies, knowledge characteristics, and goals. 

 

Acknowledgements

This research was supported by a grant to the first author from the Emmons Foundation at the University of Arizona College of Nursing and a dissemination grant from Beta Mu Chapter, Sigma Theta Tau.