Evaluation of the Effectiveness of In-Service Training Based on the Kirkpatrick Model: A Case Study of a Cardiopulmonary Resuscitation (CPR) Course for Nurses in Afzalipour Hospital, Kerman, Iran

AUTHORS

Mohammad Reza Amiresmaili 1 , Moghaddameh Mirzaee 2 , Mohsen Aminizadeh 3 , Rohaneh Rahimisadegh 4 , *

AUTHORS INFORMATION

1 Medical Informatics Research Center, Institute of Futures Studies in Health, Kerman University of Medical Sciences, Kerman, Iran

2 Department of Biostatistics and Epidemiology, School of Public Health, Kerman University of Medical Sciences, Kerman, Iran

3 Health in Emergency and Disaster Research Center and Department, University of Social Welfare and Rehabilitation Sciences, Tehran, Iran

4 Health Services Management Research Center, Institute of Futures Studies in Health, Kerman University of Medical Sciences, Kerman, Iran

ARTICLE INFORMATION

Strides in Development of Medical Education: 15 (1); e89066
Published Online: January 28, 2019
Article Type: Research Article
Received: April 23, 2017
Revised: March 10, 2018
Accepted: April 8, 2018
Crossmark

Crossmark

CHEKING

READ FULL TEXT
Abstract

Background: Training of human resources, especially nurses, is a profitable investment for hospitals with major economic return if properly planned and implemented.

Objectives: The present study aimed to evaluate the effectiveness of a cardiopulmonary resuscitation (CPR) course as an in-service training program, based on the Kirkpatrick model.

Methods: This interventional study was conducted at Afzalipour Hospital of Kerman, Iran from October 2014 to May 2015. The study sample consisted of 45 nurses, including 20 nurses in the case group and 25 nurses in the control group. The case group participated in a four-hour CPR training workshop. The Kirkpatrick model was used to determine the effectiveness of the CPR course. Data were collected using three questionnaires and hospital records.

Results: The participants were satisfied with the training course, and a significant difference was observed in the mean score of three intervals of learning levels evaluation (P < 0.0001). Based on the findings, CPR training affected the learning level of nurses from the case group; however, the average learning score was not significantly different between the two groups (P = 0.26). In addition, the difference in the mean score of behavior level was not significant before and after training (P = 0.91). The results of Chi-square test also showed that CPR training did not affect the forth level (P = 0.54). Finally, the overall effectiveness of the CPR training course was estimated at 32.51%.

Conclusions: This study indicated that effectiveness of in-service training is not at a desirable level. Since organizations allocate a lot of their resources to such training courses every year, it is essential to reconsider planning and implementation processes.

Keywords

Evaluation Education Effectiveness Cardiopulmonary Resuscitation Kirkpatrick Mode

Copyright © 2019, Strides in Development of Medical Education. This is an open-access article distributed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/) which permits copy and redistribute the material just in noncommercial usages, provided the original work is properly cited.
1. Background

Training and development of human resources is a profitable investment, which can have major economic return for organizations provided if that it is properly planned and implemented (1). Continuous in-service training is a strategy used to deal with uncertain, complex, and dynamic conditions of organizations through upgrading the employees’ knowledge and skills which quickly become obsolete (2). The importance of in-service training has been also highlighted for the nursing staff in the past few years (3). In fact, nurses should be familiar with the latest advances in clinical care because of their critical role in patient care (4). Accordingly, hospitals are investing a lot of time and money on nursing training each year (5).

Considering the importance and probable effects of training on organizations, besides the high cost of employee training and development, the effectiveness of educational programs, including nursing training courses, needs to be highlighted and evaluated (6). Despite previous studies on this subject (7-13), different aspects of in-service training remain underexplored, such as effectiveness, evaluation methods, instruments, and time of application (3).

There are controversies about the processes and assessment methods of training effectiveness. Various approaches and models have been proposed for evaluating the effectiveness of training (11). In this regard, Worthen and Sanders analyzed more than fifty evaluation models (10). Some of these models involve evaluation, while others only present a general view about assessment. These models are formed based on objectives and conditions, fundamental philosophical underpinnings, and certain viewpoints in the definition of evaluation. Each view pertains to certain aspects of evaluation and sets a specific pattern.

Several models have focused on the evaluation of learner’s behavior, training materials, or related methods (11). Some models for evaluating the effectiveness of training programs include Kirkpatrick model, Tyler’s objective model with a behavioral objectives approach, return on investment (ROI) model, context, input, reaction, and outcome (CIRO) model, success case method, and goal-free evaluation. Overall, selection of a suitable model for evaluation is very important, as different models can present different results (12).

Today, the most common method for the evaluation of training in organizations is the Kirkpatrick’s framework, consisting of four levels (13, 14): Reaction, learning, behavior, and results (15). Most previous studies in Iran have not used any specific models or frameworks and only evaluated knowledge (7), skills (8), or both (9, 10) pre- and post-intervention. On the other hand, some studies have applied models, such as Kirkpatrick model, for evaluating training effectiveness at four levels in the health sector (11-13).

Most of the mentioned studies have underlined the level of learning, while only a few have applied a broad approach incorporating higher levels of learning. Overall, effectiveness of nursing training programs remains unexplored when higher levels of the learning hierarchy are taken into account. Therefore, the present study aimed to evaluate the effectiveness of a cardiopulmonary resuscitation (CPR) training program, using the Kirkpatrick model.

2. Objectives

The results of this study can be helpful for managers and planers to promote CPR training courses in the future based on nurses’ needs and to improve ROI.

3. Methods

An interventional pretest-posttest design was used to evaluate the effectiveness of the CPR training course among nurses, based on the Kirkpatrick model in Afzalipour Hospital (Kerman, Iran) from October 2014 to May 2015. According to Kirkpatrick’s recommendation to incorporate a control group in the effectiveness evaluation of learning, behavior, and results levels (16), nurses were divided into case and control groups of approximately equal size (case group, 20; control group, 25).

The nurses of case group, were required to participate in the CPR course, which was held in form of a four-hour theoretical and practical workshop by experts at Afzalipour Hospital. On the other hand, nurses in the control group did not receive any CPR training; therefore, the learning and behavior levels in this group were only measured once, and there were no pretest-posttest comparisons. The inclusion criterion for nurses in the case and control groups, was lack of participation in a CPR training course in the past months.

The validity of the questionnaires was approved by the faculty members of nursing and emergency experts. The reliability of four-point scale questionnaire of learning level, was down by test-retest (ICC = 0.8). Moreover, to assess the reliability of the Likert scale, Cronbach’s alpha coefficient was measured to be 0.92, which confirms the reliability of the questionnaire.

The reaction level was not measured in the control group, as they did not participate in the CPR training course. On the other hand, the reaction level was evaluated in the case group immediately after the training course. The level of learning was measured three times in the case group, i.e., before, immediately after, and six months after the training course, and only once in the control group.

In the case group, the behavior level was evaluated in two intervals: Before the training course and six months after the course (16). On the other hand, the behavior level of the control group was only evaluated once. The number of successful CPR attempts at the hospital was considered as a measure of the results (forth) level. Therefore, the number of successful and unsuccessful CPRs was determined three months before and after training. To measure the total effectiveness score of the CPR training course, we first scaled the scores to 100 and then assigned a relative importance coefficient to each level. The overall effectiveness score was determined by measuring the average effectiveness score, corresponding to the relative importance coefficients.

A questionnaire was completed by each nurse from the case and control groups, containing demographic data, such as age, sex, marital status, educational background, work experience, and information about previous attendance in CPR courses (e.g., instructor and time of the course). In addition, three instruments were designed for evaluating three levels of the Kirkpatrick model.

The first questionnaire was designed to measure the reaction and satisfaction of nurses in the case group about the instructor, content, and amenities of the course. It consisted of 30 questions, scored on a five-point Likert scale, with one representing “strongly disagree” and five representing “strongly agree”. The highest score was 150, and the lowest score was 30; there was no reverse coding.

The second questionnaire measured the knowledge of nurses in both groups with regard to CPR. It included a total of 20 four-option questions. The participants were given one point for each correct response and zero for each incorrect response. The highest score was 20, while the lowest score was zero.

Finally, the third questionnaire aimed to measure the behavioral skills of nurses in both groups with respect to CPR. This questionnaire was completed before the training course and six months after the course in the case group and only once in the control group. It contained a total of 40 questions, scored on a two-point scale. The participants were given zero points for each incorrect response and two points for each correct response regarding the implementation of CPR. The highest possible score was 80, while the lowest score was zero.

After data entry, SPSS version 22 was used to analyze the data, using repeated measures ANOVA and Mann-Whitney test.

Regarding ethical considerations, an official letter was obtained from the Faculty of Management and Information of Kerman University of Medical Sciences, and permission was obtained from the administrators of Afzalipour Hospital for conducting the training course and evaluations. Also, informed consents were collected from the nurses. This study was conducted after explaining the study objectives, ways of collaboration, methods of data collection, and confidentiality of personal information (e.g., name) (ethical code: Ir.kmu.rec.1393.301).

4. Results

Comparison of the case and control groups in terms of individual characteristics showed that the two groups were similar with respect to different characteristics, including age, work experience, gender, marriage, frequency of attendance in CPR courses, educational level, place of service, and history of participation in CPR courses, and no significant difference was observed (P > 0.05).

Analysis of the relationship between the demographic characteristics of nurses and the mean scores of training evaluation levels showed the significant correlation of learning and behavior levels with history of participation in CPR courses and frequency of attendance in CPR courses in the case and control groups. Analysis of the reaction level revealed that nurses were very satisfied with the course, and the highest level of satisfaction was attributed to the content of the course (Table 1).

Table 1. The Reaction Level of Nurses in the Case Group
DimensionsMaximum ScoreMean ± SD
Content3531.6 ± 3.47
Instructor6554.85 ± 10.54
Amenities5034.9 ± 6.24
Total150121.35 ± 14.6

The learning level scores of nurses in the case group were 8.5 ± 2.74, 13.45 ± 2.28, and 10.15 ± 1.70 (out of 20) before, immediately after, and six months after the training course, respectively. The difference was significant according to the repeated measures ANOVA (P < 0.0001).

Based on the multiple comparisons and repeated measures ANOVA, it was clear that the differences in the scores of learning level were significant before and immediately after the course (P < 0.0001), before and six months after the course (P < 0.0001), and immediately after and six months after the course (P = 0.04). As the findings indicated, CPR training affected the learning level of nurses in the case group (Table 2).

Table 2. Comparison of Learning Level Scores in the Case Group at Three Intervals
IntervalsMean ± SDP ValueP Value for Multiple Comparisons
First IntervalSecond IntervalThird Interval
Before the training course (first interval)8.5 ± 2.74< 0.0001-< 0.0001< 0.0001
Immediately after the training course (second interval)13.45 ± 2.28--0.04
Six months after the training course (third interval)10.15 ± 1.7---

The results of Mann-Whitney test indicated that the mean scores of before the learning level were not significantly different between the two groups (P = 0.26). On the other hand, the mean scores of learning level were significantly different between the case group immediately and after the course (P > 0.0001) and six months after the course (P = 0.002) with the control group (Table 3).

Table 3. Comparison of Learning Level in the Case and Control Groupsa
IntervalsCase GroupControl GroupP Value
Learning level before the training course8.5 ± 2.747.4 ± 3.160.26
Learning level immediately after the training course13.45 ± 2.28< 0.0001
Learning level six months after the training course10.15 ± 1.700.002

aValues are expressed as mean ± SD.

The average scores of behavior level in the case group were 17.05 ± 12.42 and 17.2 ± 12 (out of 80) before and after training, respectively. The results of Wilcoxon test showed that the mean score of behavior level was not significantly different before and after training (P = 0.91). In addition, the mean scores of behavior level were not significantly different between the case group before (P = 0.94) and after (P = 0.77) the course with control group (Table 4).

Table 4. Comparison of Behavior Level in the Case and Control Groupsa
IntervalsCase GroupControl GroupP Value
Behavior level before the training course17.05 ± 12.4217 ± 13.10.94
Behavior level six months after the training course17.2 ± 120.77

aValues are expressed as mean ± SD.

The results of chi-square test showed that CPR training (number of successful CPRs) was not effective for hospital nurses (P = 0.54) (Table 5).

Table 5. The Results Level in the Case Group Before and After the Training Coursea
CPR PerformanceBeforeAfterTotal
Successful CPR16 (25)17 (21)33 (22.6)
Unsuccessful CPR48 (75)65 (79)113 (77.4)
Total64 (100)82 (100)146 (100)

aValues are expressed as No. (%).

The overall effectiveness score of the CPR training course was 32.51% that illustrated this course was not effectiveness (Table 6).

Table 6. The Total Effectiveness of the CPR Training Course
Levels of EvaluationMean Score After the Training Course in the Case Group (Scaled to 100)Relative Importance CoefficientMultiplied Percentage CoefficientPercentage of Overall Effectiveness of CPR Training Course
Reaction67 ± 10.4716732.51%
Learning49.58 ± 9.03299.16
Behavior25 ± 12.38375
Results21 ± 3.69484
5. Discussion

The current study aimed to evaluate the effectiveness of a CPR training course among nurses, based on the four evaluation levels of Kirkpatrick model. The results indicated that satisfaction of nurses with the training course was acceptable. The highest level of satisfaction was related to the course content, while the lowest satisfaction was attributed to the course amenities. Our findings met our expectations, as we used the most recent educational content unlike most other courses, and our course incorporated practical training, in addition to theoretical education.

In this regard, the results of a study by Alinaghian et al. showed that the participants in the electroshock training course had the highest and lowest levels of satisfaction with the amenities and content of the training course, respectively; these results are reverse with our findings (17). In addition, the results reported by Mohan and Prasad revealed that nurses in the case group had 84% satisfaction with the course instructor in the reaction level, similar to our study (18).

The mean score of learning level showed a significant increase immediately after the course, while it decreased by 17% after six months. This can be explained by the fact that nurses try to pass the evaluation exam only to receive a certificate and discard everything after the course. The results of a study by Mokhtari Nori et al. also showed that the mean score was the lowest before the CPR training course, while the mean score was the highest immediately after the course. Meanwhile, the knowledge score showed a considerable decline two years after the training course (8).

In addition, Mohan and Prasad reported a significant difference in the nurses’ knowledge scores before and after the training course, which shows that training was effective and could improve the learning level of nurses (18). Mohamed and Alias in their study showed that the level of awareness and knowledge increased after a training course (19). Moreover, Ghorbanshiroudi et al. (2012) showed that the crisis-management training course was only effective in the learning level (20); similarly, the results of our study confirmed that the CPR training course affects the learning level.

In another study, Borimnejad et al. reported a significant difference in the mean scores between the evaluated intervals (i.e., before, immediately after, and a few months after the CPR workshop). These results indicate that although CPR workshops can meet the needs of nurses to some extent, it is necessary to repeat these workshops at regular intervals (21).

The mean score of behavior level in the case group was very low before the training course and not significantly different from the mean score reported six months after the course. This finding reveals that the behavioral skill of nurses is very low and has not improved. Also, the level of behavioral skills returned to the initial level after six months, as most CPR training courses held in hospitals are theoretical and do not integrate practical assessment. In fact, nurses showed poor performance in translating knowledge into practice and could not represent their knowledge in their behavioral skills. Based on the findings, the behavior scores were very low before and six months after the training course in both groups; the scores decreased to the baseline after six months.

Moreover, the results of a study by Mokhtari Nori et al. showed that the mean score of knowledge was at the lowest level before the CPR training course; meanwhile, it significantly decreased after two years. The results were similar for practical and theoretical skills, while the score of practical skills significantly decreased after two years (8). The low level of nurses’ knowledge and skills in CPR, effectiveness of training in knowledge promotion, and necessity of regular CPR training courses for maintaining the nurses’ knowledge and skills are the most important findings reported by Shahrakivahed et al. in 2015 (22), which are in agreement with our results.

A study by Hojjati et al. showed that the overall effectiveness of a training course for nurses based on the Kirkpatrick model was 85.14%, which is significantly different from the overall effectiveness reported in our study (23). In another study, Mollahoseini and Farjad showed that the score of results level in the Kirkpatrick model was 3.06 out of five, which is almost acceptable (20).

In our study, the number of successful CPRs in the case group not only did not increase after the training course, but also showed a 4% decline. This can be explained by the fact that holding CPR training courses could not improve the necessary skills for performing a successful CPR. Therefore, this training course is not effective at the behavior level and is certainly ineffective at the results level. Other possible explanations can be related to the nature of CPR, which is a team-based activity depending on the agility and interactions of team members.

There was a significant relationship between variables, such as history and frequency of participation in CPR training courses, and scores of reaction, learning, and behavior levels. Despite the poor effectiveness of the training course, nurses with a history of participation in training had higher scores in reaction, learning, and behavior levels, compared with other nurses. Therefore, we should increase the number of CPR training courses at hospitals and incorporate the practical aspects. In addition, nurses should exercise CPR using moulage kits to identify their weaknesses; in fact, the internalized CPR skills of nurses will increase the number of successful CPRs in hospitals.

In another study, Bakhsha suggested periodic training of practical skills for the staff (9). The findings of a study by Raeisi et al. also confirm the effectiveness of periodic training, which should be taken into account in the design of short-term and low-cost courses for promoting advanced CPR skills (24). According to the present results, it is necessary to hold such training courses continuously and practically for nurses. It is also recommended to perform a primary evaluation at the behavior level among nurses to identify errors and defects in behaviors and to focus on these issues in practice.

5.1. Conclusions

Based on the present findings, effectiveness of in-service training for nurses is not at a satisfactory level, especially at higher levels of Kirkpatrick model. Therefore, a major revision is necessary in the content and processes of these training courses, as well as the methods of evaluation. Traditionally, on-the-job training focuses on the reaction level for evaluating training courses, which does not seem to be effective. Unless the training and evaluation methods are revised, hospitals will lose their scarce resources by holding training courses without any practical outcomes. Additionally, if hospital managers and planers require up-to-date, creative, and motivated nurses, it is necessary to devise and practice suitable policies considering their prospects; these policies should focus on in-service training for nurses.

It is suggested to incorporate a successful case evaluation model (a qualitative model) and interview nurses, who have successfully applied their learning in the work environment, along with those who have been unable to apply it in a useful way for some reason; in addition, it is important to determine the factors which are effective or ineffective in training. Considering the importance of evaluating the effectiveness of educational courses for improving their quality, the faculty members of universities or even headquarters should evaluate the effectiveness of training courses at four levels in other sections of Kerman University of Medical Sciences, such as other hospitals.

Acknowledgements
Footnotes
References
  • 1. Eidi A, Alipour M, Abdolahi J. [Evaluating the effectiveness of educational cources]. J Tadbir. 2009;200(19):26-32. Persian.
  • 2. Bazargan A, Kharazi K, Jamalifar M. [The comparative study of continuing education for empowerment of human resources]. J Labour Soc. 2012;150:4-12. Persian.
  • 3. Griscti O, Jacono J. Effectiveness of continuing education programmes in nursing: Literature review. J Adv Nurs. 2006;55(4):449-56. doi: 10.1111/j.1365-2648.2006.03940.x. [PubMed: 16866840].
  • 4. Lin PC, Chiang HW, Chiang TT, Chen CS. Pain management: Evaluating the effectiveness of an educational programme for surgical nursing staff. J Clin Nurs. 2008;17(15):2032-41. doi: 10.1111/j.1365-2702.2007.02190.x.
  • 5. Habibi S, Rezaei Hachesoo P, Tabaghi R. [Enhancing information literacy as a base of developing evidence-based nursing]. Health Inform Manag. 2010;7(3):371-8. Persian.
  • 6. Blake SC, Kohler SS, Culler SD, Hawley J, Rask KJ. Designing effective healthcare quality improvement training programs: Perceptions of nursing and other senior leaders. J Nurs Educ Pract. 2013;3(5):66-77. doi: 10.5430/jnep.v3n5p66.
  • 7. Arthur W Jr, Bennett W Jr, Edens PS, Bell ST. Effectiveness of training in organizations: A meta-analysis of design and evaluation features. J Appl Psychol. 2003;88(2):234-45. doi: 10.1037/0021-9010.88.2.234. [PubMed: 12731707].
  • 8. Mokhtari Nori J, Saghafinia M, Kalantar Motamedi MH, Khademol Hosseini SM. CPR training for nurses: How often is it necessary? Iran Red Crescent Med J. 2012;14(2):104-7. [PubMed: 22737563]. [PubMed Central: PMC3372042].
  • 9. Bakhsha F. Assessing the need and effect of updating the knowledge about cardio-pulmonary resuscitation in experts. J Clin Diagn Res. 2010;4(3):2512-4.
  • 10. Khorasani A, Dosti H. [Measuring training effectiveness of in service engineering education by ROI methodology]. Iran J Eng Educ. 2012;13(52):103-22. Persian. doi: 10.22047/IJEE.2012.1799.
  • 11. Jafarzadeh M. [Return of capital, a tool for evaluating of educational effectiveness]. J Tadbir. 2007;18(180):35-9. Persian.
  • 12. Sabaghyan Z, Akbari S. Organizational comperehensive education (adult education approach). Tehran: Samt; 2010. 628 p.
  • 13. AlYahya MS, Norsiah BM. Evaluation of effectiveness of training and development: The Kirkpatrick model. Asian J Bus Manag Sci. 2013;2(11):14-24.
  • 14. Bates R. A critical analysis of evaluation practice: The Kirkpatrick model and the principle of beneficence. Eval Program Plann. 2004;27(3):341-7. doi: 10.1016/j.evalprogplan.2004.04.011.
  • 15. Khorasani A, Zarrabi HH. [Exploring the evaluation patterns reviewing the patterns of Kirk Patrick and Helton]. Proceedings of the First Virtual Training Sessions. Tehran, Iran. 2013. p. 1-12. Persian.
  • 16. Kirkpatrick DL, Kirkpatrick JD. Implementing the four levels: A practical guide for effective evaluation of training programs. 1st ed. San Francisco, California: Berrett-Koehler Publishers; 2009.
  • 17. Alinaghian M, Safdarian A, Alinaghian M. Evaluation of training course effect of working with electro shock device to nursing cadre of Feiz Therapeutic Educational Center based on Kirkpatrick model in 2014. Int J Rev Life Sci. 2015;5(7):651-7.
  • 18. Mohan DR, Prasad MV. Impact of training session on biomedical waste management for a group of nurses-an analysis. Narayana Med J. 2012;1(1):25-8.
  • 19. Mohamed R, Alias AAS. Evaluating the effectiveness of a training program using the four level Kirkpatrick model in the banking sector in malaysia. International Conference on Business and Economic Research. Bandung, Indonesia. 2012.
  • 20. Mollahoseini A, Farjad S. Assessment effectiveness on the job training in higher education (case study: Takestan University). Procedia-Social and Behavioral Sciences. 2012;47:1310-4. doi: 10.1016/j.sbspro.2012.06.817.
  • 21. Borimnejad L, Rasoli M, Nikbakht Nasrabadi A, Mohammadi Mohammadi H, Ahmadzadeh M. [The effect of cardiopulmonary resuscitation workshop on nurses’ sustained learning]. Iran J Med Educ. 2008;7(2):209-15. Persian.
  • 22. Shahrakivahed A, Masinaienezhad N, Shahdadi H, Arbabisarjou A, Asadibidmeshki E, Heydari M. The effect of CPR workshop on the nurses’ level of knowledge and skill. Int Arch Med. 2015;8(108):1-10. doi: 10.3823/1707.
  • 23. Hojjati H, Mehralizadeh Y, Farhadirad H, Alostany S, Aghamolaei M. [Assessing the effectiveness of training outcome based on Kirkpatrick model: Case study]. Q J Nurs Manag. 2013;2(3):35-42. Persian.
  • 24. Raeisi H, Torabizadeh K, Makoulati N. [Studying the effect of CPR knowledge and skill retention reeducation course on nursing students]. J Rescue Relief. 2012;4(1):11-9. Persian.
  • COMMENTS

    LEAVE A COMMENT HERE: