| Abstract|| |
INTRODUCTION: Hemovigilance is a series of surveillance involving the collection and evaluation of all kinds of unwanted and unexpected effects arising from the use of blood products, including the whole transfusion chain, including the follow-up after blood collection and prevention of re-occurrence.
OBJECTIVE: In this study, it is aimed to evaluate the learning effectiveness of the hemovigilance education program developed in Süleyman Demirel University Faculty of Medicine and the overall program.
MATERIALS AND METHODS: The study was designed as a quasi-experimental pretest posttest study in quantitative research design. In the evaluation of the program, pretest and posttest prepared by the research group were applied, and feedback was received regarding the training program.
RESULTS: The data of 114 participants who received hemovigilance training were evaluated in the study (n = 114). The pretest mean was calculated as 67.42 ± 12.43 (minimum: 31, maximum: 93), and the posttest mean was 86.38 ± 11.83 (minimum: 55, maximum: 100). The realization of learning was evaluated in the second step of the model. There was a statistically significant difference between the pretest and the posttest (P = 0.000). In addition to this finding, according to the generalizability theory, the item-time (pretest/posttest application) variance percentage was calculated as 32.5%.
CONCLUSIONS: In the current literature, many studies are planned to be developed for the development of hemovigilance activities. It is recommended to organize in-service training programs for most of these studies. In evaluating the measurement tool with the generalizability theory, it was determined that education is effective in increasing the level of knowledge, but it is not sufficient and should be improved in the sampling of the information universe because it is an easy level and weak discrimination tool. In evaluating the first step of the program evaluation model, it was seen that the majority of the participants expressed their satisfaction with education. The realization of learning was evaluated in the second step of the model. According to the generalizability theory, there was a statistically significant difference between the pretest and the posttest and the high percentage of variance in the item-time (pretest/posttest application) showed that the learning took place. We believe that the training program within the scope of the study significantly increases the level of knowledge of the participants and contributes to learning. In addition, a more valid and reliable measurement will be provided with the development of the measurement tool, and the program will be developed to contribute to hemoviral surveillance.
Keywords: Classical test theory, generalizability theory, hemovigilance, measurement, program development, program evaluation
| Introduction|| |
Hemovigilance is a series of surveillance that encompasses the entire transfusion chain starting from the collection of blood to giving it to the recipient and the follow-up process, which involves collecting and evaluating information about all kinds of undesirable and unexpected effects arising from the use of blood products, including prevention of these problems from occurring and recurring.
In our country, there are studies at the master's and doctorate level related to the level of knowledge of hemovigilance, in which the level of knowledge of hemovigilance was evaluated in various populations and the need for education was emphasized.,,,,,, In addition, in the “National Hemovigilance Guide” developed within the scope of “Technical Support Project for Strengthening the Blood Supply System,” organizing education in many areas is recommended such as traceability of blood and blood components, service and product standards, quality assurance, capacity building activities for administrative and technical personnel, and appropriate clinical use of blood for clinicians. To strengthen the blood supply system in line with these requirements, it is necessary to organize training programs and evaluate training activities for health professional working in health institutions. To measure educational effectiveness, studies with quasi-experimental designs and program development studies should be designed.
In this study, it was aimed to evaluate the learning effectiveness of the hemovigilance training program developed in the Faculty of Medicine of Süleyman Demirel University to assess the overall program.
| Materials and Methods|| |
The study was designed as a quasi-experimental pretest posttest study in quantitative research design. Six steps of Kern were employed in the design of the training program., For the training program, a research group consisting of 1 hematologist, 2 medical educators, and 2 hemovigilance nurses was formed. First of all, the problem was defined, and in the general need analysis performed, the lower number of reactions than expected that developed due to the transfusion of blood and blood products in Süleyman Demirel University Research and Application Hospital in 2019 was evaluated. In the analysis of the needs of the participants, the needs analyses in the current literature and the training needs of the research and application hospital staff were evaluated. The aim of the training program was determined “to improve the knowledge level of the healthcare professionals about the reactions developing as a result of the transfusion of blood and blood products and reporting them.” One-hour large group training sessions were preferred as a training strategy. The training was carried out in the training halls of the department. In the evaluation of the program, pretest and posttest prepared by the research group were applied, and feedback was received regarding the training program.
One of the most important applications in the development of training programs is the program evaluation. The first two steps of Kirkpatrick's program evaluation model were preferred as the program evaluation model for the development of the training program of the hemovigilance unit. In the first step, feedback was received related to the reaction of the participants, and pretest posttest was applied to evaluate the learning.
MSExcel, JASP (https://jasp-stats.org/) (free), and EduG (https://www.irdp.ch/institut/english-program-1968.html) (free) programs were used to evaluate the data. In the analysis of the data, descriptive analyses of the measurement tool, pretest-posttest comparison, reliability analyses of the measurement tool through classical test theory and generalizability theory, and descriptive analyses of the feedback scale for program evaluation were performed.
| Results|| |
In the study, the data of 114 participants who received hemovigilance training were evaluated (n = 114). The pretest mean was calculated as 67.42 ± 12.43 (minimum: 31, maximum: 93), and the posttest mean was 86.38 ± 11.83 (minimum: 55, maximum: 100). Difficulty index, item variance, and discrimination indices for the items of the measurement tool were calculated according to the classical test theory. According to the item difficulty index, questions 1, 2, 4, and 10 were evaluated to be moderately difficult, questions 3, 5, 6, 7, 8, 9, and 11 were easy, and questions 12 and 13 were determined to be difficult. According to the item discrimination indices, questions 1, 3, 4, and 12 were evaluated to be very good and good, and questions 2, 5, 6, 7, 8, 9, 11, and 13 were found to be weak and very weak. The test variance was calculated as 4.21 [Table 1].
Generalizability theory enables to evaluate sources of error in a measurement., In the evaluation of the measurement tool with the generalizability theory, in the two-facet crossed design (pxixt), the percentage of persons' variance was 2.8%, the variance percentage of the items was 3.4%, the variance percentage of time (of the pretest/posttest application) was 0.0%, person-item variance percentage was 2.4%, person-time (pretest/posttest application) variance percentage was 0.0%, item-time variance percentage was 32.5%, and person-item-time (pretest/posttest application) variance percentage was calculated as 58.9%. Variance values and details are specified in [Table 2].
In the evaluation of the measurement tool used in the training according to the classical test theory, Cronbach's alpha value was calculated as 0.63. Generalizability theory is a more extended version of classical test theory from different perspectives: it deals with multiple variance sources in a single analysis, allows determining the size of each variance source, and enables to calculate the reliability coefficient (G coefficient) related to making both relative decisions and absolute decisions based on the persons' performance., In the evaluation of the measurement tool used in the training according to the generalizability theory, the G coefficient was calculated as 0.54. In the evaluation of the scale used for feedback according to the classical test theory, Cronbach's alpha value was calculated as 0.86.
In evaluating the program, the feedback of the participants was evaluated for satisfaction, which is the first step of the Kirkpatrick model (n = 74). Participants' response rates for the “agree and totally agree” choice were as follows: 87.8% for the training period, 89% for supplementary tools and equipment used in the training, 54% for the time chosen for the training program (pretest/posttest application), 89.3% for the venue where the training program was held, 88.2% for the subjects covered in the training program meeting the needs, 94.7% for training handouts, 94.7% for trainers, 96% for educators' providing satisfactory answers to their questions, and 82.7% for the level of knowledge at the end of the training [Table 3].
The achievement of learning was evaluated in the second step of the model. A statistically significant difference was found between the pretest and the posttest (P = 0.000). In addition to this finding, the item-time (pretest/posttest application) variance percentage was calculated as 32.5% according to the generalizability theory.
| Discussion/Conclusions|| |
In the current literature, many studies are planned for the improvement of hemovigilance activities.,, In most of these studies, it is recommended to organize in-service training programs.,,,,, It is also recommended to use a program development model in the development of training programs. In our study, Kern's six-step program development model was preferred for program development. The measurement tool developed for the evaluation of the training program developed under this model was evaluated according to the classical test theory. It was planned to remove the weak and very weak ones (questions 5, 6, or 11) from the measurement tool questions, to add 1 question at difficult level, 2 questions at moderate difficulty level, and to form the item set consisting of three groups as 4 easy level questions, 5 moderate difficulty level questions, and 4 difficult level questions to increase the difficulty level of the measurement tool from easy level to moderate difficulty level.
In the evaluation of the measurement tool with the generalizability theory, the discrimination of the test was low due to the low percentage of person variance in the two-facet crossed design (PxIxT).  The low variance for the items showed that there was no difference between the difficulty levels of the questions. In addition, the zero variance between the pretest and posttest indicated that the time (pretest/posttest application) did not affect the measurement tool negatively. It was determined that the low person-item variance percentage was indicative of persons' similar evaluation of the items, and the person-time variance percentage (pretest/posttest) being 0.00% showed that the effectiveness of the measurement tool did not change depending on time. The high percentage of item-time (pretest/posttest) application variance shows that there was a change in the scores of persons with the effect of the training. The high percentage of person-item-time (pretest/posttest application) variance shows that systematic/nonsystematic error sources could not be controlled with this design.
In evaluating the measurement tool with the generalizability theory, it was determined that the training was effective in increasing the level of knowledge, but that it was not sufficient in sampling the knowledge universe and it should be improved as it was an easy level tool with weak discrimination capability.
In evaluating the first step of the program evaluation model, it was seen that the majority of the participants expressed their satisfaction with the training. The achievement of learning was evaluated in the second step of the model. According to the generalizability theory, a statistically significant difference between the pretest and the posttest and the high percentage of item-time variance (pretest/posttest application) shows that learning took place. In the third step of the Kirkpartick program evaluation model, the transfer of education to business life is evaluated. In the hospital where the training was given, 24 reactions were detected in 2019 before the training. This number has been determined as a reference value for evaluating the effectiveness of the training. In the upcoming period, the effectiveness of the program will be evaluated according to the change in this number, and with the effect of future training, this number is aimed to increase first with awareness raising and then decrease with behavioral change.
We believe that the training program held within the scope of the study significantly increased the knowledge level of the participants and contributed to learning. In addition, with the improvement of the measurement tool, a more valid and reliable measurement will be provided, and with the development of a program, a contribution to hemovigilance surveillance will be made.
We would like to thank Ersen Bilgiç and Özlem Çapkunoğlu, the hemovigilance nurses who took part in the trainings where this study was conducted and who meticulously managed the process.
Conflicts of interest
There is no conflicts of interest.
| References|| |
de Vries RR, Faber JC, Strengers PF. Network M of the B of the IH. Haemovigilance: An effective tool for improving transfusion practice. Vox Sang 2011;100:60-7.
Gün R, Semra Ö, Altindis S, Uyutan Y, Koroglu M, Altindis M. Hemovigilance nursing and contribution to transfusion safety. Turkish Bull Hyg Exp Biol 2019;76:405-14.
Kalındemirtaş C. Evaluation and Comparison of Blood Products Transfusion Knowledge Levels of Healthcare Professionals (Thesis). Istanbul: HSU. Şişli Hamidiye Etfal Training and Research Hospital; 2017.
Şahin H. Nurses' Level of Knowledge (Thesis) About Blood Transfusions, and The Effect of Education on This. Afyon: Kocatepe University; 2006.
Encan B. Regarding Blood Transfusion Practices of Nurses Evaluation of Knowledge Levels. Istanbul: Istanbul Science University; 2017.
Gezer E. Evaluation of Knowledge, Attitudes and Behaviors of Emergency Medicine Specialists and Emergency Medicine Research Assistants in Medical Faculties and Education Research Hospitals about Blood Transfusions. Edirne: Trakya University; 2015.
Güleryüz M A. Blood and Blood Products Transfusion Determining the Knowledge Level of Nurses on Its Subject, Near East University; 2015.
Karabela ŞN, Altungayular S, Taşpolat İ, Baydili KN, Yaşar KK. Monitoring and transfusion nursing practices with electronic record in blood transfusion process management. Haseki Tip Bul 2019;57:310-8.
Republic of Turkey Ministry of Health. National Hemovigilance Guide, 2016.
Sweet LR, Palazzi DL. Application of kern's six-step approach to curriculum development by global health residents. Educ Heal Chang Learn Pract 2015;28:138-41.
Kern DE, Thomas PA. Curriculum Development for Medical Education: A Six-Step Approach. 2nd
ed. Baltimore: The John's Hopkins University Press; 2009.
Kirkpatrick DL. Evaluating Training Programs: The Four Levels. 1st
ed. San Francisco, Emeryville, CA: Berrett-Koehler, Publishers Group West; 1994.
Atılgan H. Generalizability Theory and Application. 1st ed. Ankara. Anı Publications; 2019.
Güler N, Eroğlu Y, Akbaba S. Reliability of criterion-dependent measurement tools according to generalizability theory: Application in the case of eating skills. Abant İzzet Baysal University Journal of Education Faculty 2014;14:217-32.
Shavelson RJ, Webb NM. Generalizability Theory: A Primer. Thousand Oaks, CA, US: Sage Publications, Inc.; 1991.
Cronbach JL, Gleser GC. The Dependability of Behavioral Measurements: Theory of Generalizability for Scores and Profiles. New York: John Wiley and Sons; 1972.
Doğu Ö , Atasoy M , Altındiş S , Solaz N , Altındiş M . Transfusion Safety in Nursing Practices; Education is a must. Middle East Medical Journal. 2015;7:161-6.
Kolcu G, Başer Kolcu M İ. Relıabılıty Analysıs Of The Feedback Scale Of A Course Wıth Classıcal Test Theory And Generalızabılıty Theory. Turkish Journal of Health Science and Life, 2020;3:20-24.
Department of Medical Education and Informatics, Süleyman Demirel Universty, SDÜ Tip Fakültesi Dekanligi Morfoloji Binasi Dogu Kampüsü 32260 Çünür, Isparta
Source of Support: None, Conflict of Interest: None
[Table 1], [Table 2], [Table 3]