Chapter 10: Evaluation
Tomica Bozic, Hajrija Seleskovic, Soeren, Huwendiek, Martina Kadmon
1. Introduction
Evaluation is an essential part of the education process. It includes evaluation of teaching of the individual teacher and of the program. It provides improved teaching and its results in better learning for the students, improved care for the patient and a better educational program.
Purpose of evaluation:
- To ensure teaching is meeting student´s learning needs
- To identify areas of improving teaching
- To allocate faculty resources
- To provide feedback for teachers
- To support promotions by teachers
- To identify the values by medical schools
- To support developement of the curriculum
When an evaluation is planned we are asking?
- What are the goal of the evaluation?
- From whom and in what will data be collected?
- Who will collect and analyse data?
- What type of analysis will be used?
- Who will see the results of the evaluation?
Ideal evaluation has the following characteristics:
- Reliability
- Validity
- Acceptability to evaluator and to persons being evaluated
- Inexpensiveness
Reliability: the teaching evaluation needs to be linked to curriculum developement and needs criteria which are consistent with a teacher´s job. The evaluation may incorporate: the teachers themselves, learners, peers, supervisors, other health professionals, program directors and patients.
Validity: where possible, more then one source of information should be used and evaluation should include objective and subjective methods and quantitative and qualitative information.
Acceptability: is necessary to provide opportunities for teaching practice with feedback with rewards and having appropriate confidence. Good evaluation system is easy to administer and to be applied to all levels of teachers. Also good evaluation systems need to produce useful results which should be linked to professional development. These may be educational outcomes (e.g. teacher performance, students learning), practice outcomes (e.g. change in trainee practice) and health outcomes (e.g. effect on patient health).
Evaluation by students is limited to areas in which they are competent to judge.
They need to feel competent for an evaluation and in its developement.
Finally, students need to know that their opinions are valued. They also have to be informed about the results of the evaluation.
Self evaluation: academic staff evaluate their own practice; self evaluation provides to change behavior; teachers have to find it useful to use different devices during teaching( videotapes, logbooks).
Peer evaluation: teachers´s peers can provide an informed, valuable and diagnostic evaluation.
All teachers and residents should be evaluated in all domains : knoweledge, scholarship, clinical competence, teaching effectiveness and professional attributes.
2. Evaluation Methods
Evaluation includes subjective and objective measures and qualitative and quantitative approaches. They are:
- Interviews: the group interview or group focus can be used to explore the opinions of several individuals simultaneously from students and residents. An individual interview may be used for a teacher who has received low ratings and the reasons are unknown and not clear.
- Surveys: peer review of teaching provides valuable and diagnostic evaluation of the clinical teacher. These include faculty peers and educational consultants. This evaluation leads to faculty development.
- OSTE: For evaluation and assessment of teaching skills simulation of teaching encounters can be used. With OSTE (objective structured teaching encounter) clinical teaching activities can be explored. The faculty can score teaching attributes. Aim of evaluation of teaching effectiveness is improvement of learner outcomes.
Evaluation of teaching from different perspectives
2.1. The teacher´s perspective
The evaluation of teaching and teachers may be done directly or indirectly affected by their clinical, research and administrative competencies.
Knowledge of clinical discipline: Many countries prefer continuing professional development in the relevant domain of practice. For teachers of clinical medicine an adequate up-to-date knowledge base is an essential characteristic. Post specialization training certification and interest in life long learning activities can be used to reflect this knowledge base.
Values and models scholarship: This includes the development and implementation of a personal lifelong learning, critical appraisal of sources of medical information. Teacher´s knowledge, scholarship and approach to lifelong learning can be evaluated by peer and learner review of practice and teaching.
Assessment of learners: For the feedback of teachers this is an essential element for teaching. Assessment of a teacher´s ability to evaluate is done by learners. Peers and allied health professionals may participate in the trainee evaluation.
Improving teaching skills: Almost all academic health centers provide programs for the improvement of teaching. The clinical teachers have to be aware of a continuing effort to improve teaching skills. This process is determined by reviewing the teaching dossier or the evaluation by peers involved in residency programs in medical school.
Professionalism: It includes knowledge and skills, professional ethics and humanistic qualities. Humanistic qualities include integrity, respect, compassion, professional responsibility, courtesy, sensitivity to the patient´s needs and professional attitudes towards colleagues and learners. Professional ethics include a high standard of moral and ethical behavior. Students, postgraduate trainees, patients, nurses and peers evaluate professional behavioral and humanistic skills of teaching physicians.
Collegiality and collaboration: Collaboration is a teacher´s ability to work with patients and multidisciplinary health professionals to provide optimal patient care, education and research. Interdisciplinary activities are essential for teaching and research in medicine. Students, residents, peers, allied health professionals and patients evaluate teacher´s collegiality and collaboration.
Competent clinical skills and role modeling of good clinical skills: Patients, allied health professionals, learners (students and residents), peers who contribute in the care of the patient evaluate the performance of physicians.
2.2. The learner's perspective
The clinical teachers have to provide learners with knowledge and skills for their future professional life. From the learner´s perspective excellent teachers have to be capable of assessing the level and needs of the learner. The good teachers have to create a positive learning environment, involving all student in the learning process, encouraging mutual respect for all learners, discussing teachers and learners expectations, accepting what he or she is doing and why. Characteristics of a teacher which are important for the learner include enthusiasm, clarity, knowledge of the discipline, and demonstrating clinical skills.
Quantitative data to evaluate are obtained by using questionnaires by learners. Confidentiality and anonymity have to be assured. Detailed qualitative data can be obtained from interviews or focus groups. Finally learners have to perceive that their evaluations are taken seriously.
2.3. The patient's perspective
Mostly patients lack the knowledge to assess technical skills or medical knowledge, but they are able to judge whether they have been treated in an ethical and compassionate manner. Patients can be involved in the evaluation of clinical teaching in three ways.
First, they can assess aspects of clinical teaching encounter.
Second, for patients their overall clinical care is important.This evaluation may be related in part to a demonstration of teaching skills.
Third, for patients it is important that their physicians are courteous, that they create a relaxed atmosphere, listen closely, understand, take patient complaints seriously, and explain what they are doing throughout the examination. Patients can evaluate clear and competent physician´s explanations, their involvement in the planning of treatment, physician´s respect for patient dignity, and outcomes of treatment from patient´s perspective.
High ratings have been given by older patients, males, less educated and healthier patients.
The methodology of patient feedback is still evolving. - High ratings have been given by older patients, males, less educated and healthier patients -. But the data can be helpful as a feedback to the individual physician/teachers regarding to the nature of the all over evaluation of the clinics and may be helpful for program administrators and faculty development planners.
2.4. The perspective of customers and stakeholders
The traditional stakeholders are governments and health insurance companies. The best arrangements may exist between insurance companies, local health care organizations and individual clinicians. They all are interested in high quality health care and the support of medical education. The government programs are interested in teaching evaluations. Health insurance companies are interested in minimizing costs for health care measures.
Deans of medical schools have to be interested in hospitals as an academic environment with high quality for teaching and research. Research activity levels, teaching evaluations and student´s satisfaction may be attractive for more students and motivated and experienced staff.
Different expectations of customers and stakeholders on outcomes of clinical teaching:
| Government: | High quality of health care to meet the needs of the population |
| Hospital directors: | High quality in clinical service High quality of clinicians/teachers More patients Higher status Higher income |
| Deans: | Successful research programs High quality teaching International acknowledgement Quality in health care with good outcomes |
| Corporations: | Cost efficiency |
|
Health insurance Companies: |
Cost control |
| Clinicians: | Balance between clinical and teaching roles Reward for good teaching |
| Students: | Quality of teaching Support in all fields of education and research |
| Patients: | Confidentiality Respect No time delay Quality concerning outcomes of treatment No increase in cost |
3. Collecting Information
- Kind of information
- Length of questionnaire
- Approach of evaluation
Questionnaires offer a collection of information about poeple´s knowledge, beliefs, behavior, expectations, respects (and vice versa), quality (i.e. education), efficiency (i.e. cost).
Questionnaires can be used as the only measure ( as in cross sectional survey) or within clinical trials or epidemiological studies.
Questionnaires need to be defined according to their length (dependent of kind of information which have to be asked about) and how it will be administered.
Questionnaires have to be defined according to the approach. Evaluation may involve objective and subjective measures and qualitative and quantitative approaches.
Subjective measurements include qualitative (focus groups and interviews) and quantitative (surveys) measurements.
Objective measures include qualitative (analysis of tasks completed in an objective structured clinical examination - OSCE- station).and quantitative measures (trend in examination scores).
Examination core is a comparison for students in the previous cohort with those participating in the teaching session.
4. Designing Evaluation
Suitability of Questionnaires
Questionnaires will be appropriate only if used within a mixed methodological study. This is important for participants since answers must be meaningful.
Using existing instruments
For actual measurements previously validated and published questionnaires should be used. In this manner it will be possible to compare the actual data with those from other studies. This procedure should not exclude the own input and achievement of the proposed evaluation.
If there is no questionnaire available a new one has to be established.
In health service research designed standard questionnaires are used so that the data can be compared across multi center studies. This is important for validity approach. Health status and quality of life lose their validity when used beyond this context (for example clinical trials).
Reliability and and validity of questionnaire
Valid questionnaires measure what it claims to measure. Reasons for failing can be questionnaires developed too short of time without competence, in different countries or cultures with different ethical background.
Questionnaires are reliable when results are consistent from repeated samples and different researchers over time. Differences in results may come from differences between participants.
Standard questionnaires increase its reliability. Participants are asked same question in an identical format and responses recorded in a uniform manner.
Manner of question presentation
Two words which are often used are frequently and regularly. „Frequently“ means frequency (frequency based rating scale). „Regularly“ means a pattern. To avoid answers without clear statements like: commonly, usually, many, some and hardly ever, questions like „ strongly agree“ through to „ strongly disagree“ should be used.
We can use ticking a particular box or saying yes, no or maybe. Also we can insert a free text box at the end of the questionnaire. Participants need instructions how to complete free text items.
If open ended questions or free text comments are used qualified experts analysing these data are necessary.
We need to realize that some respondents ( known as yes sayers) want to agree rather than disagree. If one question is affirmative, another question must be in the negative.
What else questionnaire should include?
It is important to provide an introductory letter or information sheet for participants and to take it away after completing the questionnaire,
How questionnaires should look like
In principle, questions should be short, around 12 words or less. But short questions can be perceived as inconvenient and longer sentences are preferred.
Selecting a sample
Sample size of questionnaires is determined according to the expected outcome. If we are collecting quantitative data we will use statistical advice on the minimum sample size.
For qualitative subjective evaluation reliable information can be obtained from 100 student, we don´t need more participants.
For evaluation we need more than one source and type of information.
Approval is needed before we strart
In principle a study is unethical if it is scientifically unsound, causes undue offence or trauma, breaches confidence or wastes people´s time or money.
Questionnaires formally have to be approved by the responsible institution (ethic committee, university, faculty, responsible supervisor etc.) before they can used. The responsible institutes will require details of the study design and copies of the questionnaire.
Besides these instructions it is evident:
- Questionnaire studies often fail to produce high quality data.
- When possible use previously validated questionnaires
- Questions have to be phrased appropriately for the target audience and information required.
- Good explanations and design will improve response rates.
Evaluation involves answering questions « how did I do» and « what did the students learn» .
Teachers can be evaluated in different ways.
If the students are using a source of feedback, the following methods are useful:
- We can ask students to read their lecture notes and we have information what students have learned and understood.
- We can ask for verbal feedback from individual students
- We can ask students to complete a one-minute paper
- We can ask the students to complete an evaluation questionnaire
Peers can be used for a feedback to evaluate teaching style and delivery.
- Ask a colleague to observe part or all a lecture and what aspects of the lecturing are for evaluations: clarity, logical flow, effectiveness.
5. Evaluation Form Examples
6. References
Petra MB,Trisha G. Selecting, designing and developing questionnaire.BMJ 2004;328:1312-1315.
Linda S,Susan T,Steven H, Richard H,John N, Katinka P,Artur R,Richard R. A review of the evaluation of clinical teaching: new perspectives and challenges.Medical education 2000;34:862-870.
Jill M. ABC of learning and teaching in medicine: Evaluation.BMJ 2003 Feb 15;326(7385):385-7.Review
Cantillon P.Teaching large groups.BMJ 2003 Feb 22;326(7386):437.Review.