Chapter 10: Evaluation

Tomica Bozic, Hajrija Seleskovic, Soeren, Huwendiek, Martina Kadmon

1. Introduction

Evaluation is an essential part of the education process. It includes evaluation of teaching of the individual teacher and of the program. It provides improved teaching and its results in better learning for the students, improved care for the patient and a better educational program.

Purpose of evaluation:

When an evaluation is planned we are asking?

Ideal evaluation has the following characteristics:

Reliability: the teaching evaluation needs to be linked to curriculum developement and needs criteria which are consistent with a teacher´s job. The evaluation may incorporate: the teachers themselves, learners, peers, supervisors, other health professionals, program directors and patients.

Validity: where possible, more then one source of information should be used and evaluation should include objective and subjective methods and quantitative and qualitative information.

Acceptability: is necessary to provide opportunities for teaching practice with feedback with rewards and having appropriate confidence. Good evaluation system is easy to administer and to be applied to all levels of teachers. Also good evaluation systems need to produce useful results which should be linked to professional development. These may be educational outcomes (e.g. teacher performance, students learning), practice outcomes (e.g. change in trainee practice) and health outcomes (e.g. effect on patient health).
Evaluation by students is limited to areas in which they are competent to judge.
They need to feel competent for an evaluation and in its developement.
Finally, students need to know that their opinions are valued. They also have to be informed about the results of the evaluation.

Self evaluation: academic staff evaluate their own practice; self evaluation provides to change behavior; teachers have to find it useful to use different devices during teaching( videotapes, logbooks).

Peer evaluation: teachers´s peers can provide an informed, valuable and diagnostic evaluation.
All teachers and residents should be evaluated in all domains : knoweledge, scholarship, clinical competence, teaching effectiveness and professional attributes.

2. Evaluation Methods

Evaluation includes subjective and objective measures and qualitative and quantitative approaches. They are:

Evaluation of teaching from different perspectives

2.1. The teacher´s perspective

The evaluation of teaching and teachers may be done directly or indirectly affected by their clinical, research and administrative competencies.

Knowledge of clinical discipline: Many countries prefer continuing professional development in the relevant domain of practice. For teachers of clinical medicine an adequate up-to-date knowledge base is an essential characteristic. Post specialization training certification and interest in life long learning activities can be used to reflect this knowledge base.

Values and models scholarship: This includes the development and implementation of a personal lifelong learning, critical appraisal of sources of medical information. Teacher´s knowledge, scholarship and approach to lifelong learning can be evaluated by peer and learner review of practice and teaching.

Assessment of learners: For the feedback of teachers this is an essential element for teaching. Assessment of a teacher´s ability to evaluate is done by learners. Peers and allied health professionals may participate in the trainee evaluation.

Improving teaching skills: Almost all academic health centers provide programs for the improvement of teaching. The clinical teachers have to be aware of a continuing effort to improve teaching skills. This process is determined by reviewing the teaching dossier or the evaluation by peers involved in residency programs in medical school.

Professionalism: It includes knowledge and skills, professional ethics and humanistic qualities. Humanistic qualities include integrity, respect, compassion, professional responsibility, courtesy, sensitivity to the patient´s needs and professional attitudes towards colleagues and learners. Professional ethics include a high standard of moral and ethical behavior. Students, postgraduate trainees, patients, nurses and peers evaluate professional behavioral and humanistic skills of teaching physicians.

Collegiality and collaboration: Collaboration is a teacher´s ability to work with patients and multidisciplinary health professionals to provide optimal patient care, education and research. Interdisciplinary activities are essential for teaching and research in medicine. Students, residents, peers, allied health professionals and patients evaluate teacher´s collegiality and collaboration.

Competent clinical skills and role modeling of good clinical skills: Patients, allied health professionals, learners (students and residents), peers who contribute in the care of the patient evaluate the performance of physicians.

2.2. The learner's perspective

The clinical teachers have to provide learners with knowledge and skills for their future professional life. From the learner´s perspective excellent teachers have to be capable of assessing the level and needs of the learner. The good teachers have to create a positive learning environment, involving all student in the learning process, encouraging mutual respect for all learners, discussing teachers and learners expectations, accepting what he or she is doing and why. Characteristics of a teacher which are important for the learner include enthusiasm, clarity, knowledge of the discipline, and demonstrating clinical skills.

Quantitative data to evaluate are obtained by using questionnaires by learners. Confidentiality and anonymity have to be assured. Detailed qualitative data can be obtained from interviews or focus groups. Finally learners have to perceive that their evaluations are taken seriously.

2.3. The patient's perspective

Mostly patients lack the knowledge to assess technical skills or medical knowledge, but they are able to judge whether they have been treated in an ethical and compassionate manner. Patients can be involved in the evaluation of clinical teaching in three ways.

First, they can assess aspects of clinical teaching encounter.
Second, for patients their overall clinical care is important.This evaluation may be related in part to a demonstration of teaching skills.

Third, for patients it is important that their physicians are courteous, that they create a relaxed atmosphere, listen closely, understand, take patient complaints seriously, and explain what they are doing throughout the examination. Patients can evaluate clear and competent physician´s explanations, their involvement in the planning of treatment, physician´s respect for patient dignity, and outcomes of treatment from patient´s perspective.
High ratings have been given by older patients, males, less educated and healthier patients.
The methodology of patient feedback is still evolving. - High ratings have been given by older patients, males, less educated and healthier patients -. But the data can be helpful as a feedback to the individual physician/teachers regarding to the nature of the all over evaluation of the clinics and may be helpful for program administrators and faculty development planners.

2.4. The perspective of customers and stakeholders

The traditional stakeholders are governments and health insurance companies. The best arrangements may exist between insurance companies, local health care organizations and individual clinicians. They all are interested in high quality health care and the support of medical education. The government programs are interested in teaching evaluations. Health insurance companies are interested in minimizing costs for health care measures.

Deans of medical schools have to be interested in hospitals as an academic environment with high quality for teaching and research. Research activity levels, teaching evaluations and student´s satisfaction may be attractive for more students and motivated and experienced staff.

Different expectations of customers and stakeholders on outcomes of clinical teaching:

Government: High quality of health care to meet the needs of the population
Hospital directors: High quality in clinical service
High quality of clinicians/teachers
More patients
Higher status
Higher income
Deans: Successful research programs
High quality teaching
International acknowledgement
Quality in health care with good outcomes
Corporations: Cost efficiency
Health insurance
Cost control

Clinicians: Balance between clinical and teaching roles
Reward for good teaching

Students: Quality of teaching
Support in all fields of education and research

Patients: Confidentiality
No time delay
Quality concerning outcomes of treatment
No increase in cost

3. Collecting Information

  1. Kind of information
  2. Length of questionnaire
  3. Approach of evaluation

Questionnaires offer a collection of information about poeple´s knowledge, beliefs, behavior, expectations, respects (and vice versa), quality (i.e. education), efficiency (i.e. cost).
Questionnaires can be used as the only measure ( as in cross sectional survey) or within clinical trials or epidemiological studies.

Questionnaires need to be defined according to their length (dependent of kind of information which have to be asked about) and how it will be administered.
Questionnaires have to be defined according to the approach. Evaluation may involve objective and subjective measures and qualitative and quantitative approaches.

Subjective measurements include qualitative (focus groups and interviews) and quantitative (surveys) measurements.
Objective measures include qualitative (analysis of tasks completed in an objective structured clinical examination - OSCE- station).and quantitative measures (trend in examination scores).
Examination core is a comparison for students in the previous cohort with those participating in the teaching session.

4. Designing Evaluation

Suitability of Questionnaires

Questionnaires will be appropriate only if used within a mixed methodological study. This is important for participants since answers must be meaningful.

Using existing instruments

For actual measurements previously validated and published questionnaires should be used. In this manner it will be possible to compare the actual data with those from other studies. This procedure should not exclude the own input and achievement of the proposed evaluation.

If there is no questionnaire available a new one has to be established.
In health service research designed standard questionnaires are used so that the data can be compared across multi center studies. This is important for validity approach. Health status and quality of life lose their validity when used beyond this context (for example clinical trials).

Reliability and and validity of questionnaire

Valid questionnaires measure what it claims to measure. Reasons for failing can be questionnaires developed too short of time without competence, in different countries or cultures with different ethical background.
Questionnaires are reliable when results are consistent from repeated samples and different researchers over time. Differences in results may come from differences between participants.

Standard questionnaires increase its reliability. Participants are asked same question in an identical format and responses recorded in a uniform manner.

Manner of question presentation

Two words which are often used are frequently and regularly. „Frequently“ means frequency (frequency based rating scale). „Regularly“ means a pattern. To avoid answers without clear statements like: commonly, usually, many, some and hardly ever, questions like „ strongly agree“ through to „ strongly disagree“ should be used.
We can use ticking a particular box or saying yes, no or maybe. Also we can insert a free text box at the end of the questionnaire. Participants need instructions how to complete free text items.

If open ended questions or free text comments are used qualified experts analysing these data are necessary.
We need to realize that some respondents ( known as yes sayers) want to agree rather than disagree. If one question is affirmative, another question must be in the negative.

What else questionnaire should include?

It is important to provide an introductory letter or information sheet for participants and to take it away after completing the questionnaire,

How questionnaires should look like

In principle, questions should be short, around 12 words or less. But short questions can be perceived as inconvenient and longer sentences are preferred.

Selecting a sample

Sample size of questionnaires is determined according to the expected outcome. If we are collecting quantitative data we will use statistical advice on the minimum sample size.
For qualitative subjective evaluation reliable information can be obtained from 100 student, we don´t need more participants.
For evaluation we need more than one source and type of information.

Approval is needed before we strart

In principle a study is unethical if it is scientifically unsound, causes undue offence or trauma, breaches confidence or wastes people´s time or money.
Questionnaires formally have to be approved by the responsible institution (ethic committee, university, faculty, responsible supervisor etc.) before they can used. The responsible institutes will require details of the study design and copies of the questionnaire.

Besides these instructions it is evident:

Evaluation involves answering questions « how did I do» and « what did the students learn» .
Teachers can be evaluated in different ways.
If the students are using a source of feedback, the following methods are useful:

Peers can be used for a feedback to evaluate teaching style and delivery.

5. Evaluation Form Examples

6. References

Petra MB,Trisha G. Selecting, designing and developing questionnaire.BMJ 2004;328:1312-1315.

Linda S,Susan T,Steven H, Richard H,John N, Katinka P,Artur R,Richard R. A review of the evaluation of clinical teaching: new perspectives and challenges.Medical education 2000;34:862-870.

Jill M. ABC of learning and teaching in medicine: Evaluation.BMJ 2003 Feb 15;326(7385):385-7.Review

Cantillon P.Teaching large groups.BMJ 2003 Feb 22;326(7386):437.Review.