Chapter 6Training Evaluation Training doesn¶t cost; it pays. HRD is an investment, not an expense. Training is one of the most important necessities in any business for which the companies are willing to invest their resources. Naturally the outcomes are to be measured. Organizations can¶t manage what they don¶t measure. Thus it is important to establish the right performance measures for all key investments. Step 4 Evaluate the Training Training process Step 1 Identify the Needs Step 2 Design the Training Step 3 Implement the Training what was intended to be achieved by a particular programme and what is really achieved now.Meaning Training evaluation refers to activities aimed at finding out the effectiveness of training programmes after they are conducted. . Training evaluation techniques give us solutions to answer questions like. where was the capability level of learners before the programme and where is it now. It enables HRD functionaries to prove why they should not be retrenched from service even during a market downturn. and what is the monetary value of training outcome against the cost incurred for conducting the said training programme. Cont«. objectivity. accountability and credibility to HRD by insisting on tangible and verifiable outcomes. Training evaluation brings rationality. against the objectives for which such programmes were organized. learning. organizational outputs and societal outcomes. Identifies five categories of variables and the relationships among them: secondary i nfluences. 1987. skill based. A classification scheme that specifies three categories of learning outcomes (cognitive. application. ability/enabling elements. Business results.Sl. 5 6 Systems approach (Bushnell. acquisition. 1970) Brinkerhoff (1987) Evaluation criteria Four levels: reaction. Ford. Process. intermediate or usage outcomes and impacts and worth. 1994) CIPP (Galvin. No. reaction a nd outcomes Six stages: goal setting. Applied learning on the job. 1 2 3 4 Author and year Kirkpatrick (1967. Five levels: Reaction and Planned Action. outcomes. environmental elements. program implementation. Return on investment. job behaviour and results Four levels: context. Out puts and Outcomes. immediate outcomes. program design. input. motivation elements. 1983) CIRO (Warr. Learning. Five levels: Enabling and reaction. process and product Context. input. 1990) Kraiger. and Salas (1993) 7 8 Kaufman and Keller (1994) Holton (1996) 9 Phillips (1996) Different Evaluation Models . affective) suggested by the literature and proposes evaluation measures appropriate for each category of outcomes. Four sets of activities: Inputs. has helped HRD professionals worldwide to a great extent in solving the myths and mysteries of understanding training outcome.Learning Level III .Donald Kirkpatrick¶s Evaluation Model The four level training evaluation model advocated about half a century ago by Donald Kirkpatrick (1967).Results Cont«.Reaction Level II . The four levels of Kirkpatrick¶s (1967). Donald Kirkpatrick .Behaviour Level IV . model are: Level I . Level . the style. This is useful information. verbal reaction which can be noted and analyzed. timing. and their personal reactions to the training or learning experience. Examples of reaction level: typically µhappy sheets¶ feedback forms based on subjective personal reaction to the training experience. . etc? Level of participation. Evaluation at this level will convey to us only the satisfaction level of the trainees and not what they have learnt.1 Reaction: At reaction level. level of effort required to make the most of the learning perceived practicability and potential for applying the learning. subsequent verbal or written Cont«. ease and comfort of experience. reports given by delegates to managers back at their jobs. grading by delegates. post-training surveys or questionnaires. evaluation is focused on how the trainees felt. For example: did the trainees like and enjoy the training? Did they consider the training relevant? Was it a good use of their time? Did they like the venue. Level 5 What to look for in training evaluation? Level 4 Results Level 3 Behaviour Level 2 Learning Level 1 Reaction Return on Investment Cont«. Indicators of training success . which many people in the organization would expect an effective training programme to satisfy. interview or observation can be used before and after although this is time-consuming and can be inconsistent. Cont«. online tests or interview style assessments are all possible. Reliable. assessments or tests before and after the training. Hard copy. Typically. whether the trainees learn what is expected of a particular programme? This is an important criterion. Measuring the learning may involve a quiz or a test.2 Learning: At the learning level. electronic. Methods of assessment need to be closely related to the aims of the learning. Evaluation at this level is based on.Level . so as to limit the risk of inconsistent assessment. . clear scoring and measurements need to be established. evaluation is aimed at the measurement of increase in knowledge or intellectual capability after the training. knowledge. peers. Did the trainees put their learning into effect when they returned on the job? Were the relevant skills and knowledge used? Was there a noticeable and measurable change in the activity and performance of the trainees? Was the change in behaviour and new level of knowledge sustained? Would the trainee be able to transfer his learning to another person? Is the trainee aware of changes in his behaviour. We have all come across many employees who know how to do a job well. Measuring at this level may involve observing employees¶ behaviour at work or the Cont«. bosses.Level ± 3 Behaviour: Behaviour evaluation is the extent to which the trainees applied the learning and changed their work place behaviour. but chose not to do. feedback from customers. etc. depending upon the situation. suppliers. . and skill? This is also a critical measure of training success. If learning does not result in positive workplace behaviour of the trainees then the training efforts would be a waste. This can be seen immediately or several months after the training. return on investment. and better able to serve its clients or customers as a result of the training programme? Meeting this norm is considered as the bottom line. the business data and financial data are analyzed to evaluate the training. the number of complaints. Cont«. growth. . Is the organization more efficient. accreditations. timescales. attrition. more profitable. and other quantifiable aspects of organizational performance. wastage.Level ± 4 Results: At this level. At this level. retention etc. non-compliance. Evaluation at this level aims at finding out whether the training initiative has improved the organization¶s performance effectiveness. failures. Key performance indicators are volumes. achievement of standards. For instance. percentages. quality ratings. staff turnover. Measures would typically be business or organizational. given that many things beyond employee performance can affect organizational performance. It is also the most challenging level to assess. the evaluation focuses on the business or environment resulting from the improved performance of the trainee ² it is the acid test. values. Level ± 5 Organizations expect much more than results from training. ack Phillips Cont«. . Robinson (1989). The inclusion and relevance of a fifth level is therefore arguably only relevant if the assessment of Return On Investment might otherwise be ignored or forgotten when referring simply to the µResults¶ level. whose writing redirected the attention of trainers to business impact. Jack Phillips (1996) has suggested that evaluation must go beyond Level IV and focus on real measurement of return on investment. exhorts trainers to become ³performance consultants´ and de-emphasizes training as an intervention. Robert Brinkerhoff (1987) uses data gathering and evaluation to make the training function more customer-focused and practice of continuous improvement. Thus. Many trainers are of the view that ROI can easily be included in Kirkpatrick¶s original fourth level µResults¶. Return on investment got or not Calculate what was the cost of training and what is the monetary value of performance outcome resulted on account of the said training. who are familiar with the post -training performance of the trainee. give a written test at the end of the programme or at the end of each session.Results produced or not Look for the outcomes like increase in sales/productivity. peers. or improvement in product quality/customer service or profitability.Table gives a quick idea about how to find outcome of training at the five levels discussed above. customers. or use a feedback form at the end of the programme or at the end of each day/ session Level 2-Learning occurred or not Ask questions to trainee s orally. . Cont«. or subordinates. Techniques for finding training outcomes Level of Evaluation Level 1-Reaction of the trainees How to find the Outcome? Enquire from trainees orally. Level 5. Level 4. Level 3-Behaviour changed or not Observe on the job or seek report from the supervisor. customer satisfaction. job satisfaction Business coaching: Productivity/output. losses Diversity: Turnover. time savings. conflicts. costs. quality. employee satisfaction Compliance programmes: Penalties/fines. customer satisfaction Career development/career management: recruiting expense. Absenteeism control/reduction: Absenteeism. productivity. stress. charges. efficiency. losses Cont«. employee Compensation plans: Costs. promotions. quality. settlements. employee satisfaction Communications: satisfaction Errors. settlements. absenteeism. Turnover. complaints. . charges. employee satisfaction. The words in italics indicate the name of the training programme.Phillips (2005) suggests in this regard the following dimensions to assess ROI. productivity. quality improvement. efficiency. wallet share. market share. absenteeism. quality. employee satisfaction. productivity/output. time savings. Leadership development: Productivity/output. customer loyalty. error reductions. Orientation: Early turnover. employee satisfaction. productivity improvement. employee satisfaction Employee benefits plans: Costs. time savings. absenteeism. productivity. quality. turnover Labour-management cooperation programmes: grievances. cost of sales. training time. cost/time savings. and customer satisfaction Cont«. employee satisfaction. employee satisfaction Employee relations programme: Turnover. productivity . E-learning: Cost savings. engagement Marketing and advertising: Sales. employee satisfaction Work stoppage. cycle times. customer satisfaction Meeting planning: Sales. engagement Gain sharing plans: Production costs. employee satisfaction Sexual harassment satisfaction prevention: Complaints. first-aid treatments Selection tool (new): Early turnover. productivity. rework. Personal productivity/Time management: Time savings. employee satisfaction Safety incentive plan: Accident frequency rates. employee satisfaction Project management: Time savings. employee Six Sigma: Defects. turnover. cycle time. early turnover Retention management: Turnover. quality. yield. quality improvement. stress reduction. response time. training time. engagement. accident severity rates. budgets Recruiting source (new): Costs. absenteeism. productivity Self-directed teams: Productivity/output. costs Skill-based pay: Labour costs. . customer satisfaction. turnover. turnover. absenteeism Cont«. medical costs. employee satisfaction Technology implementation: Cycle times. customer service. turnover. . cycle times. time. absenteeism Cont«. customer satisfaction Wellness/fitness: Turnover. absenteeism. employee satisfaction Stress management: Medical costs. accidents. productivity. market share. turnover. error rates. sales. cost savings. absenteeism. quality. costs. customer service. quality/service levels. Strategy/policy: Productivity/output. efficiency. sales. job satisfaction Technical training (job-related): Productivity. turnover. . absenteeism. costs. project committees.Davidson (1998) also suggests a similar approach to measure ROI as per Table. rework. absenteeism (pay for performance) Productivity. time. turnover. productivity Cont«. yield (percentage of candidates recruited). quality. quality. employee satisfaction. time -to-fill ratios Defects. etc. The table indicates some of the areas to look when trying to demonstrate results. customer satisfaction. sales. employee satisfaction Labour costs. allowing the sales staff to generate more revenue and reduce ³down time. time to deliver Cost per hire.) satisfaction.´ Measuring ROI in HR HRD Programmes Training Programmes Compensation Programmes Modified Work Structures Recruiting Programmes Total Quality Management Employee Support Programmes Possible Measurements Productivity. response time Absenteeism. turnover absenteeism. employee referrals. employee satisfaction. which mean positions would get filled more quickly. customer (teams. For example: Improved recruiting programmes for sales could lead to improved time-to-fill ratios. 10. The ROI process must account for other factors that have influenced output variables. 8. 3. The assumptions.Phillips and Whalen (2000). 5. 9. The ROI process must include the costs of the programme. 7. without complex formulas. 6. 4. and outcomes must be credible. The ROI process must be economical with the capacity to be implemented easily. 2. The ROI process must have a flexibility to be applied on a pre-programme basis as well as a post-programme basis. the ROI process must be theoretically sound. . lengthy equations and complicated methodologies. The ROI process must have a successful tract record in a variety of applications. have suggested certain criteria for effective ROI process as under: 1. The ROI process must be appropriate to a variety of programmes. methodology. The ROI process must be simple. From a research perspective. The ROI process must be applicable to all types of data including hard data and soft data. Such questionnaire should ask questions on the subject expertise as well as methodology of teaching expertise of the trainers. Table 4 shows the major source and techniques of data collection. sessionwise. It is not necessary that evaluation has to be done at all the four or five levels from reaction to return on investment. . There are different types of data available for training evaluation like individual performance details. The other questions will include the adequacy and quality of seating. lunch. Thus. At reaction level. stationary. and the relevance of the training. evaluation at any one or two levels would be sufficient. collecting appropriate and valid data using scientific methods will help in doing acceptable evaluation. audio-visual equipments Cont«. tea.Data collection for Training Evaluation Good evaluation depends upon good data. the data can be collected by oral reaction of the respondents. Depending upon the convenience and purpose. study material. performance details of an entire department or group and the increase in the economic value of the organization. or it can be collected through a questionnaire during the last session of the training programme. return rate of questionnaire is difficult to control. quickly processed. close relation to the job performance Reliable. Possible inaccurate data. trained observers needed. variety of options Non -threatening. wide sampling possible Reliable. simulation of development cost. honest y increased if it is anonymous. readily scored. respondents set varying paces. may be unreliable. ten difficult. easy to review. easily administered. objective. information system discrepancies. May be threatening. possible cultural bias. minimal reactive effects Limitations High reactive effects. . face to face threat potential. high cost. reactive effect possible. objective. excellent way to measure behaviour change Low purchase cost. job based. labour -intensive. personal interaction Low cost. respondent sets pace. reliance on norms may distort individual performance. Possibly disruptive. opportunity for clarification.Methods of data collection for training evaluation Method Interview Advantages Flexible. on the job responding conditions are not controlled. depth possible. possible low relation to job performance. Time -consuming. high Questionnaire Direct observati on Written test Performance test Performance data Lack of knowledge of criteria for keeping or discarding records. and time consuming. . performance tests etc. Randomized non-equivalent control group design: It involves testing and comparing the scores of two groups once before the training and again after the training. a test is conducted prior to the training to assess the existing level of knowledge. oral test. skill or attitude of the trainees.Designs of Training Evaluation One shot case study: This method involves evaluating the trainees only once at the end of the programme. Randomized equivalent control group design: In this method. One group pre-test/post-test design: In this method. participants are selected from identical groups and allotted randomly to control and experiment groups. Another test is conducted after the training and the actual effectiveness of training is found. Post-test only control group design: This method helps in preventing the contamination effects of pre-test sensitivities. the trainees are tested for more than once before and after the training using different types of tests like paper-pencil. One group time series: In this method. Suggestions for Better Evaluation Plan your metrics before writing survey questions Ensure the measurement process is replicable and scalable Ensure measurements are internally and externally comparable Use industry accepted measurement approaches Define value in the eyes of your stakeholders Manage the change associated with measurement Ensure the metrics are well balanced Leverage automation and technology Crawl. walk. run Ensure your metrics have flexibility .