Workshop 11
EVALUATION
by Dr. Nancy deLay
Eli Lilly and Company

Measuring results — how to measure success, failure, and utilize the data to improve effectiveness of the telework program

CONTENTS

Introduction What is evaluation and why do it?
Process overview
Planning & development

• Measurement design
• Program objectives/goals
• hen to measure
• What to measure
• TheTelework Model
• From whom to collect data
• How to collect data

INTRODUCTION

In the past 10 lessons we have shared what we know about telework implementation;

• making a decision about whether it is right for your organization
• selling it to management
• designing the telework program
• announcing the roll-out
• implementing and monitoring the program.

The final step in the process is the evaluation of the successes and challenges presented by the telework program.

The focus of this session will be to educating you about the evaluation of a telework program. There is one issue that needs to be addressed before we begin – the timing for designing the evaluation. Don’t be fooled by the placement of this lesson (dead last!). The process of evaluation begins from the inception of the telework program. You should be thinking about evaluation from DAY ONE!

A general point, regardless of popular opinion (which often is: "it’s just a survey, it’s not rocket science) evaluation IS a science (people actually receive Ph.D.’s in evaluation) and, as an aside - evaluation is not always in survey format. A poorly designed or incompletely thought- through evaluation process is more harmful than not conducting an evaluation at all. Identification and inclusion of a measurement expert, either internal or external, will ensure the success of your evaluation process, thus providing information that you can actually use.

In keeping with a previous lesson, let’s begin this guide to evaluation with a very basic question:

WHAT IS EVALUATION AND WHY DO IT?

Evaluation is often thought of as expressing the value of something (e.g. as the success of a telework program) numerically (quantitatively). This is a rather restrictive view of evaluation. There is also merit in gathering qualitative information that can be used to enhance "the numbers". The bottom line is, once the information (data) is gathered and synthesized it will paint a clear picture of what actually took place during the program’s roll out. Clarity of the picture is determined by the methodology used, and the competence of the measurement expert involved.

To expand on this metaphor - if, as an artist, you are using a charcoal pencil that is flattened on the tip -- the resulting picture will have form and depth but not sharp edges. If, however, you use a sharpened pencil, the lines will be crisp and definitive. A good evaluation uses both the sharpened and flattened tip of charcoal.

Qualitative data provides an outline of the intended analysis when used before the quantitative data is collected. This is typically done in order to understand how to broadly define your data gathering. It can also be used to enhance and give depth to numerical data during the data synthesis stage of evaluation. Properly gathered, quantitative data provides the numbers that are so often sought to provide the evaluation needed to justify the investment in the program.

Why bother with the evaluation process? While the process of designing and implementing an evaluation for the telework project can be resource intensive, the benefits are substantial. Organizational, group and/or individual benefits must be demonstrated in order for any project to be provided with the resources required for it to be properly executed. If you have clearly defined these benefits (in such a way that they can be measured, then the determination of success or identification of areas of concern can be identified through a proper evaluation methodology. The end product of a properly designed and implemented evaluation can

• confirm or disaffirm long held beliefs about the benefits of teleworking
• provide information regarding which parts of the program are working well and which parts might need attention
• help identify organizational, group, and individual benefits and
• become a source of data for cross organizational research.

PROCESS OVERVIEW

Typically, evaluation is a five-phase process: Planning, Development, Administration, Data Analysis/Reporting and finally, Action Planning. I will provide a brief description of each of these phases in this section with more in-depth explanations to follow.

PLANNING:

Planning is the most important phase of evaluation. During this stage stakeholders should be identified, an appropriate match between data collection methodology and the type of data needed should be determined, and, finally, timing and milestones should be established.

Development of the data collection process is driven by what you want to measure and the number of individuals you have available from whom to collect information. For example: if you have a small teleworking project, conducting face-to-face or phone interviews may be the most appropriate way to gather information. If, however, you have a large-scale program in place, designing a questionnaire will likely be the preferred mode of data collection.

There are many ways to administer a data collection process. The type of data you are collecting and the required reporting structure drive the administration phase. The administration "tool" can be any one or more of the following: face-to-face interview, phone interview, paper-based questionnaire or electronic questionnaire. The administration tool used drives the resources needed for this phase to be successful.

Analysis of data can require skills that run the full spectrum from no formal training needed to the highest level of research sophistication. Data analysis also requires subject matter expertise. Reporting structure should have been determined during the planning phase and depends on the organizational structure. Reports can take various forms, running the gamut from a detailed technical paper to a PowerPoint presentation with a limited number of bullet points presenting "just the facts".

Finally, taking action based on the data is fundamental. If you do not take action, all of your investments (time, money, and resources) become moot and the integrity of the evaluation program is irreparably damaged. Basically, action planning requires reinforcing those aspects of the program that work well and modifying those parts of the program that have been indicated to require improvement.

PLANNING & DEVELOPMENT:

Measurement Design

The place to begin the planning and development process for evaluation is by developing a "Players Map". Dr.’s O’Hara and Solomons walked you through the process of stakeholder identification in Lesson 2. Please refer back as that lesson provides the information needed. Just to reiterate, the map will include Program Champions, Sponsors, Owners and various other stakeholders (target groups). Evaluation design must take all those involved into consideration. The fundamentals are simple; you will be gathering information from one or all of the stakeholder groups and feeding the results back to those who provided data and other interested/vested parties.

Program objectives/goals:

Dr. Joice (Lesson 1) mentioned the difference between Telework Drivers and Telework Benefits. When an organization makes the decision to implement a telework program, as part of the evaluation, it is important to have a clear understanding of the benefits it is striving to achieve and the criteria for success. Is the driver increased productivity or Reducing work/life conflict or recruiting and retention? Development of the evaluation tool(s) must yield information that is meaningful to the stakeholders (primarily the program champion, decision makers, sponsors and owners) that will help shape and guide the success of the program.

When to Measure:

In the real-estate business, location is everything. In evaluation, timing has the equivalent impact. There are two aspects of timing to be considered: when and how often to administer.

Things to consider on the "when" aspect:

• Do you conduct "pre" benchmarking and "post" analysis for comparison purposes?
• How often do you have new members joining the program?
• Are you measuring globally (don’t measure in Europe in August, don’t measure in the Middle East/Asia in January, etc.)?

Things to consider on the "how often" aspect:

• How easy is it to get your measurement tool into the hands of and back from respondents?
• How long does it take to implement program changes and see an impact after such an adjustment?

What to measure:

Assessment procedures are selected in order to answer specific questions. Thus, the first step in determining what to measure is to clarify the questions to be answered. The most obvious place to begin is by revisiting the program objectives and goals. How will you know when you have reached these? What type of information will indicate success given the original objectives of the program?

Next, you must determine from whom to gather the information (i.e. teleworkers, teleworking & teleworker managers, co-workers, and clients)? It is important to think about all the people that your telework program impacts. You may be interested in understanding how teleworkers compare to their co-workers, or if clients of teleworkers are satisfied, etc. The scope of your measurement may be limited to teleworkers only, or may include all five of the previously mentioned groups.

The TELEWORK MODEL:

The model below is presented to help frame your thoughts for the "what" of measurement. Essentially, success of any program is driven by input variables (lead indicators) and determined by assessing outcome variables (lag indicators). In order to provide actionable data, measurement must occur around these inputs and outcomes.

Inputs:

Input variables should simply be thought of in terms of those things that impact the success or failure of a program. In the world of telework, you essentially have four categories that those items will fall under:

• Organization Culture/Climate
• Job Characteristics
• Program Structure
• People/Relationships.

Outcomes:

Outcomes, or results from implementation of a telework program, can be measured on four levels: Contextual, Organizational, Team/Group/Unit, and Individual. Although I won’t go into detail about the types of data to be collected in each of these areas, it is important to match the level of assessment to the original goals and purpose of the program. Thus if the primary goal of your program is to enhance recruiting and retention capabilities you would track the data that reflects those at an organizational level. If the goal is increased productivity you should track multiple levels – individual, team/group/unit and organizational. If you have multiple goals (which is highly likely) then you must identify all of the indicators that track the needed information. Keep in mind that each goal will provide unique, and possibly multiple and different tracking metrics.

From whom to collect data:

Once stakeholders have been identified you must consider from whom you want to gather data. There are four constituents available to you in the realm of telework (refer to People/Relationship aspect of the Telework model):

• Teleworkers
• Telework managers
• Non-teleworking coworkers
• Clients.

The program’s objectives/goals will help define from which of the possible respondent pools you should gather data. A comprehensive evaluation will gather data from all four constituencies in an integrated fashion.

This is the time to consider whether you want to sample or conduct a census process. Sampling is only practical when you have a large pool of respondents from which to choose greater than 200) and is an acceptable solution to make a study more manageable. Census is required when you need information from all people in the pool.

The level at which you are planning to report the data will also influence how many people you need to have in your study. If, for example, your organization has four functional units and you want to report results within those four levels, it will require a larger sample size than if you were reporting only at the organizational level. This is why it is crucial to agree whether the objectives are to be department by department or on an overall basis prior to data collection.

How to collect data:

Below is a grid of the most commonly used data collection methods in organizations. Many things impact the decision concerning which method to use. The important thing to remember is to match the method used to the organization and program being measured.

Method	N	Confidentiality	Level of Expertise Required	Data Generated
Interview (either face-to-face or voice-to-voice)	One-on-One Small Sample	Can guarantee confidentiality (and if third party vendor, anonymity)	Design – high Process - Moderate to High Interpretation – Moderate to High	Qualitative (possibly some quantitative)
Focus Group	Groups of 10-12 people Small to Medium Sample	Can guarantee confidentiality (and if third party vendor, anonymity)	Design – high Process – Moderate to High Interpretation – Moderate to High	Qualitative (possibly some quantitative)
IVR [is what??]	Groups Medium to Large Sample	Can guarantee anonymity	Design – High Process – Low Interpretation – Moderate to High	Quantitative (possibly some qualitative)
Paper-based Questionnaire	Groups Small to Large Sample	Can guarantee anonymity	Design-Moderate to High Process-Low to High Interpretation – Moderate to High	Quantitative and Qualitative
Electronic Questionnaire	Groups Small to Large	Can guarantee confidentiality and in some cases anonymity	Design- High Process-Low to High Interpretation – Moderate to High	Quantitative and Qualitative

ADMINISTRATION:

The data collection tool and the audience determine the logistics of administration. Regardless of the type of tools used, there are some fundamental keys to making the administration phase go well. Basically the three keys are as follows:

• Have a plan,
• Communicate the plan,
• Implement the plan.

Communicate with the program sponsor, key stakeholders, process coordinators, respondents and vendor (if you have one). Many a measurement process has gone sour over lack of good communication.

DATA ANALYSIS AND REPORTING:

Depending on the type and amount of data to be collected and analysed, the expertise required for data analysis and reporting can range from minimal to an extremely high level. This is the most likely place to bring an expert on board, internal or external. A word of warning here: if you choose to use a measurement expert only for the data analysis, give them the opportunity to review your measurement tools prior to administration. If your expert has been involved from the beginning (which is the best case scenario) s/he will have little difficulty at the analysis stage.

Reporting structure and content need to be determined prior to data analysis. The number of reports and needs of each recipient audience may be significantly different. Determine this prior to report presentation. Reporting data is not the same as data interpretation. Data reporting and interpretation requires two skill-sets; raw data analysis skills and content expertise, which requires an understanding of the activity being researched. These may or may not come housed in the same individual.

TURNING DATA INTO ACTION:

You can have an awesome data collection and reporting strategy in place and still fail. Failure often comes when trying to turn data into action. Several things can happen at this stage to derail the entire evaluation process:

• Lack of understanding of the meaning of the data
• Inappropriate audience for the information
• "Data coma" (being overwhelmed by the amount of data)
• Inability to prioritize issues
• Lack of resources to implement action

In order to move your organization beyond the data, a number of tasks must occur. First, help report recipients understand the importance of particular scores. Next clarify what low scores mean. At this point, you must prioritize issues. Once you have identified priority issues develop strategies and policies to improve the program. After actions have been delineated, identify resources to implement the selected actions. Once actions have been implemented you should continue to monitor progress on these initiatives.

The ultimate goal of the measurement process is to develop action plans that address the issues affecting the majority of participants. Essentially, this is the phase where "the rubber meets the road." As part of action planning it is imperative to communicate, communicate, and communicate some more. Let all parties know the results and the actions taken based on those results. This ensures the integrity of the measurement process and will impact the success of your actions.

CONCLUSION

It is estimated that 50% of telework programs fail. The reasons for failure are wide-ranging. An appropriately designed evaluation can help your organization identify issues before they disrupt and doom your program to failure. Evaluation is necessary to determine which aspects are successful and so should be reinforced, and which need to be improved by modification.