|
|
Cooperative Extension Service Departments
Program and Staff Development
Publications
The "Nuts and Bolts" of Evaluation Planning Printer Friendly (PDF)
University of Arkansas
Cooperative Extension Service
Evaluation Planning Quick Reference Guide
Primary Uses for Evaluation Data
- To collect results for different strategies, approaches, methods
- To identify what works and what doesn’t to adapt programs locally
- To provide feedback for quality improvement
- To provide accountability for stakeholders
- To reduce the incidence of program errors and loss of credibility
- To manage more effectively
- To test assumptions
- To decide a program’s future
Key Considerations
- Why use a Logic Model for program planning and evaluation?
1) It is simple.
2) It is proven.
3) It gives us a common program planning and evaluation language.
4) It can provide key evaluation information to test our assumptions, to assist us in identifying the most successful strategies within a program, and it can allow faculty to improve, modify or discontinue programs.
5) It provides data for faculty to plan from an informed perspective.
- Begin program planning through a needs assessment driven by an examination of the local environment and situation, and the assumptions.
1) What is the need/problem/opportunity?
2) What demographic data do you have or can you find related to the state and local indicators?
3) What is your current local situation?
4) What assumptions can be made?
- Begin evaluation planning with the end in mind…what outcomes do you expect?
1) What is meaningful?
2) What is realistic?
3) Who needs to know?
- Identify what changes you want to see as a result of your program.
1) Knowledge
2) Attitudes
3) Behavior
4) Economic conditions
5) Environmental conditions
6) Policies
7) Production practices
8) Yield or Input cost changes
Selection of Evaluation Methods
- Identify what you CAN do based on what you NEED to know based on the cost
involved, the time required, available resources, skills required of persons
involved, potential reliability or bias of methods, confidentiality issues and
ethical considerations. There are benefits and disadvantages for all methods.
This quick reference guide will provide you with available methods, sources of
information and tools to support your evaluation planning and management.
In building a Logic Model for program planning and evaluation you will need to concisely and simply identify the:
- Situation
- Environment
- Assumptions
- Program inputs
- Program outputs
- Desired outcomes
The Logic Model helps you answer the critical accountability question…
SO WHAT?
It provides a template to develop an evaluation that is SIMPLE, Logical and
Meaningful. It helps the user stay focused on what can and should be measured.
It is usually helpful to use the one-page planning guide to construct your
initial Logic Model. This allows you to see how one component of the plan is
connected to the other.
The INPUTS or necessary program investments can include:
- Faculty/Staff time
- Funding for materials, travel, etc.
- Volunteer time
- Time committed from community partners
- Equipment costs
- Facility costs
- Curricula
- Evaluation resources
The OUTPUTS or program strategies, methods and target audiences can include:
- Who do you want to reach? Desired target audiences…how many?
1) Participants
2) Customers
3) Producers
4) Members
5) Volunteers
6) Agencies/Organizations
7) Age Groups
8) Ethnic Groups
- What do you want to do? How do you plan to reach your goal? What are your
planned strategies and methods?
1) Workshops, Seminars, Classes
2) Research
3) Demonstrations
4) Media Efforts
5) Product Development
6) Distance Education
7) Counseling
8) Facilitation
9) Curriculum Development
10) Fund Raising/Grant Writing
The OUTCOMES or impacts of a program identify what short, medium and/or
long-term results you are expecting from a program. These outcomes can include:
|
Examples of Outcomes |
Conditions
Longer-term |
Action
Medium |
Learning
Shorter-term |
|
• Economic
|
• Behavior
|
• Awareness
|
|
• Environmental
|
• Practice
|
• Knowledge
|
|
• Social
|
• Decisions
|
• Attitudes
|
|
• Civic
|
• Skills
|
• Skills
|
|
• Community
|
• Policies
|
• Opinions
|
|
• Policy
|
• Social Action
|
• Intentions
|
|
• Health
|
• Policies
|
• Motivation
|
|
Chain of Outcomes |
Conditions
Longer-term |
Action
Medium |
Learning
Shorter-term |
| Quality of area ground-water is improved as
demonstrated through ADH water monitoring results at participating homes. |
Citizens close and seal 12 unused wells and
participant in ADH water testing program. |
Citizens increase knowledge of water
contamination risks. |
| Participants reduce debt and increase
savings. |
Participants establish financial goals and
use a spending plan. |
Participants increase knowledge and skills in
financial management. |
| Family child care needs met. |
Residents and employers meet to discuss and
identify options and implement a plan. |
Community increases awareness of child care
needs. |
| Residents save money and enjoy a greater
sense of community. |
Residents convert an empty inner-city lot to
a community garden. |
Youth and adults learn gardening skills,
increase nutrition knowledge. |
The Logic Model makes the complex simple if you simply invest a staff meeting
or two to brainstorm and identify these key inputs, outputs and outcomes while
developing a program or evaluation.
The content in this publication provides the critical “nuts and bolts” for
your program planning and evaluation “tool kit.”
These resources include:
- Sources of Evaluation Information
- Methods for Collecting Evaluation Information
- A Logic Model Planning Worksheet
- Evaluation Methods: Advantages/Challenges and Resources Needed
- Internet Sources of Secondary Data and Information
Sources of Evaluation Information
Existing Information
- program documents: newsletters, work plans, accomplishment reports,
statistical reports, receipts, logs, minutes of meetings, enrollment records,
personnel records, proposals, project and grant records
- existing data bases: census, housing, industry, school census data
- research reports, published literature
- histories: county, program, life histories
- media records
- public service and business records, for example, farm records; fertilizer
sales at local dealers; employment statistics; justice, social and health agency
data; DNR and SCS data; local government plans; student performance records
- other evaluations of the same or similar programs
People
- participants, beneficiaries: those people who benefit directly or
indirectly from the program
- nonparticipants, proponents, critics, victims
- key informants: anyone who has particular knowledge about the program or
program benefits participants, for example, teachers, parents, religious
leaders, previous participants
- people with special expertise, for example, judges, faculty from a nearby
college
- county residents, local leaders, influentials
- program staff, administrators, volunteers
- collaborators, competitors
- funders
- policy makers, legislators, federal or state agency staff
Pictorial Records and Observations
- before and after pictures such as photos of streets before and after a
clean-up effort, photos of the garage before and after it became a youth center
or the empty lot before and after the garden project
- art done by children which illustrates perceptions of or responses to their
environment, for example, violence, drugs
- videotape of a group meeting which illustrates order of business,
leadership and collective decision making skills
- slide series of over-time changes, for example, lakefront development,
downtown restoration, grazing management systems or changes in participant
skills, for example, training a pet or speaking before a public audience
- videotape excerpts which demonstrate participant reactions and learning
taking place
- video or photos of program activities showing the diversity of participants
- observations of events and activities to record numbers and characteristics
of participants, practices or behaviors in action, interaction patterns and
skill development
- observations of practices such as manure management practices, erosion
control, lawn care practices
- observations of verbal and nonverbal behavior such as people reacting to a
nutrition display, working together in a team process, attending a
cross-cultural experience
Material adapted from Sources of Evaluation Information, Ellen
Taylor-Powell. University of Wisconsin-Extension, Cooperative Extension,
Madison, WI, 1-99.
Methods for Collecting Evaluation Information
- Survey: collecting standardized information through structured
questionnaires to generate quantitative data. Surveys may be mailed, sent
electronically, completed on-site or through face-to-face or telephone
interviews. Sample surveys use probability sampling while informal surveys do
not.
- Interviews: information collected by talking with and listening to
people, either face-to-face or over the telephone. Interviews range on a
continuum from tightly structured (as in a survey) to free-flowing,
conversational interviews.
- Observation: collecting information through “seeing” and
“listening.” Observations may be structured or unstructured.
- Document Analysis: use of content analysis and other techniques to
analyze and summarize printed material and existing information.
- Case Study: in-depth examination of a particular case (program,
group of participants, single individual, site/location). Case studies use
multiple sources of information and methods to provide as complete a picture as
possible.
- Group Assessment: use of the group process to collect evaluation
information such as nominal group technique, focus group, Delphi, brainstorming
and community forums.
- Expert or Peer Review: examination by a review committee, panel of
experts or peers.
- Portfolio Reviews: collection of materials, including samples of
work that encompass the breadth and scope of the program/ activity being
evaluated.
- Testimonials: individual statements by people indicating personal
responses and reactions.
- Tests: use of established standards to assess knowledge, skill or
performance as in pen-and-pencil tests or skills tests.
- Photographs, Slides or Videos: use of photography to capture visual
images.
- Diaries or Journals: recording of events over time revealing the
personal perspective of the writer/recorder.
- Logs: recording of chronological entries which are usually brief and
factual.
- Other:
Action Cards: use of index cards on which participants record what they
did – the “action” – and when they reach their goal, primarily used in
self-assessment.
Simulations: use of models or mock-ups to solicit perceptions and
reactions.
Problem Stories: narrative accounts of past, present or future situations
as a means of identifying perceptions using fictional characters to externalize
the problem situation.
Creative Expression: use of art forms to represent people’s ideas and
feelings through stories, drama, dance, music and art.
Unobtrusive Measures: the gathering of information without the knowledge
of the people in the setting such as the wear and tear on a “planted” mat in
front of a display.
Material adapted from Methods for Collecting Evaluation Information,
Ellen Taylor-Powell. University of Wisconsin-Extension, Cooperative Extension,
Madison, WI, 1-99.
| Evaluation Methods, Advantages / Challenges and
Resources Needed |
| Method |
Purpose |
Advantages and Challenges |
Resources Needed |
| Document Review |
To gather information from preexisting written
materials, such as meeting minutes, reports, budgets, newspaper articles,
etc |
| + |
Most common method for evaluation |
| + |
Good method for reviewing materials on how a program was
implemented |
| + |
uses information that is already available |
|
| + |
Can be done by program staff themselves |
| - |
Takes a lot of time to review existing files, especially
if program is complex and of long duration |
|
| Observation |
To gather information on how the program
operates, or on visual changes that have resulted
|
| + |
Good for process evaluation, because you can
get an immediate impression about how the program is progressing |
| - |
information can be difficult to interpret |
| - |
May be biased by how staff interpret the
information |
| - |
Events that are seen as can be time
dependent, and not representative of what is really going on |
| - |
If people know they are being observed, they
may act differently than usual |
|
| + |
Can be done by program staff themselves |
| - |
takes a lot of time to review existing files, especially
if program is complex and of long duration |
|
| Interviews |
To obtain information from
individuals about their experiences, or to learn more about their answers to
surveys |
| + |
Excellent for both process and outcome evaluation,
because you can get in-depth information from participants |
| + |
Can ask sensitive questions that require confidentiality |
| - |
Data can be difficult and complex to analyze |
| - |
Interviewer can bias the response |
|
| - |
Can take a lot of time to conduct |
| - |
May require data inputting and analysis program |
| - |
May require assistance from a consultant to design the
interview protocol and develop the database program |
|
| Focus Groups |
To hold discussions with
groups of people (10-12) to understand in-depth what they believe were the
effects of the program, or how they saw the program implemented |
| + |
Good for outcome evaluation, because you can ask people
to explain how the program affected them |
| + |
Can identify a lot of issues and effects |
| + |
Can give staff better understanding of the program from
participant's own words |
|
| + |
Can be done relatively quickly (1-2 hours per focus
group) |
| - |
Requires a good facilitator |
| - |
Takes time to analyze and interpret the discussion |
| - |
may require extra resources for facilitator's time and
participant incentives |
|
| Case Studies |
To describe a program or
experience in depth, often using some or all of the other evaluation methods
to construct a case study |
| + |
Good for combining process and outcome evaluation |
| + |
Can be a powerful way to describe the program |
| + |
Can depict personal experiences, quotes, and unique
program processes |
|
| - |
Can take a lot of time to collect information, organize
and describe the program |
|
| Surveys and Questionnaires |
To get information from
individual people about their changes in tobacco use knowledge, attitudes,
and behavior |
| + |
Excellent for outcome evaluation |
| + |
Can get information from a lot of people |
| + |
Can be done confidentially or anonymously, so may be
more valid |
| + |
Can use questions from existing surveys |
| - |
More effective when using yes / no or true / false
questions. |
| - |
Are more impersonal for participants, and usually not
good for getting quotes in participants' own words |
|
| - |
takes time to develop questionnaire |
| - |
May require database program to manage and analyze a lot
of data |
| - |
May require extra resources for participant incentives
and data analysis |
|
|
Source:
Asian Pacific Partners for Empowerment and Leadership (APPEAL),
Integrating Evaluation into Tobacco Programs for Asian American and Pacific
Islander Communities, 2001.
www.appealforcommunities.org
|
Internet Sources of Secondary Data and Information
Agriculture
National Agricultural Statistical Service (NASS)
www.nass.usda.gov
NASS - County Data
www.nass.usda.gov/indexcounty.htm
Education
Arkansas Department of Education
www.as-is.org/
U.S. Department of Education
www.ed.gov/stats.html
National Center for Education Statistics
http://nces.ed.gov
Employment
Arkansas Employment Security Department
www.accessarkansas.org/esd/labormarketinformation.htm
Bureau of Economic Analysis
www.bea.doc.gov/bea/regional/data.htm
Bureau of Census - 1997 Economic Census
www.census.gov/epcd/www/econ97.html
Family
Arkansas Advocates for Children and Youth
www.aradvocates.org
Government (local)
Bureau of Economic Analysis
www.bea.doc.gov
Health
Arkansas Department of Health
www.healthyarkansas.com/data/data/html
National Health Information Center
www.health.gov/nhic
Households
Bureau of Census - 2000 Census
www.census.gov/dmd/www/databank.html
Housing
Bureau of Census - 2000 Census
www.census.gov/dmd/www/databank.html
Bureau of Census
www.census.gov/hes/www/housing.html
Income
Bureau of Economic Analysis
www.bea.doc.gov/bea/regional/data.html
Bureau of Census
www.census.gov/hhes/www/income.html
Bureau of Census - 1997 Economic Census
www.census.gov/eped/www/econ97.html
Population
Arkansas State Data Center - 2000 Census
www.ualr.edu/
Bureau of Census - 2000 Census
www.census.gov/dmd/www/databank.html
Bureau of Census
www.census.gov
Poverty
Bureau of Census
www.census.gov/hhes/www/poverty.html
Retail Sales
Bureau of Census - 1997 Economic Census
www.census.gov/epcd/www/econ97.html
Transportation
Bureau of Economic Analysis
www.bts.gov
Arkansas Highway & Transportation Department
www.bts.gov
Youth Risk Factors
UALR Census Data Center, Children’s Data
Center Project
www.aiea.ualr/csdc/csdcinfor/child.html
Back to Program and Staff Development
Publications
|
