Cooperative Extension Service Departments
Program and Staff Development
Programs - FSLL Initiative Evaluation
 Evaluation of Programs and Activities for
Educators
Evaluation of Participant Changes
This page contains:
Evaluation Recommendations and Suggestions (including evaluation tools for
those who are conducting FSLL-based educational programs with clientele).
The following links to additional sites containing
program evaluation tips and suggestions:
Back to Evaluation of Programs and Activities for Educators
FSLL Performance Goals:
To increase the number of Initiative participants who:
-
engage in activities which increase their financial literacy related to
later life issues
-
utilize recommended practices in managing their use of credit in light of
their long-term goals for later life
-
initiate contributions to a retirement savings plan or increase contributions
to retirement plans
-
determine retirement income needs and/or future income needs
-
develop a plan to achieve retirement and/or future income goals
-
establish or revise investment goals
-
participate in employer-provided retirement plans
-
increase their contributions to employer-provided retirement plans
-
increase their knowledge of risks, costs and financing options for health,
including long-term care
-
develop a plan for managing long-term health care needs
-
develop an integrated plan for accumulating, protecting, and distributing/transferring
assets
FSLL Participant Outcome:
Levels of Participant Changes
The example evaluation tools and suggested methods are presented based on the
level of the goal or outcome in relation to the Targeting Outcomes of Programs
(TOP) Model developed by Bennett and Rockwell. If you have specific
questions related to evaluating your FSLL-related programs, work with the individual
in your organization whose responsibilities include program evaluation.
Knowledge Change
Knowledge Change FSLL Goals:
"To increase the number of Initiative participants who engage in activities
which increase their financial literacy related to later life issues."
"To increase the number of Initiative participants who increase their
knowledge of risks, costs and financing options for health, including long-term
care."
Pre-Test/Post-Test Methods:
The use of a pre-test/post-test model may or may not be a good option for use
in measuring knowledge change as the result of an educational program. To use
a pre-test/post-test procedure to measure knowledge change, you must develop
a valid test of the subject matter knowledge based on what will be covered in
the program. This usually means a test instrument that has quite a few items.
In developing the pre-test/post-test instrument(s), the educator must carefully
review the program materials and curricula being used to make sure the test
instruments reflect what is going to be taught. A review of the test by a “panel
of experts” in the subject matter area is also usually recommended to
ensure that the knowledge content is adequately covered by the test items and
that the items are written in an appropriate way to measure the knowledge level
of individuals in that program. A field test of the instrument, prior to its
use, with individuals similar to those who will be in the program is also recommended.
The use of a pre-test/post-test procedure with a well-designed test instrument
can provide objective data about changes in participants’ knowledge.
In addition to the effort needed to develop a valid knowledge test instrument,
a down side of using a pre-test/post-test method is that you have to give the
test twice. Another concern is that some adult audiences may be turned off by
having to take a “test,” and choose not to continue participation
in the program.
Note: If you are going to ask program participants to self-report their perceptions
of their level of knowledge about the program subject matter (e.g., “What
is your level of knowledge about program topic/issue?”), it is recommended
that you NOT do so using a pre-test/post-test procedure.
Why?
Many times people will not have accurate perceptions of how much they really
know (or don’t know) about a subject until after the completion of the
educational program. If you ask someone to indicate their level of knowledge
about a topic or issue on a pre-test, they may underestimate or, worse, overestimate
their level of knowledge. By the end of the program, participants realize that
they either knew more about the topic than they originally thought or really
didn’t know as much about the subject as they originally thought. In either
case, the response on the post-test survey at the end of the program would either
indicate more knowledge change than what really took place or less knowledge
change (or possibly no change or even negative change) from pre-test to post-test.
So, what should you do?
If you are going to ask participants to self-report perceptions of their knowledge
level change from pre-program to post-program, ask them to do so after the completion
of the program. You can use either a post-program survey or a post-then-pre
survey. (See examples below) At the end of the program, participants will have
a better perspective of how their knowledge has changed during the program.
Post-Program Methods:
Post-program evaluations of knowledge change can be conducted at the completion
of the total program or, if the program involves a number of different sessions
or activities, at the completion of different segments of the program. A number
of different scales can be used for reporting changes in knowledge level. Here
are a few examples:
Post-Then-Pre Program Methods:
Like the post-program evaluation, the post-then-pre method of evaluating knowledge
change can be conducted at the completion of the total program or, if the program
involves a number of different sessions or activities, at the completion of
different segments of the program. Likewise, a number of different post-then-pre
scales can be used for determining changes in knowledge level. The post-then-pre
instrument asks participants to indicate what they think is their level of knowledge
about the program topic or issue after the completion of the program. Then participants
are asked to think back to before the program and indicate what they feel their
level of knowledge was before participating in the program. Asking these questions
after the program is completed gives the participants the perspective to indicate
if they feel that their knowledge has changed as a result of the program. Participants’
knowledge change is represented by the difference between the knowledge level
prior to the program and the knowledge level after the program. Here are a few
examples:
Participant Aspirations for Change
Aspiration FSLL Goals:
"Initiative participants will aspire to implement recommended behavior/
practice changes."
What are Aspirations?
Aspirations are really self-reported perceptions of participants as to the
likelihood that they will do something or change the way they do something as
a result of participating in the program. Aspirations can be measured at the
end of a program or, in the case of a multi-segmented program, at the completion
of segments of the program using a written end-of-program survey or verbally
through a face-to-face interview at the completion of the program's delivery.
(Note: Aspirations are NOT the same as Practice Changes! Aspirations are indications
that participants say that they intend or plan to use or adopt what was learned,
not that they have already used or adopted it.
Response choices used for aspiration questions/items may be
simply: "Yes/No" or "Check All That Apply." Specific practices/behaviors that
are included in the evaluation instrument will be those that were addressed in
the program in which the individuals participated.
Participant Practice/Behavior Changes
Practice/Behavior Change FSLL Goals:
"Increase the number of Initiative participants who:
-
utilize recommended practices in managing their use of credit in light
of their long-term goals for later life
-
initiate contributions to a retirement savings plan or increase contributions
to retirement plans
-
determine retirement income needs and/or future income needs
-
develop a plan to achieve retirement and/or future income goals
-
establish or revise investment goals
-
participate in employer-provided retirement plans
-
increase their contributions to employer-provided retirement plans
-
develop a plan for managing long-term health care needs
-
develop an integrated plan for accumulating, protecting, and distributing/transferring
assets"
What are Practice/Behavior Changes?
Practice/behavior changes are measurements of ACTUAL changes in behaviors or
practices by participants that occur as a result of participation in a program.
Change can be defined as either the adoption of a new practice/behavior or the
increased utilization of a current behavior/practice. For example, someone may
already have had an existing retirement savings plan and have increased their
monthly contribution to this plan as a result of the FSLL program.. If we ask
that person if they initiated a retirement savings program as a result of the
program, they would honestly answer NO. However, if we ask them if they “have
initiated or increased contributions to a retirement savings plan,” they
could answer YES.
Actual practice/behavior change cannot, in most cases, be determined immediately
at the conclusion of the program delivery. The program participants must be
given an opportunity to implement the changes. Therefore, measuring practice/behavior
changes usually involves some type of follow-up survey of participants. The
length of time between the completion of instruction and the follow-up measurement
will vary by the specific practice/behavior change. Some changes may take place
in a matter of weeks, while other changes may take months to implement. You,
the educator, need to determine what is an appropriate period of time to allow
before conducting a follow-up with program participants about possible practice/behavior
changes.
Evidence of Practice/Behavioral Changes
Evidence of practice/behavior changes can be measured through actual observation
of the behavior/practice change or through participants’ self-reports
of practice/behavior changes. Observed practice/behavioral changes related to
financial security in later life might include physical evidence (e.g., a written
budget or retirement plan) or documentation that provides evidence of practice
changes (e.g., retirement fund statements showing increased contributions).
These types of evidence may be available if you are working with individuals
more one-on-one and they feel comfortable in sharing their records.
Methods of Measuring Practice/Behavioral Changes
In most cases, practice/behavior changes will need to be measured through some
type of survey or interview with participants that asks them to self-report
any changes that have occurred as a result of the program. Such a survey may
be conducted face-to-face, by telephone or by mail.
Another way to conduct a follow-up would be to hold an additional program meeting/activity
that is part of the original program schedule that can be used as an opportunity
to collect follow-up information about practice/behavior changes. This event
would have to be held after a suitable amount of time has passed to give participants
opportunities to make changes. In addition to using the event as an evaluation
data collection opportunity, you could also plan some additional activities
for the event, such as additional educational programming or a “celebration”
activity to celebrate the successes of participants. This event would also be
an excellent time to do some needs assessment to get ideas for future programs.
Asking questions about actual practice/behavior changes is very similar to
the way you would ask participations about their aspirations to make changes.
The difference is that, rather than asking them if they plan to change practices/behaviors,
you are now asking them if they actually did make those changes. The same formats
(i.e.,checklists, YES/NO) can also be used for questions asking participants
about actual changes.
Economic Changes
Economic Change FSLL Goals:
"Initiative participants will experience increased financial security
throughout later life."
The ultimate goal of the Financial Security in Later Life Initiative is for
participants to increase their financial security through the adoption or increased
use of recommended practices taught through Extension educational programs.
Increased financial security can be operationally defined for FSLL evaluation
purposes as participants’ self-reported perceptions of increased financial
security and/or evidence of positive financial changes (e.g., increased dollars
saved, amount of debt reduced, new dollars invested for later life needs, etc.)
that have resulted from participation in a FSLL-related program.
The objectives in the FSLL Evaluation and Accountability Database related to
economic change and possible ways to measure these outcomes are as follows:
FSLL Indicator 14. Number of participants reporting/demonstrating increased
financial security.
Evidence demonstrating increased financial security might include records or
documents indicating new or increased contributions to investment plans, retirement
accounts, closed credit card accounts, long-term health care plans, etc. If
you are working with clientele one-on-one or in small groups, you may have opportunities
to observe these types of evidence of increased financial security. Observation
of these changes in financial security records or documents by the educator
can be documented through the educator’s program notes or the use of a
checklist by the educator. Report the number of individuals who have such evidence
Program participants can also self-report changes in their financial security
as a result of participating in a FSLL-related program. This is a softer measure,
but may be the only way to capture evidence of financial security change with
some audiences.
Participants' Reactions
Reactions are the responses of clientele as to the value or quality of programs,
teachers, teaching methods used, materials, facilities and other factors involved
in the program. For FSLL evaluation, we are interested in participants’
perceived value of the educational program as a whole and their perceived value
of program materials. Whether or not an individual would recommend a program
to someone else is another indicator of the value of a program. Reactions should
not be confused with knowledge or skill changes. A person indicating that the
program was well presented and met their needs is NOT the same as documentation
for knowledge or skill changes.
Participants’ valuation of a program or of program materials can be measured
using a YES/NO format or a Likert-type scale of multiple levels of agreement
(ex. STRONGLY AGREE to STRONGLY DISAGREE).
Other “Reactions”-type questions you might want to include in end-of-program
evaluation instruments can help provide you with information you can use for
program improvement. These questions can also be used to elicit ideas for future
program topics and ideas. Here are some examples:
What were the three things that you liked the most about the program?
How could the program have been improved?
What topics would you like to see addressed in future programs?
*To read and access our information you must have
Adobe Acrobat Reader loaded on your computer. To
download this free program select the "Get Acrobat Reader" icon.
Once you have Acrobat Reader loaded on your computer, just select
the file you wish to read.
Back to Evaluation of Programs and Activities for Educators
|
