|
PLS 506 |
|
Program
Evaluation |
| |
| Course Objectives |
| Program
evaluation is a common form of policy analysis that is practiced widely
in the public and nonprofit sector. Program evaluations supply
information that program managers and policymakers use to make decisions
about which programs to fund, modify, expand, or eliminate. In other
words, its focus is on helping public managers and policymakers
determine whether the policy or program is working. Therefore, it is an
important accountability tool used by program managers and funders.
Thus, it is important for public managers and policy analysts to be well
versed in the basic concepts and techniques used in the evaluation of
program processes and impacts. |
| |
| All evaluations
have weaknesses, and some have more than others. A great deal of money
is also spent on evaluations and they are often very difficult to
conduct successfully. The primary goal of this course is to help
students become informed consumers of the products of evaluation
research and to learn the basic skills needed to design and conduct high
quality evaluations. Students will be exposed to the common forms of
policy and program evaluation such as needs assessments, monitoring
social programs, and assessing the effectiveness or efficiency of their
impacts. At its core, program evaluation requires the use of social
science research methodology to determine whether policies and programs
make a difference. Accordingly, much of the course content revolves
around methodological issues related to proper research design.
However, evaluations often raise important social, political, and
ethical issues. These will be discussed as well. Specifically, the
course is designed to: |
| |
 | Develop a
familiarity with the range of methods and research designs employed
in program evaluation in order to critically evaluate the quality of
a program evaluation; |
| Improve your
ability to articulate program objectives and logic models that can
be used in process or outcome evaluations; |
| Develop your
expertise in framing evaluation questions and developing a research
design that is most appropriate for answering these questions;
|
| Develop your
appreciation of the complexity of the methodological challenges and
political, social, ethical, and practical issues associated with
evaluating policies and programs; |
| Learn how to
communicate the results of your program evaluations to managers,
decisionmakers, and the general public; and, |
| Ultimately,
help you become a more effective policy analyst or public manager. |
|
| |
| Course objectives
are met by exposing you to the theories and concepts associated with
policy and program evaluation as well as practical analytical tools and
techniques derived from these theories. We will also discuss the
political and ethical implications associated with program evaluations
the use of different research designs. The course will develop your
critical thinking abilities by analyzing case studies and actual program
evaluations. Your oral communication skills will be enhanced through
class presentations. Writing assignments will be used to improve your
ability to communicate your ideas and arguments in written form. Thus,
the course helps develop a wide range of the MPA program’s learning
competencies. |
| |
|
Readings |
|
The following books are required for this course and can be purchased at
local bookstores or through vendors on the Internet: |
| |
|
Rossi, Peter H., Mark W. Lipsey, and Howard E. Freeman. 2004.
Evaluation: A Systematic Approach. Seventh Edition. Thousand
Oaks, CA: Sage. ISBN 0761908943 |
|
Wholey, Joseph S., Harry P. Hatry, and Kathryn E. Newcomer (eds.).
2010. Handbook of Practical Program Evaluation.
Third
Edition. San Francisco, CA: John Wiley & Sons
Inc.. ISBN 9780470522479 |
|
| |
|
Additional required and recommended readings have been placed on reserve
in the library or have links on the class web site. Copies of the
lecture notes and overheads used in class can also be found on the class
web site. |
| |
|
Expectations & Approach |
|
This is not a lecture-dominated class where the instructor speaks and
the students passively listen. While new theories and concepts may be
introduced through lectures, the majority of the course is structured as
a seminar. Students are expected to do the reading in advance and
course time will be spent discussing and applying the readings by
discussing program evaluations or case studies. Course readings are
supplemented with videos or occasional guest speakers. Individual and
group exercises will also be used to analyze the theories and concepts
in the readings. While part of your learning will also be cognitive or
factual in nature, much of what you learn in this course will involve
developing your analytical skills on your own and enhancing your
self-awareness of the organizations in which you work or the programs
that are of interest to you. I encourage you to share your professional
experiences with others in the class during formal and informal
discussion. Supplemental information to stimulate your learning
experience is available on the class web site. I encourage you to make
full use of these resources and suggest other web sites that can be
shared with your fellow students via the course web site. I also
encourage you to discuss the cases and program evaluations with your
fellow students outside of class. In short, students are expected to
take an active part in their learning and course discussions will
largely be shaped by the issues in the readings that are of most
interest to students. Thus, you are encouraged to prepare yourself for,
and be open to, the variety of ways that you can learn during this
course. |
| |
|
Course Requirements |
|
Your grade in this class will be based on your ability to understand and
apply the theories discussed in the course readings. More specifically,
your grade will depend on your performance on the following course
requirements: |
| |
|
Class Participation |
|
Students are expected to complete the assigned readings and come to
class prepared to discuss all readings, handouts, and assignments that
are due. To ensure that students are prepared to discuss the readings,
I will randomly ask students to summarize a reading, describe a theory,
or discuss a case. Each student should also be prepared to share with
the class questions that the readings raised and be able to identify the
most significant point or contribution that the author makes to the
practice of public administration. |
| |
|
While attendance will not be taken on a regular basis, poor classroom
attendance will result in a failing participation grade since you cannot
participate if you are not in class. Therefore, you should notify the
instructor when you are unable to attend class. If you should miss a
session when a video is shown, it may be possible to make arrangements
to view the film privately at the library or to borrow the video. In
certain circumstances it may also be possible to make-up other in-class
activities with a written assignment if frequent absences have become a
problem. Your participation grade will also be lowered due to the
frequent inability to respond to the instructor’s questions, poor class
preparation, lack of enthusiastic participation in class exercises, or
the failure to treat others in the classroom with respect (e.g., talking
while others are talking, ridiculing other students, etc.). Simply put,
in order to receive an “A” in this class you must be a civil, active
contributor to class and not merely score well on exams and assignments. |
| |
|
Exam |
|
There will be one exam at the end of the class during final exam time.
It will be an objective exam consisting of some combination of
multiple-choice, short answer, and essay questions. Several readings
may be provided in advance of the exam that will be used for exam
questions. |
| |
|
Critique of Evaluation Reported in the Popular Press (Individual) |
|
It
is not uncommon that the findings of some major evaluation are reported
in the popular press. However, it is also common that the findings are
misreported or exaggerate the positive or negative findings from the
evaluation. This assignment is designed to help you learn how to read
the news related to such findings with a critical eye. For this
assignment you are to find a report of a major policy or program
evaluation that was reported in the popular press (e.g., New York Times,
Washington Post, Wall Street Journal, Newsweek, Time, etc.) and critique
the evaluation in a 2 – 3 page memo. Since virtually every evaluation
has a weakness, you should approach your review assuming that the
evaluation was poorly done and try to look for possible weaknesses to
critique. Does the news report suggest any problems? Does it
potentially oversell the findings? In short, be critical. Your memo
should provide a concise summary and critique of the study. Your
critique should be based strictly on the information contained (or not
contained) in the article. It should also identify the relevant
information that should have been included for the reader to make a
proper judgment and interpretation of the evaluation findings. You are
not to read the original study. You are required to submit a
copy of the article (preferably in electronic form) that you reviewed
that is suitable for distribution to your classmates at a later date.
|
| |
|
Case Analysis (Individual) |
|
It
is not uncommon for evaluations to raise a variety of social, political,
and ethical issues. In order to explore these issues, we will analyze a
case study. Since case analysis is essentially self-learning through
simulated experience, its success depends on the lively exchange of
information, ideas, and opinions during case discussions. Students must
come prepared to discuss all aspects of the case, apply the concepts in
the readings to the case, and be able to answer questions posed by the
instructor or fellow classmates or it will adversely affect your class
participation grade. Your analysis of the case will be in the form of a
memo limited to no more than 3 pages in length. The course website will
provide additional guidance for completing this assignment. |
| |
| Responsible Conduct of Research (RCR) Training
(Individual) |
| RCR training
has become more and more commonplace in graduate schools across the
country, not just in the sciences. In fact, the National Science
Foundation will soon mandate RCR training for any graduate student who
is supported on an NSF grant and many instructors
require this training as part of regular course requirements.
There is no better way for you to familiarize yourselves with the
responsibilities for the responsible conduct of evaluation research than
to have you complete the same training module used by thousands of
faculty and students around the country. This assignment requires completing the RCR online training course
offered through the Collaborative Institutional Training
Initiative (CITI) at the University of Miami.
The directions for registering for and completing the training can be
found at the bottom of the following webpage:
http://www.uncw.edu/orssp/Responsible-Conduct.html. You then need to submit a copy of your completion
certificate to get a grade for this assignment. |
| |
|
Critique of a Program Evaluation (Group) |
|
Even if you never do a program evaluation, it is important for public
managers and policy analysts to be a good consumer of evaluations and be
able to communicate complicated evaluation findings to a lesser informed
audience of policymakers or the public. For this assignment, your group
must identify a major evaluation of a policy or program implemented in
the public or nonprofit sector. The evaluation should be from a
reputable governmental or nongovernment organization (e.g., Brookings,
Urban Institute, etc.) and be available for download on the internet
(you must provide the web address). It
must also employ an experimental or quasi-experimental design.
Your group should prepare a short concise critique of the evaluation in
the form of a 4 - 5 page memorandum. The critique should describe the
treatment (i.e., program), its intended targets, the intended benefits,
the program’s causal model/program logic/theory, the research design,
methods, and conclusions. You should then critique the evaluation’s
major strengths and weaknesses with a critical eye and highlight any
major sources of potential bias or problems related to the study’s
validity. You will present your findings during the second half of the
semester. The week before your presentation you should circulate to the
rest of class your memo, a hyperlink to the final report, and if
available a link to an executive summary. Your presentation is limited
to 12 minutes and should be prepared as if you were briefing an audience
of policymakers and the public. |
| |
|
Program Evaluation
Project (Individual) |
|
In
order to learn how to apply the theories and concepts from the readings,
students will also complete a Program Evaluation Project. You have three options for completing the assignment: |
| |
|
Conduct an empirical study that evaluates a policy or program; or |
|
Research Design for a policy or program; or |
|
Conduct a meta-analysis of evaluation studies. |
|
| |
|
The course website describes the requirements associated with each
option in greater detail. Regardless of the option selected, students
will be required to complete two short written assignments based on the
policy or program. First, students will be required to submit a short 1
-2 page Program Memo that summarizes the program, problem, intervention,
and intended beneficiaries. The Measurement Memo uses the program’s
causal model/program logic/theory developed in the first memo to specify
the research questions, operational definitions, and specific measures
used to gauge program success. While ungraded, your effort on these
assignments will be factored into your participation grades. The memos
will also provide a chance to provide constructive feedback prior to
submitting the final report. |
| |
|
All students will then prepare a final report consistent with the
options chosen. It should be professional in appearance and include a
cover, table of contents, executive summary, the main body of the
evaluation report (e.g., discussion of program, the causal model/program
logic/theory, research questions/hypotheses, research design, data
collection, data analysis procedures, and where applicable findings and
conclusions), supporting tables and figures, a bibliography, and any
necessary appendices or supporting materials. The report is expected to
be clear, concise, and professional. It should communicate your
findings effectively to your client. Final reports will probably be
around 25 double-spaced pages in length but will vary based on the
option, program, and analysis. All students will presenter their
evaluation reports in class. Students will have 10 minutes for your
presentation. Your presentation should be designed as if you were
briefing a potential funder or evaluation sponsor. |
| |
|
Grading |
|
All written work will be evaluated based on your analysis of the
readings and cases, the organization of your ideas, the strength and
substance of your arguments, your ability to properly apply the concepts
discussed in class, and the quality of your writing (e.g., spelling,
grammar, punctuation, etc.). Failure to complete an assignment or
failing to follow directions will result in a 0 for the assignment. The
final course grade will be calculated based upon the following weights: |
| |
| Class Participation (10%) |
100 points |
| Critique of Popular Press Article
(Individual) (10%) |
100 points |
| Responsible Conduct of Research
(Individual) (10%) |
100 points |
| Critique of Program Evaluation (Group)
(10%) |
100 points |
| Case Study (Individual) (10%) |
100 points |
| Program Evaluation Project (Individual) (25%) |
250 points |
| Exam (25%) |
250 points |
|
| |
|
Course grades will be calculated based on the following: A (920 – 1000),
A- (900 – 919), B+ (880 – 899), B (820 – 879), B- (800 – 819), C+ (780 –
799), C (720 – 779), C- (700 – 719), D+ (680 – 699), D (620 – 679), D-
(600 – 619), and F (0 – 599). There is no rounding up or down. In the
event that an assignment is given a letter grade, points are allocated
by using the midpoint of the grade range and multiplying it the number
of possible points. For example, an A for a 5% memo would equal 95% of
50 points or 47.5 while a B+ would equal 89% of 50 points or 44.5.
There is no rounding up or down. |
| |
| All members of
UNCW’s community are expected to follow the academic Honor Code. Please
read the UNCW Honor Code carefully (as covered in the UNCW Student
Handbook). Academic dishonesty in any form will not be tolerated in
this class. Be advised that I have a zero tolerance policy for cheating
or plagiarism. Anyone caught cheating or plagiarizing an assignment
will receive a failing grade for the course. |
| |
|
Late Assignments/Incompletes |
|
Tentative due dates for course assignments are listed on the attached
course schedule and are subject to change based on class progress and
student preferences. You should check the course schedule and
announcements pages on the course web site periodically for changes in
due dates. Students may bring forward requests for changing the dates
of major assignments for class approval with a minimum of one-week
advance notice to the class. |
| |
|
Unless I have agreed in advance, all assignments are due at the end of
the class period they are due. Informing the instructor of your
intention to be absent does not waive your obligation to submit the work
that is due. Late assignments lose one full letter grade per class they
are late. An incomplete will only be granted when there are serious
extenuating circumstances that occur after the withdrawal period. |
| |
|
Disabilities, Respect, and Harassment |
| Students with
diagnosed disabilities should contact the Office of Disability Services
(962-7555). Please give me a copy of the letter you receive from Office
of Disability Services detailing class accommodations you may need. If
you require accommodation for test-taking please make sure I have the
referral letter no less than three days before the test. |
| |
| UNCW has recently
instituted a Respect Compact to affirm our commitment to a civil
community, characterized by mutual respect. That Compact will soon be
affixed to the wall of each classroom and can be accessed at:
http://www.uncw.edu/stuaff/pdc/documents/SeahawkRespectCompact.pdf.
Students are expected to treat others in this class in accordance with
this campus policy. |
| |
| UNCW practices a
zero tolerance policy for any kind of violent or harassing behavior. If
you are experiencing an emergency of this type contact the police at 911
or UNCW CARE at 962-2273. Resources for individuals concerned with a
violent or harassing situation can be located at http://www.uncw.edu/wsrc/crisis.html. |
| |
|
Other Class Policies |
| Cell phone use
will not be tolerated in this class. Turn off your cell phones prior to
class. If your phone rings, the Professor reserves the right to answer
the call and/or to confiscate the phone. |
| |
| Laptops may be
used in this class. Laptop users should sit in the back row to avoid
distracting others. Laptops will be banned from the classroom for the
rest of the semester the first time any laptop user is seen engaging in
non-classroom related activity. |
| |
|
Download a PDF Version of the
Syllabus by Clicking Here |
