Identifying relevant outcomes

Measuring and monitoring program outcomes

Magnitude of program effects

Techniques for detecting effects

Assessing the practical significance of program effects

Qualitative analysis

Techniques for quantitative Analysis

Sampling designs and procedures

Measurement and reliability

Meta-analysis

Rossi, et al. Chapters 7, 10

Wholey, et al. Chapters 19, 20, 22

Reserve #2, #7, #8, #9

Read this interesting article from The Atlantic (12/11) on hidden biases and how they can complicate the use of surveys

It is also important for evaluators to not only think about whether something is statistically significant but also in terms of the substantive significance of the relationships or(e.g., outcomes or impacts) of a program.  The problem of course is that people often have trouble understanding the meaning of numbers.  Watch this short clip from David Letterman for some examples of the problems people often have in terms of interpreting numbers:

http://www.youtube.com/watch?v=2dJanAgydps

Studies can reach faulty conclusions in many ways.  One obvious problem is confusing correlations with causality.  See this interesting article with examples of spurious correlations from Business Insider (5/14).  Sometimes, closer examination of the role that sample size plays raises other issues.  See this interesting article from Reuters (3/16/14) looking at the plunge in pre-school obesity.  Was it really due to government intervention of bad math?

Download as a PDF file (Notes related to reliability & validity)

Figures on reliability & validity

Want to learn more about conducting a meta-analysis? The Cochrane Collaboration has a set of online training modules for researchers considering conducting meta-analysis.  Of particular interest might be this page on what is a meta-analysis and this page describing the difference between a systemic review and a meta-analysis. 

Meta-Evaluation bibliography

Report - Program Evaluation: Experienced Agencies Follow a Similar Model For Prioritizing Research Needs - GAO-11-176

Agranoff, R. & Radin, B. A. (1991). The comparative case study approach in public administration. In Research in Public Administration JAI Press. pp. 203 – 231.

Babbie, Earl. 1989. The Practice of Social Research. Fifth Edition. Belmont, CA: Wadsworth Publishing Co.

Dillman, Don A. 2000. Mail and internet surveys: the tailored design method. Second Edition. New York, NY: John Wiley and Sons.

Glaser, B. G. & Strauss, A. L. 1967. The discovery of grounded theory: Strategies for qualitative research. New York, NY: Aldine.

Long, J. Scott (1997), Regression Models for Categorical and Limited Dependent Variables (Thousand Oaks, CA: SAGE Publications)

Maxwell, J. A. 1996. Qualitative research design: An interactive approach. Thousand Oaks, CA: SAGE Publications.

Miles, M. B. & Huberman, A. M. 1994. Qualitative data analysis: An expanded sourcebook, Second Edition. Thousand Oaks, CA: SAGE Publications.

O’Sullivan, E. & G. Rassel. 1989. Research methods for public administrators. New York, NY: Longman.

Patton, Michael Quinn. 1990. Qualitative Evaluation and Research Methods. Second Edition. Newbury Park, CA: SAGE Publications.

Singleton, Royce A., Bruce C. Straits, Margaret Miller Straits. 1993. Approaches to Social Research. Second Edition. New York, NY: Oxford University Press.

Strauss, A. & Corbin, J. 1990. Basics of qualitative research: Grounded theory procedures and techniques. Newbury Park, CA: SAGE Publications.

Yin, Robert K.  2003.  Case Study Research:  Design and Methods 3^rd ed.  Thousand Oaks, CA:  Sage Publications.