Information Resources on

Program Evaluation

Program evaluation is the systematic study using objective measures to analyze how well a program is working. (GAO-02-923, pg. 6)

Center for Evaluation Methods and Issues, USGAO

December 2003

NOTE: While not exhaustive, this is a list of resources that Center staff have found useful for learning about program evaluation. The list is periodically updated, but please note that Internet addresses often change. For further information or additional suggestions, contact Stephanie Shipman at (202) 512-2700.

INFORMATION RESOURCES ON PROGRAM EVALUATION

The categories of resources included are professional associations and networking groups, local continuing education opportunities in the Washington, DC area, evaluation web sites with multiple links, GAO publications, periodicals, and books and reports.

PROFESSIONAL ASSOCIATIONS AND NETWORKING GROUPS

American Evaluation Association (AEA) – the profession’s national association. Offers annual national conference, electronic mailing list, peer-reviewed journals, training workshops, and topical interest groups.

AEA Central Office

16 Sconticut Neck Road #290

Fairhaven, MA 02719

Phone/Fax 888-232-2275 (toll free in US and some of Canada)

E-mail: AEA@kistcon.com Website: http://www.eval.org

LISTSERV: EVALTALK, to subscribe send e-mail to listserv@bama.ua.edu

Washington Evaluators – Washington, DC local affiliate of the American Evaluation Association. Holds meetings, offers electronic mailing list, and has website that advises members of upcoming meetings, employment opportunities, and other relevant news. http://www.washeval.org

Contact wursterr@state.gov for membership information, job and event postings.

Eastern Evaluation Research Society – affiliate of AEA serving DE, MD, NJ, PA.

Contact wursterr@state.gov. or see http://www.eers.org

Inspectors General, President’s Council on Integrity and Efficiency (IG- PCIE) –- Inspection and Evaluation Committee. Offers a monthly newsletter and roundtable meetings. Website lists publications, membership, and employment opportunities. The IG-related IG Auditors Training Institute offers evaluation-related courses. http://www.ignet.gov/

Contact Johnnie Frazier, Inspector General, Department of Commerce, at jfrazier@oig.doc.gov.

Brown Bag Lunch Series on Performance Measurement and Evaluation –

Monthly discussions jointly sponsored by GAO, George Washington University, and the Center for Accountability and Performance of the American Society of Public Administration (ASPA). http://www.aspanet.org/cap

To attend, contact Kathryn Newcomer at newcomer@gwis2.circ.gwu.edu.

LOCAL CONTINUING EDUCATION OPPORTUNITIES

U.S. Department of Agriculture, Graduate School – Courses include: Introduction to Program Evaluation; Applying Impact Evaluation Methods to Audits; Performance Measurement and Budgeting. It also offers a Certificate in Program and Management Analysis. Call toll free at 888-744-GRAD. See http://grad.usda.gov.

The Evaluators’ Institute – Summer institute held in DC offers short-term professional development courses for practicing evaluators. Course topics include: the basics of program evaluation, evaluation models and theory, qualitative methods to show causal connections, and performance measurement. For program announcement, see http://www.EvaluatorsInstitute.com or contact Registrar@EvaluatorsInstitute.com or call 302-644-0990 (fax 302-644-0991).

Howard University Evaluation Training Institute – Offers training courses on math and science project evaluation, contact vthomas@howard.edu.

Local universities – also offer individual undergraduate and graduate level courses and degrees.

EVALUATION WEB SITES

AEA, American Evaluation Association. Provides links to AEA’s Guiding Principals for Evaluators and AEA’s Position Statement on High Stakes Testing, its Topical Interest Groups, local affiliates, and to international evaluation sites, graduate and professional development programs, and evaluation tools and handbooks. http://www.eval.org

Canadian Evaluation Society. Provides links to regional affiliates, position papers, resources and "data banks" of evaluation reports by 14 Canadian federal agencies and 2 private sector organizations. http://www.evaluationcanada.ca/site.cgi?section=1&_lang= The society is also developing a list of potential benefits derived from evaluation, such as support accountability for program performance and spending. http://consultation.evaluationcanada.ca/englishhome.htm

The Evaluation Center at Western Michigan University. Provides information on their Evaluation Checklist Projects and links to their other evaluation reports. http://www.wmich.edu/evalctr. The center also maintains a directory of self-identified evaluators developed with support from the National Science Foundation. Source information on the “Directory of Evaluators” is listed under Reports.

The Evaluation Exchange. The Harvard Family Research Project’s evaluation periodical is a free newsletter providing an exchange of ideas and practices in the evaluation of family support services and community development.

http://gseweb.harvard.edu/~hfrp/eval.html

Pennsylvania State University, College of Agricultural Sciences, Cooperative Extension. Program Evaluation. Provides evaluation planning tools including Tipsheets Marginof Error: How to Explain it; Reduce Evaluation Burden; Which

Program Should I Evaluate?; Increasing Response to Surveys. University Park, PA, September 2001. http://www.extension.psu.edu/evaluation.

Bill Trochim’s Center for Social Research Methods. Has materials including on-line hypertext courses on applied social research methods such as Program Evaluation and Research Design. http://trochim.human.cornell.edu.

The Treasury Board of Canada (functional equivalent to the U.S. OMB). Provides links to Canadian federal policies, guides and standards for program evaluation and international evaluation groups. http://www.tbs-sct.gc.ca/eval/home_acceuil_e.asp (keep a line between home and acceuil and e.asp.)

U.S. Agency for International Development, Center for Development Information and Evaluation. Provides evaluation and performance monitoring tips, and agency evaluation reports. http://www.dec.org/usaid_eval. (Keep a line between usaid and eval.) Its new EvalWeb offers resources and a collaborative online community. http://www.dec.org/partners/evalweb.

U.S. Department of Education,

- Policy and Program Studies Service (PPSS). Provides links to its evaluation reports and offers subscriptions to various electronic newsletters and journals on education. http://www.ed.gov/about/offices/list/ous/ppss/evaluation.html

- New Institute of Education Sciences (IES). Lists current research funding opportunities. http://www.ed.gov/about/offices/list/ies/index.html?src=oc

IES sponsors the “What Works Clearinghouse.” http://www.w-w-c.org/

U.S. Department of Health and Human Services,

- Centers for Disease Control and Prevention Evaluation Working Group. Provides numerous links to public health evaluation and logic model resources. http://www.cdc.gov/eval/resources.htm. This group developed a framework that summarizes and organizes the basic elements of program evaluation in public health, see “The Framework for Program Evaluation in Public Health.”

- Inspector General, Office of Inspections & Evaluation. Has links to their reports and resources for conducting evaluations and inspections. http://oig.hhs.gov

- Office of the Assistant Secretary for Planning and Evaluation (ASPE). Offers a periodic newsletter,“PIC Highlights,” summarizes recently completed HHS sponsored studies. The studies are listed in the Policy Information Center (PIC) database of about 7,000 abstracts of completed and on-going evaluations. http://aspe.hhs.gov/pic/hilites/index.shtml Links to the Performance Improvement annual report listing agency program evaluations are available at http://aspe.hhs.gov/progsys/index.shtml.

U.S. Department of State, Bureau of Education and Cultural Affairs, Office of Policy and Evaluation. Has links to its completed and on-going outcome evaluations. http://exchanges.state.gov/education/evaluations/index.htm

United Way Outcome Measurement Resource Network. Provides links to measuring community outcomes, evaluation resources and their evaluation guide, "Measuring Program Outcomes: A Practical Approach." http://national.unitedway.org/outcomes

University of Wisconsin-Cooperative Extension Service. Program Development and Evaluation. Provides evaluation planning tools, sample questionnaires, data collection and analysis guidance, samples and guidance on logic models, and reporting ideas. It uses extension-related examples. http://www.uwex.edu/ces/pdande/.

GAO PUBLICATIONS

Ordering Information: Telephone (202) 512-6000; send mail to U.S. General Accounting Office, P.O. Box 37050, Washington, DC 20013; or visit GAO’s web site http://www.gao.gov and access these publications under “Other Publications” and “GAO Reports.” The .pdf files need to be read in Acrobat.

GAO Guides and Methods Transfer Papers:

·Case Study Evaluations. GAO/PEMD-10.1.9, November 1990. http://www.gao.gov/special.pubs/pe1019.pdf

·Content Analysis: A Methodology for Structuring and Analyzing Written Material. GAO/PEMD-10.3.1, September 1996.

·Designing Evaluations. GAO/PEMD-10.1.4, March 1991. http://www.gao.gov/special.pubs/pe1014.pdf

·Developing and Using Questionnaires. GAO/PEMD-10.1.7, October 1993.

·Performance Measurement and Evaluation: Definitions and Relationships. GAO/GGD-98-26, April 1998. http://www.gao.gov/special.pubs/gg98026.pdf

·Prospective Evaluation Methods: The Prospective Evaluation Synthesis. GAO/PEMD-10.1.10, November 1990.

·Quantitative Data Analysis: An Introduction. GAO/PEMD-10.1.11, June 1992. http://www.gao.gov/special.pubs/pe10111.pdf

·Record Linkage and Privacy: Issues in Creating New Federal Research and Statistical Information. GAO-01-126SP, April 2001. http://www.gao.gov/cgi-bin/getrpt?rptno=gao-01-126sp Very large file.

·The Evaluation Synthesis. GAO/PEMD-10.1.2, March 1992. http://www.gao.gov/special.pubs/pe1012.pdf

·The Results Act: An Evaluator’s Guide to Assessing Agency Annual Performance Plans. GAO/GGD-10.1.20, April 1998. http://www.gao.gov/special.pubs/gg10120.pdf

·Using Statistical Sampling. GAO/PEMD-10.1.6, May 1992.

·Using Structured Interviewing Techniques. GAO/PEMD-10.1.5, July 1991. http://www.gao.gov/special.pubs/pe1015.pdf

GAO Reports:

Program Evaluation: An Evaluation Culture and Collaborative Partnerships Help Build Agency Capacity (GAO-03-454, May 2, 2003). http://www.gao.gov/new.items/d03454.pdf

Program Evaluation: Strategies for Assessing How Information Dissemination Contribute to Agency Goals (GAO-02-923, Sept. 29, 2002). http://www.gao.gov/new.items/d02923.pdf

Program Evaluation: Studies Helped Agencies Measure or Explain Program Performance (GAO/GGD-00-204, Sept. 29, 2000). http://www.gao.gov/cgi-bin/fetchrpt?rptno=ggd-00-204

Managing for Results: Views on Ensuring the Usefulness of Agency Performance Information to Congress (GAO/GGD-00-35, Jan. 26, 2000). http://www.gao.gov/cgi-bin/fetchrpt?rptno=ggd-00-35

Performance Plans: Selected Approaches for Verification and Validation of Agency Performance Information (GAO/GGD-99-139, July 30, 1999). http://www.gao.gov/cgi-bin/fetchrpt?rptno=ggd-99-139

Managing for Results: Measuring Program Results That Are Under Limited Federal Control (GAO/GGD-99-16, Dec. 11, 1998). http://www.gao.gov/archive/1999/gg99016.pdf

Grant Programs: Design Features Shape Flexibility, Accountability, and Performance Information (GAO/GGD-98-137, June 22, 1998). http://www.gao.gov/archive/1998/gg98137.pdf

Program Evaluation: Agencies Challenged by New Demand for Information on Program Results (GAO/GGD-98-53, April 24, 1998). (Results of GAO's survey on the roles and resources available for federal program evaluation and the characteristics of evaluation studies conducted.)

http://www.gao.gov/cgi-bin/fetchrpt?rptno=ggd-98-53

Managing for Results: Analytic Challenges in Measuring Performance (GAO/HEHS/GGD-97-138, May 30, 1997). http://www.gao.gov/archive/1997/h297138.pdf

Block Grants: Issues in Designing Accountability Provisions (GAO/AIMD-95-226, Sept. 1995). (Under GAO Reports search GAO Publications for AIMD-95-226.)

Program Evaluation: Improving the Flow of Information to the Congress (GAO/PEMD-95-1, Jan. 30, 1995). (Under GAO Reports search GAO Publications for PEMD-95-1).

PERIODICALS

· American Journal of Evaluation (A publication of AEA. AEA membership includes receipt of this journal. Published by Pergamon Press, Elsevier Publishing, http://www.elsevier.com).

· New Directions for Evaluation (A publication of AEA. AEA membership includes receipt of this journal. Published by Jossey-Bass, http://www.jbp.com).

· Evaluation and Program Planning (Published by Pergamon Press, Elsevier Publishing).

· Evaluation Review (Published by Sage Publications, Inc., http://www.sagepub.com).

· Evaluation: The International Journal of Theory, Research and Practice (Published by Sage Publications in association with The Tavistock Institute).

BOOKS AND REPORTS

AEA. Guiding Principles for Evaluators. American Evaluation Association. No date. http://www.eval.org/Publications/publications.html

Centers for Disease Control and Prevention. “Framework for Program Evaluation in Public Health.” MMWR 1999; 48 (No. RR-11).

Chelimsky, E.. and Shadish, W.R. Jr., editors. (1997) Evaluation for the 21^st Century. Thousand Oaks, CA: Sage Publications.

Denzin, N.K. and Lincoln, Y.S., editors. (2000) Handbook of Qualitative Research. (2^nd ed.). Thousand Oaks, CA: Sage Publications.

“Directory of Evaluators.” The Evaluation Center at Western Michigan University.

http://www.wmich.edu/evalctr

Hatry, H. (1999) Performance Measurement: Getting Results. Washington, DC: Urban Institute Press.

Hatry, H., van Houten, T., Plantz, M., and Greenway, M.T. (1996) Measuring Program Outcomes: A Practical Approach. United Way of America. http://national.unitedway.org/outcomes/library/pgmomres.cfm

Mark, M.M., Henry, G.T., & Julnes, G. (2000) Evaluation: An Integrated Framework for Understanding, Guiding, and Improving Policies and Programs. San Francisco, CA: Jossey-Bass.

Miles, M.B. & Huberman, M. (1994) Qualitative Data Analysis: An Expanded Sourcebook (2^nd ed.). Thousand Oaks, CA: Sage Publications.

National Science Foundation. “User-Friendly Handbook for Mixed Methods Evaluation.” Directorate for Education and Human Resources, Division of Research,

Evaluation and Communication. August 1997. NSF Report # 97-153. http://www.ehr.nsf.gov/EHR/REC/pubs/NSF97-153/pdf/mm_eval.pdf.

National Science Foundation “User-Friendly Handbook for Project Evaluation.” Directorate for Education and Human Resources, Division of Research, Evaluation and Communication. No date. NSF Report # 93-152. http://www.nsf.gov/cgi-bin/getpub?nsf93152

Owen, J.M. & Rogers. P.J. (1999) Program Evaluation: Forms and Approaches. Thousand Oaks, CA: Sage Publications.

Patton, M.Q. (1997) Utilization Focused Evaluation: The New Century Text (3^rd ed.). Thousand Oaks, CA: Sage Publications.

Posavac, E.J. & Carey, R.G. (1997) Program Evaluation: Methods and Case Studies. (5^th ed.). Upper Saddle River, NJ: Prentice-Hall.

Rossi, P.H., Freeman, H.E. & Lipsey, M.W. (2003) Evaluation: A Systematic Approach (7^th ed.). Thousand Oaks, CA: Sage Publications.

Sonnichsen, Richard C. (1999) High Impact Internal Evaluation: A Practitioner’s Guide to Evaluating and Consulting Inside Organizations. Thousand Oaks, CA: Sage Publications.

Shadish, W.R., Cook, T.D. & Leviton, L.C. (1991) Foundations of Program Evaluation. Thousand Oaks, CA: Sage Publications.

The Joint Committee on Standards for Educational Evaluation. (1994) The Program Evaluation Standards. Thousand Oaks, CA: Sage Publications. (Also available at http://www.eval.org/Publications/publications.html)

U.S. Department of Health and Human Services, Office of the Inspector General. Practical Evaluation for Public Managers: Getting the Information You Need. Nov. 1994. http://oig.hhs.gov/organization/OEI/other/pepm.pdf

Weiss, C.H. (1998) Evaluation: Methods for Studying Programs and Policies (2^nd ed.). Upper Saddle River, NJ: Prentice-Hall.

Wholey, J. S., Hatry, H.P. & Newcomer, K.E. (1994) Handbook of Practical Program Evaluation. San Francisco, CA: Jossey-Bass.

W.K. Kellogg Foundation. (1998) W.K. Kellogg Foundation Evaluation Handbook. Item # 1203. http://www.wkkf.org/Programming/ResourceOverview.aspx?CID=281

Worthen, B.R., Sanders, J.R. & Fitzpatrick, J.L. (1997) Program Evaluation: Alternative Approaches and Practical Guides (2^nd ed.). New York: Longman.