EDN 523. Research in Education

Howard Coleman and Martin A. Kozloff

Spring, 2006

Dr. COLEMAN - Phone – 962-7287

Office Hours: Mondays – 2:00 – 6:00 and Wednesdays – 2:00 – 6:00

M-F email or call to schedule appointments

Course Description

This course will teach students the essential features of valid, credible, useful research in education. These features include:

1. The logic of research.

2. Distinguishing between research that is well-designed, yields logically valid and therefore credible and useful findings vs. research that is ill-designed, and therefore of questionable validity, credibility, and usefulness.

3. Using knowledge of curriculum design and research to evaluate curricula, instruction, and professional development/supervision.

4. Connecting the issues to be examined or the problem to be solved, the appropriate strategies (quantitative and/or qualitative; survey, ethnographic, and experimental) to use, and the methods for conducting research; e.g., defining concepts, developing instruments, collecting information, analyzing data, and drawing conclusions.

We will organize the course around developing a district or school Reading First or Striving Readers proposal. To do this, we will focus on teaching reading, on reading research, and on research in general.

Most materials are on the web. Some materials will be handed at as the needed.

Reading and other assignments are specified in the course outline (below).

Please complete assignments by the time material is scheduled for class.

Course grades will be based on the total points earned for assignments.

Unit I. Core Ideas

This unit is an overview of the core components,choices, and requirements of useful research. We will generally follow theoutline below

Validity Credibility Usability

Feasibility

Acceptability

A. Kinds of research questions.

1. What is the state of things; e.g., achievement by grade level, sex, ethnicity, social class, teachers?

This question focuses on a cross-section at one time. Comparing achievement by ethnicity and sex of students vs. ethnicity and sex of teachers may suggest relationships. What would the data look like if there were/were no relationship?

2. Are things improving, worsening, staying the same? Under what conditions are they doing so?

Both questions require longitudinal research. The second one is looking for relationships. Causal? Mere correlation but no causal link? What would you have to know in order credibly to infer causation?

[See B, below] Read pp. 1-8, on propositions, in http://www.uncw.edu/people/kozloffm/The Logic of Theory and Research.doc

3. Which factors, if any, promote, weaken, have no effect (e.g., on teacher satisfaction)? What would you have to know in order credibly to infer causation?

4. Which methods work better (e.g., supervision and assistance)? What would you have to know in order credibly to infer causation?

B. B. Rules for Inductive Inferences (identifying relationships, whether causal or merely co-variation).

1. Establish time order, so as not to reverse cause-effect.

2. Obtain evidence of co-variation in variables.

3. Compare and contrast samples (the same samples over time and changing conditions [longitudinal], or different samples in different conditions [cross-sectional]) so as to use methods of inductive inference: concomitant variation, difference, agreement, joint agreement and difference, residues (John Stuart Mill).

4. Weaken rival hypotheses (threats to internal and external validity).

See http://www.uncw.edu/people/kozloffm/The Logic of Theory and Research.doc—pp. 24-28, the sections on what it takes to draw a causal inference, deductive fallacies (e.g., affirming the consequent), and informal fallacies (e.g., hasty generalization).

Here, too… http://people.uncw.edu/kozloffm/fallacies.doc pp. 3-5 on deduction (see especially, the fallacy of affirming the consequence); informal fallacies (see especially, 10 [false cause], 11 [wrong direction}, 13 [hasty generalization], 14 [equivocation]).

And here. "Experimental Designs" http://people.uncw.edu/kozloffm/experimentaldesigns.doc

the section on internal and external validity, beginning on page 5.

C. Validity of what?

1. Definitions.

a. Too broad (include what should not be included)?

b. Too narrow (exclude what should be included)?

c. Clear terms (refer to observable events)?

d. Operational definitions properly derived from conceptual definitions?

See http://www.uncw.edu/people/kozloffm/The Logic of Theory and Research.doc

Concepts (p. 1)

Conceptual and operational definitions (pp. 21-22)

2. Measures.

a. Levels: nominal (male, female), ordinal (Stinko --> So so --> Wow), interval

(10, 20, 30, 40 degrees), ratio (10 episodes; 77%; 50 correct words per minute).

b. Questions (open-ended; with rating scales); rating scales; event recording; narrative recording.

b. Validity: face, construct, criterion, pragmatic.

See http://people.uncw.edu/kozloffm/EDN523levelsofanal.html

http://people.uncw.edu/kozloffm/creatingqandint.doc

http://people.uncw.edu/kozloffm/datacollection.doc

3. Instrumentation/data collection devices

a. Interview, questionnaire, event rating scales in direct observation, event recording (ratio level) in direct observation, narrative recording, content analysis.

See http://people.uncw.edu/kozloffm/creatingqandint.doc

http://people.uncw.edu/kozloffm/datacollection.doc

b. Validity: face, construct, criterion, pragmatic.

c. Objective?

d. Field tested?

4. Design.

a. Case study, survey, document analysis (e.g., official statistics), experiment (natural, intervention).

See http://people.uncw.edu/kozloffm/datacollection.doc

b. Phase, extent, basic question. Pilot project (worth repeating?), replication (fluke or reliable?), demonstration (how it works), dissemination (try it on a larger scale).

See http://people.uncw.edu/kozloffm/typesofresearch.html

http://people.uncw.edu/kozloffm/experimentaldesigns.doc

c. Controls threats to internal and external validity; e.g., via comparison groups, reliability checks, pre- and post measures, samples (random, representativeness—convenience, probability samples, bias?), random allocation or matching?

See "Experimental Designs" http://people.uncw.edu/kozloffm/experimentaldesigns.doc

the section on internal and external validity, beginning on page 5.

5. Data.

a. Accurate?

b. Reliable? (inter-observer and intra-observer).

c. Properly recorded on forms.

6. Data analysis. See http://www.ecs.org/html/educationIssues/Research/primer/understandingtutorial.asp

a. Proper descriptive and inferential statistics.

b. If qualitative, proper identification of classes of events, sequences, connections, time order, conditions.

c. Proper display; e.g., graphs, tables, diagrams

7. Findings.

a. Statements supported by data?

8. Conclusions.

a. Implications properly derived from findings? Warranted? Neither too narrow nor too broad nor too firm?

b. Next steps (action, research) properly derived from findings and implications?

Unit II. Evaluating Research

This unit focuses on (1) principles of logic that (2) provide criteria for evaluating research (3) which we will use to evaluate articles and data bases and that also (4) serve as guidelines for designing research and making instructional decisions; e.g., how to tell whether it is reasonable to use "Mel and Ned's Fenomenal Fonics" in your school or to toss the brochure in the trash, and how to design a pilot test if that seems reasonable.

Step 1. Read the following papers and extract (1) issues or considerations and then (2) criteria and guidelines (do this, don't do that) for evaluating and conducting research and for making instructional decisions. For example,

Consideration: If a new method is tested on a small scale but is advocated for adoption on a wide scale, there is the strong possibility of committing the fallacy of hasty generalization. The population is sure to have characteristics that are not in the small sample. Therefore, findings from the study of the small sample may not apply to the population (that is, the findings are not externally valid). Therefore, the method is not likely to work uniformly. But without knowledge of what population characteristics affect effectiveness (intervening variables), you can't predict when the method will or won't work.

Guideline: Innovations must be replicated in increasingly large studies that approximate the larger population. Intervening variables that affect outcomes must be identified.

Write your entries in a form you might use to make a handout or give a presentation to school staff. Call it something like, "What is scientifically based research?" (In other words, this is an assignment.)

1. From

http://www.uncw.edu/people/kozloffm/The Logic of Theory and Research.doc—pp. 24-28, the sections on what it takes to draw a causal inference.

And http://people.uncw.edu/kozloffm/fallacies.doc pp. 3-5 on deduction (see especially, the fallacy of affirming the consequence); informal fallacies (see especially, 10 [false cause], 11 [wrong direction}, 13 [hasty generalization], 14 [equivocation]).

2. From "Experimental Designs" http://people.uncw.edu/kozloffm/experimentaldesigns.doc —the section on internal and external validity, starting on page 5.

3. " Identifying and implementing educationalpractices supported by rigorousevidence."http://people.uncw.edu/kozloffm/RigorousEvidence.pdf

4. "What research says about reading." http://people.uncw.edu/kozloffm/whatresearchsays.htm

5. "Using research and reason in education." http://people.uncw.edu/kozloffm/Research%20and%20Reason.pdf

6. "A policy makers primer on educational research." [Basic statistics made easy.] http://www.ecs.org/html/educationIssues/Research/primer/understandingtutorial.asp

7. From Report of the National Reading Panel, Addendum, Methodology.

http://www.nichd.nih.gov/publications/nrp/findings.htm

8. From the IDEA website on Analysis of Reading Assessment Instruments for K-3. http://idea.uoregon.edu:16080/assessment/

Criteria by which they evaluated instruments.

Step 2. Evaluate a research report using your paper (above) on scientifically based research. (This is another written assignment.) Get an article from the library or on line. Good choices include writings on whole language, brain based learning, multiple intelligence, Reading Recovery, Four Blocks, learning styles, Accelerated Reader, and developmentally appropriate practices. How does the report stack up with respect to the criteria you identified? What should the researchers have done instead. You might use two columns.

Positive/negative regarding criteria How to improve

1. Authors claim that their teaching method is effective, 1. Compare pre-and post-test scores.

based on students' scores on a post-test, as compared 2. Use comparison groups to control for

to post-test scores of last year's class. However, there rival explanations.

is no pre-test (to show where students started); 3. Use randomized groups.

the past year's or the current class may have

characteristics that account for the differences in

achievement apart from the teaching method;

and there is no comparison group (to show if the new

method is associated with higher achievement).

Step 3. What makes a good research base? Evaluating a research base involves evaluating individual studies (as in step 2, above) and evaluating the quality of the whole sample of studies that constitute the base. Important issues here are:
1. Were researchers independent of the product being tested/sold?

2. Were there comparison groups?

3. Were there replications? Where the replications such that we can have confidence in internal validity and external validity?

Advocates and publishers make claims such as "Our curriculum has been extensively field tested." Or "Our curriculum is based on scientific research." Too often, "testing" and "research" mean (1) qualitative research (questionable reliability of data and findings, not controlled comparisons); (2) pre-post designs (no comparison groups), short time period (not longitudinal), using instruments that do not adequately measure the phenomenon; (3) testimony of "experts" (fallacy of appeal to authority); or (4) cascades of irrelevant citations (e.g., citing research or theory about how speaking is learned without systematic instruction, and therefore reading instruction, too, requires little systematic instruction). [The illogic, here, is the fallacy of overgeneralization--if one aspect of language behavior is learned without systematic instruction, so are other aspects of language behavior.] Still, administrators and teachers are taken in by these claims, and implement pernicious trash that wastes time and money, students' learning opportunities, and teachers' hard work and good will.

Use both the above (step 1) writings, especially Report of the National Reading Panel, and the following writings to develop considerations and guidelines for evaluating a research base. (This is your third written assignment.)

1. "A synth esisof research on reading."http://www.nrrf.org/synthesis_research.htm

2. Review of Research on Reading Recovery. [Note sources of invalidity.]
http://people.uncw.edu/kozloffm/readingrecovery.doc

3. Grossen, The research base for Reading Mastery. http://darkwing.uoregon.edu/~adiep/rdgtxt.htm

4. Research base for Corrective Reading. http://www.sraonline.com/index.php/home/curriculumsolutions/di/correctivereading/correctivereadresear/666

5. Research base for Success for All. http://www.successforall.net/research/research.htm#1

6. Research base for Open Court. http://www.sraonline.com/index.php/home/curriculumsolutions/reading/ocr2002/ocrresearch/190

6. Borman et al, Comprehensive school reform and student achievement. [Note criteria for inclusion, and cautions, in conclusion section.]

You MUST see the whole document at http://www.csos.jhu.edu/crespar/techReports/report59.pdf

Additional research bases and reviews include the following. [You should put these in your school, and they should be used as part of the final authority.]

1. Cotton, K. (1995). Effective school practices: A research synthesis 1995 Update.

http://www.nwrel.org/scpd/esp/esp95.html

2. Walberg, H.J. & Paik, S. (2000). Effective education practices. Educational Practices Series. International Academy of Education. International Bureau of Education.

http://www.ibe.unesco.org/International/Publications/EducationalPractices/EducationalPracticesSeriesPdf/prac06e.pdf

3. Kozloff, M.A. (2002). Sufficient Scaffolding, Organizing and Activating Knowledge, and Sustaining High Engaged Time.

http://www.uncwil.edu/people/kozloffm/scaffolding.pdf

4. Rosenshine, B. (1997). Advances in Research on Instruction.

http://epaa.asu.edu/barak/barak.html

5. Rosenshine, B. (1997). The Case for Explicit, Teacher-led, Cognitive Strategy Instruction. http://epaa.asu.edu/barak/barak1.html

6. Ellis, E.S., & Worthington, L.A. (1994). Research Synthesis on Effective Teaching Principles and the Design of Quality Tools for Educators.

http://idea.uoregon.edu/~ncite/documents/techrep/tech05.pdf

7. Anderson, J.R., Reder, L.M., & Simon, H.A. Applications and Misapplications of Cognitive Psychology to Mathematics Education. Department of Psychology. Carnegie Mellon University. Pittsburgh, PA 15213. Online at http://act.psy.cmu.edu/personal/ja/misapplied.html\

12. Dixon, R. "Review of High Quality Experimental Mathematics Research." University of Oregon. National Center to Improve the Tools of Educators.

On-line at http://idea.uoregon.edu/~ncite/documents/math/math.html

Unit III. Planning Research

This unit examines kinds of research, steps in conducting research, and (building on unit I) design considerations

Read

1. Types of Research in a Sequence. http://people.uncw.edu/kozloffm/typesofresearch.html

2. Outline of Steps in Planning Research and Writing Proposals. http://people.uncw.edu/kozloffm/outlineofstepsinplanning.html

3. Steps in Planning Research Projects and Writing Proposals. http://people.uncw.edu/kozloffm/EDN523StepsinResearch.html

4. Reading First proposal form. http://people.uncw.edu/kozloffm/schoolRFproposal.htm

6. Reading First materials. http://people.uncw.edu/kozloffm/resources_for_reading_first.htm

Also see

a. Big ideas in beginning reading. Eugene, OR: University of Oregon. http://reading.uoregon.edu/

b. Moats, L.C. (1999). Teaching reading is rocket science. Washington, DC: American Federation of Teachers.

On line at http://people.uncw.edu/kozloffm/rocketsci.pdf

c. Moats, L.C. (1995). The missing foundation in teacher education. American Educator, 19(2), 43-51.

On-line at http://www.greenwoodinstitute.org/resources/resmiss.html.

d. Oregon Reading First Center (2003). Review of Comprehensive Programs. http://reading.uoregon.edu/curricula/or_rfc_review.php

e. Analysis of Reading Assessment Instruments for K-3. http://idea.uoregon.edu:16080/assessment/

Criteria by which they evaluated instruments.

Writing.

The major course project is to write a Reading First proposal. We’ll do it as a group—with a division of labor. But first, let’s plan the main outline of a RF grant…

(1) What is the objective? (implement and evaluate new reading curriculum and professional development).

(2) What data say that change, of a certain kind, is needed? (data on reading achievement. Other?)

(3) What are the features of the intervention—(a) reading curricula (core, supplemental, intervention); (b) assessment (screening, diagnostic, progress monitoring, outcome); (c) five main reading skills (phonemic awareness, phonics, fluency, vocabulary, comprehension); (d) teacher training and supervision; (e) the “host environment” (readiness? How can you tell?)

(4) What research design will be used, and for which aspects? (Overall, an experiment? [Which design?] Also survey? [Within and/or across schools; e.g., if you gradually expanded scope?] Also ethnographic? [Classroom learning communities? How school discusses change? Interview kids and families?]

(5) What data will be used to establish baseline, assess implementation, assess progress, assess outcomes? (Eogs, standardized reading assessments, teachers’ knowledge and practice (?), school culture and social patterns)

(6) What research is there on different curricula? (See 7d, above) Also check Google for Voyager, Success for All, Open Court, Reading Mastery, etc.

Unit IV. Experimental Research

This unit examines experimental research—the type of research that most closely and most conspicuously follows the canons of scientific research.

Read.

1. http://people.uncw.edu/kozloffm/experimentaldesigns.doc

Writing.

Which aspects of a RF implementation would have experimental designs?

Why?

Unit V. Instruments and Data Collection

This unit examines the deductive logic used to create instruments and to collect data using instruments. We focus on questionnaires, interviews and interview schedules, direct observation (narrative recording and event recording), and tests.

Read.

1. Levels and scales. http://people.uncw.edu/kozloffm/EDN523levelsofanal.html

2. Measures and instruments. http://people.uncw.edu/kozloffm/creatingqandint.doc

3. Data collection. http://people.uncw.edu/kozloffm/datacollection.doc

4. See reading assessment instruments at the IDEA website.

Writing.

1. Regarding our developing research plan, why and how would you use interviews, questionnaires, and direct observation? [Triagulation? Different methods for measuring different sort of things?]

2. Identify variables to measure.

Unit VI. Putting It All Together

In this unit we will finish the RF proposal. That is, we will answer all of the questions on the proposal form and create as many instruments and data collection protocols as we can. We will also examine Corrective Reading—a remedial/intervention program that can be used as early as grade 3.