EDN 523 Research in Education
Howard Coleman
Phone – 962-7287
Email: colemanh@uncw.edu
Office Hours: Mondays – 2:00 –
6:00 and Wednesdays – 2:00 – 6:00
M-F email or call to schedule
appointments
Course
Description
This
course will teach students the essential features of valid, credible, useful
research in education. These features
include:
1. The logic of research.
2. Distinguishing between research that is well-designed, yields logically valid and therefore credible and useful findings vs. research that is ill-designed, and therefore of questionable validity, credibility, and usefulness.
3. Using knowledge of curriculum design and research to evaluate curricula, instruction, and professional development/supervision.
4. Connecting the issues to be examined or the problem to
be solved, the appropriate strategies
(quantitative and/or qualitative; survey, ethnographic, and experimental) to
use, and the methods for conducting
research; e.g., defining concepts, developing instruments, collecting
information, analyzing data, and drawing conclusions.
Most
materials are on the web. Some materials will be handed at as the needed.
Please
complete assignments by the time material is scheduled for class.
Course grades will be based on the total points earned for assignments.

Unit I. Core Ideas
This unit is an overview of the core components,choices, and requirements of useful research. We will generally follow theoutline below
Validity Credibility Usability
![]()
Feasibility
Acceptability
A. Kinds of research questions.
1. What is the state of things; e.g., achievement by grade level, sex, ethnicity, social class, teachers?
This question focuses on a cross-section at one time. Comparing achievement by ethnicity and sex of students vs. ethnicity and sex of teachers may suggest relationships. What would the data look like if there were/were no relationship?
2. Are things improving, worsening, staying the same? Under what conditions are they doing so?
Both questions require longitudinal research. The second one is looking for relationships. Causal? Mere correlation but no causal link? What would you have to know in order credibly to infer causation?
[See B, below] Read pp. 1-8, on propositions, in http://www.uncw.edu/people/kozloffm/The Logic of Theory and Research.doc
3. Which factors, if any, promote, weaken, have no effect (e.g., on teacher satisfaction)? What would you have to know in order credibly to infer causation?
4. Which methods work better (e.g., supervision and assistance)? What would you have to know in order credibly to infer causation?
B. B. Rules for Inductive Inferences (identifying relationships, whether causal or merely co-variation).
1. Establish time order, so as not to reverse cause-effect.
2. Obtain evidence of co-variation in variables.
3. Compare and contrast samples (the same samples over time and changing conditions [longitudinal], or different samples in different conditions [cross-sectional]) so as to use methods of inductive inference: concomitant variation, difference, agreement, joint agreement and difference, residues (John Stuart Mill).
4. Weaken rival hypotheses (threats to internal and external validity).
See http://www.uncw.edu/people/kozloffm/The Logic of Theory and Research.doc—pp. 24-28, the sections on what it takes to draw a causal inference, deductive fallacies (e.g., affirming the consequent), and informal fallacies (e.g., hasty generalization).
Here, too… http://people.uncw.edu/kozloffm/fallacies.doc pp. 3-5 on deduction (see especially, the fallacy of affirming the consequence); informal fallacies (see especially, 10 [false cause], 11 [wrong direction}, 13 [hasty generalization], 14 [equivocation]).
And here. "Experimental Designs" http://people.uncw.edu/kozloffm/experimentaldesigns.doc
the section on internal and external validity, beginning on page 5.
1. Definitions.
a. Too broad (include what should not be included)?
b. Too narrow (exclude what should be included)?
c. Clear terms (refer to observable events)?
d. Operational definitions properly derived from conceptual definitions?
See http://www.uncw.edu/people/kozloffm/The Logic of Theory and Research.doc
Concepts (p. 1)
Conceptual and operational definitions (pp. 21-22)
2. Measures.
a. Levels: nominal (male, female), ordinal (Stinko --> So so --> Wow), interval
(10, 20, 30, 40 degrees), ratio (10 episodes; 77%; 50 correct words per minute).
b. Questions (open-ended; with rating scales); rating scales; event recording; narrative recording.
b. Validity: face, construct, criterion, pragmatic.
See http://people.uncw.edu/kozloffm/EDN523levelsofanal.html
http://people.uncw.edu/kozloffm/creatingqandint.doc
http://people.uncw.edu/kozloffm/datacollection.doc
3. Instrumentation/data collection devices
a. Interview, questionnaire, event rating scales in direct observation, event recording (ratio level) in direct observation, narrative recording, content analysis.
See http://people.uncw.edu/kozloffm/creatingqandint.doc
http://people.uncw.edu/kozloffm/datacollection.doc
b. Validity: face, construct, criterion, pragmatic.
c. Objective?
d. Field tested?
4. Design.
a. Case study, survey, document analysis (e.g., official statistics), experiment (natural, intervention).
See http://people.uncw.edu/kozloffm/datacollection.doc
b. Phase, extent, basic question. Pilot project (worth repeating?), replication (fluke or reliable?), demonstration (how it works), dissemination (try it on a larger scale).
See http://people.uncw.edu/kozloffm/typesofresearch.html
http://people.uncw.edu/kozloffm/experimentaldesigns.doc
c. Controls threats to internal and external validity; e.g., via comparison groups, reliability checks, pre- and post measures, samples (random, representativeness—convenience, probability samples, bias?), random allocation or matching?
See "Experimental Designs" http://people.uncw.edu/kozloffm/experimentaldesigns.doc
the section on internal and external validity, beginning on page 5.
5. Data.
a. Accurate?
b. Reliable? (inter-observer and intra-observer).
c. Properly recorded on forms.
6. Data analysis. See http://www.ecs.org/html/educationIssues/Research/primer/understandingtutorial.asp
a. Proper descriptive and inferential statistics.
b. If qualitative, proper identification of classes of events, sequences, connections, time order, conditions.
c. Proper display; e.g., graphs, tables, diagrams
7. Findings.
a. Statements supported by data?
8. Conclusions.
a. Implications properly derived from findings? Warranted? Neither too narrow nor too broad nor too firm?
b. Next steps (action, research) properly derived from findings and implications?
Unit II. Evaluating Research
Step 1. Read the following papers and extract (1) issues or considerations and then (2) criteria and guidelines (do this, don't do that) for evaluating and conducting research and for making instructional decisions. For example,
Consideration: If a new method is tested on a small scale
but is advocated for adoption on a wide scale, there is the strong possibility
of committing the fallacy of hasty generalization. The population is sure to have characteristics
that are not in the small
sample. Therefore, findings from the
study of the small sample may not apply to the population (that is, the
findings are not externally valid).
Therefore, the method is not likely to work uniformly. But without knowledge of what population
characteristics affect effectiveness (intervening variables), you can't predict
when the method will or won't work.
Guideline: Innovations must be replicated
in increasingly large studies that approximate the larger population.
Intervening variables that affect outcomes must be identified.
Write your entries in a form you might use to make a handout or give a presentation to school staff. Call it something like, "What is scientifically based research?" (In other words, this is an assignment.)
1. From
http://www.uncw.edu/people/kozloffm/The Logic of Theory and Research.doc—pp. 24-28, the sections on what it takes to draw a causal inference.
And http://people.uncw.edu/kozloffm/fallacies.doc pp. 3-5 on deduction (see especially, the fallacy of affirming the consequence); informal fallacies (see especially, 10 [false cause], 11 [wrong direction}, 13 [hasty generalization], 14 [equivocation]).
2. From "Experimental Designs" http://people.uncw.edu/kozloffm/experimentaldesigns.doc —the section on internal and external validity, starting on page 5.
3. " Identifying and implementing educationalpractices supported by rigorousevidence."http://people.uncw.edu/kozloffm/RigorousEvidence.pdf
4. "What research says about reading." http://people.uncw.edu/kozloffm/whatresearchsays.htm
5. "Using research and reason in education." http://people.uncw.edu/kozloffm/Research%20and%20Reason.pdf
6. "A policy makers primer on educational research." [Basic statistics made easy.] http://www.ecs.org/html/educationIssues/Research/primer/understandingtutorial.asp
7. From Report of the National
http://www.nichd.nih.gov/publications/nrp/findings.htm
8. From the IDEA website on Analysis of Reading Assessment Instruments for K-3. http://idea.uoregon.edu:16080/assessment/
Criteria by which they
evaluated instruments.
Step 3. What makes a good research
base?
Evaluating a research base involves evaluating individual studies (as in step
2, above) and evaluating the quality of the whole sample of studies that
constitute the base. Important issues here are:
1. Were researchers independent of the product being tested/sold?
2. Were there comparison groups?
3. Were there replications? Where the replications such that we can have confidence in internal validity and external validity?
Advocates and publishers make claims such as "Our curriculum
has been extensively field tested." Or "Our curriculum is based on
scientific research." Too often, "testing" and
"research" mean (1) qualitative research (questionable reliability of
data and findings, not controlled comparisons); (2) pre-post designs (no
comparison groups), short time period (not longitudinal), using instruments
that do not adequately measure the phenomenon; (3) testimony of
"experts" (fallacy of appeal to authority); or (4) cascades of
irrelevant citations (e.g., citing research or theory about how speaking is
learned without systematic instruction, and therefore reading instruction, too, requires
little systematic instruction). [The
illogic, here, is the fallacy of overgeneralization--if one aspect of language
behavior is learned without systematic instruction, so are other aspects of
language behavior.] Still,
administrators and teachers are taken in by these claims, and implement
pernicious trash that wastes time and money, students' learning opportunities,
and teachers' hard work and good will.
Use both the above (step 1) writings, especially Report of the National Reading Panel,
and the following writings to develop considerations and guidelines for
evaluating a research
base. (This
is your third written assignment.)
1. "A synth esisof research on reading."http
://www.nrrf.org/synthesis_research.htm3. Grossen, The research base for Reading Mastery. http://darkwing.uoregon.edu/~adiep/rdgtxt.htm
5. Research base for Success for All. http://www.successforall.net/research/research.htm#1
You MUST see the whole document at http://www.csos.jhu.edu/crespar/techReports/report59.pdf
1. Cotton, K. (1995). Effective school
practices: A research synthesis 1995 Update.
http://www.nwrel.org/scpd/esp/esp95.html
http://www.uncwil.edu/people/kozloffm/scaffolding.pdf
4.
Rosenshine,
B. (1997). Advances in Research on Instruction.
http://epaa.asu.edu/barak/barak.html
http://idea.uoregon.edu/~ncite/documents/techrep/tech05.pdf
On-line at http://idea.uoregon.edu/~ncite/documents/math/math.html
a. Big ideas in beginning reading. Eugene, OR: University of Oregon. http://reading.uoregon.edu/
On line at http://people.uncw.edu/kozloffm/rocketsci.pdf
c. Moats, L.C. (1995). The missing foundation in teacher education. American Educator, 19(2), 43-51.
On-line at http://www.greenwoodinstitute.org/resources/resmiss.html.
e. Analysis of Reading Assessment Instruments for K-3. http://idea.uoregon.edu:16080/assessment/
Criteria by which they evaluated
instruments.
Which
aspects of a RF implementation would have experimental designs?
1. Levels and scales. http://people.uncw.edu/kozloffm/EDN523levelsofanal.html
2. Measures and instruments. http://people.uncw.edu/kozloffm/creatingqandint.doc
3. Data collection. http://people.uncw.edu/kozloffm/datacollection.doc
4.
See reading assessment instruments at the IDEA website.
2. Identify variables to measure.