Foundations of Research
Quantitative Research
Numerical data
Experimental
Casual Comparative
Single Subject Experiment
Multiple Regression
Correlation
Descriptive/Survey
Basic and Applied Research (for theories)
Basic use to develop or refine a theory
Applied use to apply or test theory
Education Research (Formative and Summative)
Formative: use to improve a program/product under development.
Summative: use to evaluate the overall quality of a program/product to make decisions
Research and Development (R&D)
Use to research consumer needs and develop a product to fulfill those needs.
*Is not to test a theory.
Action Research
Use to solve everyday problems.
*not to test theory and result are not to generalized to other settings.
choice of paradigms (quanitative, qualitative, or action)
not a matter of personal preferences; rather, it is a matter of the research question that is being answered.
Which research design is best for educators?
The type of research one chooses is really based on the research question they want to answer. That is the first, most important, consideration.
paradigm
a perspective based on a set of assumptions, concepts, and values that are held by a community or researchers.
quantitative variables
vary in degree or amount (e.g., annual income varies from zero to very high income)
independent variables
the presumed cause of another variable
example: amount of studying affects test grade.
Dependent variables
the presumed effect or outcome.
are influenced by one or more independent variables.
intervening variables
also called mediator or mediating variables.
are variables that occur between two other variables.
example, tissue damage is an intervening variable in the smoking and lung cancer relationship.
Smoking>Tissue Damage>Lung Cancer
random assignment
which creates "equivalent" groups
used in the strongest experimental research designs
active manipulation of an independent variable
it is only in experimental research that "manipulation" is present
extraneous variable
a variable that may compete with the independent variable in explaining the outcome. Remember this, if you are ever interested in identifying cause and effect relationships you must always determine whether there are any extraneous variables you need to worry about.
moderator variable
delineates how a relationship of interest changes under different conditions.
Example:
Perhaps the relationship between studying(IV) and test grades (DV) changes according to different levels of use of a drug such as Ritalin (MV).
nonexperimental research
no manipulation of the independent variable. There also is no random assignment of participants to groups.
if you ever see a relationship between two variables in nonexperimental research you cannot jump to a conclusion of cause and effect because there will be too many other alternative explanations for the relationship.
case of correlational research
one quantitative IV and one quantitative DV.
Example: Selfesteem (IV) and class performance (DV).
· You would look for the relationship by calculating the correlation coefficient.
· The correlation coefficient is a number that varies between 1 and +1, and 0 stands for no relationship. The farther the number is from 0, the stronger the relationship.
· If the sign of the correlation coefficient is positive (e.g., +.65) then you have a positive correlation, which means the two variables move in the same direction (as one variable increases, so does the other variable). Education level and annual income are positively correlated (i.e., the higher the education, the higher the annual income).
· If the sign of the correlation coefficient is negative (e.g., .71) then you have a negative correlation, which means the two variables move in opposite directions (as one variable increases, the other decreases). Smoking and life expectancy are negatively correlated (i.e., the higher the smoking, the lower the life expectancy).
"basic case" of causalcomparative research
one categorical IV and one quantitative DV
Example: Gender (IV) and class performance (DV).
· You would look for the relationship by comparing the male and female average performance levels.
3 Necessary Conditions for Causation
To conclude changes in variable A cause changes in variable B.
Condition 1:Relationship conditon  Variable A and Variable B must be related.
Condition 2: Temporal Antecedent Condition  proper time order must be established.
Condition 3: Lack of alternative explanation condition the relationship between variable A and B must not be due to some confounding extraneous or third variable
Qualitative Research Methods
1 Phenomenology
2 Ethnography
3 Case study research
4 Grounded theory
5 Historical research
Phenomenology
a form of qualitative research in which the researcher attempts to understand how one or more individuals experience a phenomenon. For example, you might interview 20 widows and ask them to describe their experiences of the deaths of their husbands.
Ethnography
is the form of qualitative research that focuses on describing the culture of a group of people. Note that a culture is the shared attitudes, values, norms, practices, language, and material things of a group of people. For an example of an ethnography, you might decide to go and live in a Mohawk communities and study the culture and their educational practices.
Case study research
is a form of qualitative research that is focused on providing a detailed account of one or more cases. For an example, you might study a classroom that was given a new curriculum for technology use.
Grounded theory
is a qualitative approach to generating and developing a theory form data that the researcher collects. For an example, you might collect data from parents who have pulled their children out of public schools and develop a theory to explain how and why this phenomenon occurs, ultimately developing a theory of school pullout.
Historical research
research about events that occurred in the past. An example, you might study the use of corporeal punishment in schools in the 19th century.
Mixed Research Methods
general type of research in which quantitative and qualitative methods, techniques, or other paradigm characteristics are mixed in one overall study.
Advantages of Mixed Research
First of all, we advocate the use of mixed research when it is feasible. We are excited about this new movement in educational research and believe it will help qualitative and quantitative researchers to get along better and, more importantly, it will promote the conduct of excellent educational research
Mixed method research
is research in which the researcher uses the qualitative research paradigm for one phase of a research study and the quantitative research paradigm for another phase of the study. For example, a researcher might conduct an experiment (quantitative) and after the experiment conduct an interview study with the participants (qualitative) to see how they viewed the experiment and to see if they agreed with the results. Mixed method research is like conducting two ministudies within one overall research study.
Mixed model research
is research in which the researcher mixes both qualitative and quantitative research approaches within a stage of the study or across two of the stages of the research process. For example, a researcher might conduct a survey and use a questionnaire that is composed of multiple closedended or quantitative type items as well as several openended or qualitative type items. For another example, a researcher might collect qualitative data but then try to quantify the data.
Methods of Knowing
Experience
Authority (someone we know and trust tells us)
Reasoning/Thinking
*All limited so as researchers we use Scientific Method
Inductive Reasoning
Developing generalizations based on observation of a limited number of related events or experiences.
Ex:
An instructor reviews research methods texts and notices sampling content in each text and concludes that all research methods texts have sampling
A teacher knows several student athletes who are good students and concludes that all student athletes are good students
Deductive Reasoning
Arriving at specific conclusions based on general principles, observations, or experiences.
Ex:
All research textbooks contain a chapter on sampling. The book you are reading is a research text and therefore must contain a sampling chapter.
All student athletes are good students. Mary is a student athlete and therefore she must be a good student.
Steps of the Scientific Method
Recognition and identification of a problem
Formulation of hypothesis
Data collection
Data analysis
Statement of conclusions
Confirm or disconfirm hypothesis
Limitations of the Scientific Method
The scientific method can not answer all questions, especially those of a philosophical or ethical nature.
•Application of the scientific method can never capture the full richness of the context.
•Measurement error is a limitation of the scientific method.
goal of educational research
describe, explain, predict, and control situations involving human beings.
Selection and definition of a problem
A problem is a question of interest.
•The problem can be tested or the question answered through the collection and analysis of data.
•Generally researchers use a review of the existing literature to generate hypotheses related to their question.
Execution of research procedures
Research procedures are dictated by the research problem and the identified variables.
•Procedures include activities related to collecting data about the problem.
Analysis of Data
Data are analyzed such that the researcher can test the hypothesis or answer the research question.
•Data analysis often includes statistical techniques.
Drawing and stating conclusions
Conclusions are based upon analyses of our data and are stated in terms of the original hypothesis or research question.
Conclusions should indicate whether the hypothesis was rejected or supported.
For those studies that include synthesis of verbal data, conclusions may be more tentative.
Educational research is often broadly categorized as qualitative or quantitative.
Both quantitative and qualitative researchers collect and analyze data.
Both quantitative and qualitative researchers derive conclusions and interpretations.
Approaches to Educational ResearchQuantitative
Numerical data
Describes, predicts or controls variables of interest
The world is relatively uniform, stable, and predictable
Researchers state hypothesis, specify research procedures, and control context.
Researchers identify large samples.
Researchers are interested in statistical significance
Approaches to Educational ResearchQualitative
Narrative and visual data
Gains insights into phenomenon of interest
Knowledge is situated and contextual. There are different perspectives
Researchers form foreshadowed problems. They don't control but explain context.
A large amount of data is collected over an extended time in a natural setting.
Data are categorized and organized into patterns
Example of Quantitative
Are there differences in the amount of discipline referrals between girls and boys with Autism Spectrum Disorders?
What variable best predicts whether first generation college students stay continually enrolled in college until graduation?
Do students exposed to animated science materials learn more if the animation is accompanied by sound or no sound?
Example of Qualitative
What are the social experiences of middle school girl with an Autism Spectrum Disorder?
What challenges do first generation college students from small rural schools experience as they enroll in college?
How do students describe their experiences learning from animated science materials?
Survey Research
determines and reports the 'current status' of the subject of study.
often collects numerical data to test hypotheses or answer questions.
may examine preferences, attitudes, or opinions
Ex:
How do parents feel about national teacher certification?
To what degree do students report test anxiety before the SAT exam?
Correlational Research
provides a quantitative measure of relationship between or among variables.
This measure is expressed as a correlation coefficient.
Ranges from 1 to 1; 0 indicates no relationship
do not indicate causeeffect relations among variables.
Ex:
A high correlation between selfconcept and achievement does not mean that selfconcept causes higher achievement.
CausalComparative Research
attempts to determine the cause or reason for existing differences.
The grouping variable is the potential cause.
The dependent variable is the effect.
Ex:
Are there differences in final exam performance between students given a practice exam and those given more time for independent study?
Are there differences between elderly men and women who recently experienced a stroke in the amount of minutes of exercise during rehabilitation prior to release from hospital?
CausalComparative Research
provides limited causeeffect data but true causeeffect findings can be determined only through experimental research.
Sometimes it is impossible to conduct true experiments such as when grouping variables:
Cannot be manipulated.
e.g., year in school, age, gender
Should not be manipulated.
e.g., attended rehabilitation or not; exposed to verbal abuse or not
Experimental Research
includes at least one independent variable and the effect is measured on at least one dependent variable.
In experimental research extraneous variables are controlled.
researchers select participants, group participants, administer treatments, control the research setting, control the length of treatment exposure, select research measures, and are able to draw causeeffect conclusions.
Examples of Experimental Research
Sometimes experimental research is not possible in educational contexts because of difficulties with random selection and random assignment.
Examples:
Are there differences in achievement between students randomly assigned to one of two problem solving strategy conditions?
The independent variable is problem solving strategy conditions.
The dependent variable is achievement.
Is there an effect of corrective feedback on students' achievement on final physics test scores and interest in physics?
The independent variable is corrective feedback.
The dependent variables are final test scores and interest.
SingleSubject Research
used to study behavior change within a person or a group as they are exposed to an intervention or a treatment.
In singlesubject research the size of the sample is one.
Ex:
The effects of graphing intervention on offtask behavior.
The effects of a cessation program on smoking.
Qualitative Research
seek rich understandings.
conducted through sustained indepth, incontext, research.
numerous approaches: historical research, symbolic interaction, grounded theory, ethology, phenomenology
Narrative Research
study how different people experience their world.
allows for people to tell the story of their lives.
often focus on one person and collect stories of that person's life.
establish a trusting personal relationship with their participants.
The narrativeis the story of the phenomenon being investigated and also the method of inquiry.
Narrative ResearchExamples
What is the experience of students in a new coed dormitory?
How does a winning coach react when faced with a less skilled team and a losing season?
Ethnographic Research
the study of cultural patterns and perspectives of participants in their natural settings.
avoid interpretations early and enter the setting slowly.
have personal relationships with their participants.
collect data in waves, reentering the site several times.
resultant ethnography is a narrative that presents participants' everyday events.
During data collection the ethnographer identifies categories and enters themes into these categories.
Ethnographic Research Examples
How does the returning student population adjust to technology enhanced instruction in entry level college courses?
Case study
qualitative approach that examines a bounded system for study.
includes specific design, data collection techniques, and data analysis strategies.
Ex:
What are the challenges faced by a school board with shrinking district population and mandated facility renovations?
Qualitative Research Process
1.Identifying a research topic
•The initial topic is often narrowed.
2.Reviewing the literature
•Previous research is examined to garner important information.
3.Selecting participants
•Generally fewer purposely sampled participants than in quantitative studies.
4.Collecting data
•Generally interviews, observations, and artifacts serve as data
5.Analyzing and interpreting data
•Researchers analyze for themes and generate interpretations.
6.Reporting and evaluating the research
•Researchers summarize and integrate the data in narrative and visual form.
Qualitative Research
1.Qualitative researchers spend a great amount of time in the field and engage in persontoperson interactions.
2.Qualitative data are analyzed inductively.
3.Qualitative researchers avoid making assumptions and remain open to alternative explanations.
Classification of Research
The approach used in a study is determined by the research problem.
Often the same general problem can be approached from several different types of study.
Research methods are selected after the topic or question is determined
Example of how to choose which type of research to use...
Given the general topic of yearround schooling, suggest how this topic can be addressed differently by each one of the types of research :
Survey: What are parents' opinions about yearround schooling?
Correlational: Is there a relationship between standardized test scores and yearround schooling?
CausalComparative: Are there differences in amount of content mastered between children enrolled in yearround schooling versus those in traditional schooling?
Experimental: Are there differences in selfesteem between those students randomly assigned to districts with yearround schooling versus those assigned to traditional schooling?
Single subject: What are the effects of yearround schooling on the vocabulary use of students who are reading disabled?
Research by Purpose
Basic and applied research
Basic research is conducted to develop or refine theory.
Applied research is conducted to apply or test a theory.
Evaluation research
Evaluation research is a form of applied research that involves data collection for decision making.
Evaluation research may be formative or summative
Research and Development (R & D)
Research and development is the process of researching needs and developing products to fit those needs.
Action Research
Action research is a systematic inquiry conducted by teachers, principals, or other stakeholders.
Ethics of Educational Research
Ethical considerations are an important part of research.
Researchers must be aware of and attend to ethical considerations.
Two main overarching ethical rules guide researchers.
Participants should not be harmed.
Researchers obtain participants' informed consent.
Researchers must submit their proposal for review and approval.
Deception poses an ethical dilemma. If the participants know the purpose of a study, it may change their behavior.
e.g., gender, race, attitudes, medical status
When a study must use deception it causes problems for informed consent. These types of studies must undergo strict ethical review.
Ethics of Educational Researchsocieties
behavioral researchers have similar codes for ethical research largely due to the National Research Act of 1974 which Created a code for the protection of human subjects.
Ethics of Educational Researchinstitutions
have review groups that assure participant protections.
IRB (Institutional Review Board) or HSRC (Human Subjects Review Committee)
Participants should not be harmed.
Physically, mentally, socially
Research participants freely agree to participate
Researchers ensure freedom from harm
No undue risks
Personal privacy and confidentiality
Limit access of data to those who 'need to know'
Participants' involvement should not be reported
Anonymity
Study participants have complete anonymity when their identities are unknown to the researcher.
Confidentiality
Study participants are known to researcher but are not disclosed.
e.g., removing names from data
The Buckley Amendment (The Family Educational Rights and Privacy Act of 1974)
Data that identifies a student is not available without written permission
Must describe what data, for what purposes, and to whom
Ethics of Qualitative Research
In addition to ethics of educational research generally, qualitative research often poses additional ethical challenges.
The nature of the research changes so informed consent is challenging.
The close relationship between the researcher and the participant may allow the researcher to know personal and perhaps illacts of the participant that may pose ethical challenges.
Ethical Guideposts
A researcher should have an ethical perspective with regard to the research that is very close to her personal ethical position.
Informed consent should be obtained through a dialogue between the researcher and the participants
Be cognizant of the broader social principles that define your ethical stance. Potential results do not drive ethical standards.
Minimize the potential for harm to your participants.
Attend to confidentiality and omit deception.
Gaining Entry to the Research Site
Researchers need cooperation to conduct their studies.
Identify and follow procedures for gaining approval at any given site.
e.g., superintendent or school board
Procedures generally require filling out forms that describe the study.
Researchers may need permission from principals or teachers.
Written permission from parents is often required.
Gaining entry and obtaining permission often takes considerable time.
Schools and other research communities may request something in return for their participation in your study.
e.g., a final report prior to dissemination, professional development, parent education
What are some advantages of using quantitative data?
Some examples include that you can analyze vast amount of information quickly, the data can more easily be gathered, and you can more easily compare the results of different groups
What are examples of qualitative research data collection instruments?
Examples of qualitative data collection instruments are surveys; interviews notes , recordings, and transcripts; and field observation protocols.
What is quantitative data?
Quantitative data are anything that provides a quantity or numerical score, such as the scores from a test or rubric.
Survey Research
involves collecting data
to test hypotheses or to answer questions
about people's opinions on some topic or
issue.
an instrument to collect data that describes characteristics of a population.
challenging to conduct
and findings from survey research are not always easily interpreted.
Types of Survey Research
A sample survey is designed to sample participants and generalize to a population.
A census survey samples every member of the population.
Census surveys are generally done with small and accessible populations.
CrossSectional Surveys
Example: A study that examines the selfconcept of all preservice teachers at a given university.
Data are collected from selected participants at a single point in time.
These studies provide a snapshot of current beliefs or opinions.
These survey studies are not intended to provide insight into trends or changes over time.
Longitudinal Surveys
Data are collected at two or more time points.
One of the challenges when conducting longitudinal studies is attrition
Several types of longitudinal survey designs.
Trend studies
Cohort studies
Panel studies
Followup studies
Trend studies
examine changes over time in a particular population defined by some trait.
e.g., entering kindergartners
Cohort studies
examine one population selected at a particular time period.
Ex:
e.g., A researcher could study children that received speech and language support as first graders in 2007. She might examine some participants from the population in 2008 and then sample others from that population in 2010.
Panel studies
involve the same sample of individuals over a set time.
e.g., A researcher might conduct a fiveyear panel study of the first class of law students at a new institution. He might survey them each spring for five years.
Followup studies
investigate change in a previously studied population.
e.g., A researcher might be interested in a followup study in 2015 of those participants involved in the cohort study of firstgraders in speech and language support during 2007.
Ways of Conducting Survey Research
Questionnaires are a written collection of survey questions answered by a group of participants.
Interviews are oral, inperson question and answer sessions between the researcher and a respondent.
Conducting a Questionnaire Study
Stating the problem
Developing a questionnaire requires skill and time.
Researchers should plan content and format of the questionnaire.
Most surveys include one of two types of items.
Stating the problem in questionnaire study
Participants are more likely to respond to questionnaires that they perceive address a sufficiently relevant topic or problem.
Researchers should set objectives for the type of information desired from a questionnaire.
surveys include one of two types of items.
Structured items or closedended items for which participants choose among possible responses (e.g., Likert scale).
Unstructured items in which participants have freedom of response (e.g., fillin answer).
Guidelines to consider when constructing a questionnaire
Include only items that relate to the objectives of the study.
Focus each question on a single concept.
Collect demographic information.
Define terms.
Include a point of reference or comparison for participants.
Avoid leading questions.
Avoid sensitive questions to which respondents might not answer honestly.
Don't ask a question that assumes something that may not be true.
Organize items from general to specific.
Have others read your instrument and provide feedback.
Write directions for the respondents.
Include the purpose of the study at the top of the instrument.
Pilot test the questionnaire.
Conducting a Questionnaire StudyPilot testing the questionnaire
Pilot test with members of the intended sample population.
Ask respondents to make suggestions for any content to add or to delete.
Incorporate feedback from your pilot administration and make any appropriate changes.
Conducting a Questionnaire StudyPreparing the cover letter
Cover letters should accompany all surveys.
Cover letters should be brief.
Cover letters should be addressed to a specific person.
The cover letter should explain the purpose of the study.
The cover letter should include a statement regarding the importance and significance of the study.
The cover letter should include how results from the study will be shared with respondents and interested others.
Pilot test the cover letter.
Administering the Questionnaire
Selecting participants
Appropriate sampling strategies should be followed.
Distributing the questionnaire
Email
Mail
Telephone
Personal administration
Interview
Administering the QuestionnaireMail
Advantages
Inexpensive
Can be confidential or anonymous
Standardized items and procedures
Easy to score most items
Disadvantages
Response rate may be small
Cannot follow up items
Response sets possible
Limited to those who read
Administering the QuestionnaireEMail
Advantages
Easy to target respondents
Quickly administered
Inexpensive
Can be confidential or anonymous
Standardized items and procedures
Easy to score most items
Disadvantages
Not everyone has email
Multiple replies from same participant possible
Response rate may be small
May get sorted to trash
Cannot follow up items
Response sets possible
Limited to those who read
Administering the QuestionnaireTelephone
Advantages
High response rates
Quick data collection
Can reach a range of locations and respondents
Can use 800 call in numbers to increase response rates
Disadvantages
Requires phone number lists
Difficult to get indepth data
Administrators must be trained
Cost
Cell phones versus landlines
Administering the QuestionnairePersonal Administration
Advantages
Efficient when respondents are in close proximity
Personal connection with respondents
Disadvantages
Time consuming
Training
Cost
Personal connection with respondents
Administering the QuestionnaireInterviews
Advantages
Can followup responses and probe for additional information or clarity
May be recorded
Response rate
Personal connection with respondents
Disadvantages
Time consuming
Training
Cost
Interviewer bias possible
Unstructured data analysis
Personal connection with respondents
Conducting FollowUp Activities
The higher the return rate the better your study.
Low response rates limit generalizability.
Send out a reminder postcard to increase response rates.
Send out a second complete mailing to increase response rates.
Consider phonecall reminders to increase response rates.
Response rates vary.
First mailings usually return about 30%50%.
Second mailings add an additional 20%.
Most times additional mailings are not costeffective.
Researchers should try to determine possible reasons for nonresponse.
Nonresponse may be at the survey or item level.
Researchers should carefully look for patterns of nonresponse.
Tabulating Questionnaire Responses
Use scannable answer sheets if possible.
Accuracy
Time
Enter answers into spreadsheet or statistical package.
Code openended answers.
Consider available qualitative software analysis packages.
Analyzing Results
Report total sample size.
One way to report results from surveys is to share overall percentage of return and response rates per item.
e.g., Percentage that answered 'yes' and percentage that answered 'no'
Alternative reports can include total scores or mean scores by cluster or area of a survey. Such as comparisons by participant characteristics.
e.g., Percentage of men or women with a particular response.
Reports of survey studies will vary by the underlying purpose of the survey.
Correlational Research
determine the nature of relations among variables or to use these relations to make predictions.
often examine numerous variables believed to be related to complex variables (e.g., achievement).
Unrelated variables are discarded from future studies while those related may be examined further through causalcomparative or experimental studies.
High correlations among variables do not imply causation (e.g., selfconcept and achievement).
Correlational procedures are also used to examine reliability and validity.
Correlational Research Process
Problem selection
Participant and instrument selection
Design and procedure
Data analysis and interpretation
Correlation coefficients
Correlational Research ProcessProblem selection
Correlational studies are designed to explore whether and how variables are related.
Correlational studies are designed to test hypotheses regarding expected relations among variables.
Correlational Research ProcessParticipant and instrument selection
Samples are derived from acceptable sampling methods.
Sample must include at least 30 participants.
When reliability and validity of instrumentation is lower, sample size must be larger.
Correlational Research ProcessDesign and procedure
Correlational studies share a simple design. Scores for two or more variables of interest are obtained for each member of the sample and these scores are then correlated (e.g., selfconcept and achievement).
Correlational Research ProcessData analysis and interpretation
The meaning of a correlation coefficient will vary depending upon purpose.
Sample size effects the strength of a correlation coefficient.
Some correlation coefficients that are statistically significant may not represent meaningful significance.
Correlation coefficients do not represent percentage of relation between variables.
The square of the correlation coefficient indicates the amount of variance shared by the variables (shared variance).
e.g., a correlation of .50 indicates 25% shared variance.
Correlational Research ProcessCorrelation coefficients
Correlation coefficients range from 1 to 1.
A correlation of 0 indicates no relationship.
Correlation coefficients between +.35 and .35 represent a weak relationship or norelationship.
Correlation coefficients between +.35 and +.65 or between .35 and .65 represent moderaterelationships.
Correlation coefficients between .65 and 1.0 or between .65 and 1.0 represent strongrelationships.
Relationship Studies
gain insight into variables or factors that are related to a complex variable (e.g., retention, academic achievement).
help researchers to determine which variables may be suitable for future research.
provide insight into which variables should be controlled for in future causalcomparative or experimental studies.
Relationship StudiesData collection
Researchers first identify variables to be correlated.
Variables should be purposely identified.
A smaller number of carefully identified variables is preferable to a larger number (e.g., shotgun approach).
After identifying variables the researcher next identifies the appropriate population and sampling procedure to select participants for the study.
In some relationship studies data are collected all at one time or in several sessions conducted in close succession.
Relationship StudiesData analysis and interpretation
In relationship studies scores from one variable are correlated with scores for another variable; or scores for several variables are correlated with a particular variable of interest.
The result is a single correlation coefficient or a number of correlation coefficients.
The method of calculating a correlation coefficient depends upon the nature of the data.
Relationship StudiesData analysis and interpretation
Correlation coefficients
Correlation coefficients
The Pearson r coefficient is the most common and most precise coefficient. Pearson r is used for continuous variables.
The Spearman rho coefficient is appropriate to use when one of the variables are represented by rankorder data.
The phicoefficient is used when both variables are expressed as a categorical dichotomy.
Other correlation coefficients are appropriate given characteristics of the data collected, sample size, and underlying data distribution.
e.g., Kendall's tau, Biserial, Point biserial, Tetrachoric, Intraclass, eta.
Relationship StudiesData analysis and interpretationinaccurate estimates of relation among variables.
Several factors may contribute to inaccurate estimates of relation among variables.
Underlying relationships that are curvilinear will effect coefficients.
Attenuation occurs when measures have low reliability and may provide inaccurate correlation coefficients.
Restricted range in scores generally leads to underestimates of relations.
Prediction Studies
When two variables are highly related, scores on one variable can be used to predict scores on the other variable.
The variable used to predict is called the predictor.
The variable that is predicted is called the criterion.
*used to determine which variables are the most highly correlated with a criterion variable.
More than one variable can be used to make predictions.
Prediction StudiesData collection
In prediction studies all measures should be valid measures.
It is especially critical that the criterion variable be validly measured.
In prediction studies, sometimes the predictor variables are administered prior to the criterion variable (e.g., SAT and university GPA).
Attrition is a problem in some prediction studies.
Shrinkage, or the tendency to find less accuracy in predicting criterion variables in subsequent samples, is often noted in prediction studies.
Crossvalidation is the process of conducting subsequent prediction studies with new samples to verify effects found in an initial prediction study
Attrition
The action or process of gradually reducing the strength or effectiveness of someone or something through sustained attack or pressure.
a reduction or decrease in numbers, size, or strength
Prediction StudiesData analysis and interpretation
In prediction studies, data analysis involves correlating predictor variables with the criterion variable.
Multiple regression is used when a combination of variables is used to predict a criterion variable (e.g., Success in Algebra may be predicted by prior knowledge, prior achievement, aptitude, etc.)
Intervening variables may lower prediction accuracy (e.g., teacher).
The amount of common variance shared by predictors is the squared correlation of the predictors and the criterion and is referred to as the coefficient of determination.
Prediction StudiesData analysis and interpretationsingle variable prediction equation
Y= a+bX
Y= Predicted criterion score for an individual
X= An individual's score on the predictor variable
a= A constant calculated from scores of all participants
b= A coefficient that indicates the contribution of the predictor variable to the criterion variable
Other CorrelationBased Analyses
In discriminate function analysis, continuous variables are used to predict a categorical variable.
Canonical analysis produces a correlation based upon a group of predictor variables and a group of criterion variables.
Path analysis provides a diagram that illustrates how variables are related to one another.
Structural equation modeling, or LISREL, extends path analysis and predicts relations among variables with added precision.
Factor analysis is used to decrease the number of variables under consideration by grouping variables into clusters called factors.
Interpreting Correlation Coefficients
Was the proper correlation method used?
Do the variables have high reliabilities? Low reliabilities lower the chance of finding significant relations.
Is the validity of the variables strong? Invalid variables produce meaningless results.
Is the range of scores to be correlated restricted or extended? Narrow or restricted score ranges lower correlation coefficients, whereas broad or extended score ranges raise them.
How large is the sample? The larger the sample the smaller the value needed to reach statistical significance. Large samples may yield correlations that are statistically significant but practically unimportant.
CausalComparative Research
researcher attempts to determine the cause or reason for existing differences in groups or individuals.
Retrospective casualcomparative research studies start with effects and investigate causes
Prospective casualcomparative research studies start with the causes and investigate the effects.
does not establish causeeffect relations.
generally includes more than two groups and at least one dependent variable.
the independent variable is not manipulated by the researcher.
CausalComparative Research
can be conducted when variables cannot or should not be experimentally manipulated.
can facilitate decision making.
provide insight into conducted or potential experimental studies.
generally less costly than are experimental studies.
CausalComparative ResearchEx of retrospective causes and investigate the effects..
More common in educational research.
e.g., A researcher interested in the benefits of an exercise program on reducing stress may select a group of people who had enrolled in a stressreduction exercise class and those who had not and compares their stress levels.
CausalComparative ResearchEx of prospective causes and investigate the effects.
What is the effect of X?
e.g., A researcher may hypothesize that those children that attend dance classes during elementary school have higher selfesteem when in middle school. She would identify a group of middleschool children who had dance classes in elementary school and a group of those who did not, and compare their selfesteem.
CausalComparative Researchindependent variable is not manipulated
The independent variable has occurred or is already formed.
Independent variable in causalcomparative studies is often referred to as the grouping variable.
Examples of variables investigated in Causalcomparative studies:
Organismic variables (e.g., age, ethnicity, sex)
Ability variables (e.g., achievement)
Personality variables (e.g., selfconcept)
Familyrelated variables (e.g., SES)
Schoolrelated variables(e.g., type of school, size of school)
CausalComparative Research Example
Ethnicity, birth order, spatial ability, teacher education level, goal orientation
Ethnicity (organismic), birth order (familyrelated), spatial ability (ability), teacher education level (schoolrelated), goal orientation (personality)
Limitations of causalcomparative research.
The experimenter has limited control.
Caution in interpretation is necessary as causeeffect relations cannot be established.
Only relations are established.
Conducting a CausalComparative Study
The basic causalcomparative design involves selecting two groups that differ on a variable of interest and comparing them on a dependent variable.
Definition and selection of comparison groups is critical in causalcomparative research.
Grouping variables must be operationally defined (e.g., training versus no training).
Researchers should test for differences between groups (e.g., prior knowledge).
Basic causalcomparative designs
There are several control procedures that researchers can employ to strengthen their causalcomparative designs.
Conducting a CausalComparative Studytest for differences between groups
The more similar the groups are on extraneous variables, the fewer alternative explanations there may be for research findings.
Conducting a CausalComparative StudyBasic causalcomparative designs
In one design: One group is exposed to an independent variable while the other group is not. Both groups are measured on a dependent variable.
(E) (X) O
(C) O
In a second design: Two groups are exposed to different independent conditions. Both groups are then measured on a dependent variable.
(E) (X1) O
(C) (X2) O
Conducting a CausalComparative Studycontrol procedures that researchers can employ to strengthen their designs.
Matching: Researchers can attempt to equate groups and control for one or more variables.
For example, a researcher comparing two types of instruction might control for prior achievement. To do this, he would do pairwise matching and would place an equal number of high achieving students in each condition.
Comparing homogeneous groups or subgroups: Researchers can also compare groups that are homogeneous with respect to an extraneous variable.
For example, the researcher may select only highachieving students for his study.
Analysis of Covariance (ANCOVA): Researchers can use this statistical technique to adjust scores on a dependent variable for initial differences on a related variable.
For example, the researcher could measure prior knowledge and use those scores as a covariate.
Conducting a CausalComparative Study
Data analysis and interpretation
Descriptive and inferential statistics are used to analyze data from causalcomparative studies.
Conducting a CausalComparative Study
Descriptive and inferential statistics
Descriptive statistics often include the mean and the standard deviation.
Inferential tests used include ttests, analyses of variance, and chi square.
Experimental Research
the only type of research that can test hypotheses to establish causeeffect relations.
In experimental research studies the independent variable is also called the treatment, causal, or experimental variable.
In experimental research studies the dependent variable is also called the criterion, effect, or posttest variable
Experimental Research
the most structured of all research.
can provide evidence for causeeffect relations.
Several experimental studies taken together can provide support for generalization of results.
Experimental Researchtest hypotheses to establish causeeffect relations
The researcher manipulates at least one independent variable and controls other relevant variables, and observes the effect on one or more dependent variables.
The researcher manipulates the treatment.
The researcher has control over selection and assignment.
Experimental ResearchThe experimental process
The steps in the experimental research process are the same as in other types of research.
Selecting and defining a problem
Selecting participants and measuring instruments
Preparing a research plan
Executing procedures
Analyzing the data
Formulating conclusions
Experimental ResearchThe experimental process
In experimental studies, the researcher controls selection and assignment.
Experimental studies often examine comparisons between or among groups.
Comparison of approaches (A versus B)
Comparison of an approach to an existing approach (A versus no A)
Comparison of different amounts of a single approach (A little of A versus a lot of A)
Experimental ResearchThe experimental process
In experimental research studies the group that receives the treatment is the experimental group and the group that does not receive the treatment is called the control group.
Sometimes groups are comparison groups that receive alternative treatments (e.g., two types of instruction in a content area).
Experimental Researchchallenges for experimental studies in educational settings
A lack of sufficient exposure to treatments (i.e., treatments are too short or diffuse).
Failure to make treatments significantly different from one another (e.g., an experimental instructional program in math may not be different enough from the comparison math instructional program).
Experimental ResearchManipulation and control
In experimental studies, researchers control or remove the influence of extraneous variables.
Participant variables
Organismic (e.g., age)
Intervening (e.g., interest)
Environmental variables (e.g., school or teacher effects)
Experimental ResearchThreats to Experimental Validity
Internal validity refers to the degree to which observed differences in the dependent variable are a direct result of manipulation of the independent variable and not some other variable.
Internal validity is concerned with rival explanations for an effect.
External validity, sometimes referred to as ecological validity, is the degree to which the results from a study are generalizable to other groups.
When researchers increase the internal validity of their study, they decrease their external validity.
When researchers are concerned with external validity, their ability to control important extraneous variables suffers.
When there is a choice, researchers should err on the side of control and maximize internal validity.
Experimental ResearchThreats to Internal Validity
History
Maturation
Testing
Instrumentation
Statistical regression
Differential selection of participants
Mortality
Selectionmaturation and interactive effects
Experimental ResearchThreats to Internal ValidityHistory
Any event occurring during a study that is not part of the experimental treatment but that may effect the dependent variable represents a history threat.
Longerlasting studies are more prone to history threats.
Ex:
In a study of the effects of instructional simulations in learning chemistry content, a history threat would be demonstrated if students in the study were exposed to simulations in a different setting, such as when learning geography, while the study was being conducted.
Experimental ResearchThreats to Internal ValidityMaturation
Maturation refers to physical, intellectual, and emotional changes that naturally occur within participants over a period of time.
Ex:
In studies of interventions that are designed to increase children's theory of mind, if the interventions lasted more than a couple of weeks at critical time points, participants may gain critical theory of mind awareness simply due to cognitive development and not due to the treatment.
Experimental ResearchThreats to Internal ValidityTesting
Testing as a threat to internal validity is demonstrated when taking a pretest alters the result of a posttest.
Experimental ResearchThreats to Internal ValidityInstrumentation
Instrumentation is a threat to internal validity when the instrumentation is either unreliable or is changed between preand posttesting.
Experimental ResearchThreats to Internal ValidityStatistical regression
Extremely high or low scores tend to regress to the mean on retesting.
Ex:
If students perform poorly on a pretest it is difficult to determine if the gain in their scores is due to treatment effects.
Experimental ResearchThreats to Internal ValidityDifferential selection of participants
Participants in the control and experimental groups differ in ways that influence the dependent measure.
Experimental ResearchThreats to Internal ValidityMortality
Participants drop out of the study at differential rates across conditions.
Experimental ResearchThreats to Internal ValiditySelection can interact with other threats to internal validity (i.e., history, maturation, instrumentation).
Participants selected into the treatment and control conditions have different experiences or maturation rates or instrumentation varies across conditions.
Experimental ResearchExternal validity threats can be divided into two categories:
'Generalizing to whom' threats
Threats affecting groups to which the study can be generalized
'Generalizing to what' threats
Threats affecting the settings, conditions, variables, and contexts to which the results can be generalized
Pretesttreatment interaction
Multipletreatment interference
Selectiontreatment interaction
Specificity of variables
Experimenter effects
Reactive arrangements
Experimental ResearchThreats to External ValidityPretesttreatment interaction
This threat occurs when participants respond differently to a treatment because they have been exposed to a pretest.
Pretest may alert participants.
Selfreport measures are often susceptible to pretesttreatment interaction effects.
At times, unobtrusive measures can be used as pretests, to limit this threat to validity (e.g., using previously administered standardized assessments to measure ability in science instead of using a pretest).
Experimental ResearchThreats to External ValidityMultipletreatment interference
This threat occurs when previous treatments crossover into a current experiment. This makes it challenging to determine the effectiveness of the later treatment.
This threat may occur in studies that access participants who have been exposed to other research studies (e.g., university participant pools).
Experimental ResearchThreats to External ValiditySelectiontreatment interaction
When a study's findings only apply to the groups selected and are not representative of other groups.
This may happen in nonrandomly assigned studies where a treatment is less or more effective for certain demographics (e.g., ability levels)
Experimental ResearchThreats to External ValiditySpecificity of variables
When researchers do not adequately define their variables, instruments, or population, it makes it difficult to determine how well the findings will generalize to an alternative population.
Experimental ResearchThreats to External ValidityExperimenter effects
Experimenter effects occur when characteristics or behaviors of the experimenter influence the participants' responses.
Experimental ResearchThreats to External ValidityReactive arrangements
These threats are also referred to as participant effects. These threats are associated with differences in participants' behavior, feelings, and attitudes because they are in a study.
Hawthorne effect
John Henry effect (Compensatory rivalry)
Placebo effect
Novelty effect
Experimental ResearchThreats to External ValidityReactive arrangementsHawthorne effect
Any situation in which participants' behavior is affected because they are in a study.
Experimental ResearchThreats to External ValidityReactive arrangementsJohn Henry effect (Compensatory rivalry)
Members of the control group compete with the experimental group.
Experimental ResearchThreats to External ValidityReactive arrangementsPlacebo effect
To combat compensatory rivalry, researchers attempt to give control groups a placebo, not the experimental treatment, but something to decrease the perception that they are in the control group. Participants should perceive they are all getting the same thing.
Experimental ResearchThreats to External ValidityReactive arrangementsNovelty effect
When participants are engaged in something different this may increase attention, interest, behavior, learning, etc., just because it is something new.
Experimental ResearchGroup Experimental Designs
The validity of an experiment is a function of the degree to which extraneous variables are controlled.
Randomization is the best mechanism to control for extraneous variables.
Randomization distinguishes experimental designs.
Randomization should be used whenever possible.
If groups cannot be randomly formed, variables should be held constant when at all possible (e.g., time of day, which researcher is present).
Participant variables can be controlled and held constant
Experimental ResearchGroup Experimental Designs
Participant variables can be controlled and held constant
Matching can equate groups through random assignment of pairs.
Comparing homogeneous groups allows the researcher to control for extraneous variables.
Participants can serve as their own controls to control for participant differences.
Analysis of covariance (ANCOVA) is a statistical procedure that can be used to control for participant variables.
Experimental ResearchTypes of Group DesignsSinglevariable designs
Preexperimental designs do not adequately control for extraneous variables and should be avoided.
Trueexperimental designs offer a very high degree of control and are always preferred designs.
Quasiexperimental designsdo not control as well as experimental designs but are preferable over preexperimental designs
Experimental ResearchTypes of Group DesignsFactorial designs
any design that involves two or more independent variables, at least one that is manipulated.
Experimental ResearchPreExperimental Designs
The oneshot case study involves a single group that is exposed to a treatment (X) and then posttested (O).
X O
Threats to validity are not adequately controlled with this design.
Do notuse this design
Experimental ResearchPreExperimental Designs
The onegroup pretestposttest design involves a single group that is pretested, exposed to treatment, and then tested again.
O X O
The success of the treatment is determined by comparing pretest and posttest scores.
This design does not control for history, testing, instrumentation, regression, or maturation.
Statistical regression is not controlled nor is pretesttreatment interaction.
Experimental ResearchPreExperimental Designs
The staticgroup comparison designinvolves at least two nonrandomly formed groups. One group receives an experimental treatment and the other group receives the traditional treatment. Both groups are posttested.
X1 O
X2 O
Experimental ResearchPreExperimental Designs
The number of groups can be expanded beyond two.
The groups are better described as comparison, not experimental and control.
This design does not control for maturation, selection effects, selection interactions, and mortality.
There is some control for history in this design.
This design is sometimes used in exploratory studies.
Experimental ResearchTrue Experimental Designs
The pretestposttest control group design requires at least two groups.
Groups are formed by random assignment.
Both groups are administered a pretest, each group receives a different treatment and both groups are posttested.
The design may be extended to include additional groups.
Experimental ResearchTrue Experimental Designs
R O X1 O
R O X2 O
R O X3 O
The combination of random assignment and the presence of a pretest and a control group serve to control for all threats to internal validity.
Experimental ResearchTrue Experimental Designs
The only potential weakness in this design is a possible interaction between the pretest and the treatment.
Researchers should report assess and report the probability of a pretesttreatment interaction.
Experimental ResearchTrue Experimental Designs
There are a few variations on the basic pretestposttest control group design.
One variation includes random assignment of matched pairs to the treatment groups.
There is little advantage to this variation.
Another variation of this design involves one or more additional posttests.
R O X1 O O
R O X2 O O
Experimental ResearchTrue Experimental Designs
The posttestonly control groupdesign is the same as the pretestposttest control group design except that it lacks a pretest.
This design is often expanded to include more than two groups.
R X1 O
R X2 O
The posttestonly control group design is best used when there is likelihood of a pretesttreatment interaction threat.
As with the pretestposttest control group design, the addition of a matched random assignment does not represent an increased advantage.
Experimental ResearchTrue Experimental Designs
The Solomon FourGroup Designis a combination of the pretestposttest control group design and the posttestonly control group design.
R O X1 O
R O X2 O
R X1 O
R X2 O
The analysis of the Solomon fourgroup design is a 2 x 2 factorial analysis of variance.
This analysis tests if those who received the treatment performed differently than those who did not.
This analysis can assess if there is a testing effect.
This analysis assesses for pretestinteraction effects.
Experimental ResearchTrue Experimental Designs
The Solomon fourgroup design requires a large number of participants.
The Solomon fourgroup design may not always be the best design.
The design selected should be based upon potential threats and the nature of the proposed study.
Experimental ResearchQuasiExperimental Designs
When it is not possible to assign participants to groups randomly, researchers can use quasiexperimental studies.
In the nonequivalent control group design, two or more treatment groups are pretested, administered a treatment, and posttested.
O X1 O
O X2 O
Experimental ResearchQuasiExperimental Designs
The nonequivalent control group designinvolves the random assignment of groups not individuals.
The lack of random assignment introduces validity threats (e.g., regression, and selection interaction effects).
To reduce threats when using this design researchers often assure groups are as equivalent as possible (e.g., use ANCOVA).
Experimental ResearchQuasiExperimental Designs
The timeseries designis an elaboration of the pretestposttest design.
One group is repeatedly pretested until pretest scores are stable. The group is then exposed to a treatment and after treatment is repeatedly posttested.
Experimental ResearchQuasiExperimental Designs
O O O O X O O O O
A variation of the timeseries design is the multiple timeseries design that includes a control group.
This variation eliminates the history and instrumentation threats.
O O O O X1 O O O O
O O O O X2 O O O O
Experimental ResearchQuasiExperimental Designs
History is a threat with timeseries designs.
Instrumentation may also be a threat if testing changes.
Pretest threats are problematic with timeseries designs. However, it is relatively easy to establish the degree of the threat given data from repeated testing.
Experimental ResearchQuasiExperimental Designs
In a counterbalanced design, all groups receive all treatments but in a different order and all groups are posttested after each treatment.
Counterbalanced designs can include any number of groups.
The number of groups is equal to the number of treatments.
Treatment order is randomly assigned.
A unique threat with a counterbalanced design is multiple treatment interaction.
X1 O X2 O X3 O
X3 O X1 O X2 O
X2 O X3 O X1 O
Experimental ResearchFactorial Designs
Factorial Designs are elaborations on singlevariable experimental designs to permit investigation of two or more variables, at least one of which is manipulated by the researcher.
Factorial designs are often employed after an independent variable has first been investigated individually.
Experimental ResearchFactorial Designs
The purpose of a factorial design is to determine whether the effects of an independent variable are generalizable across all levels.
One example of a factorial design is the 2 X 2 design.
Type of instruction (computerbased or paper and pencil) by gender.
Many factors (independent variables) studies are possible to address specific research questions.
SingleSubject Experimental Designs
designs are applied when the sample size is one or when a number of individuals are considered to be one group.
These designs are used to study behavior change in response to treatment.
Each participant serves as her own control.
SingleSubject Experimental Designs
Each participant is exposed to both treatment and control phases and is measured during each phase.
Singlesubject designs are often used for research in special education, communication science disorders, and clinical psychology.
Applications of these designs, however, are appropriate to many additional areas of research.
SingleSubject Experimental Designs
Design representation in singlesubject research
The nontreatment phase is represented by A.
The treatment phase is represented by B.
Example:
if we were to study the inclass swearing behavior of Melissa, we could make several observations of her in class, recording her swearing behavior, introduce an intervention for several classes and observe her swearing behavior, stop the intervention and observe her for several classes. This would be represented as ABA.
SingleSubject Experimental Designs
When research results are intended to generalize to other groups, single subject designs are not appropriate.
Sometimes, when it may not be ethical to conduct a group design, a singlesubject design is appropriate.
Singlesubject designs are appropriate when the aim is to improve functioning of an individual.
SingleSubject Experimental Designs
Singlesubject designs suffer from low external validity.
Replication is the key to generalizability in singlesubject designs.
The effect of the baseline phase may pose an external validity threat in singlesubject designs.
Similar to the pretesttreatment interaction threat in group designs.
SingleSubject Experimental Designs
Internal validity
Repeated and reliable measurement
Baseline is established in singlesubject designs through multiple observations.
This helps to control for maturation.
Data are also collected during the treatment phases in singlesubject designs.
This helps to control for history.
Instrumentation is a threat to internal validity in singlesubject designs.
Intraobserver reliability and interobserver reliability are critical for effective singlesubject studies.
Treatment should be explained in enough detail to permit replication.
Baseline stability
It is often difficult to know how many data points are necessary for baseline. This is a critical decision in singlesubject designs.
Minimum of three data points are required but more are often necessary
Length of the treatment phase parallels baseline
Singlevariable rule
Only one variable at a time should be
SingleSubject Experimental Designs
Types of SingleSubject Designs
ABA withdrawl designs alter phases of baseline and treatment.
There are a number of variations of ABA designs.
AB designs
OOOO XOXOXOXO
Baseline Phase Treatment Phase
A B
SingleSubject Experimental Designs
Types of SingleSubject Designs
Additive designs
ABA Design
OOOO XOXOXOXO OOOO
Baseline Phase Treatment Phase Baseline Phase
A B A
Changing criterion design
Baseline is followed by successive treatment phases where in each a more stringent criterion is required.
SingleSubject Experimental Designs
Types of SingleSubject Designs
ABAB Design
OOOO XOXOXOXO OOOO XOXOXOXO
Baseline Treatment Baseline Treatment
A B A B
SingleSubject Experimental Designs
Types of SingleSubject Designs
MultipleBaseline Designs
Multiplebaseline designs entail the systematic addition of behaviors, subjects, or settings for intervention.
Multiple behaviors for one subject
One behavior for several subjects
One behavior and one subject for several settings
Multiplebaseline designs can often be used when it is not ethical to remove a treatment or to reverse a treatment.
SingleSubject Experimental Designs
Data Analysis and Interpretation
Data analysis in singlesubject research is typically based on visual inspection and analysis of a graphic representation of results.
First, the researcher evaluates the adequacy of the design.
Second, if the design is deemed valid, the researcher assesses treatment effectiveness.
Effectiveness is evaluated based upon clinical effectiveness not statistical effectiveness.
Although available, inferential statistical procedures are not often used in singlesubject designs.
External validity in singlesubject designs is established through replication.
SingleSubject Experimental Designs
Data Analysis and Interpretation
Replication
Direct:Replication by the same investigator in a specific setting with the same or different participants.
Simultaneous:Replication with a number of participants with the same problem at the same location and same time.
Systematic:Follows direct replication with different investigators, behaviors, and settings.
Clinical replication involves the development of a treatment package composed of two or more interventions that have been found to be individually successful.
Descriptive Statisticsdefinition
Statisticsis a set of procedures for describing, synthesizing, analyzing, and interpreting quantitative data.
The mean is an example of a statistic.
Descriptive StatisticsCalculated
One can calculate statistics by hand or can use the assistance of statistical programs.
Excel, SPSS, and many other programs exist. Some programs are also available on the Web to analyze datasets.
Descriptive StatisticsPreparing Data for Analysis
After data are collected, the first step toward analysis involves converting behavioral responses into a numerical system or categorical organization.
It is critical that all data are scored accurately and consistently.
Data scoring should be doublechecked for consistency and accuracy (i.e., at least 25% of all cases should be checked).
Openended items should be scored by two scorers to check reliability.
All data scoring and coding procedures should be documented and reported in the written report.
Descriptive StatisticsPreparing Data for Analysis
After instruments are scored, the resulting data are tabulated and entered into a spreadsheet.
Tabulation involves organizing the data systematically (e.g., by participant).
In this potential dataset,
ID represents participant number
Cond. is the experimental condition (1 or 2)
Gender is represented by female=1; male=2
Achievement (Ach) and motivation (Mot) were also variables assessed
Descriptive StatisticsTypes of Descriptive Statistics
After data are tabulated and entered, the next step is to conduct descriptive statistics to summarize data.
In some studies, only descriptive statistics will be conducted.
If the indices are calculated for a sample, they are referred to as statistics.
If indices are calculated for the entire population, they are referred to as parameters.
Descriptive StatisticsTypes of Descriptive
Frequencies
The frequency refers to the number of times something occurs.
Frequencies are often used to describe categorical data.
We might want to have frequency counts of how many males and females were in a study or how many participants were in each condition.
Frequency counts are not as helpful in describing interval and ratio data.
Descriptive StatisticsMeasures of Central Tendency
Measures of central tendency are indices that represent a typical score among a group of scores.
Measures of central tendency provide a way to describe a dataset with a single number.
Descriptive StatisticsMeasures of Central Tendency three most common
Mean: Appropriate for describing interval or ratio data
Median: Appropriate for describing ordinal data
Mode: Appropriate for describing nominal data
Descriptive StatisticsMeasures of Central TendencyMean
The mean is the most commonly used
measure of central tendency.
The formula for the mean is:
X= ΣX/n
To calculate the mean, all the scores
are summed and then divided by the
number of students.
Descriptive StatisticsMeasures of Central TendencyMedian
The median is the midpoint in a distribution: 50% of the scores are above the median and 50% of the scores are below the median.
To determine the median, all scores are listed in order of value.
If the total number of scores is odd, the median is the middle score.
If the total number of scores is even, the median is halfway between the two middle scores.
Median values are useful when there is large variance in a distribution.
Descriptive StatisticsMeasures of Central TendencyMode
The mode is the most frequently occurring score in a distribution.
The mode is established by looking at a set of scores or at a graph of scores and determining which score occurs most frequently.
The mode is of limited value.
Some distributions have more than one mode (e.g., bimodal, or multimodal distributions)
Descriptive StatisticsDeciding among measures of central tendency
Generally the mean is most preferred.
The mean takes all scores into account.
The mean, however, is greatly influenced by extreme scores.
When there are extreme scores present in a distribution, the median is a better measure of central tendency.
Descriptive StatisticsMeasures of Variability
Measures of variability provide an index of the degree of spread in a distribution of scores.
Measures of variability are critical to examine and report because some distributions may be very different but yet still have the same mean or median.
Descriptive StatisticsMeasures of VariabilityThree common measures
range, quartile deviation, and standard deviation.
Descriptive StatisticsMeasures of VariabilityThree common measuresRange
The difference between the highest and lowest score.
The range is not a stable measure.
The range is quickly determined.
Descriptive StatisticsMeasures of VariabilityThree common measuresQuartile Deviation
One half the difference between the upper quartile and the lower quartile in a distribution.
By subtracting the cutoff point for the lower quartile from the cutoff point for the upper quartile and then dividing by two we obtain a measure of variability.
A small number indicates little variability and illustrates that the scores are close together.
Descriptive StatisticsMeasures of VariabilityThree common measuresVariance
The amount of spread among scores. If the variance is small the scores are close together. If the variance is large the scores are spread out.
Calculation of the variance shows how far each score is from the mean.
The formula for the variance is:
Σ(XX)2/n
Descriptive StatisticsMeasures of VariabilityThree common measuresStandard deviation
The score root of the variance.
The standard deviation is used with interval and ratio data.
The standard deviation is the most commonly used measure of variability.
If the mean and the standard deviation are known, the distribution can be described fairly well.
SDrepresents the standard deviation of a sample and the symbol (i.e., the Greek lower case sigma) represents the standard deviation of the population.
Descriptive StatisticsThe Normal Curve
If a variable is normally distributed then several things are true about the distribution of the variable.
Fifty percent of the scores are above the mean and 50% are below the mean.
The mean, median, and mode have the same value.
Most scores are near the mean.
34.13% of the scores fall between the mean and one standard deviation above the mean and 34.13% of scores fall below the mean and one standard deviation below the mean.
That is, 68.26% of the scores fall within one standard deviation of the mean.
More than 99% of the scores fall within three standard deviations above and below the mean.
Descriptive StatisticsThe Normal Curve
Skewed distributions
When a distribution is not normally distributed, it is said to be skewed.
A skewed distribution is not symmetrical.
The mean, median, and mode are not the same value.
The farther apart the mean and the median, the more skewed the distribution.
A negatively skewed distribution has extreme scores at the lower end of the distribution.
Mean<Median<Mode
A positively skewed distribution has extreme scores at the higher end of the distribution.
Mean>Median>Mode
Descriptive StatisticsMeasures of Relative Position
Measures of relative position indicate where a score falls in the distribution relative to all the other scores.
Measures of relative position indicate how well an individual has scored in comparison to others in the distribution.
Measures of relative position express different scores on a common scale.
Descriptive StatisticsMeasures of Relative Positionmost frequently used measures of relative position
percentile ranks
standard scores.
Descriptive StatisticsMeasures of Relative Positionpercentile ranks
Percentile ranks indicate the percentage of scores that fall at or below a given score.
Percentile ranks are appropriate for ordinal data and are also used for interval data.
A percentile rank of 50 indicates the median score.
Descriptive StatisticsMeasures of Relative Positionstandard scores
A standard score uses standard deviation units to express how far an individual student's test score is from the mean.
i.e., a standard score reports how many standard deviations a given score is from the mean of a distribution.
Standard scores allow scores from different tests to be compared on a common scale and therefore allow for calculations on those data.
.
Descriptive StatisticsMeasures of Relative Positionstandard scoresZ score
A zscore is the most basic and most often used standard score.
A zscore is directly tied to the standard deviation.
A score that represents the mean has a zscore of 0.
A score at 1 standard deviation above the mean has a zscore of 1
To convert a raw score to a zscore we use the following formula where Xis the raw score.
Z =XX
SD
The characteristics of the normal distribution can be used to approximate where a score falls based upon a standard score.
Descriptive StatisticsMeasures of Relative Positionstandard scoresTscore
A Tscore is a standard score sometimes used instead of a zscore.
A Tscore is calculated by multiplying a zscore by 10 and adding 50.
Tscores transform the scores such that there are not negative values.
Descriptive StatisticsMeasures of Relationship
Measures of relationship indicate the degree to which two sets of scores are related.
There are several statistical measures of relationship.
The nature of a given dataset dictates which measure is used.
Descriptive StatisticsMeasures of Relationship
Pearson ris statistic used to calculate relationship for interval or ratio data.
Pearson r takes into account every score.
Pearson ris the most stable measure of relationship.
Spearman rho is used to calculate relationship with ordinal data.
There are other tests of relationship for ordinal data (e.g., Gamma, Kendall's tau).
Spearman rho is most popular.
Descriptive StatisticsGraphing Data
a critical step in analyzing any dataset.
All variables should be graphed.
Graphing helps researchers to know their data.
Statistical packages make graphing data an easy task.
Graphing helps to identify any errors in a dataset.
Inferential Statistics
data analysis techniques for determining how likely it is that results obtained from a sample, or samples, are the same results that would be obtained from the entire population.
Descriptive statistics show how often or how frequent an event or score occurred.
Inferential statistics help researchers know whether they can generalize their findings to a population based upon their sample of participants.
Inferential statistics use data to assess likelihood—or probability.
Inferential Statistics
Inferences about populations are based on information from samples.
There is very little chance that any sample is identical to the population.
The expected variance among sample means and the population mean is referred to as sampling error.
Sampling error is expected.
Sampling error tends to be normally distributed.
Inferential StatisticsStandard Error
A distribution of sample means is normally distributed and has it's own mean and standard deviation.
The standard deviation of the sample means is referred to as the standard error of the mean.
Our ability to estimate standard error of the mean is affected by size of sample.
As the sample size increases the standard error of the mean decreases.
Inferential StatisticsStandard Error
Our ability to estimate the standard error of the mean is also affected by the size of the population standard deviation.
The standard error of the mean can be calculated by:)=(SE)= SD
X N1
SEx= the standard error of the mean
SD= the standard deviation for a sample
N= the sample size
Inferential StatisticsHypothesis Testing
Hypothesis testing is the process of decision making in which researchers evaluate the results of a study against their original expectations.
Research hypothesis: Predicting a difference in scores
Null hypothesis: Predicting no difference in scores
Inferential StatisticsHypothesis Testing
We want to assure differences we observe between groups are 'real' differences and did not occur by chance.
If the groups are significantly different we reject the null hypothesis.
We do not accept a research hypothesis, we can't prove our hypothesis.
We instead report that our research hypothesis was supported.
If there are not expected differences, we report that the null hypothesis was not rejected; and that our research hypothesis was not supported.
Inferential StatisticsTests of Significance
allow us to inferentially test if differences between scores in our sample are simply due to chance or if they are representative of the true state of affairs in the population.
Inferential StatisticsTests of Significanceconduct a test of significance
determine a preselected probability level, known as level of significance (alphaor α).
Usually educational researchers use alpha.05 or 5 out of 100 chances that the observed difference occurred by chance (α=.05).
Inferential StatisticsTests of SignificanceTwotailed and onetailed tests
Tests of significance are almost always twotailed.
Researchers will select a onetailed test of significance only when they are quite certain that a difference will occur in only one direction.
It is 'easier' to obtain a significant effect when predicting in one direction.
Inferential StatisticsTests of SignificanceType I and Type II Errors
Based upon a test of significance the researcher will either reject or not reject the null hypothesis.
The researcher makes a decision that the observed effect is or is not due to chance.
This decision is based upon probability, not certainty.
Sometimes the researcher will erroneously reject the null hypothesis or will erroneously retain the null hypothesis.
Inferential StatisticsTests of SignificanceType I Errors
When the researcher incorrectly rejects the null hypothesis she has committed a Type Ierror.
Inferential StatisticsTests of Significance Type II Errors
When the researcher incorrectly fails to reject the null hypothesis but a true difference exists, she has committed a Type II error.
Inferential StatisticsTests of SignificanceDegrees of Freedom
After determining whether the significance test will be twotailed or onetailed and selecting a probability level (i.e., alpha), the researcher selects an appropriate statistical test to conduct the analysis.
Degrees of freedom are the number of observations free to vary around a parameter.
Each test of significance has its own formula for determining degrees of freedom (df ).
The value for the df is important in determining whether the results are statistically significant.
Inferential StatisticsTests of Significance Choosing which to use
The use of a specific significance test is determined by several factors.
Scale of measurement represented by the data
Participant selection
Number of groups being compared
Number of independent variables
Significance tests applied incorrectly can lead to incorrect decisions.
The first decision in selecting an appropriate test is to determine whether a parametric or nonparametric test will be used.
Parametric tests are generally more powerful and are generally preferred.
Inferential StatisticsTests of Significance Parametric tests
require that the data meet several assumptions.
Variable must be normally distributed
Interval or ratio scale of measurement
Selection of participants is independent
Variance of the comparison groups is equal
Most parametric tests are fairly robust.
If assumptions are violated, nonparametric tests should be used.
Inferential StatisticsTests of Significance t test
The ttest is used to determine whether two groups of scores are significantly different from one another.
The ttest compares the observed difference between means with the difference expected by chance.
Inferential StatisticsTests of Significance t test for independent samples
The ttest for independent samples is a parametric test of significance used to determine if differences exist between the means of two independent samples.
Independent samples are randomly formed.
The assumption is that the means are the same at the outset of the study but there may be differences between the groups after treatment.
Inferential StatisticsTests of Significance t test for nonindependent samples
The ttest for nonindependent samples is a parametric test of significance used to determine if differences exist between the means of two groups that are formed through matching.
When scores are nonindependent, they are systematically related.
Inferential StatisticsTests of Significance t test
The comparison of gain or difference scores are not generally tested with a ttest.
There are other better strategies for analyzing such data.
e.g., ttest on posttest scores (if there are no differences on pretest scores).
e.g., Analysis of covariance (if there are differences in pretest scores).
Inferential StatisticsTests of Significance Analysis of Variance
A simple (oneway) analysis of variance (ANOVA) is a parametric test used to determine whether scores from two or more groups are significantly different at a selected probability level.
ANOVA is used to avoid the error rate problems of conducting multiple tests.
Inferential StatisticsTests of Significance Analysis of Variance F ratio
An F ratio is computed to determine if sample means are significantly different.
The F ratio is calculated based upon variance between groups/variance within groups.
The larger the Fratio the more likely there are differences among groups.
Inferential StatisticsTests of Significance Analysis of Variance
If there are significant differences among groups based upon an ANOVA; the researcher then must determine where these differences exist.
Multiple comparisons are used to determine where differences between groups are statistically significant.
Comparisons planned before collecting data are referred to as a priori.
Comparisons after are referred to as post hoc.
When a factorial design is used and there are two or more independent variables analyzed, a factorial or multifactor analysis of variance is used to analyze the data.
MANOVA is an analytic procedure used when there is more than one dependent variable and multiple independent variables.
Inferential StatisticsTests of Significance Analysis of Variance
ANCOVA is a form of ANOVA that allows for control of extraneous variables and also is used as a means for increasing power of an analysis.
Power is increased in an ANCOVA because the withingroup error variance is decreased.
When a study has two or more dependent variables, and a covariate, MANCOVA is used.
Inferential StatisticsTests of Significance Analysis of VarianceMultiple Regression
Multiple regression is used to determine the amount of variance accounted for in a dependent variable by interval and ratio level independent variables.
Multiple regression combines variables that are known to predict the criterion variable into an equation.
Stepwise regression allows the researcher to enter one variable at a time.
Multiple regression is also the basis for path analysis.
Path analysis begins with a predictive model.
Path analysis determines the degree to which predictor variables interact with each other and contribute to variance in the dependent variables.
Inferential StatisticsTests of Significance Chi Square
Chi Square is a nonparametric test used to test differences between groups when the data are frequency counts or percentages or proportions converted into frequencies.
A true category is one in which persons naturally fall.
An artificial category is one that is defined by the researcher.
Inferential StatisticsOther Statistical Procedures
Data mining uses analytical tools to identify and predict patterns in datasets.
Factor analysis is a statistical procedure used to identify relations among variables in a correlation matrix.
Factor analysis is often used to reduce instruments to scales or subscales.
Structural Equation Modeling (SEM)
Structural equation modeling is similar to a combination of path analysis and factor analysis.
SEM is a powerful analytic tool.
SamplingWhat happens if I don't have a good sample?
If one doesn't have a good sample, then the results will not be generalizable and therefore will have less importance.
SamplingWhat are some examples of sampling techniques for quantitative research?
Random sampling techniques might include simple random sampling, systematic sampling and cluster sampling. Examples of nonrandom sampling include convenience sampling, quota sampling, and snowball sampling.
SamplingWhy is sampling important in quantitative research?
Because one of the purposes of quantitative research is to generalize the results obtained from a small sample to its larger population, you want to make sure that your sample represents the population as much as possible.
Quantitative Sampling
Sampling: The process of selecting a number of participants for a study.
Sample: A sample is made up of individuals, events, or items, selected from the population.
a sample is wellselected the results will be generalizableto the population
Quantitative SamplingDefining a Population
Populations may be any size and cover large geographical distances.
The entire population is rarely available.
Quantitative SamplingTarget population
Population to which the researcher would ideally like to generalize results.
Quantitative SamplingAccessible population (Available population)
Population from which researcher can realistically select participants.
Quantitative SamplingTarget & Accessible Populations Example
Consider that you are interested in high school juniors' perceptions of admission criteria for Ivy League schools. Your target population would be high school juniors in the United States but the accessible population would be much different. Although you could conduct the study with the target population, the study would require extensive resources and as such, an accessible population would likely be studied instead.
Quantitative SamplingSelecting a Random SampleProbability Sampling
techniques permit the researcher to specify the probability that each member of the population has in being selected for the sample.
Simple Random Sampling
Stratified Sampling
Cluster Sampling
Systematic Sampling
Quantitative SamplingSelecting a Random SampleProbability Sampling
All individuals in the defined population have an equal and independent chance of being selected for the sample.
Random sampling is the best way to possibly obtain a representative sample.
Quantitative SamplingSteps in Simple Random Sampling
1.Identify and define the population.
2.Determine the desired sample size.
3.List all members of the population.
4.Assign all individuals a consecutive number from zero to the required number.
5.Select an arbitrary number in the table of random numbers.
6.For the selected number look at only the necessary number of digits.
7.The number that is selected represents that person in the population.
8.Go to the next number in the column and repeat steps 6 and 7.
Quantitative SamplingRandom Sampling ExampleSteps 13
We are interested in selecting a random sample of participants for a study of students' nutrition.
1.Identify and define the population:
The population is middle school children in the school district in which we live.
Middle schools in our district include children in sixth through eighth grade.
There are 820 middle school children in our district.
2.Determine desired sample size.
We would like to sample 325 students.
3.List all members of the population.
We generate a comprehensive list of all the middle school students.
Quantitative SamplingRandom Sampling ExampleSteps 46
4.Each member of the population is given a number.
We give each student a sequential number.
5.We consult a table of random numbers and pick an arbitrary number.
The text has a table of random numbers.
Statistical packages can generate a table.
Websites are available with tables of random numbers.
6.For the arbitrary number we have selected, we look at the last 3 digits because our needed sample contains 3 digits.
Quantitative SamplingRandom Sampling ExampleSteps 78
7.We include in our sample the person with the number corresponding to the number we have selected.
If the number selected is greater than 325 we ignore it and go to the next number.
8.We continue to go through steps 6 and 7 until we have selected our entire sample of participants.
Quantitative SamplingStratified Sampling
guarantees desired representation of relevant subgroups within the sample.
Populations can be divided into subgroups (strata).
should be used when a goal is to compare behavior of participants from subgroups.
e.g., In our nutrition study we may be interested in gender or SES or ethnicity or other potential subgroups.
Quantitative SamplingStratified SamplingProportional stratified sampling
Identified subgroups are represented in the sample in the same proportion as in the population.
e.g., If 28% of a population of advanced science students are girls then the sample should be represented by 28% girls.
e.g., In our nutrition study we may be interested in representation by SES.
Stratified sampling can be used to select equalsized, nonproportional, samples from subgroups when subgroup comparisons are important.
e.g., gender, race, ability level, SES.
Quantitative SamplingStratified Sampling for EqualSized Groups 13
1.Identify and define the population.
Our 820 middle school students are our population.
2.Determine desired sample size.
We would like to sample 100.
3.Identify the variable and subgroups (strata).
We would like an equal number of students representing athletes and nonathletes.
Quantitative SamplingStratified Sampling for EqualSized Groups 45
4.Classify all members of the population as one of your identified groups.
We identify each student as either an athlete or nonathlete.
5.Randomly select an equal number of participants from each subgroup.
We use the random number table to select 50 from each group.
Quantitative SamplingProportional Stratified Sampling
We can use this same procedure to select a proportional stratified sample.
If for example our population includes 12% athletes we may not want equal groups but instead may desire that our sample is 12% athletes.
After we classify students into our subgroups we simply randomly select the appropriate percentage from each subgroup.
Quantitative SamplingCluster Sampling
intact groups but not individuals, are randomly selected.
e.g., An example cluster could be schools, classrooms, communities, or counties.
may be used when it is not possible to list all members of a population or when the population is spread out or large.
Quantitative SamplingCluster Sampling
may be more convenient and may take less time than random or stratified sampling.
found in many studies when it is hard to sample individuals within a group.
Quantitative SamplingCluster Sampling drawbacks
Cluster sampling is not representative.
Oftenused statistical methods are not appropriate for data from studies that employed cluster sampling.
Quantitative SamplingCluster Sampling Example Steps 13
1.Identify and define population.
Middle school students in our state are our population.
2.Determine desired sample size.
We would like to sample 1500 students.
3.Identify and define a logical cluster
Schools are a logical cluster.
Quantitative SamplingCluster Sampling Example Steps 46
4.List all clusters
We obtain a list of all the middle schools in the State.
5.Estimate average number of participants per cluster.
•In our State the average number of students in each middle school is 250.
6.Determine number of clusters needed
•If our desired sample is 1500 we need to randomly select 6 schools.
Quantitative SamplingCluster Sampling Example Steps 78
7.Randomly select the needed number of clusters.
•We randomly select the six schools.
8.Include in your study all the participants in each selected cluster.
•We include all the students in each of the six schools in our study.
Quantitative SamplingSystematic Sampling
we select every Kthindividual from a list.
The list includes all the members of a population.
K is determined by dividing the number of individuals by the number of participants desired in a sample.
Randomly ordered lists can be used in systematic sampling.
Quantitative SamplingSystematic Sampling Example Steps 13
1.Identify and define population.
Middle school students in our school are our population.
2.Determine desired sample size.
We would like to sample 50 students.
3.We list all students in the school
We might randomly order the list.
Avoid bias that may be introduced by other listing procedures.
Quantitative SamplingSystematic Sampling Example Steps 47
4.We divide the number of students in our population by the number of participants that we need in our sample.
e.g., 350 in the school/50 students therefore K=7
5.We start in a random place on the list.
6.We take every Kth (in this case 7th) name on the list until we have selected our 50 participants.
7.If we come to the end of the list then we simply go to the top and continue using our strategy of every Kthname.
Quantitative SamplingDetermining Sample Size
It is not easy to determine how large a sample should be.
If the sample is too small it will limit generalizability.
If a population is very large, you need less percentage of the population for your sample to be representative.
If the population is small, sample the entire population.
If the sample is very large, 400 participants can be sampled.
For populations between 100 and 5000, sample a percentage of the population.
Generally, sample as much of a population as you are able to sample.
Quantitative SamplingSampling Error and Sampling Bias
Sampling Error is chance variation that occurs when a sample does not represent the population.
Sampling Bias is systematic sampling error
e.g., In a study of principals' opinions of effective leadership styles, sampling bias may occur if you sampled principals at a leadership conference.
Quantitative SamplingSelecting a NonRandom Sample
NonProbability sampling or Nonrandom sampling: At times researchers can not use random sampling.
When nonrandom sampling is used it often becomes challenging to describe the sample.
Bias may also be a concern in nonrandom sampling.
Quantitative SamplingTypes of NonRandom Sampling Convenience Sampling
Convenience sampling is also called haphazard or accidental sampling.
Convenience sampling selects whoever is available.
Convenience sampling introduces bias. Those available may not be representative
e.g., In a study of community members' opinions about public transportation, depending upon where you sampled people from they might have very different opinions. If you sampled people at a bus stop, for example, you might not get representative data.
Quantitative SamplingTypes of NonRandom Sampling Purposive Sampling
Purposive sampling is also called judgment sampling.
Purposive sampling entails selecting a sample believed to be representative.
Purposive sampling introduces bias because the researcher may be incorrect in her belief that the sample is representative.
e.g., If a researcher wanted to study differences between expert and novice teachers and selected participants based upon judgments regarding which teachers were deemed expert.
Quantitative SamplingTypes of NonRandom SamplingQuota Sampling
Quota sampling occurs when the researcher selects a sample based on a required number of individuals with particular characteristics.
Quota sampling is often used in widescale survey research.
Bias is introduced in several ways when quota sampling is used, including through accessibility.
e.g., A researcher may need to have 300 men ages 4050 in a study. He would sample until he had obtained the 300 men.
Qualitative Sampling
In qualitative sampling a small number of participants are selected that will be good informants.
Good informants are thoughtful and reflective.
Good informants communicate effectively.
Good informants are comfortable with the researcher and within the research site.
Qualitative Sampling
Qualitative samples are smaller and less representative.
Generalizability and representativeness are not the same concern in qualitative research as they are in quantitative research.
Sample size in qualitative research is determined by the following:
The range of participants available and the degree to which range in participants is consistent with what is found in the population.
Qualitative Samplingdata saturation
Participants may be sampled until data saturation. Data saturation occurs when additional participants are not providing new information.
Qualitative SamplingHomogeneous sampling
occurs when the researcher selects participants who represent a similar experience.
e.g., A researcher might select only middle class women whose mother's stayed at home during the school day.
Qualitative SamplingIntensity sampling
occurs when the researcher selects participants who represent different levels of the research topic.
e.g., A researcher might select some new administrators and some more experienced ones for his study
Qualitative SamplingCriterion sampling
occurs when the researcher selects cases that fit a certain criteria.
e.g., A researcher may select only those participants who are first generation law school students.
Qualitative SamplingSnowball sampling
occurs when the researcher selects new participants based upon other participant's suggestions regarding who else might be studied.
e.g., A researcher might be interested in high school students who are not in school activities but are involved in outofschool activities. Once she has identified some participants she may ask them if they know of others who may fit her criteria.
Qualitative SamplingRandompurposive sampling
Randompurposive sampling occurs when the researcher generates a larger than needed potential sample and randomly selects from that sample.
e.g., A researcher might purposively sample teachers within a specific district and then randomly select 3 teachers to represent the sample.
Qualitative SamplingTypes
Homogeneous sampling
Intensity sampling
Criterion sampling
Snowball sampling
Randompurposive sampling
Qualitative ResearchWhat are the benefits of qualitative research?
Qualitative research allows you to look at data in context and draw from it more meaningful findings.
Qualitative ResearchWhat are the characteristics of qualitative research?
Qualitative research studies are conducted through sustained indepth, incontext, research and seek to provide rich understanding to research questions.
Qualitative ResearchWhat are types of qualitative research?
historical research
case study
grounded theory
ethnology
phenomenology
Qualitative ResearchCharacteristics
1. A total or complete picture is sought.
2. Seeks to understand people's interpretations.
3. Data are perceptions of the people in the environment.
4. Investigations are conducted under natural conditions.
5. The focus is on design and procedures to gain "real," "rich," and "deep" data
6. Small sample sizes
7. No controls for samples
8. Inductive
9. Useful in studying cultures, historical data, and studying complex issues.
10. Narrative results rather than numerical.
Quantitative researchCharacteristics
1. Mathematical or statistical models are used to analyze the data.
2. It is objective and statistical.
3. Variables can be manipulated.
4. Uses large samples for more accurate data.
5. Researcher has little or no interaction with participants.
6. Shows cause and effect relationships
7. Deductive
8. Begins with a hyposthesis tha can be proven or disproved
Qualitative data collection  fieldwork.
Fieldwork includes materials gathered, recorded, and compiled during the study.
Fieldwork requires the researcher to immerse himself in the setting over time.
The researcher collects as much data she can as unobtrusively as possible.
Qualitative researchWhat are the main data collection methods for qualitative research?
Qualitative data collection is referred to as fieldwork.
Qualitative data is narrative and visual.
Observation
Interviews
Questionnaires
Examining Records
Quantitative researchObservation
The researcher obtains data by watching participants.
Observational data is often less subject to participant bias.
The researcher attempts not to change the setting
Types:
Participant observation
Nonparticipant observation
Quantitative researchParticipant Observation
The researcher becomes part of and a participant in the situation being observed.
The researcher participates while observing and collecting data.
Quantitative researchNonparticipant observation
The researcher is not directly part of the situation being observed.
The researcher observes and records but does not interact with the participants.
Nonparticipant observation is a less intrusive form of observation.
Quantitative researchRecording Observations 1
Field notes include descriptive information about what the observer has directly seen and heard on site.
Field notes also include reflective information that captures an observer's personal reactions and thoughts related to the observations.
The researcher avoids evaluative terms in field notes but instead describes behaviors.
Observational protocols are often used.
Protocols provide the researcher with a focus during the observation.
Protocols also provide a framework for the field notes.
Quantitative researchRecording Observations 2
Start slowly. Do not assume that you know what you are looking for until you have experience in the setting and have spent time with the participants.
Try to enter the field with no preconceptions. Recognize and dismiss your assumptions and remain open.
Write your field notes as soon as possible. Don't discuss the observation until you have written field notes.
Quantitative researchRecording Observations 3
Include the date, site, and time on notes. Use large margins and write impressions in the margins. Draw diagrams.
List key words related to the observation and outline what you saw and heard. Use the keywords and the outline to write your notes.
Keep the descriptive and reflective parts of your field notes distinct.
Write down your hunches, questions, and insights after each observation.
Number the lines or paragraphs of your field notes to help you find sections when needed.
Enter your field notes into a computer program for later examination and analysis.
Quantitative researchInterviews
Interviews are purposeful interactions in which one person obtains information from another person.
Interviews allow for data not available through observation alone.
Interviews may be formal and planned or informal and unplanned.
Quantitative researchUnstructured
Interview
Unstructured interviews are similar to conversations.
Unstructured interviews are commonly used to gain more personal information.
Quantitative researchStructured
Interview
include predetermined questions.
Phrasing structured interviews can be challenging.
Include both openended and closed questions.
Pilot test the questions.
Quantitative researchGuidelines for Interviewing
Listening is the most important part of interviewing.
Don't interrupt. Wait.
Tolerate silence. The participant may be thinking.
Avoid leading questions.
Keep participants focused and ask for details.
Followup on what participants say and ask questions when you don't understand.
Don't be judgmental about participants' views or beliefs.
Don't debate with participants.
Quantitative researchQuestionnaires
Interviews are time consuming.
Some researchers use questionnaires and then followup questionnaires with interviews.
Questionnaires allow for larger amounts of data collection.
The nature of the data collected with questionnaires is different than data from observations.
Quantitative researchQuestionnaire Guidelines
Make questionnaire attractive.
Carefully proofread questionnaires.
Avoid lengthy questionnaires.
Do not ask unnecessary questions.
Use structured items with a variety of responses
Include a section that allows respondents to include 'other comments'.
Determine if respondents' identities are necessary and if so, develop a mechanism to track respondents.
Quantitative researchExamining Records
Qualitative researchers use a variety of available documents.
Archival documents
Journals
Maps
Videotape and audiotape
Artifacts
Quantitative researchValidity and Reliability
Validity in qualitative research addresses whether the data accurately measures what it was intended to measure.
Trustworthiness and understanding are terms used to describe validity in qualitative research.
Trustworthiness can be established by:
Credibility:The report addresses problems that are not easily explained.
Transferability:The description provided is such that others can identify with the setting.
Dependability:The stability of the data is addressed.
Confirmability:The neutrality and objectivity of the data are apparent.
Quantitative researchValidity
Criteria for qualitative research validity.
Descriptive validity: factual accuracy of the account
Interpretive validity:researcher accurately interprets participants' behaviors and actions
Theoretical validity: how well the report relates to broader theory
Evaluative validity: whether the report was created without researcher's judgment
Generalizability (Internal and External): the degree to which research is generalizable within and outside the setting
Quantitative researchValidity
Strategies for ensuring the validity of qualitative research
Prolong participation at the study site
Persistently observe
Use peer debriefing
Collect additional artifacts
Conduct member checks
Establish structural corroboration or coherence
Establish referential adequacy
Collect detailed descriptive data
Develop detailed descriptions of the context
Establish an audit trail
Practice triangulation
Practice reflexivity
Quantitative researchValidity
Practical options to assure trustworthiness
Talk little; listen a lot
Record observations accurately
Begin writing early
Let readers 'see' for themselves
Report fully
Be candid
Seek feedback
Write accurately
Quantitative researchReliability
Reliability
Qualitative researchers address reliability by examining the techniques they are using to collect data.
Generalizability is less a concern for qualitative researchers than it is for quantitative researchers. Qualitative researchers are more concerned with relevance.
Narrative Research
Narrative research in education has been influenced by several factors.
The increased emphasis in teacher reflection and action research
The increased emphasis on teacher knowledge
The increased emphasis on empowering teachers
Narrative Research Analysis
Narrative analysis (Current focus)
Researcher collects descriptions of events through interviews and observations and synthesizes them into narratives or stories.
Story is the outcome of the study.
Analysis of narrative
Researcher collects stories as data and seeks to understand underlying themes from the stories.
Narrative Research Process Considerations
The narrative research process is highly personal and requires care and sensitivity.
The narrative researcher must be careful with ethical guidelines as narrative research requires a very close relationship between the researcher and the participant.
Equality of voice is required so that the participant feels empowered to tell the story.
The narrative researcher must determine if she has adequate time and the necessary personal characteristics to conduct this form of research.
Narrative Research Process
Identify purpose of the research study and a phenomenon to explore.
Identify an individual who can help you learn about the phenomenon.
Develop initial narrative research questions.
Consider the researcher's role.
Develop data collection methods, paying particular attention to interviewing, and collect the data.
Collaborate with the research participant to construct the narrative and to validate the accuracy of the story.
Write the narrative account.
Narrative Research Types
Creswell (1995) suggests that the numerous types of narrative research can be categorized based upon several characteristics.
Who authored the account
Scope of the narrative
Who provides the story
Theoretical/conceptual framework
Whether the elements are included in one narrative
Narrative ResearchExamples
Autobiographies
Life writing
Personal narratives
Life histories
Narrative interviews
Narrative ResearchCharacteristics
A focus on the experiences of individuals.
A concern with the chronology of individuals' experiences.
A focus on the construction of life stories based on data collected through interviews.
Restorying as a teachnique for constructing the narrative account.
Inclusion of context and place in the story.
A collaborative approach that involves the researcher and the participants in the negotiation of the final text.
A narrative constructed around the question, "And then what happened?".
Narrative ResearchData Collection Techniques
Restorying
Oral history
Photographs, memory boxes and other artifacts
Storytelling
Letter writing
Autobiographical and biographical writing
Ethnographic Research
study of cultural patterns and perspectives of participants in their natural settings.
engage in longterm study of particular phenomena to situate understandings in context.
engage in intensive participant observation.
Ethnographic Research Goal
describe, analyze, and interpret the culture of a group, over time, in terms of the group's shared beliefs, behaviors, and language.
Ethnographic Research Process
first determine that he or she has the time, access, experience, personal style, and commitment to undertake an ethnography.
identify the purpose of the research.
demonstrates the relevance of the proposed study.
next decides on the site and sample for the study.
negotiate and gain entry into the research site.
attend carefully to confidentiality and anonymity.
establish rapport with the collaborators.
begins data collection.
last steps in the ethnographic research process include analysis and interpretation of the data and writing the ethnographic report.
Ethnographic Research Characteristics
natural setting
facetoface interaction
accurate reflection
inductive, interactive, and repetitious collection of unstructured data and analytic strategies.
fieldwork data collection
multiple methods for data collection, including conducting interviews and observations and reviewing documents, artifacts, and visual materials.
sociopolitical and historical context
small number of cases
Ethnographic Research Types
Critical ethnography (political)
Realist ethnography
Ethnographic case study
Case Study Research
allencompassing method that includes specific design, data collection, and data analysis.
also refers to the product of the research.
yields concrete knowledge that readers personally relate to their own experiences.
qualitative
appropriate when the researcher wants to answer a descriptive or an explanatory question.
Case Study ResearchCharacteristics
focus on a particular phenomenon.
include thick description of the case and include many variables and analyses of their interactions.
provide new insights to the researcher and the participants.

Case Study ResearchDesigning
Determine the research questions.
Define the case under study.
Determine the role of theory development in case selection.
Determine the theoretical and conceptual framework of the case study.
Determine whether a single case study; a multiple case study; or a collective case study is appropriate.
Case Study Researchsampling
Case study research generally employs purposive sampling.
Cases are selected because they can provide information regarding the research problem.
Screening is used to determine if a potential participant has the necessary experience and knowledge of the phenomenon under investigation.
Case Study ResearchData Collection
Case study researchers use the same data collection techniques as do other qualitative researchers.
The data collection techniques are selected to answer 'how' and 'why' questions.
Action ResearchCharacteristics
Action research is persuasive and authoritative.
Action research is relevant.
Action research is accessible.
Action research challenges intractability of reform of the educational system.
Action research is not a fad.
Action ResearchTypes
Democratic: Enables participation of all people
Equitable: Acknowledges people's equality of worth
Liberating: Provides freedom from oppressive, debilitating conditions
Enhancing: Enables expression of people's potential
Action ResearchFour steps:
Identifying an area of focus
Data collection
Data analysis and interpretation
Action planning
Literature Review Purpose
Definition, purpose, and scope
The review of related literature involves systematic identification, location, and analysis of documents containing information related to the research problem.
Some meaningful studies are conducted in areas where little previous research exists, and some meaningful problems remain in heavilyresearched areas of study.
Literature Review Qualitative Research
Qualitative researchers may not review the literature indepth before they begin a research study.
Qualitative research topics emerge over time.
Reviewing too much literature may compromise the inductive process of qualitative research.
Demonstrates underling assumptions critical to the research questions
Demonstrates that the researcher is knowledgeable about related research
Helps researcher identify gaps in the body of literature and may provide rationale for additional research
Assists in refining research questions and guides working hypotheses
Literature ReviewSteps for Conducting
1.Identify and make a list of keywords to guide your literature search.
2.Using your keywords, locate primary and secondary sources.
3.Evaluate the quality of your sources.
4.Abstract your sources.
5.Analyze and organize your sources using a literature matrix.
6.Write the literature review.
Literature ReviewEvaluating your sources:
1.What was the problem statement of the study?
2.Who was studied?
3.Where was the source published?
4.When was the research conducted?
5.How was the study conducted?
CARSWhat are the elements of the CARS evaluation method?
The elements are Credibility, Accuracy, Reasonableness, and Support.
SourcesWhat is the difference between primary and secondary sources?
Primary sources are written by those that have a firsthand account or experience with the source. Secondary sources are those written by people who did not conduct the study or live the experience.
Research topic
There are, however, some commonly used words that relate to a certain type of research. For example, if a topic investigates opinions or attitudes, it is very likely a survey or descriptive research. If it is investigating the relation between two variables, it is a correlational study. If it is investigating the effect of a method or type of instruction, it is most likely an experimental research. If the purpose of a study is to investigate if a method or type of instruction would fix a daily classroom problem, it is action research.
Research topic Narrowing
Narrow quantitative topics at the start of the research process.
Qualitative researchers often narrow their topic after they are in the field.
Research topic Characteristics
1.The topic is interesting.
2.The topic is researchable.
3.The topic has theoretical or practical importance.
4.The topic is ethical.
5.The topic is manageable for you given
your current skills, resources, and time available.
Research topic Quantitative
7Stating the Research Topic
Quantitative research topics
A topic statement describes the variables of interest, relations among those variables, and aspects of the sample.
Research topicQualitative
Stating the Research Topic
Qualitative research topics are often stated in more general language at the outset of a study because the focus of the study will likely emerge after time in the field.
Hypotheses Types
Inductive Hypothesis: A generalization based upon observations
Deductive Hypothesis:
Derived from theory and provides evidence that supports, expands, or even contradicts theory
Nondirectional Hypothesis:
States that a relationship or difference exists among variables
Directional Hypothesis:
States the expected direction of the relationship or difference among variables
Null Hypothesis:
States that there is no significant relationship or difference among variables.
HypothesesNull Hypothesis
Null hypotheses are stated when there is little existing research or theoretical support for a hypothesis.
Null hypotheses are also more conservative than directional hypotheses in statistical tests.
Most studies are not based in the null hypothesis.
Hypothesis
A good hypothesis:
is clearly and concisely stated.
states the relation or difference among variables.
defines variables in measurable terms.
Hypotheses Stating
Model for hypotheses:
P=The participants
X=The treatment, the causal or independent variable (IV)
Y=The study outcome, the effect or dependent variable (DV)
HypothesesQualitative
The qualitative researcher does not state formal hypotheses before conducting studies.
Qualitative researchers may develop guiding hypotheses for the proposed research.
Qualitative researchers often generate new hypotheses during the course of their study.
Qualitative researchers may generate research questions from their guiding hypotheses
Errors What is a Type II error?
A type II error is when you accept the null hypothesis and it is actually false. Or in other words, you think there is no difference when there really is a difference. This is also known as a false negative.
Errors What is a Type I error?
A type I error is when you reject the null hypothesis and it is actually true. Or in other words, you think there is a difference but there really is no difference. This is also known as a false positive.
VariablesWhat is an independent variable in a quantitative research study?
The independent variable is the one that is manipulated, such as the instruction offered. It is also called the treatment or the cause (Y variable).
VariablesWhat is a dependent variable in a quantitative research study?
The dependent variable is where you look for change based on what you did, such as an improvement in test scores. It is usually referred to as learning outcome; the effect (or Y) variable.
VariablesConstruct
A construct is an abstraction that cannot be observed directly but is invented to explain behavior.
e.g., intelligence, motivation, ability
VariablesConstructs & Variables
Constructs must be operationallydefined to be observable and measurable.
Variables are operationallydefined constructs.
Variables are placeholders that can assume any one of a range of values.
Variables may be measured by instruments.
VariablesNominal variables
Nominal variables describe categorical data.
e.g., gender, political party affiliation, school attended, marital status
Nominal variables are qualitative.
VariablesQuantitative
Quantitative variables range on a continuum with ordinal, interval, and ratio variables.
Measurement scalesOrdinal variables
Ordinal variables describe rank order with unequal units.
e.g., order of finish, ranking of schools or groups as levels
Measurement scalesInterval variables
Interval variables describe equal intervals between values.
e.g., achievement, attitude, test scores
Measurement scalesRatio variables
Ratio variables describe all of the characteristics of the other levels but also include a true zero point.
e.g., total number of correct items on a test, time, distance, weight
VariablesDependent
Dependent variables are those believed to depend on or to be caused by another variable.
Dependent variables are also called criterion variables.
VariablesIndependent
Independent variables are the hypothesized cause of the dependent variable. There must be at least two levels of an independent variable.
Independent variables are also called an experimental variables, manipulated variables, or treatment variables.
Collect data
There are three major ways for researchers to collect data.
A researcher can administer a standardized test.
e.g., an achievement test
A researcher can administer a selfdeveloped instrument.
e.g., a survey you might develop
A researcher can record naturallyoccurring events or use already available data.
e.g., recording offtask behavior of a student in a classroom
Collect dataQuantitative
Quantitative researchers often use paper and pencil (or electronic) methods.
Selection methods: The respondent selects from possible answers (e.g., multiple choice test).
Supply methods: The respondent has to provide an answer (e.g., essay items).
Measurement scalesInterpreting Instrument Data
Raw Score
Normreferenced scoring
Criterionreferenced scoring
Selfreferenced scoring
Measurement scalesInterpreting Instrument DataRaw Score
Number or point value of items correct (e.g., 18/20 items correct).
Measurement scalesinterpreting Instrument DataNormreferenced scoring
Student's performance is compared with performance of others (e.g., grading on a curve).
Measurement scalesInterpreting Instrument DataCriterionreferenced scoring
Student's performance is compared to preset standard (e.g., class tests).
Measurement scalesInterpreting Instrument DataSelfreferenced scoring
How individual student's scores change over time is measured (e.g., speeded math facts tests).
Measurement scalesCognitive tests
measure intellectual processes (e.g., thinking, memorizing, calculating, analyzing).
Standardized tests measure individual's current proficiency in given areas of knowledge or skill.
Standardized tests are often given as a test battery (e.g., Iowa test of basic skills, CTBS).
Diagnostic tests provide scores to facilitate identification of strengths and weaknesses (e.g., tests given for diagnosing reading disabilities).
Aptitude tests measure prediction or potential versus what has been learned (e.g., Wechsler Scales).
Measurement scalesAffective tests 1
Affective tests measure affective characteristics (e.g., attitude, emotion, interest, personality).
Attitude scales measure what a person believes or feels.
Likert scales measure agreement on a scale.
Strongly agree, Agree, Undecided, Disagree, Strongly disagree
Measurement scalesAffective tests 2
Semantic differential scales require the individual to indicate attitude by position on a scale.
FairUnfair
32 1 0 1 2 3
Rating scales may require a participant to check the most appropriate description.
5=always; 4=almost always, 3=sometimes...
The Thurstone Scale & Guttman Scales are also used to measure attitudes.
Measurement scalesAdditional Inventories 1
Interest inventories assess personal likes and dislikes (e.g., occupational interest inventories).
Values tests assess the relative strength of a person's values (e.g., Study of Values instrument).
Measurement scalesAdditional Inventories 2
Personality inventories provide participants with statements that describe behaviors characteristic of given personality traits and the participant answers each statement (e.g., MMPI).
Projective tests were developed to eliminate some of the concerns with selfreport measures. These tests are ambiguous so that presumably the respondent will project true feelings (e.g., Rorschach).
Measurement scalesCriteria for Good Instruments
Validity refers to the degree that the test measures what it is supposed to measure.
Validity is the most important test characteristic.
Measurement scalesCriteria for Good Instruments validity
There are numerous established validity standards.
Content validity
Criterionrelated validity
Concurrent validity
Predictive validity
Construct validity
Consequential validity
ValidityCriterionRelated Concurrent
The scores on a test are correlated to scores on an alternative test given at the same time (e.g., two measures of reading achievement).
ValidityCriterionRelated Predictive
The degree to which a test can predict how well a person will do in a future situation, e.g., GRE, (with predictor represented by GRE score and criterion represented as success in graduate school).
ValidityContent
Content validity addresses whether the test measures the intended content area.
Content validity is an initial screening type of validity.
Content validity is sometimes referred to as Face Validity.
Content validity is measured by expert judgment (content validation).
ValidityContent2 areas
Content validity is concerned with both:
Item validity: Are the test items measuring the intended content?
Sampling validity: Do the items measure the content area being tested?
One example of a lack of content validity is a math test with heavy reading requirements. It may not only measure math but also reading ability and is therefore not a valid math test.
ValidityConstruct
Construct validity is the most important form of validity.
Construct validity assesses what the test is actually measuring.
It is very challenging to establish construct validity.
ValidityConstruct evidence
Construct validity requires confirmatory and disconfirmatory evidence.
Scores on tests should relate to scores on similar tests and NOT relate to scores on other tests.
For example, scores on a math test should be more highly correlated with scores on another math test than they are to scores from a reading test.
ValidityConsequential
Consequential validity refers to the extent to which an instrument creates harmful effects for the user.
Some tests may harm the test taker.
For example, a measure of anxiety may make a person more anxious.
Validityfactors that threaten
Unclear directions
Confusing or unclear items
Vocabulary or required reading ability too difficult for test takers
Subjective scoring
Cheating
Errors in administration
Reliability
Reliability refers to the consistency of an instrument to measure a construct.
Reliability is expressed as a reliability coefficient based upon a correlation.
Reliability coefficients should be reported for all measures.
Reliability affects validity.
There are several forms of reliability.
Reliability Types
TestRetest (Stability)
Alternate forms (Equivalence)
Internal Consistency
Scorer and rater reliabilities
Reliability TypesTestRetest
reliability measures the stability of scores over time.
To assess testretest reliability, a test is given to the same group twice and a correlation is taken between the two scores.
The correlation is referred to Coefficient of Stability.
Reliability TypesAlternate forms
measures the relationship between two versions of a test that are intended to be equivalent.
To assess alternate forms reliability, both tests are given to the same group and the scores on each test are correlated.
The correlation is referred to as the Coefficient of Equivalence.
Reliability TypesInternal Consistency
represents the extent to which items in a test are similar to one another.
Splithalf: The test is divided into halves and a correlation is taken between the scores on each half.
Coefficient alpha and KuderRichardson measure the relationship between and among all items and total scale of a test.
Reliability TypesScorer and rater reliabilities
reflect the extent to which independent scorers or a single scorer over time agree on a score.
Interjudge (interrater) reliability: Consistency of two or more independent scorers.
Intrajudge (intrarater) reliability: Consistency of one person over time.
ReliabilityStandard Error of Measurement
an estimate of how often one can expect errors of a given size in an individual's test score.
SEm=SD * SQT 1r
Sem=Standard error of measurement
SD=Standard deviation of the test scores
r=the reliability coefficient
Selecting a Test Where to find?
These are a good place to start when selecting a test.
MMY: The Mental Measurements Yearbook is the most comprehensive source of test information
ProEd Publications
ETS Test Collection Database
Professional Journals
Test publishers and distributors