How can we help?

You can also find more resources in our Help Center.

424 terms

WGU RFC1

Foundations of Research
STUDY
PLAY
Quantitative Research
Numerical data
Experimental
Casual Comparative
Single Subject Experiment
Multiple Regression
Correlation
Descriptive/Survey
Mixed Method Research
Action Research
Both Quantitative and Qualitative
Qualitative Research
Ethnography
Case Study
Historical
Grounded Theory
Ethnography
The scientific description of the customs of peoples and cultures.
Basic and Applied Research (for theories)
Basic- use to develop or refine a theory
Applied- use to apply or test theory
Education Research (Formative and Summative)
Formative: use to improve a program/product under development.
Summative: use to evaluate the overall quality of a program/product to make decisions
Research and Development (R&D)
Use to research consumer needs and develop a product to fulfill those needs.
*Is not to test a theory.
Action Research
Use to solve everyday problems.
*not to test theory and result are not to generalized to other settings.
choice of paradigms (quanitative, qualitative, or action)
not a matter of personal preferences; rather, it is a matter of the research question that is being answered.
Which research design is best for educators?
The type of research one chooses is really based on the research question they want to answer. That is the first, most important, consideration.
paradigm
a perspective based on a set of assumptions, concepts, and values that are held by a community or researchers.
Variables
something that takes on different values or categories
the opposite of constants
constants
something that cannot vary, such as a single value or category of a variable
quantitative variables
vary in degree or amount (e.g., annual income- varies from zero to very high income)
categorical variables
vary in type or kind (e.g., gender).
independent variables
the presumed cause of another variable
example: amount of studying affects test grade.
Dependent variables
the presumed effect or outcome.
are influenced by one or more independent variables.
Example of Independent and dependent variables
Independent:Smoking
Dependent:Lung Cancer
intervening variables
also called mediator or mediating variables.
are variables that occur between two other variables.
example, tissue damage is an intervening variable in the smoking and lung cancer relationship.
Smoking---->Tissue Damage---->Lung Cancer
purpose of experimental research
to study cause and effect relationships
random assignment
which creates "equivalent" groups
used in the strongest experimental research designs
active manipulation of an independent variable
it is only in experimental research that "manipulation" is present
extraneous variable
a variable that may compete with the independent variable in explaining the outcome. Remember this, if you are ever interested in identifying cause and effect relationships you must always determine whether there are any extraneous variables you need to worry about.
confounding variable
it has confused or confounded the relationship we are interested in.
moderator variable
delineates how a relationship of interest changes under different conditions.
Example:
Perhaps the relationship between studying(IV) and test grades (DV) changes according to different levels of use of a drug such as Ritalin (MV).
nonexperimental research
no manipulation of the independent variable. There also is no random assignment of participants to groups.
if you ever see a relationship between two variables in nonexperimental research you cannot jump to a conclusion of cause and effect because there will be too many other alternative explanations for the relationship.
case of correlational research
one quantitative IV and one quantitative DV.
Example: Self-esteem (IV) and class performance (DV).

· You would look for the relationship by calculating the correlation coefficient.

· The correlation coefficient is a number that varies between -1 and +1, and 0 stands for no relationship. The farther the number is from 0, the stronger the relationship.

· If the sign of the correlation coefficient is positive (e.g., +.65) then you have a positive correlation, which means the two variables move in the same direction (as one variable increases, so does the other variable). Education level and annual income are positively correlated (i.e., the higher the education, the higher the annual income).

· If the sign of the correlation coefficient is negative (e.g., -.71) then you have a negative correlation, which means the two variables move in opposite directions (as one variable increases, the other decreases). Smoking and life expectancy are negatively correlated (i.e., the higher the smoking, the lower the life expectancy).
"basic case" of causal-comparative research
one categorical IV and one quantitative DV
Example: Gender (IV) and class performance (DV).

· You would look for the relationship by comparing the male and female average performance levels.
3 Necessary Conditions for Causation
To conclude changes in variable A cause changes in variable B.
Condition 1:Relationship conditon - Variable A and Variable B must be related.
Condition 2: Temporal Antecedent Condition - proper time order must be established.
Condition 3: Lack of alternative explanation condition- the relationship between variable A and B must not be due to some confounding extraneous or third variable
Qualitative Research Methods
1- Phenomenology
2- Ethnography
3- Case study research
4- Grounded theory
5- Historical research
Phenomenology
a form of qualitative research in which the researcher attempts to understand how one or more individuals experience a phenomenon. For example, you might interview 20 widows and ask them to describe their experiences of the deaths of their husbands.
Ethnography
is the form of qualitative research that focuses on describing the culture of a group of people. Note that a culture is the shared attitudes, values, norms, practices, language, and material things of a group of people. For an example of an ethnography, you might decide to go and live in a Mohawk communities and study the culture and their educational practices.
Case study research
is a form of qualitative research that is focused on providing a detailed account of one or more cases. For an example, you might study a classroom that was given a new curriculum for technology use.
Grounded theory
is a qualitative approach to generating and developing a theory form data that the researcher collects. For an example, you might collect data from parents who have pulled their children out of public schools and develop a theory to explain how and why this phenomenon occurs, ultimately developing a theory of school pull-out.
Historical research
research about events that occurred in the past. An example, you might study the use of corporeal punishment in schools in the 19th century.
Mixed Research Methods
general type of research in which quantitative and qualitative methods, techniques, or other paradigm characteristics are mixed in one overall study.
Advantages of Mixed Research
First of all, we advocate the use of mixed research when it is feasible. We are excited about this new movement in educational research and believe it will help qualitative and quantitative researchers to get along better and, more importantly, it will promote the conduct of excellent educational research
Mixed method research
is research in which the researcher uses the qualitative research paradigm for one phase of a research study and the quantitative research paradigm for another phase of the study. For example, a researcher might conduct an experiment (quantitative) and after the experiment conduct an interview study with the participants (qualitative) to see how they viewed the experiment and to see if they agreed with the results. Mixed method research is like conducting two mini-studies within one overall research study.
Mixed model research
is research in which the researcher mixes both qualitative and quantitative research approaches within a stage of the study or across two of the stages of the research process. For example, a researcher might conduct a survey and use a questionnaire that is composed of multiple closed-ended or quantitative type items as well as several open-ended or qualitative type items. For another example, a researcher might collect qualitative data but then try to quantify the data.
Types of Quantitative Research
1- Experimental
2- Nonexperimental
Methods of Knowing
-Experience
-Authority (someone we know and trust tells us)
-Reasoning/Thinking
*All limited so as researchers we use Scientific Method
Inductive Reasoning
Developing generalizations based on observation of a limited number of related events or experiences.
Ex:
An instructor reviews research methods texts and notices sampling content in each text and concludes that all research methods texts have sampling
A teacher knows several student athletes who are good students and concludes that all student athletes are good students
Deductive Reasoning
Arriving at specific conclusions based on general principles, observations, or experiences.
Ex:
All research textbooks contain a chapter on sampling. The book you are reading is a research text and therefore must contain a sampling chapter.
All student athletes are good students. Mary is a student athlete and therefore she must be a good student.
Steps of the Scientific Method
Recognition and identification of a problem
Formulation of hypothesis
Data collection
Data analysis
Statement of conclusions
Confirm or disconfirm hypothesis
Hypothesis
an explanation for the occurrence of certain behaviors, phenomena, or events
Limitations of the Scientific Method
The scientific method can not answer all questions, especially those of a philosophical or ethical nature.
•Application of the scientific method can never capture the full richness of the context.
•Measurement error is a limitation of the scientific method.
goal of educational research
describe, explain, predict, and control situations involving human beings.
Selection and definition of a problem
A problem is a question of interest.
•The problem can be tested or the question answered through the collection and analysis of data.
•Generally researchers use a review of the existing literature to generate hypotheses related to their question.
Execution of research procedures
Research procedures are dictated by the research problem and the identified variables.
•Procedures include activities related to collecting data about the problem.
Analysis of Data
Data are analyzed such that the researcher can test the hypothesis or answer the research question.
•Data analysis often includes statistical techniques.
Drawing and stating conclusions
-Conclusions are based upon analyses of our data and are stated in terms of the original hypothesis or research question.
Conclusions should indicate whether the hypothesis was rejected or supported.
-For those studies that include synthesis of verbal data, conclusions may be more tentative.
Educational research is often broadly categorized as qualitative or quantitative.
Both quantitative and qualitative researchers collect and analyze data.
Both quantitative and qualitative researchers derive conclusions and interpretations.
Approaches to Educational Research-Quantitative
Numerical data
Describes, predicts or controls variables of interest
The world is relatively uniform, stable, and predictable
Researchers state hypothesis, specify research procedures, and control context.
Researchers identify large samples.
Researchers are interested in statistical significance
Approaches to Educational Research-Qualitative
Narrative and visual data
Gains insights into phenomenon of interest
Knowledge is situated and contextual. There are different perspectives
Researchers form foreshadowed problems. They don't control but explain context.
A large amount of data is collected over an extended time in a natural setting.
Data are categorized and organized into patterns
Example of Quantitative
Are there differences in the amount of discipline referrals between girls and boys with Autism Spectrum Disorders?
What variable best predicts whether first generation college students stay continually enrolled in college until graduation?
Do students exposed to animated science materials learn more if the animation is accompanied by sound or no sound?
Example of Qualitative
What are the social experiences of middle school girl with an Autism Spectrum Disorder?
What challenges do first generation college students from small rural schools experience as they enroll in college?
How do students describe their experiences learning from animated science materials?
Survey Research
-determines and reports the 'current status' of the subject of study.
-often collects numerical data to test hypotheses or answer questions.
-may examine preferences, attitudes, or opinions
Ex:
How do parents feel about national teacher certification?
To what degree do students report test anxiety before the SAT exam?
Correlational Research
-provides a quantitative measure of relationship between or among variables.
-This measure is expressed as a correlation coefficient.
Ranges from -1 to 1; 0 indicates no relationship
-do not indicate cause-effect relations among variables.
Ex:
A high correlation between self-concept and achievement does not mean that self-concept causes higher achievement.
Causal-Comparative Research
-attempts to determine the cause or reason for existing differences.
-The grouping variable is the potential cause.
-The dependent variable is the effect.
Ex:
Are there differences in final exam performance between students given a practice exam and those given more time for independent study?
Are there differences between elderly men and women who recently experienced a stroke in the amount of minutes of exercise during rehabilitation prior to release from hospital?
Causal-Comparative Research
-provides limited cause-effect data but true cause-effect findings can be determined only through experimental research.
-Sometimes it is impossible to conduct true experiments such as when grouping variables:
Cannot be manipulated.
e.g., year in school, age, gender
Should not be manipulated.
e.g., attended rehabilitation or not; exposed to verbal abuse or not
Experimental Research
includes at least one independent variable and the effect is measured on at least one dependent variable.
In experimental research extraneous variables are controlled.
researchers select participants, group participants, administer treatments, control the research setting, control the length of treatment exposure, select research measures, and are able to draw cause-effect conclusions.
Examples of Experimental Research
Sometimes experimental research is not possible in educational contexts because of difficulties with random selection and random assignment.
Examples:
Are there differences in achievement between students randomly assigned to one of two problem solving strategy conditions?
The independent variable is problem solving strategy conditions.
The dependent variable is achievement.
Is there an effect of corrective feedback on students' achievement on final physics test scores and interest in physics?
The independent variable is corrective feedback.
The dependent variables are final test scores and interest.
Single-Subject Research
used to study behavior change within a person or a group as they are exposed to an intervention or a treatment.
In single-subject research the size of the sample is one.
Ex:
The effects of graphing intervention on off-task behavior.
The effects of a cessation program on smoking.
Qualitative Research
-seek rich understandings.
-conducted through sustained in-depth, in-context, research.
-numerous approaches: historical research, symbolic interaction, grounded theory, ethology, phenomenology
Narrative Research
-study how different people experience their world.
-allows for people to tell the story of their lives.
-often focus on one person and collect stories of that person's life.
-establish a trusting personal relationship with their participants.
-The narrativeis the story of the phenomenon being investigated and also the method of inquiry.
Narrative Research-Examples
What is the experience of students in a new co-ed dormitory?
How does a winning coach react when faced with a less skilled team and a losing season?
Ethnographic Research
-the study of cultural patterns and perspectives of participants in their natural settings.
-avoid interpretations early and enter the setting slowly.
-have personal relationships with their participants.
-collect data in waves, re-entering the site several times.
-resultant ethnography is a narrative that presents participants' everyday events.
-During data collection the ethnographer identifies categories and enters themes into these categories.
Ethnographic Research Examples
How does the returning student population adjust to technology enhanced instruction in entry level college courses?
Case study
-qualitative approach that examines a bounded system for study.
-includes specific design, data collection techniques, and data analysis strategies.
Ex:
What are the challenges faced by a school board with shrinking district population and mandated facility renovations?
Qualitative Research Process
1.Identifying a research topic
•The initial topic is often narrowed.
2.Reviewing the literature
•Previous research is examined to garner important information.
3.Selecting participants
•Generally fewer purposely sampled participants than in quantitative studies.
4.Collecting data
•Generally interviews, observations, and artifacts serve as data
5.Analyzing and interpreting data
•Researchers analyze for themes and generate interpretations.
6.Reporting and evaluating the research
•Researchers summarize and integrate the data in narrative and visual form.
Qualitative Research
1.Qualitative researchers spend a great amount of time in the field and engage in person-to-person interactions.
2.Qualitative data are analyzed inductively.
3.Qualitative researchers avoid making assumptions and remain open to alternative explanations.
Classification of Research
The approach used in a study is determined by the research problem.
Often the same general problem can be approached from several different types of study.
Research methods are selected after the topic or question is determined
Example of how to choose which type of research to use...
Given the general topic of year-round schooling, suggest how this topic can be addressed differently by each one of the types of research :
Survey: What are parents' opinions about year-round schooling?
Correlational: Is there a relationship between standardized test scores and year-round schooling?
Causal-Comparative: Are there differences in amount of content mastered between children enrolled in year-round schooling versus those in traditional schooling?
Experimental: Are there differences in self-esteem between those students randomly assigned to districts with year-round schooling versus those assigned to traditional schooling?
Single subject: What are the effects of year-round schooling on the vocabulary use of students who are reading disabled?
Research by Purpose
Basic and applied research
Basic research is conducted to develop or refine theory.
Applied research is conducted to apply or test a theory.
Evaluation research
Evaluation research is a form of applied research that involves data collection for decision making.
Evaluation research may be formative or summative
Research and Development (R & D)
Research and development is the process of researching needs and developing products to fit those needs.
Action Research
Action research is a systematic inquiry conducted by teachers, principals, or other stakeholders.
Ethics of Educational Research
Ethical considerations are an important part of research.
Researchers must be aware of and attend to ethical considerations.
Two main overarching ethical rules guide researchers.
Participants should not be harmed.
Researchers obtain participants' informed consent.
Researchers must submit their proposal for review and approval.
Deception poses an ethical dilemma. If the participants know the purpose of a study, it may change their behavior.
e.g., gender, race, attitudes, medical status
When a study must use deception it causes problems for informed consent. These types of studies must undergo strict ethical review.
Ethics of Educational Research-societies
behavioral researchers have similar codes for ethical research largely due to the National Research Act of 1974 which Created a code for the protection of human subjects.
Ethics of Educational Research-institutions
have review groups that assure participant protections.
IRB (Institutional Review Board) or HSRC (Human Subjects Review Committee)
Participants should not be harmed.
Physically, mentally, socially
Research participants freely agree to participate
Researchers ensure freedom from harm
No undue risks
Informed consent
: Researchers provide information about the study and any potential dangers
Personal privacy and confidentiality
Limit access of data to those who 'need to know'
Participants' involvement should not be reported
Anonymity
Study participants have complete anonymity when their identities are unknown to the researcher.
Confidentiality
Study participants are known to researcher but are not disclosed.
e.g., removing names from data
The Buckley Amendment (The Family Educational Rights and Privacy Act of 1974)
Data that identifies a student is not available without written permission
Must describe what data, for what purposes, and to whom
Ethics of Qualitative Research
In addition to ethics of educational research generally, qualitative research often poses additional ethical challenges.
The nature of the research changes so informed consent is challenging.
The close relationship between the researcher and the participant may allow the researcher to know personal and perhaps ill-acts of the participant that may pose ethical challenges.
Ethical Guideposts
A researcher should have an ethical perspective with regard to the research that is very close to her personal ethical position.
Informed consent should be obtained through a dialogue between the researcher and the participants
Be cognizant of the broader social principles that define your ethical stance. Potential results do not drive ethical standards.
Minimize the potential for harm to your participants.
Attend to confidentiality and omit deception.
Gaining Entry to the Research Site
Researchers need cooperation to conduct their studies.
Identify and follow procedures for gaining approval at any given site.
e.g., superintendent or school board
Procedures generally require filling out forms that describe the study.
Researchers may need permission from principals or teachers.
Written permission from parents is often required.
Gaining entry and obtaining permission often takes considerable time.
Schools and other research communities may request something in return for their participation in your study.
e.g., a final report prior to dissemination, professional development, parent education
What are some advantages of using quantitative data?
Some examples include that you can analyze vast amount of information quickly, the data can more easily be gathered, and you can more easily compare the results of different groups
What are examples of qualitative research data collection instruments?
Examples of qualitative data collection instruments are surveys; interviews notes , recordings, and transcripts; and field observation protocols.
What is quantitative data?
Quantitative data are anything that provides a quantity or numerical score, such as the scores from a test or rubric.
Survey Research
-involves collecting data
to test hypotheses or to answer questions
about people's opinions on some topic or
issue.
-an instrument to collect data that describes characteristics of a population.
-challenging to conduct
and findings from survey research are not always easily interpreted.
Types of Survey Research
A sample survey is designed to sample participants and generalize to a population.
A census survey samples every member of the population.
Census surveys are generally done with small and accessible populations.
Survey Research Designs
-Cross-Sectional Surveys
-Longitudinal Surveys
Cross-Sectional Surveys
Example: A study that examines the self-concept of all pre-service teachers at a given university.
Data are collected from selected participants at a single point in time.
These studies provide a snapshot of current beliefs or opinions.
These survey studies are not intended to provide insight into trends or changes over time.
Longitudinal Surveys
Data are collected at two or more time points.
One of the challenges when conducting longitudinal studies is attrition
Several types of longitudinal survey designs.
-Trend studies
-Cohort studies
-Panel studies
-Follow-up studies
Trend studies
examine changes over time in a particular population defined by some trait.
e.g., entering kindergartners
Cohort studies
examine one population selected at a particular time period.
Ex:
e.g., A researcher could study children that received speech and language support as first graders in 2007. She might examine some participants from the population in 2008 and then sample others from that population in 2010.
Panel studies
involve the same sample of individuals over a set time.
e.g., A researcher might conduct a five-year panel study of the first class of law students at a new institution. He might survey them each spring for five years.
Follow-up studies
investigate change in a previously studied population.
e.g., A researcher might be interested in a follow-up study in 2015 of those participants involved in the cohort study of first-graders in speech and language support during 2007.
Ways of Conducting Survey Research
-Questionnaires are a written collection of survey questions answered by a group of participants.
-Interviews are oral, in-person question and answer sessions between the researcher and a respondent.
Conducting a Questionnaire Study
-Stating the problem
Developing a questionnaire requires skill and time.
Researchers should plan content and format of the questionnaire.
Most surveys include one of two types of items.
Stating the problem in questionnaire study
Participants are more likely to respond to questionnaires that they perceive address a sufficiently relevant topic or problem.
Researchers should set objectives for the type of information desired from a questionnaire.
surveys include one of two types of items.
Structured items or closed-ended items for which participants choose among possible responses (e.g., Likert scale).
Unstructured items in which participants have freedom of response (e.g., fill-in answer).
Guidelines to consider when constructing a questionnaire
Include only items that relate to the objectives of the study.
Focus each question on a single concept.
Collect demographic information.
Define terms.
Include a point of reference or comparison for participants.
Avoid leading questions.
Avoid sensitive questions to which respondents might not answer honestly.
Don't ask a question that assumes something that may not be true.
Organize items from general to specific.
Have others read your instrument and provide feedback.
Write directions for the respondents.
Include the purpose of the study at the top of the instrument.
Pilot test the questionnaire.
Conducting a Questionnaire Study-Pilot testing the questionnaire
Pilot test with members of the intended sample population.
Ask respondents to make suggestions for any content to add or to delete.
Incorporate feedback from your pilot administration and make any appropriate changes.
Conducting a Questionnaire Study-Preparing the cover letter
Cover letters should accompany all surveys.
Cover letters should be brief.
Cover letters should be addressed to a specific person.
The cover letter should explain the purpose of the study.
The cover letter should include a statement regarding the importance and significance of the study.
The cover letter should include how results from the study will be shared with respondents and interested others.
Pilot test the cover letter.
Administering the Questionnaire
Selecting participants
Appropriate sampling strategies should be followed.
Distributing the questionnaire
Email
Mail
Telephone
Personal administration
Interview
Administering the Questionnaire-Mail
Advantages
Inexpensive
Can be confidential or anonymous
Standardized items and procedures
Easy to score most items
Disadvantages
Response rate may be small
Cannot follow up items
Response sets possible
Limited to those who read
Administering the Questionnaire-E-Mail
Advantages
Easy to target respondents
Quickly administered
Inexpensive
Can be confidential or anonymous
Standardized items and procedures
Easy to score most items
Disadvantages
Not everyone has email
Multiple replies from same participant possible
Response rate may be small
May get sorted to trash
Cannot follow up items
Response sets possible
Limited to those who read
Administering the Questionnaire-Telephone
Advantages
High response rates
Quick data collection
Can reach a range of locations and respondents
Can use 800 call in numbers to increase response rates
Disadvantages
Requires phone number lists
Difficult to get in-depth data
Administrators must be trained
Cost
Cell phones versus landlines
Administering the Questionnaire-Personal Administration
Advantages
Efficient when respondents are in close proximity
Personal connection with respondents
Disadvantages
Time consuming
Training
Cost
Personal connection with respondents
Administering the Questionnaire-Interviews
Advantages
Can follow-up responses and probe for additional information or clarity
May be recorded
Response rate
Personal connection with respondents
Disadvantages
Time consuming
Training
Cost
Interviewer bias possible
Unstructured data analysis
Personal connection with respondents
Conducting Follow-Up Activities
The higher the return rate the better your study.
Low response rates limit generalizability.
Send out a reminder postcard to increase response rates.
Send out a second complete mailing to increase response rates.
Consider phone-call reminders to increase response rates.
Response rates vary.
First mailings usually return about 30%-50%.
Second mailings add an additional 20%.
Most times additional mailings are not cost-effective.
Researchers should try to determine possible reasons for nonresponse.
Nonresponse may be at the survey or item level.
Researchers should carefully look for patterns of nonresponse.
Tabulating Questionnaire Responses
Use scannable answer sheets if possible.
Accuracy
Time
Enter answers into spreadsheet or statistical package.
Code open-ended answers.
Consider available qualitative software analysis packages.
Analyzing Results
Report total sample size.
One way to report results from surveys is to share overall percentage of return and response rates per item.
e.g., Percentage that answered 'yes' and percentage that answered 'no'
Alternative reports can include total scores or mean scores by cluster or area of a survey. Such as comparisons by participant characteristics.
e.g., Percentage of men or women with a particular response.
Reports of survey studies will vary by the underlying purpose of the survey.
Correlational Research
-determine the nature of relations among variables or to use these relations to make predictions.
-often examine numerous variables believed to be related to complex variables (e.g., achievement).
Unrelated variables are discarded from future studies while those related may be examined further through causal-comparative or experimental studies.
High correlations among variables do not imply causation (e.g., self-concept and achievement).
Correlational procedures are also used to examine reliability and validity.
Correlational Research Process
Problem selection
Participant and instrument selection
Design and procedure
Data analysis and interpretation
Correlation coefficients
Correlational Research Process-Problem selection
Correlational studies are designed to explore whether and how variables are related.
Correlational studies are designed to test hypotheses regarding expected relations among variables.
Correlational Research Process-Participant and instrument selection
Samples are derived from acceptable sampling methods.
Sample must include at least 30 participants.
When reliability and validity of instrumentation is lower, sample size must be larger.
Correlational Research Process-Design and procedure
Correlational studies share a simple design. Scores for two or more variables of interest are obtained for each member of the sample and these scores are then correlated (e.g., self-concept and achievement).
Correlational Research Process-Data analysis and interpretation
The meaning of a correlation coefficient will vary depending upon purpose.
Sample size effects the strength of a correlation coefficient.
Some correlation coefficients that are statistically significant may not represent meaningful significance.
Correlation coefficients do not represent percentage of relation between variables.
The square of the correlation coefficient indicates the amount of variance shared by the variables (shared variance).
e.g., a correlation of .50 indicates 25% shared variance.
Correlational Research Process-Correlation coefficients
Correlation coefficients range from -1 to 1.
A correlation of 0 indicates no relationship.
Correlation coefficients between +.35 and -.35 represent a weak relationship or norelationship.
Correlation coefficients between +.35 and +.65 or between -.35 and -.65 represent moderaterelationships.
Correlation coefficients between .65 and 1.0 or between -.65 and -1.0 represent strongrelationships.
Relationship Studies
-gain insight into variables or factors that are related to a complex variable (e.g., retention, academic achievement).
-help researchers to determine which variables may be suitable for future research.
-provide insight into which variables should be controlled for in future causal-comparative or experimental studies.
Relationship Studies-Data collection
Researchers first identify variables to be correlated.
Variables should be purposely identified.
A smaller number of carefully identified variables is preferable to a larger number (e.g., shotgun approach).
After identifying variables the researcher next identifies the appropriate population and sampling procedure to select participants for the study.
In some relationship studies data are collected all at one time or in several sessions conducted in close succession.
Relationship Studies-Data analysis and interpretation
In relationship studies scores from one variable are correlated with scores for another variable; or scores for several variables are correlated with a particular variable of interest.
The result is a single correlation coefficient or a number of correlation coefficients.
The method of calculating a correlation coefficient depends upon the nature of the data.
Relationship Studies-Data analysis and interpretation
-Correlation coefficients
Correlation coefficients
The Pearson r coefficient is the most common and most precise coefficient. Pearson r is used for continuous variables.
The Spearman rho coefficient is appropriate to use when one of the variables are represented by rank-order data.
The phicoefficient is used when both variables are expressed as a categorical dichotomy.
Other correlation coefficients are appropriate given characteristics of the data collected, sample size, and underlying data distribution.
e.g., Kendall's tau, Biserial, Point biserial, Tetrachoric, Intraclass, eta.
Relationship Studies-Data analysis and interpretation-inaccurate estimates of relation among variables.
Several factors may contribute to inaccurate estimates of relation among variables.
Underlying relationships that are curvilinear will effect coefficients.
Attenuation occurs when measures have low reliability and may provide inaccurate correlation coefficients.
Restricted range in scores generally leads to underestimates of relations.
Prediction Studies
When two variables are highly related, scores on one variable can be used to predict scores on the other variable.
The variable used to predict is called the predictor.
The variable that is predicted is called the criterion.
*used to determine which variables are the most highly correlated with a criterion variable.
More than one variable can be used to make predictions.
Prediction Studies-Data collection
In prediction studies all measures should be valid measures.
It is especially critical that the criterion variable be validly measured.
In prediction studies, sometimes the predictor variables are administered prior to the criterion variable (e.g., SAT and university GPA).
Attrition is a problem in some prediction studies.
Shrinkage, or the tendency to find less accuracy in predicting criterion variables in subsequent samples, is often noted in prediction studies.
Cross-validation is the process of conducting subsequent prediction studies with new samples to verify effects found in an initial prediction study
Attrition
-The action or process of gradually reducing the strength or effectiveness of someone or something through sustained attack or pressure.
-a reduction or decrease in numbers, size, or strength
Prediction Studies-Data analysis and interpretation
In prediction studies, data analysis involves correlating predictor variables with the criterion variable.
Multiple regression is used when a combination of variables is used to predict a criterion variable (e.g., Success in Algebra may be predicted by prior knowledge, prior achievement, aptitude, etc.)
Intervening variables may lower prediction accuracy (e.g., teacher).
The amount of common variance shared by predictors is the squared correlation of the predictors and the criterion and is referred to as the coefficient of determination.
Prediction Studies-Data analysis and interpretation-single variable prediction equation
Y= a+bX
Y= Predicted criterion score for an individual
X= An individual's score on the predictor variable
a= A constant calculated from scores of all participants
b= A coefficient that indicates the contribution of the predictor variable to the criterion variable
Other Correlation-Based Analyses
In discriminate function analysis, continuous variables are used to predict a categorical variable.
Canonical analysis produces a correlation based upon a group of predictor variables and a group of criterion variables.
Path analysis provides a diagram that illustrates how variables are related to one another.
Structural equation modeling, or LISREL, extends path analysis and predicts relations among variables with added precision.
Factor analysis is used to decrease the number of variables under consideration by grouping variables into clusters called factors.
Interpreting Correlation Coefficients
Was the proper correlation method used?
Do the variables have high reliabilities? Low reliabilities lower the chance of finding significant relations.
Is the validity of the variables strong? Invalid variables produce meaningless results.
Is the range of scores to be correlated restricted or extended? Narrow or restricted score ranges lower correlation coefficients, whereas broad or extended score ranges raise them.
How large is the sample? The larger the sample the smaller the value needed to reach statistical significance. Large samples may yield correlations that are statistically significant but practically unimportant.
Causal-Comparative Research
-researcher attempts to determine the cause or reason for existing differences in groups or individuals.
-Retrospective casual-comparative research studies start with effects and investigate causes
-Prospective casual-comparative research studies start with the causes and investigate the effects.
-does not establish cause-effect relations.
-generally includes more than two groups and at least one dependent variable.
-the independent variable is not manipulated by the researcher.
Causal-Comparative Research
-can be conducted when variables cannot or should not be experimentally manipulated.
-can facilitate decision making.
-provide insight into conducted or potential experimental studies.
-generally less costly than are experimental studies.
Causal-Comparative Research-Ex of retrospective causes and investigate the effects..
More common in educational research.
e.g., A researcher interested in the benefits of an exercise program on reducing stress may select a group of people who had enrolled in a stress-reduction exercise class and those who had not and compares their stress levels.
Causal-Comparative Research-Ex of prospective causes and investigate the effects.
What is the effect of X?
e.g., A researcher may hypothesize that those children that attend dance classes during elementary school have higher self-esteem when in middle school. She would identify a group of middle-school children who had dance classes in elementary school and a group of those who did not, and compare their self-esteem.
Causal-Comparative Research-independent variable is not manipulated
The independent variable has occurred or is already formed.
Independent variable in causal-comparative studies is often referred to as the grouping variable.
Examples of variables investigated in Causal-comparative studies:
Organismic variables (e.g., age, ethnicity, sex)
Ability variables (e.g., achievement)
Personality variables (e.g., self-concept)
Family-related variables (e.g., SES)
School-related variables(e.g., type of school, size of school)
Causal-Comparative Research Example
Ethnicity, birth order, spatial ability, teacher education level, goal orientation
Ethnicity (organismic), birth order (family-related), spatial ability (ability), teacher education level (school-related), goal orientation (personality)
Limitations of causal-comparative research.
The experimenter has limited control.
Caution in interpretation is necessary as cause-effect relations cannot be established.
Only relations are established.
Conducting a Causal-Comparative Study
The basic causal-comparative design involves selecting two groups that differ on a variable of interest and comparing them on a dependent variable.
Definition and selection of comparison groups is critical in causal-comparative research.
Grouping variables must be operationally defined (e.g., training versus no training).
Researchers should test for differences between groups (e.g., prior knowledge).
Basic causal-comparative designs
There are several control procedures that researchers can employ to strengthen their causal-comparative designs.
Conducting a Causal-Comparative Study-test for differences between groups
The more similar the groups are on extraneous variables, the fewer alternative explanations there may be for research findings.
Conducting a Causal-Comparative Study-Basic causal-comparative designs
In one design: One group is exposed to an independent variable while the other group is not. Both groups are measured on a dependent variable.
(E) (X) O
(C) O
In a second design: Two groups are exposed to different independent conditions. Both groups are then measured on a dependent variable.
(E) (X1) O
(C) (X2) O
Conducting a Causal-Comparative Study-control procedures that researchers can employ to strengthen their designs.
Matching: Researchers can attempt to equate groups and control for one or more variables.
For example, a researcher comparing two types of instruction might control for prior achievement. To do this, he would do pair-wise matching and would place an equal number of high achieving students in each condition.
Comparing homogeneous groups or subgroups: Researchers can also compare groups that are homogeneous with respect to an extraneous variable.
For example, the researcher may select only high-achieving students for his study.
Analysis of Covariance (ANCOVA): Researchers can use this statistical technique to adjust scores on a dependent variable for initial differences on a related variable.
For example, the researcher could measure prior knowledge and use those scores as a covariate.
Conducting a Causal-Comparative Study
-Data analysis and interpretation
Descriptive and inferential statistics are used to analyze data from causal-comparative studies.
Conducting a Causal-Comparative Study-
Descriptive and inferential statistics
Descriptive statistics often include the mean and the standard deviation.
Inferential tests used include t-tests, analyses of variance, and chi square.
Experimental Research
-the only type of research that can test hypotheses to establish cause-effect relations.
In experimental research studies the independent variable is also called the treatment, causal, or experimental variable.
In experimental research studies the dependent variable is also called the criterion, effect, or posttest variable
Experimental Research
-the most structured of all research.
-can provide evidence for cause-effect relations.
-Several experimental studies taken together can provide support for generalization of results.
Experimental Research-test hypotheses to establish cause-effect relations
The researcher manipulates at least one independent variable and controls other relevant variables, and observes the effect on one or more dependent variables.
The researcher manipulates the treatment.
The researcher has control over selection and assignment.
Experimental Research-The experimental process
The steps in the experimental research process are the same as in other types of research.
Selecting and defining a problem
Selecting participants and measuring instruments
Preparing a research plan
Executing procedures
Analyzing the data
Formulating conclusions
Experimental Research-The experimental process
In experimental studies, the researcher controls selection and assignment.
Experimental studies often examine comparisons between or among groups.
Comparison of approaches (A versus B)
Comparison of an approach to an existing approach (A versus no A)
Comparison of different amounts of a single approach (A little of A versus a lot of A)
Experimental Research-The experimental process
In experimental research studies the group that receives the treatment is the experimental group and the group that does not receive the treatment is called the control group.
Sometimes groups are comparison groups that receive alternative treatments (e.g., two types of instruction in a content area).
Experimental Research-challenges for experimental studies in educational settings
A lack of sufficient exposure to treatments (i.e., treatments are too short or diffuse).
Failure to make treatments significantly different from one another (e.g., an experimental instructional program in math may not be different enough from the comparison math instructional program).
Experimental Research-Manipulation and control
In experimental studies, researchers control or remove the influence of extraneous variables.
Participant variables
Organismic (e.g., age)
Intervening (e.g., interest)
Environmental variables (e.g., school or teacher effects)
Experimental Research-Threats to Experimental Validity
Internal validity refers to the degree to which observed differences in the dependent variable are a direct result of manipulation of the independent variable and not some other variable.
Internal validity is concerned with rival explanations for an effect.
External validity, sometimes referred to as ecological validity, is the degree to which the results from a study are generalizable to other groups.
When researchers increase the internal validity of their study, they decrease their external validity.
When researchers are concerned with external validity, their ability to control important extraneous variables suffers.
When there is a choice, researchers should err on the side of control and maximize internal validity.
Experimental Research-Threats to Internal Validity
History
Maturation
Testing
Instrumentation
Statistical regression
Differential selection of participants
Mortality
Selection-maturation and interactive effects
Experimental Research-Threats to Internal Validity-History
Any event occurring during a study that is not part of the experimental treatment but that may effect the dependent variable represents a history threat.
Longer-lasting studies are more prone to history threats.
Ex:
In a study of the effects of instructional simulations in learning chemistry content, a history threat would be demonstrated if students in the study were exposed to simulations in a different setting, such as when learning geography, while the study was being conducted.
Experimental Research-Threats to Internal Validity-Maturation
Maturation refers to physical, intellectual, and emotional changes that naturally occur within participants over a period of time.
Ex:
In studies of interventions that are designed to increase children's theory of mind, if the interventions lasted more than a couple of weeks at critical time points, participants may gain critical theory of mind awareness simply due to cognitive development and not due to the treatment.
Experimental Research-Threats to Internal Validity-Testing
Testing as a threat to internal validity is demonstrated when taking a pretest alters the result of a posttest.
Experimental Research-Threats to Internal Validity-Instrumentation
Instrumentation is a threat to internal validity when the instrumentation is either unreliable or is changed between pre-and posttesting.
Experimental Research-Threats to Internal Validity-Statistical regression
Extremely high or low scores tend to regress to the mean on retesting.
Ex:
If students perform poorly on a pretest it is difficult to determine if the gain in their scores is due to treatment effects.
Experimental Research-Threats to Internal Validity-Differential selection of participants
Participants in the control and experimental groups differ in ways that influence the dependent measure.
Experimental Research-Threats to Internal Validity-Mortality
Participants drop out of the study at differential rates across conditions.
Experimental Research-Threats to Internal Validity-Selection can interact with other threats to internal validity (i.e., history, maturation, instrumentation).
Participants selected into the treatment and control conditions have different experiences or maturation rates or instrumentation varies across conditions.
Experimental Research-External validity threats can be divided into two categories:
'Generalizing to whom' threats
Threats affecting groups to which the study can be generalized
'Generalizing to what' threats
Threats affecting the settings, conditions, variables, and contexts to which the results can be generalized
Pretest-treatment interaction
Multiple-treatment interference
Selection-treatment interaction
Specificity of variables
Experimenter effects
Reactive arrangements
Experimental Research-Threats to External Validity-Pretest-treatment interaction
This threat occurs when participants respond differently to a treatment because they have been exposed to a pretest.
Pretest may alert participants.
Self-report measures are often susceptible to pretest-treatment interaction effects.
At times, unobtrusive measures can be used as pretests, to limit this threat to validity (e.g., using previously administered standardized assessments to measure ability in science instead of using a pretest).
Experimental Research-Threats to External Validity-Multiple-treatment interference
This threat occurs when previous treatments cross-over into a current experiment. This makes it challenging to determine the effectiveness of the later treatment.
This threat may occur in studies that access participants who have been exposed to other research studies (e.g., university participant pools).
Experimental Research-Threats to External Validity-Selection-treatment interaction
When a study's findings only apply to the groups selected and are not representative of other groups.
This may happen in non-randomly assigned studies where a treatment is less or more effective for certain demographics (e.g., ability levels)
Experimental Research-Threats to External Validity-Specificity of variables
When researchers do not adequately define their variables, instruments, or population, it makes it difficult to determine how well the findings will generalize to an alternative population.
Experimental Research-Threats to External Validity-Experimenter effects
Experimenter effects occur when characteristics or behaviors of the experimenter influence the participants' responses.
Experimental Research-Threats to External Validity-Reactive arrangements
These threats are also referred to as participant effects. These threats are associated with differences in participants' behavior, feelings, and attitudes because they are in a study.
Hawthorne effect
John Henry effect (Compensatory rivalry)
Placebo effect
Novelty effect
Experimental Research-Threats to External Validity-Reactive arrangements-Hawthorne effect
Any situation in which participants' behavior is affected because they are in a study.
Experimental Research-Threats to External Validity-Reactive arrangements-John Henry effect (Compensatory rivalry)
Members of the control group compete with the experimental group.
Experimental Research-Threats to External Validity-Reactive arrangements-Placebo effect
To combat compensatory rivalry, researchers attempt to give control groups a placebo, not the experimental treatment, but something to decrease the perception that they are in the control group. Participants should perceive they are all getting the same thing.
Experimental Research-Threats to External Validity-Reactive arrangements-Novelty effect
When participants are engaged in something different this may increase attention, interest, behavior, learning, etc., just because it is something new.
Experimental Research-Group Experimental Designs
The validity of an experiment is a function of the degree to which extraneous variables are controlled.
Randomization is the best mechanism to control for extraneous variables.
Randomization distinguishes experimental designs.
Randomization should be used whenever possible.
If groups cannot be randomly formed, variables should be held constant when at all possible (e.g., time of day, which researcher is present).
Participant variables can be controlled and held constant
Experimental Research-Group Experimental Designs
-Participant variables can be controlled and held constant
Matching can equate groups through random assignment of pairs.
Comparing homogeneous groups allows the researcher to control for extraneous variables.
Participants can serve as their own controls to control for participant differences.
Analysis of covariance (ANCOVA) is a statistical procedure that can be used to control for participant variables.
Experimental Research-Types of Group Designs
Single-variable designs
Factorial designs
Experimental Research-Types of Group Designs-Single-variable designs
Pre-experimental designs do not adequately control for extraneous variables and should be avoided.
True-experimental designs offer a very high degree of control and are always preferred designs.
Quasi-experimental designsdo not control as well as experimental designs but are preferable over pre-experimental designs
Experimental Research-Types of Group Designs-Factorial designs
any design that involves two or more independent variables, at least one that is manipulated.
Experimental Research-Pre-Experimental Designs
The one-shot case study involves a single group that is exposed to a treatment (X) and then posttested (O).
X O
Threats to validity are not adequately controlled with this design.
Do notuse this design
Experimental Research-Pre-Experimental Designs
The one-group pretest-posttest design involves a single group that is pretested, exposed to treatment, and then tested again.
O X O
The success of the treatment is determined by comparing pretest and posttest scores.
This design does not control for history, testing, instrumentation, regression, or maturation.
Statistical regression is not controlled nor is pretest-treatment interaction.
Experimental Research-Pre-Experimental Designs
The static-group comparison designinvolves at least two nonrandomly formed groups. One group receives an experimental treatment and the other group receives the traditional treatment. Both groups are posttested.
X1 O
X2 O
Experimental Research-Pre-Experimental Designs
The number of groups can be expanded beyond two.
The groups are better described as comparison, not experimental and control.
This design does not control for maturation, selection effects, selection interactions, and mortality.
There is some control for history in this design.
This design is sometimes used in exploratory studies.
Experimental Research-True Experimental Designs
The pretest-posttest control group design requires at least two groups.
Groups are formed by random assignment.
Both groups are administered a pretest, each group receives a different treatment and both groups are posttested.
The design may be extended to include additional groups.
Experimental Research-True Experimental Designs
R O X1 O
R O X2 O
R O X3 O
The combination of random assignment and the presence of a pretest and a control group serve to control for all threats to internal validity.
Experimental Research-True Experimental Designs
The only potential weakness in this design is a possible interaction between the pretest and the treatment.
Researchers should report assess and report the probability of a pretest-treatment interaction.
Experimental Research-True Experimental Designs
There are a few variations on the basic pretest-posttest control group design.
One variation includes random assignment of matched pairs to the treatment groups.
There is little advantage to this variation.
Another variation of this design involves one or more additional posttests.
R O X1 O O
R O X2 O O
Experimental Research-True Experimental Designs
The posttest-only control groupdesign is the same as the pretest-posttest control group design except that it lacks a pretest.
This design is often expanded to include more than two groups.
R X1 O
R X2 O
The posttest-only control group design is best used when there is likelihood of a pretest-treatment interaction threat.
As with the pretest-posttest control group design, the addition of a matched random assignment does not represent an increased advantage.
Experimental Research-True Experimental Designs
The Solomon Four-Group Designis a combination of the pretest-posttest control group design and the posttest-only control group design.
R O X1 O
R O X2 O
R X1 O
R X2 O
The analysis of the Solomon four-group design is a 2 x 2 factorial analysis of variance.
This analysis tests if those who received the treatment performed differently than those who did not.
This analysis can assess if there is a testing effect.
This analysis assesses for pretest-interaction effects.
Experimental Research-True Experimental Designs
The Solomon four-group design requires a large number of participants.
The Solomon four-group design may not always be the best design.
The design selected should be based upon potential threats and the nature of the proposed study.
Experimental Research-Quasi-Experimental Designs
When it is not possible to assign participants to groups randomly, researchers can use quasi-experimental studies.
In the nonequivalent control group design, two or more treatment groups are pretested, administered a treatment, and posttested.
O X1 O
O X2 O
Experimental Research-Quasi-Experimental Designs
The nonequivalent control group designinvolves the random assignment of groups not individuals.
The lack of random assignment introduces validity threats (e.g., regression, and selection interaction effects).
To reduce threats when using this design researchers often assure groups are as equivalent as possible (e.g., use ANCOVA).
Experimental Research-Quasi-Experimental Designs
The time-series designis an elaboration of the pretest-posttest design.
One group is repeatedly pretested until pretest scores are stable. The group is then exposed to a treatment and after treatment is repeatedly posttested.
Experimental Research-Quasi-Experimental Designs
O O O O X O O O O
A variation of the time-series design is the multiple time-series design that includes a control group.
This variation eliminates the history and instrumentation threats.
O O O O X1 O O O O
O O O O X2 O O O O
Experimental Research-Quasi-Experimental Designs
History is a threat with time-series designs.
Instrumentation may also be a threat if testing changes.
Pretest threats are problematic with time-series designs. However, it is relatively easy to establish the degree of the threat given data from repeated testing.
Experimental Research-Quasi-Experimental Designs
In a counterbalanced design, all groups receive all treatments but in a different order and all groups are posttested after each treatment.
Counterbalanced designs can include any number of groups.
The number of groups is equal to the number of treatments.
Treatment order is randomly assigned.
A unique threat with a counter-balanced design is multiple treatment interaction.
X1 O X2 O X3 O
X3 O X1 O X2 O
X2 O X3 O X1 O
Experimental Research-Factorial Designs
Factorial Designs are elaborations on single-variable experimental designs to permit investigation of two or more variables, at least one of which is manipulated by the researcher.
Factorial designs are often employed after an independent variable has first been investigated individually.
Experimental Research-Factorial Designs
The purpose of a factorial design is to determine whether the effects of an independent variable are generalizable across all levels.
One example of a factorial design is the 2 X 2 design.
Type of instruction (computer-based or paper and pencil) by gender.
Many factors (independent variables) studies are possible to address specific research questions.
Single-Subject Experimental Designs
designs are applied when the sample size is one or when a number of individuals are considered to be one group.
These designs are used to study behavior change in response to treatment.
Each participant serves as her own control.
Single-Subject Experimental Designs
Each participant is exposed to both treatment and control phases and is measured during each phase.
Single-subject designs are often used for research in special education, communication science disorders, and clinical psychology.
Applications of these designs, however, are appropriate to many additional areas of research.
Single-Subject Experimental Designs
-Design representation in single-subject research
The nontreatment phase is represented by A.
The treatment phase is represented by B.
Example:
if we were to study the in-class swearing behavior of Melissa, we could make several observations of her in class, recording her swearing behavior, introduce an intervention for several classes and observe her swearing behavior, stop the intervention and observe her for several classes. This would be represented as A-B-A.
Single-Subject Experimental Designs
When research results are intended to generalize to other groups, single subject designs are not appropriate.
Sometimes, when it may not be ethical to conduct a group design, a single-subject design is appropriate.
Single-subject designs are appropriate when the aim is to improve functioning of an individual.
Single-Subject Experimental Designs
Single-subject designs suffer from low external validity.
Replication is the key to generalizability in single-subject designs.
The effect of the baseline phase may pose an external validity threat in single-subject designs.
Similar to the pretest-treatment interaction threat in group designs.
Single-Subject Experimental Designs-
Internal validity
Repeated and reliable measurement
Baseline is established in single-subject designs through multiple observations.
This helps to control for maturation.
Data are also collected during the treatment phases in single-subject designs.
This helps to control for history.
Instrumentation is a threat to internal validity in single-subject designs.
Intra-observer reliability and inter-observer reliability are critical for effective single-subject studies.
Treatment should be explained in enough detail to permit replication.
Baseline stability
It is often difficult to know how many data points are necessary for baseline. This is a critical decision in single-subject designs.
Minimum of three data points are required but more are often necessary
Length of the treatment phase parallels baseline
Single-variable rule
Only one variable at a time should be
Single-Subject Experimental Designs-
Types of Single-Subject Designs
A-B-A withdrawl designs alter phases of baseline and treatment.
There are a number of variations of A-B-A designs.
A-B designs
OOOO XOXOXOXO
Baseline Phase Treatment Phase
A B
Single-Subject Experimental Designs-
Types of Single-Subject Designs
Additive designs
A-B-A Design
OOOO XOXOXOXO OOOO
Baseline Phase Treatment Phase Baseline Phase
A B A
Changing criterion design
Baseline is followed by successive treatment phases where in each a more stringent criterion is required.
Single-Subject Experimental Designs-
Types of Single-Subject Designs
A-B-A-B Design
OOOO XOXOXOXO OOOO XOXOXOXO
Baseline Treatment Baseline Treatment
A B A B
Single-Subject Experimental Designs-
Types of Single-Subject Designs
Multiple-Baseline Designs
Multiple-baseline designs entail the systematic addition of behaviors, subjects, or settings for intervention.
Multiple behaviors for one subject
One behavior for several subjects
One behavior and one subject for several settings
Multiple-baseline designs can often be used when it is not ethical to remove a treatment or to reverse a treatment.
Single-Subject Experimental Designs-
Data Analysis and Interpretation
Data analysis in single-subject research is typically based on visual inspection and analysis of a graphic representation of results.
First, the researcher evaluates the adequacy of the design.
Second, if the design is deemed valid, the researcher assesses treatment effectiveness.
Effectiveness is evaluated based upon clinical effectiveness not statistical effectiveness.
Although available, inferential statistical procedures are not often used in single-subject designs.
External validity in single-subject designs is established through replication.
Single-Subject Experimental Designs-
Data Analysis and Interpretation
Replication
Direct:Replication by the same investigator in a specific setting with the same or different participants.
Simultaneous:Replication with a number of participants with the same problem at the same location and same time.
Systematic:Follows direct replication with different investigators, behaviors, and settings.
Clinical replication involves the development of a treatment package composed of two or more interventions that have been found to be individually successful.
Descriptive Statistics-definition
Statisticsis a set of procedures for describing, synthesizing, analyzing, and interpreting quantitative data.
The mean is an example of a statistic.
Descriptive Statistics-Calculated
One can calculate statistics by hand or can use the assistance of statistical programs.
Excel, SPSS, and many other programs exist. Some programs are also available on the Web to analyze datasets.
Descriptive Statistics-Preparing Data for Analysis
After data are collected, the first step toward analysis involves converting behavioral responses into a numerical system or categorical organization.
It is critical that all data are scored accurately and consistently.
Data scoring should be double-checked for consistency and accuracy (i.e., at least 25% of all cases should be checked).
Open-ended items should be scored by two scorers to check reliability.
All data scoring and coding procedures should be documented and reported in the written report.
Descriptive Statistics-Preparing Data for Analysis
After instruments are scored, the resulting data are tabulated and entered into a spreadsheet.
Tabulation involves organizing the data systematically (e.g., by participant).
In this potential dataset,
ID represents participant number
Cond. is the experimental condition (1 or 2)
Gender is represented by female=1; male=2
Achievement (Ach) and motivation (Mot) were also variables assessed
Descriptive Statistics-Types of Descriptive Statistics
After data are tabulated and entered, the next step is to conduct descriptive statistics to summarize data.
In some studies, only descriptive statistics will be conducted.
If the indices are calculated for a sample, they are referred to as statistics.
If indices are calculated for the entire population, they are referred to as parameters.
Descriptive Statistics-Types of Descriptive
Frequencies
The frequency refers to the number of times something occurs.
Frequencies are often used to describe categorical data.
We might want to have frequency counts of how many males and females were in a study or how many participants were in each condition.
Frequency counts are not as helpful in describing interval and ratio data.
Descriptive Statistics-Measures of Central Tendency
Measures of central tendency are indices that represent a typical score among a group of scores.
Measures of central tendency provide a way to describe a dataset with a single number.
Descriptive Statistics-Measures of Central Tendency- three most common
Mean: Appropriate for describing interval or ratio data
Median: Appropriate for describing ordinal data
Mode: Appropriate for describing nominal data
Descriptive Statistics-Measures of Central Tendency-Mean
The mean is the most commonly used
measure of central tendency.
The formula for the mean is:
X= ΣX/n
To calculate the mean, all the scores
are summed and then divided by the
number of students.
Descriptive Statistics-Measures of Central Tendency-Median
The median is the midpoint in a distribution: 50% of the scores are above the median and 50% of the scores are below the median.
To determine the median, all scores are listed in order of value.
If the total number of scores is odd, the median is the middle score.
If the total number of scores is even, the median is halfway between the two middle scores.
Median values are useful when there is large variance in a distribution.
Descriptive Statistics-Measures of Central Tendency-Mode
The mode is the most frequently occurring score in a distribution.
The mode is established by looking at a set of scores or at a graph of scores and determining which score occurs most frequently.
The mode is of limited value.
Some distributions have more than one mode (e.g., bi-modal, or multi-modal distributions)
Descriptive Statistics-Deciding among measures of central tendency
Generally the mean is most preferred.
The mean takes all scores into account.
The mean, however, is greatly influenced by extreme scores.
When there are extreme scores present in a distribution, the median is a better measure of central tendency.
Descriptive Statistics-Measures of Variability
Measures of variability provide an index of the degree of spread in a distribution of scores.
Measures of variability are critical to examine and report because some distributions may be very different but yet still have the same mean or median.
Descriptive Statistics-Measures of Variability-Three common measures
range, quartile deviation, and standard deviation.
Descriptive Statistics-Measures of Variability-Three common measures-Range
The difference between the highest and lowest score.
The range is not a stable measure.
The range is quickly determined.
Descriptive Statistics-Measures of Variability-Three common measures-Quartile Deviation
One half the difference between the upper quartile and the lower quartile in a distribution.
By subtracting the cutoff point for the lower quartile from the cutoff point for the upper quartile and then dividing by two we obtain a measure of variability.
A small number indicates little variability and illustrates that the scores are close together.
Descriptive Statistics-Measures of Variability-Three common measures-Variance
The amount of spread among scores. If the variance is small the scores are close together. If the variance is large the scores are spread out.
Calculation of the variance shows how far each score is from the mean.
The formula for the variance is:
Σ(X-X)2/n
Descriptive Statistics-Measures of Variability-Three common measures-Standard deviation
The score root of the variance.
The standard deviation is used with interval and ratio data.
The standard deviation is the most commonly used measure of variability.
If the mean and the standard deviation are known, the distribution can be described fairly well.
SDrepresents the standard deviation of a sample and the symbol (i.e., the Greek lower case sigma) represents the standard deviation of the population.
Descriptive Statistics-The Normal Curve
If a variable is normally distributed then several things are true about the distribution of the variable.
Fifty percent of the scores are above the mean and 50% are below the mean.
The mean, median, and mode have the same value.
Most scores are near the mean.
34.13% of the scores fall between the mean and one standard deviation above the mean and 34.13% of scores fall below the mean and one standard deviation below the mean.
That is, 68.26% of the scores fall within one standard deviation of the mean.
More than 99% of the scores fall within three standard deviations above and below the mean.
Descriptive Statistics-The Normal Curve
-Skewed distributions
When a distribution is not normally distributed, it is said to be skewed.
A skewed distribution is not symmetrical.
The mean, median, and mode are not the same value.
The farther apart the mean and the median, the more skewed the distribution.
A negatively skewed distribution has extreme scores at the lower end of the distribution.
Mean<Median<Mode
A positively skewed distribution has extreme scores at the higher end of the distribution.
Mean>Median>Mode
Descriptive Statistics-Measures of Relative Position
Measures of relative position indicate where a score falls in the distribution relative to all the other scores.
Measures of relative position indicate how well an individual has scored in comparison to others in the distribution.
Measures of relative position express different scores on a common scale.
Descriptive Statistics-Measures of Relative Position-most frequently used measures of relative position
percentile ranks
standard scores.
Descriptive Statistics-Measures of Relative Position-percentile ranks
Percentile ranks indicate the percentage of scores that fall at or below a given score.
Percentile ranks are appropriate for ordinal data and are also used for interval data.
A percentile rank of 50 indicates the median score.
Descriptive Statistics-Measures of Relative Position-standard scores
A standard score uses standard deviation units to express how far an individual student's test score is from the mean.
i.e., a standard score reports how many standard deviations a given score is from the mean of a distribution.
Standard scores allow scores from different tests to be compared on a common scale and therefore allow for calculations on those data.
.
Descriptive Statistics-Measures of Relative Position-standard scores-Z score
A z-score is the most basic and most often used standard score.
A z-score is directly tied to the standard deviation.
A score that represents the mean has a z-score of 0.
A score at 1 standard deviation above the mean has a z-score of 1
To convert a raw score to a z-score we use the following formula where Xis the raw score.
Z =X-X
SD
The characteristics of the normal distribution can be used to approximate where a score falls based upon a standard score.
Descriptive Statistics-Measures of Relative Position-standard scores-T-score
A T-score is a standard score sometimes used instead of a z-score.
A T-score is calculated by multiplying a z-score by 10 and adding 50.
T-scores transform the scores such that there are not negative values.
Descriptive Statistics-Measures of Relationship
Measures of relationship indicate the degree to which two sets of scores are related.
There are several statistical measures of relationship.
The nature of a given dataset dictates which measure is used.
Descriptive Statistics-Measures of Relationship
Pearson ris statistic used to calculate relationship for interval or ratio data.
Pearson r takes into account every score.
Pearson ris the most stable measure of relationship.
Spearman rho is used to calculate relationship with ordinal data.
There are other tests of relationship for ordinal data (e.g., Gamma, Kendall's tau).
Spearman rho is most popular.
Descriptive Statistics-Graphing Data
a critical step in analyzing any dataset.
All variables should be graphed.
Graphing helps researchers to know their data.
Statistical packages make graphing data an easy task.
Graphing helps to identify any errors in a dataset.
Inferential Statistics
data analysis techniques for determining how likely it is that results obtained from a sample, or samples, are the same results that would be obtained from the entire population.
Descriptive statistics show how often or how frequent an event or score occurred.
Inferential statistics help researchers know whether they can generalize their findings to a population based upon their sample of participants.
Inferential statistics use data to assess likelihood—or probability.
Inferential Statistics
Inferences about populations are based on information from samples.
There is very little chance that any sample is identical to the population.
The expected variance among sample means and the population mean is referred to as sampling error.
Sampling error is expected.
Sampling error tends to be normally distributed.
Inferential Statistics-Standard Error
A distribution of sample means is normally distributed and has it's own mean and standard deviation.
The standard deviation of the sample means is referred to as the standard error of the mean.
Our ability to estimate standard error of the mean is affected by size of sample.
As the sample size increases the standard error of the mean decreases.
Inferential Statistics-Standard Error
Our ability to estimate the standard error of the mean is also affected by the size of the population standard deviation.
The standard error of the mean can be calculated by:)=(SE)= SD
X N-1
SEx= the standard error of the mean
SD= the standard deviation for a sample
N= the sample size
Inferential Statistics-Hypothesis Testing
Hypothesis testing is the process of decision making in which researchers evaluate the results of a study against their original expectations.
Research hypothesis: Predicting a difference in scores
Null hypothesis: Predicting no difference in scores
Inferential Statistics-Hypothesis Testing
We want to assure differences we observe between groups are 'real' differences and did not occur by chance.
If the groups are significantly different we reject the null hypothesis.
We do not accept a research hypothesis, we can't prove our hypothesis.
We instead report that our research hypothesis was supported.
If there are not expected differences, we report that the null hypothesis was not rejected; and that our research hypothesis was not supported.
Inferential Statistics-Tests of Significance
allow us to inferentially test if differences between scores in our sample are simply due to chance or if they are representative of the true state of affairs in the population.
Inferential Statistics-Tests of Significance-conduct a test of significance
determine a preselected probability level, known as level of significance (alphaor α).
Usually educational researchers use alpha.05 or 5 out of 100 chances that the observed difference occurred by chance (α=.05).
Inferential Statistics-Tests of Significance-Two-tailed and one-tailed tests
Tests of significance are almost always two-tailed.
Researchers will select a one-tailed test of significance only when they are quite certain that a difference will occur in only one direction.
It is 'easier' to obtain a significant effect when predicting in one direction.
Inferential Statistics-Tests of Significance-Type I and Type II Errors
Based upon a test of significance the researcher will either reject or not reject the null hypothesis.
The researcher makes a decision that the observed effect is or is not due to chance.
This decision is based upon probability, not certainty.
Sometimes the researcher will erroneously reject the null hypothesis or will erroneously retain the null hypothesis.
Inferential Statistics-Tests of Significance-Type I Errors
When the researcher incorrectly rejects the null hypothesis she has committed a Type Ierror.
Inferential Statistics-Tests of Significance- Type II Errors
When the researcher incorrectly fails to reject the null hypothesis but a true difference exists, she has committed a Type II error.
Inferential Statistics-Tests of Significance-Degrees of Freedom
After determining whether the significance test will be two-tailed or one-tailed and selecting a probability level (i.e., alpha), the researcher selects an appropriate statistical test to conduct the analysis.
Degrees of freedom are the number of observations free to vary around a parameter.
Each test of significance has its own formula for determining degrees of freedom (df ).
The value for the df is important in determining whether the results are statistically significant.
Inferential Statistics-Tests of Significance- Choosing which to use
The use of a specific significance test is determined by several factors.
Scale of measurement represented by the data
Participant selection
Number of groups being compared
Number of independent variables
Significance tests applied incorrectly can lead to incorrect decisions.
The first decision in selecting an appropriate test is to determine whether a parametric or nonparametric test will be used.
Parametric tests are generally more powerful and are generally preferred.
Inferential Statistics-Tests of Significance- Parametric tests
require that the data meet several assumptions.
Variable must be normally distributed
Interval or ratio scale of measurement
Selection of participants is independent
Variance of the comparison groups is equal
Most parametric tests are fairly robust.
If assumptions are violated, nonparametric tests should be used.
Inferential Statistics-Tests of Significance- t test
The ttest is used to determine whether two groups of scores are significantly different from one another.
The ttest compares the observed difference between means with the difference expected by chance.
Inferential Statistics-Tests of Significance- t test for independent samples
The ttest for independent samples is a parametric test of significance used to determine if differences exist between the means of two independent samples.
Independent samples are randomly formed.
The assumption is that the means are the same at the outset of the study but there may be differences between the groups after treatment.
Inferential Statistics-Tests of Significance- t test for nonindependent samples
The ttest for nonindependent samples is a parametric test of significance used to determine if differences exist between the means of two groups that are formed through matching.
When scores are nonindependent, they are systematically related.
Inferential Statistics-Tests of Significance- t test
The comparison of gain or difference scores are not generally tested with a t-test.
There are other better strategies for analyzing such data.
e.g., ttest on posttest scores (if there are no differences on pretest scores).
e.g., Analysis of covariance (if there are differences in pretest scores).
Inferential Statistics-Tests of Significance- Analysis of Variance
A simple (one-way) analysis of variance (ANOVA) is a parametric test used to determine whether scores from two or more groups are significantly different at a selected probability level.
ANOVA is used to avoid the error rate problems of conducting multiple tests.
Inferential Statistics-Tests of Significance- Analysis of Variance- F ratio
An F ratio is computed to determine if sample means are significantly different.
The F ratio is calculated based upon variance between groups/variance within groups.
The larger the F-ratio the more likely there are differences among groups.
Inferential Statistics-Tests of Significance- Analysis of Variance
If there are significant differences among groups based upon an ANOVA; the researcher then must determine where these differences exist.
Multiple comparisons are used to determine where differences between groups are statistically significant.
Comparisons planned before collecting data are referred to as a priori.
Comparisons after are referred to as post hoc.
When a factorial design is used and there are two or more independent variables analyzed, a factorial or multifactor analysis of variance is used to analyze the data.
MANOVA is an analytic procedure used when there is more than one dependent variable and multiple independent variables.
Inferential Statistics-Tests of Significance- Analysis of Variance
ANCOVA is a form of ANOVA that allows for control of extraneous variables and also is used as a means for increasing power of an analysis.
Power is increased in an ANCOVA because the within-group error variance is decreased.
When a study has two or more dependent variables, and a covariate, MANCOVA is used.
Inferential Statistics-Tests of Significance- Analysis of Variance-Multiple Regression
Multiple regression is used to determine the amount of variance accounted for in a dependent variable by interval and ratio level independent variables.
Multiple regression combines variables that are known to predict the criterion variable into an equation.
Stepwise regression allows the researcher to enter one variable at a time.
Multiple regression is also the basis for path analysis.
Path analysis begins with a predictive model.
Path analysis determines the degree to which predictor variables interact with each other and contribute to variance in the dependent variables.
Inferential Statistics-Tests of Significance- Chi Square
Chi Square is a nonparametric test used to test differences between groups when the data are frequency counts or percentages or proportions converted into frequencies.
A true category is one in which persons naturally fall.
An artificial category is one that is defined by the researcher.
Inferential Statistics-Other Statistical Procedures
Data mining uses analytical tools to identify and predict patterns in datasets.
Factor analysis is a statistical procedure used to identify relations among variables in a correlation matrix.
Factor analysis is often used to reduce instruments to scales or subscales.
Structural Equation Modeling (SEM)
Structural equation modeling is similar to a combination of path analysis and factor analysis.
SEM is a powerful analytic tool.
Sampling-What happens if I don't have a good sample?
If one doesn't have a good sample, then the results will not be generalizable and therefore will have less importance.
Sampling-What are some examples of sampling techniques for quantitative research?
Random sampling techniques might include simple random sampling, systematic sampling and cluster sampling. Examples of non-random sampling include convenience sampling, quota sampling, and snowball sampling.
Sampling-Why is sampling important in quantitative research?
Because one of the purposes of quantitative research is to generalize the results obtained from a small sample to its larger population, you want to make sure that your sample represents the population as much as possible.
Quantitative Sampling
-Sampling: The process of selecting a number of participants for a study.
-Sample: A sample is made up of individuals, events, or items, selected from the population.
-a sample is well-selected the results will be generalizableto the population
Quantitative Sampling-Defining a Population
Populations may be any size and cover large geographical distances.
The entire population is rarely available.
Quantitative Sampling-Target population
Population to which the researcher would ideally like to generalize results.
Quantitative Sampling-Accessible population (Available population)
Population from which researcher can realistically select participants.
Quantitative Sampling-Target & Accessible Populations Example
Consider that you are interested in high school juniors' perceptions of admission criteria for Ivy League schools. Your target population would be high school juniors in the United States but the accessible population would be much different. Although you could conduct the study with the target population, the study would require extensive resources and as such, an accessible population would likely be studied instead.
Quantitative Sampling-Selecting a Random Sample-Probability Sampling
techniques permit the researcher to specify the probability that each member of the population has in being selected for the sample.
Simple Random Sampling
Stratified Sampling
Cluster Sampling
Systematic Sampling
Quantitative Sampling-Selecting a Random Sample-Probability Sampling
All individuals in the defined population have an equal and independent chance of being selected for the sample.
Random sampling is the best way to possibly obtain a representative sample.
Quantitative Sampling-Steps in Simple Random Sampling
1.Identify and define the population.
2.Determine the desired sample size.
3.List all members of the population.
4.Assign all individuals a consecutive number from zero to the required number.
5.Select an arbitrary number in the table of random numbers.
6.For the selected number look at only the necessary number of digits.
7.The number that is selected represents that person in the population.
8.Go to the next number in the column and repeat steps 6 and 7.
Quantitative Sampling-Random Sampling Example-Steps 1-3
We are interested in selecting a random sample of participants for a study of students' nutrition.
1.Identify and define the population:
The population is middle school children in the school district in which we live.
Middle schools in our district include children in sixth through eighth grade.
There are 820 middle school children in our district.
2.Determine desired sample size.
We would like to sample 325 students.
3.List all members of the population.
We generate a comprehensive list of all the middle school students.
Quantitative Sampling-Random Sampling Example-Steps 4-6
4.Each member of the population is given a number.
We give each student a sequential number.
5.We consult a table of random numbers and pick an arbitrary number.
The text has a table of random numbers.
Statistical packages can generate a table.
Websites are available with tables of random numbers.
6.For the arbitrary number we have selected, we look at the last 3 digits because our needed sample contains 3 digits.
Quantitative Sampling-Random Sampling Example-Steps 7-8
7.We include in our sample the person with the number corresponding to the number we have selected.
If the number selected is greater than 325 we ignore it and go to the next number.
8.We continue to go through steps 6 and 7 until we have selected our entire sample of participants.
Quantitative Sampling-Stratified Sampling
-guarantees desired representation of relevant subgroups within the sample.
Populations can be divided into subgroups (strata).
-should be used when a goal is to compare behavior of participants from subgroups.
e.g., In our nutrition study we may be interested in gender or SES or ethnicity or other potential subgroups.
Quantitative Sampling-Stratified Sampling-Proportional stratified sampling
Identified subgroups are represented in the sample in the same proportion as in the population.
e.g., If 28% of a population of advanced science students are girls then the sample should be represented by 28% girls.
e.g., In our nutrition study we may be interested in representation by SES.
-Stratified sampling can be used to select equal-sized, nonproportional, samples from subgroups when subgroup comparisons are important.
e.g., gender, race, ability level, SES.
Quantitative Sampling-Stratified Sampling for Equal-Sized Groups 1-3
1.Identify and define the population.
Our 820 middle school students are our population.
2.Determine desired sample size.
We would like to sample 100.
3.Identify the variable and subgroups (strata).
We would like an equal number of students representing athletes and non-athletes.
Quantitative Sampling-Stratified Sampling for Equal-Sized Groups 4-5
4.Classify all members of the population as one of your identified groups.
We identify each student as either an athlete or non-athlete.
5.Randomly select an equal number of participants from each subgroup.
We use the random number table to select 50 from each group.
Quantitative Sampling-Proportional Stratified Sampling
We can use this same procedure to select a proportional stratified sample.
If for example our population includes 12% athletes we may not want equal groups but instead may desire that our sample is 12% athletes.
After we classify students into our subgroups we simply randomly select the appropriate percentage from each subgroup.
Quantitative Sampling-Cluster Sampling
-intact groups but not individuals, are randomly selected.
e.g., An example cluster could be schools, classrooms, communities, or counties.
-may be used when it is not possible to list all members of a population or when the population is spread out or large.
Quantitative Sampling-Cluster Sampling
-may be more convenient and may take less time than random or stratified sampling.
-found in many studies when it is hard to sample individuals within a group.
Quantitative Sampling-Cluster Sampling drawbacks
Cluster sampling is not representative.
Often-used statistical methods are not appropriate for data from studies that employed cluster sampling.
Quantitative Sampling-Cluster Sampling Example Steps 1-3
1.Identify and define population.
Middle school students in our state are our population.
2.Determine desired sample size.
We would like to sample 1500 students.
3.Identify and define a logical cluster
Schools are a logical cluster.
Quantitative Sampling-Cluster Sampling Example Steps 4-6
4.List all clusters
We obtain a list of all the middle schools in the State.
5.Estimate average number of participants per cluster.
•In our State the average number of students in each middle school is 250.
6.Determine number of clusters needed
•If our desired sample is 1500 we need to randomly select 6 schools.
Quantitative Sampling-Cluster Sampling Example Steps 7-8
7.Randomly select the needed number of clusters.
•We randomly select the six schools.
8.Include in your study all the participants in each selected cluster.
•We include all the students in each of the six schools in our study.
Quantitative Sampling-Systematic Sampling
-we select every Kthindividual from a list.
-The list includes all the members of a population.
-K is determined by dividing the number of individuals by the number of participants desired in a sample.
-Randomly ordered lists can be used in systematic sampling.
Quantitative Sampling-Systematic Sampling Example Steps 1-3
1.Identify and define population.
Middle school students in our school are our population.
2.Determine desired sample size.
We would like to sample 50 students.
3.We list all students in the school
We might randomly order the list.
Avoid bias that may be introduced by other listing procedures.
Quantitative Sampling-Systematic Sampling Example Steps 4-7
4.We divide the number of students in our population by the number of participants that we need in our sample.
e.g., 350 in the school/50 students therefore K=7
5.We start in a random place on the list.
6.We take every Kth (in this case 7th) name on the list until we have selected our 50 participants.
7.If we come to the end of the list then we simply go to the top and continue using our strategy of every Kthname.
Quantitative Sampling-Determining Sample Size
It is not easy to determine how large a sample should be.
If the sample is too small it will limit generalizability.
If a population is very large, you need less percentage of the population for your sample to be representative.
If the population is small, sample the entire population.
If the sample is very large, 400 participants can be sampled.
For populations between 100 and 5000, sample a percentage of the population.
Generally, sample as much of a population as you are able to sample.
Quantitative Sampling-Sampling Error and Sampling Bias
Sampling Error is chance variation that occurs when a sample does not represent the population.
Sampling Bias is systematic sampling error
e.g., In a study of principals' opinions of effective leadership styles, sampling bias may occur if you sampled principals at a leadership conference.
Quantitative Sampling-Selecting a Non-Random Sample
NonProbability sampling or Nonrandom sampling: At times researchers can not use random sampling.
When nonrandom sampling is used it often becomes challenging to describe the sample.
Bias may also be a concern in nonrandom sampling.
Quantitative Sampling-Types of Non-Random Sampling
Convenience Sampling
Purposive
Quota
Quantitative Sampling-Types of Non-Random Sampling- Convenience Sampling
Convenience sampling is also called haphazard or accidental sampling.
Convenience sampling selects whoever is available.
Convenience sampling introduces bias. Those available may not be representative
e.g., In a study of community members' opinions about public transportation, depending upon where you sampled people from they might have very different opinions. If you sampled people at a bus stop, for example, you might not get representative data.
Quantitative Sampling-Types of Non-Random Sampling- Purposive Sampling
Purposive sampling is also called judgment sampling.
Purposive sampling entails selecting a sample believed to be representative.
Purposive sampling introduces bias because the researcher may be incorrect in her belief that the sample is representative.
e.g., If a researcher wanted to study differences between expert and novice teachers and selected participants based upon judgments regarding which teachers were deemed expert.
Quantitative Sampling-Types of Non-Random Sampling-Quota Sampling
Quota sampling occurs when the researcher selects a sample based on a required number of individuals with particular characteristics.
Quota sampling is often used in wide-scale survey research.
Bias is introduced in several ways when quota sampling is used, including through accessibility.
e.g., A researcher may need to have 300 men ages 40-50 in a study. He would sample until he had obtained the 300 men.
Qualitative Sampling
In qualitative sampling a small number of participants are selected that will be good informants.
Good informants are thoughtful and reflective.
Good informants communicate effectively.
Good informants are comfortable with the researcher and within the research site.
Qualitative Sampling
Qualitative samples are smaller and less representative.
Generalizability and representativeness are not the same concern in qualitative research as they are in quantitative research.
Sample size in qualitative research is determined by the following:
The range of participants available and the degree to which range in participants is consistent with what is found in the population.
Qualitative Sampling-data saturation
Participants may be sampled until data saturation. Data saturation occurs when additional participants are not providing new information.
Qualitative Sampling-Homogeneous sampling
occurs when the researcher selects participants who represent a similar experience.
e.g., A researcher might select only middle class women whose mother's stayed at home during the school day.
Qualitative Sampling-Intensity sampling
occurs when the researcher selects participants who represent different levels of the research topic.
e.g., A researcher might select some new administrators and some more experienced ones for his study
Qualitative Sampling-Criterion sampling
occurs when the researcher selects cases that fit a certain criteria.
e.g., A researcher may select only those participants who are first generation law school students.
Qualitative Sampling-Snowball sampling
occurs when the researcher selects new participants based upon other participant's suggestions regarding who else might be studied.
e.g., A researcher might be interested in high school students who are not in school activities but are involved in out-of-school activities. Once she has identified some participants she may ask them if they know of others who may fit her criteria.
Qualitative Sampling-Random-purposive sampling
Random-purposive sampling occurs when the researcher generates a larger than needed potential sample and randomly selects from that sample.
e.g., A researcher might purposively sample teachers within a specific district and then randomly select 3 teachers to represent the sample.
Qualitative Sampling-Types
-Homogeneous sampling
-Intensity sampling
-Criterion sampling
-Snowball sampling
-Random-purposive sampling
Qualitative Research-What are the benefits of qualitative research?
Qualitative research allows you to look at data in context and draw from it more meaningful findings.
Qualitative Research-What are the characteristics of qualitative research?
Qualitative research studies are conducted through sustained in-depth, in-context, research and seek to provide rich understanding to research questions.
Qualitative Research-What are types of qualitative research?
-historical research
-case study
-grounded theory
-ethnology
-phenomenology
Qualitative Research-Characteristics
1. A total or complete picture is sought.
2. Seeks to understand people's interpretations.
3. Data are perceptions of the people in the environment.
4. Investigations are conducted under natural conditions.
5. The focus is on design and procedures to gain "real," "rich," and "deep" data
6. Small sample sizes
7. No controls for samples
8. Inductive
9. Useful in studying cultures, historical data, and studying complex issues.
10. Narrative results rather than numerical.
Quantitative research-Characteristics
1. Mathematical or statistical models are used to analyze the data.
2. It is objective and statistical.
3. Variables can be manipulated.
4. Uses large samples for more accurate data.
5. Researcher has little or no interaction with participants.
6. Shows cause and effect relationships
7. Deductive
8. Begins with a hyposthesis tha can be proven or disproved
Qualitative data collection - fieldwork.
Fieldwork includes materials gathered, recorded, and compiled during the study.
Fieldwork requires the researcher to immerse himself in the setting over time.
The researcher collects as much data she can as unobtrusively as possible.
Qualitative research-What are the main data collection methods for qualitative research?
Qualitative data collection is referred to as fieldwork.
Qualitative data is narrative and visual.
Observation
Interviews
Questionnaires
Examining Records
Quantitative research-Observation
The researcher obtains data by watching participants.
Observational data is often less subject to participant bias.
The researcher attempts not to change the setting
Types:
Participant observation
Nonparticipant observation
Quantitative research-Participant Observation
The researcher becomes part of and a participant in the situation being observed.
The researcher participates while observing and collecting data.
Quantitative research-Nonparticipant observation
The researcher is not directly part of the situation being observed.
The researcher observes and records but does not interact with the participants.
Nonparticipant observation is a less intrusive form of observation.
Quantitative research-Recording Observations 1
Field notes include descriptive information about what the observer has directly seen and heard on site.
Field notes also include reflective information that captures an observer's personal reactions and thoughts related to the observations.
The researcher avoids evaluative terms in field notes but instead describes behaviors.
Observational protocols are often used.
Protocols provide the researcher with a focus during the observation.
Protocols also provide a framework for the field notes.
Quantitative research-Recording Observations 2
Start slowly. Do not assume that you know what you are looking for until you have experience in the setting and have spent time with the participants.
Try to enter the field with no preconceptions. Recognize and dismiss your assumptions and remain open.
Write your field notes as soon as possible. Don't discuss the observation until you have written field notes.
Quantitative research-Recording Observations 3
Include the date, site, and time on notes. Use large margins and write impressions in the margins. Draw diagrams.
List key words related to the observation and outline what you saw and heard. Use the keywords and the outline to write your notes.
Keep the descriptive and reflective parts of your field notes distinct.
Write down your hunches, questions, and insights after each observation.
Number the lines or paragraphs of your field notes to help you find sections when needed.
Enter your field notes into a computer program for later examination and analysis.
Quantitative research-Interviews
Interviews are purposeful interactions in which one person obtains information from another person.
Interviews allow for data not available through observation alone.
Interviews may be formal and planned or informal and unplanned.
Quantitative research-Interview Types
Unstructured
Structured
Quantitative research-Unstructured
Interview
Unstructured interviews are similar to conversations.
Unstructured interviews are commonly used to gain more personal information.
Quantitative research-Structured
Interview
include predetermined questions.
Phrasing structured interviews can be challenging.
Include both open-ended and closed questions.
Pilot test the questions.
Quantitative research-Guidelines for Interviewing
Listening is the most important part of interviewing.
Don't interrupt. Wait.
Tolerate silence. The participant may be thinking.
Avoid leading questions.
Keep participants focused and ask for details.
Follow-up on what participants say and ask questions when you don't understand.
Don't be judgmental about participants' views or beliefs.
Don't debate with participants.
Quantitative research-Questionnaires
Interviews are time consuming.
Some researchers use questionnaires and then follow-up questionnaires with interviews.
Questionnaires allow for larger amounts of data collection.
The nature of the data collected with questionnaires is different than data from observations.
Quantitative research-Questionnaire Guidelines
Make questionnaire attractive.
Carefully proofread questionnaires.
Avoid lengthy questionnaires.
Do not ask unnecessary questions.
Use structured items with a variety of responses
Include a section that allows respondents to include 'other comments'.
Determine if respondents' identities are necessary and if so, develop a mechanism to track respondents.
Quantitative research-Examining Records
Qualitative researchers use a variety of available documents.
Archival documents
Journals
Maps
Videotape and audiotape
Artifacts
Quantitative research-Validity and Reliability
Validity in qualitative research addresses whether the data accurately measures what it was intended to measure.
Trustworthiness and understanding are terms used to describe validity in qualitative research.
Trustworthiness can be established by:
Credibility:The report addresses problems that are not easily explained.
Transferability:The description provided is such that others can identify with the setting.
Dependability:The stability of the data is addressed.
Confirmability:The neutrality and objectivity of the data are apparent.
Quantitative research-Validity
Criteria for qualitative research validity.
Descriptive validity: factual accuracy of the account
Interpretive validity:researcher accurately interprets participants' behaviors and actions
Theoretical validity: how well the report relates to broader theory
Evaluative validity: whether the report was created without researcher's judgment
Generalizability (Internal and External): the degree to which research is generalizable within and outside the setting
Quantitative research-Validity
Strategies for ensuring the validity of qualitative research
Prolong participation at the study site
Persistently observe
Use peer debriefing
Collect additional artifacts
Conduct member checks
Establish structural corroboration or coherence
Establish referential adequacy
Collect detailed descriptive data
Develop detailed descriptions of the context
Establish an audit trail
Practice triangulation
Practice reflexivity
Quantitative research-Validity
Practical options to assure trustworthiness
Talk little; listen a lot
Record observations accurately
Begin writing early
Let readers 'see' for themselves
Report fully
Be candid
Seek feedback
Write accurately
Quantitative research-Reliability
Reliability
Qualitative researchers address reliability by examining the techniques they are using to collect data.
Generalizability is less a concern for qualitative researchers than it is for quantitative researchers. Qualitative researchers are more concerned with relevance.
Narrative Research
Narrative research in education has been influenced by several factors.
The increased emphasis in teacher reflection and action research
The increased emphasis on teacher knowledge
The increased emphasis on empowering teachers
Narrative Research- Analysis
Narrative analysis (Current focus)
Researcher collects descriptions of events through interviews and observations and synthesizes them into narratives or stories.
Story is the outcome of the study.
Analysis of narrative
Researcher collects stories as data and seeks to understand underlying themes from the stories.
Narrative Research- Process Considerations
The narrative research process is highly personal and requires care and sensitivity.
The narrative researcher must be careful with ethical guidelines as narrative research requires a very close relationship between the researcher and the participant.
Equality of voice is required so that the participant feels empowered to tell the story.
The narrative researcher must determine if she has adequate time and the necessary personal characteristics to conduct this form of research.
Narrative Research- Process
Identify purpose of the research study and a phenomenon to explore.
Identify an individual who can help you learn about the phenomenon.
Develop initial narrative research questions.
Consider the researcher's role.
Develop data collection methods, paying particular attention to interviewing, and collect the data.
Collaborate with the research participant to construct the narrative and to validate the accuracy of the story.
Write the narrative account.
Narrative Research- Types
Creswell (1995) suggests that the numerous types of narrative research can be categorized based upon several characteristics.
Who authored the account
Scope of the narrative
Who provides the story
Theoretical/conceptual framework
Whether the elements are included in one narrative
Narrative Research-Examples
Autobiographies
Life writing
Personal narratives
Life histories
Narrative interviews
Narrative Research-Characteristics
A focus on the experiences of individuals.
A concern with the chronology of individuals' experiences.
A focus on the construction of life stories based on data collected through interviews.
Restorying as a teachnique for constructing the narrative account.
Inclusion of context and place in the story.
A collaborative approach that involves the researcher and the participants in the negotiation of the final text.
A narrative constructed around the question, "And then what happened?".
Narrative Research-Data Collection Techniques
Restorying
Oral history
Photographs, memory boxes and other artifacts
Storytelling
Letter writing
Autobiographical and biographical writing
Ethnographic Research
-study of cultural patterns and perspectives of participants in their natural settings.
-engage in long-term study of particular phenomena to situate understandings in context.
-engage in intensive participant observation.
Ethnographic Research- Goal
describe, analyze, and interpret the culture of a group, over time, in terms of the group's shared beliefs, behaviors, and language.
Ethnographic Research -Process
-first determine that he or she has the time, access, experience, personal style, and commitment to undertake an ethnography.
-identify the purpose of the research.
-demonstrates the relevance of the proposed study.
-next decides on the site and sample for the study.
-negotiate and gain entry into the research site.
-attend carefully to confidentiality and anonymity.
-establish rapport with the collaborators.
-begins data collection.
-last steps in the ethnographic research process include analysis and interpretation of the data and writing the ethnographic report.
Ethnographic Research -Characteristics
-natural setting
-face-to-face interaction
-accurate reflection
-inductive, interactive, and repetitious collection of unstructured data and analytic strategies.
-fieldwork data collection
-multiple methods for data collection, including conducting interviews and observations and reviewing documents, artifacts, and visual materials.
-sociopolitical and historical context
-small number of cases
Ethnographic Research -Types
Critical ethnography (political)
Realist ethnography
Ethnographic case study
Ethnographic Research -Techniques
Triangulation
Participant observation
Case Study Research
-all-encompassing method that includes specific design, data collection, and data analysis.
-also refers to the product of the research.
-yields concrete knowledge that readers personally relate to their own experiences.
-qualitative
-appropriate when the researcher wants to answer a descriptive or an explanatory question.
Case Study Research-Characteristics
-focus on a particular phenomenon.
-include thick description of the case and include many variables and analyses of their interactions.
-provide new insights to the researcher and the participants.
-
Case Study Research-Designing
Determine the research questions.
Define the case under study.
Determine the role of theory development in case selection.
Determine the theoretical and conceptual framework of the case study.
Determine whether a single case study; a multiple case study; or a collective case study is appropriate.
Case Study Research-sampling
Case study research generally employs purposive sampling.
Cases are selected because they can provide information regarding the research problem.
Screening is used to determine if a potential participant has the necessary experience and knowledge of the phenomenon under investigation.
Case Study Research-Data Collection
Case study researchers use the same data collection techniques as do other qualitative researchers.
The data collection techniques are selected to answer 'how' and 'why' questions.
Action Research-Characteristics
Action research is persuasive and authoritative.
Action research is relevant.
Action research is accessible.
Action research challenges intractability of reform of the educational system.
Action research is not a fad.
Action Research-Types
Democratic: Enables participation of all people
Equitable: Acknowledges people's equality of worth
Liberating: Provides freedom from oppressive, debilitating conditions
Enhancing: Enables expression of people's potential
Action Research-Four steps:
Identifying an area of focus
Data collection
Data analysis and interpretation
Action planning
Action Research- Techniques
Collecting, analyzing, and interpreting data
Action planning
Literature Review- Purpose
Definition, purpose, and scope
The review of related literature involves systematic identification, location, and analysis of documents containing information related to the research problem.
Some meaningful studies are conducted in areas where little previous research exists, and some meaningful problems remain in heavily-researched areas of study.
Literature Review- Qualitative Research
Qualitative researchers may not review the literature in-depth before they begin a research study.
Qualitative research topics emerge over time.
Reviewing too much literature may compromise the inductive process of qualitative research.
Demonstrates underling assumptions critical to the research questions
Demonstrates that the researcher is knowledgeable about related research
Helps researcher identify gaps in the body of literature and may provide rationale for additional research
Assists in refining research questions and guides working hypotheses
Literature Review-Steps for Conducting
1.Identify and make a list of keywords to guide your literature search.
2.Using your keywords, locate primary and secondary sources.
3.Evaluate the quality of your sources.
4.Abstract your sources.
5.Analyze and organize your sources using a literature matrix.
6.Write the literature review.
Literature Review-Evaluating your sources:
1.What was the problem statement of the study?
2.Who was studied?
3.Where was the source published?
4.When was the research conducted?
5.How was the study conducted?
CARS-What are the elements of the CARS evaluation method?
The elements are Credibility, Accuracy, Reasonableness, and Support.
Sources-What is the difference between primary and secondary sources?
Primary sources are written by those that have a first-hand account or experience with the source. Secondary sources are those written by people who did not conduct the study or live the experience.
Research topic
There are, however, some commonly used words that relate to a certain type of research. For example, if a topic investigates opinions or attitudes, it is very likely a survey or descriptive research. If it is investigating the relation between two variables, it is a correlational study. If it is investigating the effect of a method or type of instruction, it is most likely an experimental research. If the purpose of a study is to investigate if a method or type of instruction would fix a daily classroom problem, it is action research.
Research topic- Narrowing
Narrow quantitative topics at the start of the research process.
Qualitative researchers often narrow their topic after they are in the field.
Research topic- Characteristics
1.The topic is interesting.
2.The topic is researchable.
3.The topic has theoretical or practical importance.
4.The topic is ethical.
5.The topic is manageable for you given
your current skills, resources, and time available.
Research topic- Quantitative
7Stating the Research Topic
Quantitative research topics
A topic statement describes the variables of interest, relations among those variables, and aspects of the sample.
Research topic-Qualitative
Stating the Research Topic
Qualitative research topics are often stated in more general language at the outset of a study because the focus of the study will likely emerge after time in the field.
Hypotheses- Types
Inductive Hypothesis: A generalization based upon observations
Deductive Hypothesis:
Derived from theory and provides evidence that supports, expands, or even contradicts theory
Nondirectional Hypothesis:
States that a relationship or difference exists among variables
Directional Hypothesis:
States the expected direction of the relationship or difference among variables
Null Hypothesis:
States that there is no significant relationship or difference among variables.
Hypotheses-Null Hypothesis
Null hypotheses are stated when there is little existing research or theoretical support for a hypothesis.
Null hypotheses are also more conservative than directional hypotheses in statistical tests.
Most studies are not based in the null hypothesis.
Hypothesis
A good hypothesis:
is clearly and concisely stated.
states the relation or difference among variables.
defines variables in measurable terms.
Hypotheses- Stating
Model for hypotheses:
P=The participants
X=The treatment, the causal or independent variable (IV)
Y=The study outcome, the effect or dependent variable (DV)
Hypotheses-Qualitative
The qualitative researcher does not state formal hypotheses before conducting studies.
Qualitative researchers may develop guiding hypotheses for the proposed research.
Qualitative researchers often generate new hypotheses during the course of their study.
Qualitative researchers may generate research questions from their guiding hypotheses
Errors- What is a Type II error?
A type II error is when you accept the null hypothesis and it is actually false. Or in other words, you think there is no difference when there really is a difference. This is also known as a false negative.
Errors- What is a Type I error?
A type I error is when you reject the null hypothesis and it is actually true. Or in other words, you think there is a difference but there really is no difference. This is also known as a false positive.
Variables-What is an independent variable in a quantitative research study?
The independent variable is the one that is manipulated, such as the instruction offered. It is also called the treatment or the cause (Y variable).
Variables-What is a dependent variable in a quantitative research study?
The dependent variable is where you look for change based on what you did, such as an improvement in test scores. It is usually referred to as learning outcome; the effect (or Y) variable.
Measurement scales
nominal, ordinal, ratio, and interval.
Variables-Construct
A construct is an abstraction that cannot be observed directly but is invented to explain behavior.
e.g., intelligence, motivation, ability
Variables-Constructs & Variables
Constructs must be operationally-defined to be observable and measurable.
Variables are operationally-defined constructs.
Variables are placeholders that can assume any one of a range of values.
Variables may be measured by instruments.
Variables-Nominal variables
Nominal variables describe categorical data.
e.g., gender, political party affiliation, school attended, marital status
Nominal variables are qualitative.
Variables-Quantitative
Quantitative variables range on a continuum with ordinal, interval, and ratio variables.
Measurement scales-Ordinal variables
Ordinal variables describe rank order with unequal units.
e.g., order of finish, ranking of schools or groups as levels
Measurement scales-Interval variables
Interval variables describe equal intervals between values.
e.g., achievement, attitude, test scores
Measurement scales-Ratio variables
Ratio variables describe all of the characteristics of the other levels but also include a true zero point.
e.g., total number of correct items on a test, time, distance, weight
Variables-Dependent
Dependent variables are those believed to depend on or to be caused by another variable.
Dependent variables are also called criterion variables.
Variables-Independent
Independent variables are the hypothesized cause of the dependent variable. There must be at least two levels of an independent variable.
Independent variables are also called an experimental variables, manipulated variables, or treatment variables.
Collect data
There are three major ways for researchers to collect data.
A researcher can administer a standardized test.
e.g., an achievement test
A researcher can administer a self-developed instrument.
e.g., a survey you might develop
A researcher can record naturally-occurring events or use already available data.
e.g., recording off-task behavior of a student in a classroom
Collect data-Qualitative
Qualitative researchers often use interviews and observations.
Collect data-Quantitative
Quantitative researchers often use paper and pencil (or electronic) methods.
Selection methods: The respondent selects from possible answers (e.g., multiple choice test).
Supply methods: The respondent has to provide an answer (e.g., essay items).
Measurement scales-Interpreting Instrument Data
Raw Score
Norm-referenced scoring
Criterion-referenced scoring
Self-referenced scoring
Measurement scales-Interpreting Instrument Data-Raw Score
Number or point value of items correct (e.g., 18/20 items correct).
Measurement scales-interpreting Instrument Data-Norm-referenced scoring
Student's performance is compared with performance of others (e.g., grading on a curve).
Measurement scales-Interpreting Instrument Data-Criterion-referenced scoring
Student's performance is compared to preset standard (e.g., class tests).
Measurement scales-Interpreting Instrument Data-Self-referenced scoring
How individual student's scores change over time is measured (e.g., speeded math facts tests).
Measurement scales-Types
Cognitive tests
Measurement scales-Cognitive tests
measure intellectual processes (e.g., thinking, memorizing, calculating, analyzing).
Standardized tests measure individual's current proficiency in given areas of knowledge or skill.
Standardized tests are often given as a test battery (e.g., Iowa test of basic skills, CTBS).
Diagnostic tests provide scores to facilitate identification of strengths and weaknesses (e.g., tests given for diagnosing reading disabilities).
Aptitude tests measure prediction or potential versus what has been learned (e.g., Wechsler Scales).
Measurement scales-Affective tests 1
Affective tests measure affective characteristics (e.g., attitude, emotion, interest, personality).
Attitude scales measure what a person believes or feels.
Likert scales measure agreement on a scale.
Strongly agree, Agree, Undecided, Disagree, Strongly disagree
Measurement scales-Affective tests 2
Semantic differential scales require the individual to indicate attitude by position on a scale.
FairUnfair
32 1 0 -1 -2 -3
Rating scales may require a participant to check the most appropriate description.
5=always; 4=almost always, 3=sometimes...
The Thurstone Scale & Guttman Scales are also used to measure attitudes.
Measurement scales-Additional Inventories 1
Interest inventories assess personal likes and dislikes (e.g., occupational interest inventories).
Values tests assess the relative strength of a person's values (e.g., Study of Values instrument).
Measurement scales-Additional Inventories 2
Personality inventories provide participants with statements that describe behaviors characteristic of given personality traits and the participant answers each statement (e.g., MMPI).
Projective tests were developed to eliminate some of the concerns with self-report measures. These tests are ambiguous so that presumably the respondent will project true feelings (e.g., Rorschach).
Measurement scales-Criteria for Good Instruments
Validity refers to the degree that the test measures what it is supposed to measure.
Validity is the most important test characteristic.
Measurement scales-Criteria for Good Instruments- validity
There are numerous established validity standards.
Content validity
Criterion-related validity
Concurrent validity
Predictive validity
Construct validity
Consequential validity
Validity-Criterion-Related
Two types of criterion-related validity include:
Concurrent
Predictive
ValidityCriterion-Related -Concurrent
The scores on a test are correlated to scores on an alternative test given at the same time (e.g., two measures of reading achievement).
Validity-Criterion-Related -Predictive
The degree to which a test can predict how well a person will do in a future situation, e.g., GRE, (with predictor represented by GRE score and criterion represented as success in graduate school).
Validity-Content
Content validity addresses whether the test measures the intended content area.
Content validity is an initial screening type of validity.
Content validity is sometimes referred to as Face Validity.
Content validity is measured by expert judgment (content validation).
Validity-Content-2 areas
Content validity is concerned with both:
Item validity: Are the test items measuring the intended content?
Sampling validity: Do the items measure the content area being tested?
One example of a lack of content validity is a math test with heavy reading requirements. It may not only measure math but also reading ability and is therefore not a valid math test.
Validity-Construct
Construct validity is the most important form of validity.
Construct validity assesses what the test is actually measuring.
It is very challenging to establish construct validity.
Validity-Construct evidence
Construct validity requires confirmatory and disconfirmatory evidence.
Scores on tests should relate to scores on similar tests and NOT relate to scores on other tests.
For example, scores on a math test should be more highly correlated with scores on another math test than they are to scores from a reading test.
Validity-Consequential
Consequential validity refers to the extent to which an instrument creates harmful effects for the user.
Some tests may harm the test taker.
For example, a measure of anxiety may make a person more anxious.
Validity-factors that threaten
Unclear directions
Confusing or unclear items
Vocabulary or required reading ability too difficult for test takers
Subjective scoring
Cheating
Errors in administration
Validity- Types
Criterion-Related
Content
Construct
Consequential
Reliability
Reliability refers to the consistency of an instrument to measure a construct.
Reliability is expressed as a reliability coefficient based upon a correlation.
Reliability coefficients should be reported for all measures.
Reliability affects validity.
There are several forms of reliability.
Reliability- Types
Test-Retest (Stability)
Alternate forms (Equivalence)
Internal Consistency
Scorer and rater reliabilities
Reliability- Types-Test-Retest
reliability measures the stability of scores over time.
To assess test-retest reliability, a test is given to the same group twice and a correlation is taken between the two scores.
The correlation is referred to Coefficient of Stability.
Reliability- Types-Alternate forms
measures the relationship between two versions of a test that are intended to be equivalent.
To assess alternate forms reliability, both tests are given to the same group and the scores on each test are correlated.
The correlation is referred to as the Coefficient of Equivalence.
Reliability- Types-Internal Consistency
represents the extent to which items in a test are similar to one another.
Split-half: The test is divided into halves and a correlation is taken between the scores on each half.
Coefficient alpha and Kuder-Richardson measure the relationship between and among all items and total scale of a test.
Reliability- Types-Scorer and rater reliabilities
reflect the extent to which independent scorers or a single scorer over time agree on a score.
Interjudge (inter-rater) reliability: Consistency of two or more independent scorers.
Intrajudge (intra-rater) reliability: Consistency of one person over time.
Reliability-Standard Error of Measurement
an estimate of how often one can expect errors of a given size in an individual's test score.
SEm=SD * SQT 1-r
Sem=Standard error of measurement
SD=Standard deviation of the test scores
r=the reliability coefficient
Selecting a Test- Where to find?
These are a good place to start when selecting a test.
MMY: The Mental Measurements Yearbook is the most comprehensive source of test information
Pro-Ed Publications
ETS Test Collection Database
Professional Journals
Test publishers and distributors
Selecting a Test- Which to choose?
First, examine validity.
Next, consider reliability.
Consider ease of test use.
Assure participants have not been previously exposed to the test.
Assure sensitive information is not unnecessarily included.