Kazdin (2003) describes four types of group designs with random assignment of subjects to each group: a) pre- and post-test control group, b) post-test only control group, c) Solomon four-group and d) factorial designs. Although a pretest is generally very advantageous, one weakness is that it could sensitize the subject to the treatment effect. Kazdin (2003) and Cook and Campbell (1979) also describe a few different types of quasi-experimental designs where random assignment is not used: a) pre- and post- test design (one and two groups), and b) post-test only design (one, two and multiple groups). Quasi-experiments can serve as a starting ground to measure effects of manipulation and provide information for more controlled studies in the future.
The lack of a pre-test in quasi-experiments makes it difficult to estimate group differences prior to treatment (Kazdin, 2003; Cook & Campbell, 1979). Especially in designs without a control group it is hard to eliminate threats such as maturation, history, testing, statistical regression, and other third variable factors could be affecting the change between pre- and post-test. Although not ideal, when pre-tests and control conditions are not possible, researchers can minimize threats to validity and rival explanations by using known base rates, archival records, matching other correlate variables (age, sex, social class, etc.), large sample size, common sense, theory, and experience.
In observational studies, the investigator evaluates the variables of interest by selecting groups, rather than experimentally manipulating the variables of interest. The goal is to show associations between variables, but it may be possible to show causation. One limitation is that it does not allow for strong influences to be drawn about what lead to the outcome of interest.
Two types of observational studies are case control and cohort designs. The designs address a range of questions about how variables interact to produce an outcome (mediators) and how the characteristics that influence (moderators) whether or for whom the outcome occurs.
Observational studies require special attention to construct validity (the conclusions can be attributed to the constructs the researcher had in mind) rather than to other influences. In addition to specifying the construct,
As we have already read, although quasi-experimental designs have their limitations, there are ways that researchers can minimize threats to validity and rival explanations by using known base rates, archival records, matching subjects on other correlate variables (age, sex, social class, etc.), large sample size, common sense, theory, and experience.
Although qualitative research is based on descriptive and lengthy narrative accounts with little standardized measurement, it's important to note that it can be done rigorously and systematically (Kazdin, 2003). Qualitative research is a methodical way of understanding and evaluating individual experiences. The analysis is based on the researcher's interpretation and identification of important themes and ideas in the narratives of the participants. However, sometimes researchers come up with a coding scheme a priori based on a theoretical framework (deductive approach) and can re-evaluate the codes as they go through the data (Weitzman, NIH e-course). Another step that is encouraged in qualitative studies is for investigators to consult with other researchers about their identification of important themes and their interpretation of the data. Additionally, the subject's feedback is also collected and taken into consideration. These additional steps add to the triangulation process in which multiple methodologies, perspectives, and analyses are used to strengthen the conclusion of the study.
However, even with multiple methods and the elaborate details available from participants, the results are still vulnerable to misinterpretation (interpretive validity). And even with rich details of day-to-day experiences, qualitative studies usually have very small sample sizes, which can make it hard to generalize the results (external validity). Confirmability is the extent to which the results are free of the experimenter's bias and can be replicated by others. However, if the outcome of the study is based on the unique situation and experience of the participants and the subjective interpretation of the researcher, replication seems unlikely with a different set of participants and researchers.
Despite its limitations, qualitative studies can be especially useful as part of a mixed methods design that combines qualitative and quantitative research (Creswell et al., 2011). A qualitative exploratory study can generate hypothesis about key constructs that can be further evaluated with quantitative studies. Qualitative data can also be collected as a follow-up to better understand quantitative data. However, carefully designing and properly executing a mixed methods study can be a complex and cumbersome process. It requires an integrative multi-member research team with diverse expertise, specific leadership qualities, bigger budget, access to increased resources needed for multiple methodologies, continued training, additional time commitment for frequent meetings, large sample sizes, analytical expertise in mixed methods reserach, etc. Researcher should evaluate both the theoretical need and the resources available before taking on mixed methods research project.
-Qualitative studies are done systematically and with precision if done well.
-Studying subjects in context
-Confirmability: gaining consensus on data, as to do with replication.
- Coding and then classifying important themes are the first two steps.