83 terms

V379 Final


Terms in this set (...)

What evaluation questions can be answered by conducting economic evaluation?
• Relating costs to program results
• Allocation of resources
• Expand a program
• Shut a program down
What are the two primary types of economic evaluation?
• Cost-Effectiveness Analysis (CEA)
• Cost-Benefit Analysis (CBA)
Cost-Effectiveness Analysis (CEA)
- costs are monetized against a single non-monetized outcome (e.g. # of lives saved).
-Combines information about program costs (dollars) with one specific outcome (units of effectiveness) - the result is a ratio of cost to units of effectiveness
-Cost per life saved
-Cost per person returning to employment
-Cost per illness prevented
Cost-Benefit Analysis (CBA)
-Both costs and benefits to society are monetized to determine a net social benefit.

-Goes beyond CEA in monetizing the benefits of a program or intervention
-The premise is that a "bottom line" for projects or programs can be assessed
EXAMPLE - How do the costs of providing a job training program for workers who are laid off compare to the financial benefits derived from the training?
CBA practical uses
-Can provide a straight-forward "bottom line" for the efficacy of programs.
-Can be problematic to attribute ALL costs and ALL benefits to the program, as opposed to other factors.
CEA practical uses
-Can facilitate comparisons across program alternatives that share the same objective.
-There can be a tendency to create ratios based only on what is measurable, not what it most meaningful.
What are the four types of reliability?
• Instruments
• Ratings/Observations
• Instrument Items
• Construct Scales
The extent to which a measurement instrument gives the same reading if repeated in a given situation (instrument reliability).
The extent to which different raters would code an observation or rating the same way (inter-rater reliability).
Instrument Items
The extent to which related items on an instrument assess the same construct (inter-item reliability).
Construct Scales
The extent to which results are consistent across two different scales or sets of items (split-scale reliability).
What kinds of validity are relevant to data collection measures?
• Face Validity
• Content Validity
• Process Response Validity
• Internal Structure Validity
-Concurrent Validity
-Predictive Validity
-Convergent Validity
What are the four levels of measurement presented in class?
Objects cannot be ordered or added; they are labels.
Objects of a set can be rank-ordered; equivalence between objects not guaranteed.
Rank-ordered scale in which the intervals represent equal distances; no absolute zero value.
Rank-ordered scale with equal intervals and an absolute zero value.
What are the types of error are relevant to survey research?
Sampling Error
Sampling Bias
Response Error
Sampling Error
-The level of sampling error is always controlled by the sample size.
-Larger samples have less chance of producing results that don't represent the population as a whole.
Sampling Bias
-Members of a sample differ from the larger population in some systematic way.

-Selection Bias: Specific population groups are more or less likely to be included based on survey methodology
-Reducing Bias: Ensure that methodology does not favor one group of respondents over another

-Non-Response Bias: Occurs when failure to respond is disproportionate across subgroups of the population

-Reducing Bias: Ensure that methodology and instrument do not favor or exclude on group of respondents over another.

NOTE: Sampling bias is NOT controlled by sample size; instead these are systematic, occurring with samples of all sizes.
Response Error
-Differences between the answers respondents provide and the true state of things.

-Comprehension Error: When respondents do not understand a question, but provide a response anyway.

-Knowledge Error: Respondents do not know the answer or cannot recall it, but still provide an answer.

-Reporting Error: Responses given to be socially desirable or to maintain privacy.

Minimizing Response Error involves the design and layout of your survey instrument
What is Social Exchange Theory and what are its implications for survey research?
the actions of individuals are motivated by the return they are expected to bring from others.
• Rewards: what one expects to gain
• Costs: what one gives up or spends
• Trust: expectation that the reward outweighs the cost
What strategies are used to increase the rewards of responding to a survey?
• Show positive regard
• Say thank you
• Ask for advice
• Support group values
• Give tangible rewards
• Make questionnaire interesting
• Give social validation
• Inform respondents that opportunities to respond are scarce
What strategies are used to decrease the costs of responding to a survey?
• Avoid subordinating language
• Avoid embarrassment
• Avoid inconvenience
• Make questionnaires appear short and easy
• Minimize requests to obtain personal information
• Keep requests similar to other requests to which a person has already responded
What strategies are used to increase trust among survey respondents?
• Provide a token of appreciation in advance
• Sponsorship by a legitimate authority
• Make task appear important
• Invoke other exchange relationships
Response Scales: Behavior
• Per day, Per week, Per month, Per year, Never
• Always, Frequently, Seldom, Never
• Always, Sometimes, Never
• All, Most, Some A Few, None
• Often, Sometimes, Rarely, Never
Response Scales: Opinion
• Completely Satisfied, Mostly Satisfied, Somewhat Satisfied, Dissatisfied, Very Dissatisfied
• Very Important, Somewhat Important, Not too Important, Not at all Important
• Oppose, Support
• Strongly Oppose, Oppose, Support, Strongly Support
Response Scales: Knowledge
• Very Familiar, Somewhat Familiar, Not too Familiar, Not at all Familiar
• True, False
• A Lot, Some, A Little, Nothing
What types of questions are appropriate to appear as the first question on a survey instrument?
• Should always be easy to answer
• Should be VERY relevant to the stated purpose of the survey
• Should not be too personal
• Should not be open-ended
Criterion-Based Scales
-Comparison against the respondent's absolute criteria for performance
-Excellent, good, fair, poor, etc.
Norm-Referenced Scales
-Asks for a comparison with some reference group in mind
-Above average, below average, etc.
ODD # of Responses = Midpoint
(a) SA A N D SD
(b) Always Often ½ the time Seldom Never

NOTE 1: There are reasons for either choice; but the choice should be made consciously (e.g., are you measuring change - that might be from a neutral opinion?)
NOTE 2: "Not Sure" or "Don't Know" is not a midpoint - and should be included at the end of the scale, not in the middle.
EVEN # of Responses = No Midpoint
(a) SA A D SD
(b) Always Often Seldom Never

NOTE 1: There are reasons for either choice; but the choice should be made consciously (e.g., are you measuring change - that might be from a neutral opinion?)
NOTE 2: "Not Sure" or "Don't Know" is not a midpoint - and should be included at the end of the scale, not in the middle.
Where do evaluators get information on the content that should be included in a survey instrument?
-Collaborate with stakeholders (participants, advisory boards, staff members) to determine what you want to measure.
-Be sure anyone writing questions is familiar with the program (has directly observed or participated).
-Review other surveys that have been used with similar programs or with similar measures - may lead of plenty of valid items.
What are some common mistakes in writing survey questions?
-Choose simple words over specialized words
-Use complete sentences to ask questions
-Avoid vague quantifiers when more precise estimates can be provided
-Avoid specificity that exceeds the respondent's potential for having an accurate, ready-made answer
-Use equal numbers of positive and negative categories for scalar questions.
-Avoid bias from unequal comparisons
-Eliminate"check-all-that-apply" question formats to reduce primacy effects.
-Develop response options that are mutually exclusive.
-Avoid double-barreled questions
-Avoid asking respondents to say yes in order to say no.
Choose simple words over specialized words:
• Tired = exhausted
• Honest = candid
• Most important = top priority
• Work = employment
• You answers = your responses to this questionnaire
• Job concerns = work-related employment issues
When a word exceeds 6-7 letters, chances are that a shorted and more easily understood word can be substituted.
Use complete sentences to ask questions
• Number of years lived in Idaho:(Bad Choice)
How many years have you lived in Idaho?
• Your county:(Bad Choice)
In what Idaho county do you live?
Avoid vague quantifiers when more precise estimates can be provided
• How often did you attend religious services during the past year?
• Never, rarely, occasionally, regularly (Bad Choice)

• Not at all, a few times, about once a month, 2-3 times per month, about once a week, more than once a week.
Avoid specificity that exceeds the respondent's potential for having an accurate, ready-made answer
About how many books have you read for leisure during the past year?
• ______ Number of books(Bad Choice)

• None, 1-2, 3-5, 6-10, 11 or more
Use equal numbers of positive and negative categories for scalar questions.
How satisfied were you with the service you received when you bought your air conditioner?
• Completely satisfied, mostly satisfied, somewhat satisfied, neither satisfied nor dissatisfied, dissatisfied (Bad Choice)

• Completely satisfied, somewhat satisfied, neither satisfied nor dissatisfied, somewhat dissatisfied, completely dissatisfied
Avoid bias from unequal comparisons
Which of the following do you feel is most responsible for the recent outbreaks of violence in America's schools?
• Irresponsible parents, school policies, television programs. (Bad Choice)

• The way children are raised by their parents, school policies, television programs
Eliminate"check-all-that-apply" question formats to reduce primacy effects.
Which of the following characteristics would you like to see your child develop as he/she grows up? (Check all that apply) (Bad Choice)

To what extent would you like to see your child develop the following characteristics as he/she grows up?
Develop response options that are mutually exclusive.
From which of these sources did you first learn about the tornado in Derby?
• Radio
• Television
• Someone at work
• While at home (Not a Source)
• While traveling to work ( Not A Source)
Avoid double-barreled questions
• Should the city build a new swimming pool that includes lanes for swimming laps that is not enclosed for winter use? (Bad Choice)

• Should the city build a new swimming pool that includes lanes for swimming laps?
• Should the city build a new swimming pool that is not enclosed for winter use?
Avoid asking respondents to say yes in order to say no.
Do you favor or oppose not allowing the state to raise taxes without approval of 60% of the voters? (Bad Choice)

Do you favor or oppose requiring 60% approval by voters in order to raise state taxes?
According to Visual Processing Theory, what are the three phases involved in getting people to remember information presented visually?
• Pre-attention
• Working Memory
• Long-term Memory
Phase One: Pre-attention
• Humans are very adept at constantly scanning their environment for things that stand out.
Implications for Evaluation Communications:
• Color, alignment, motion, orientation, and size grab attention - when used judiciously
• Involves creating a hierarchy of information for the readers
• Involves headings, charts & tables, indents/outdents

- Involves getting the viewer's attention. To what should they pay attention?
Phase Two: Working Memory
• Attributes noticed in Phase One are held in the working memory - and might make it into long-term memory
• Research shows that we can only hold 3-5 chunks in working memory at any given time
Implications for Evaluation Communications:
• "Predigested Information" - charts/graphs
• Easily readable fonts and graphics
• Charts that show position on a common scale (e.g., bar charts versus pie charts)

- Involves showing the viewer what we want them to know? What does this information mean
Phase Three: Long-Term Memory
• New information is incorporated into existing mental schemas or networks stored in the brain
• Where comprehension and cognition occur
Implications for Evaluation Communications:
• Using one color among grey bars rather than different colors in charts
• Using faint gridlines - so as not to distract
• Placing all associated text within one eyespan

- Involves ensuring that the viewer can remember the information. How can we eliminate other distractions?
When are graphs and tables NOT necessary for communicating information?
To display a single number or two, no need to create a table or graph
• "Eloquence through simplicity"
• Can effectively convey key message, quickly
Tables are used when:
• To look up individual values
• To compare individual values
• Precise values are needed
• Multiple units are used
• Detail and summary values are needed
Use graphs when:
• The message is the shape or pattern of values
• The message is about relationships among sets of values
With what types of data are pie charts most likely to be helpful?
• Quantitative information
-avoid as much as possible, hard to decode circular areas and can't use as many data points or compare series
Bars (Columns):
helpful for comparing values.
• Nominal comparisons: Bars
• - We use nominal comparisons when we want to compare values of different categories.
• - Check your message: "This is bigger than that", This is the biggest", This is almost twice", etc.
• Ranking: Bars
• Display the order in which quantitative values relate to each other
• Check your message: "higher than", "lower than", "the highest", "the lowest", "equal to", "greater than"
: helpful for showing trends and shape of the data.
• Time series: Lines usually
• Relationship between one or more variables and time.
• -Check your message: "change", "rise", "increase", "drop", "fluctuate", "grow", "decline", "decrease", "trend".
• Distributions
• Use lines when you want to emphasizes the shape of the distribution and bars when you want to compare values.
In what ways can you "reduce clutter" in graphs created in Excel?
• Take away background lines
• Take away vertical access if possible
• Use data labels
• Labels are used sparingly
• Text is hierarchical and readable
• Delete legends
What are some strategies for using color to enhance graphics?
• Use colors that you can see when printed out
• Use a color to make data stand out
• Highlight patterns themes and contrasts
• Legible for people who are colorblind
-use color only when needed to serve a particular communication goal
What are some attributes of a useful title for a graph?
• Have a title and a subtitle
• 6 word descriptive title
• If your graph is a standalone piece, your title should convey the four W: Who, What, Where and When.
-Questions may vary between respondents (as will the data)
-Very dependent upon the interviewer's conversational skills
-STRENGTH: responsive to individual and situational differences
-LIMITATION: requires a lot of time to gather systematic information
-LIMITATION: data are difficult to analyze
-interview guide
-Involves a list of questions that will be explored during an interview
-List serves as a guide to ensure that all topics are covered, but exact wording and sequence can vary
-STRENGTH: predetermined information to be collected
-STRENGTH: retains a fair amount of flexibility in individual interviews
-LIMITATION: variability between interviews can make data analysis challenging
Standardized Interview Protocol
-interview protocol
-Consists of a set of questions that are carefully worded and arranged; every respondent experiences the same interview
-Can be useful when a large number of interviews are being conducted and analyzed
-STRENGTH: potential interviewer effects are reduced
-STRENGTH: data are often easier to analyze
-LIMITATION: does not permit researcher to pursue topics that spontaneously emerge
-LIMITATION: individual differences and circumstances are harder to take into account
• Descriptive data
• Key stakeholder perspectives
• Instrument development
• Respondents easy to access
• Respondents respond better verbally
• Standardized data from large numbers of respondent
• Quantitative data
• Respondents not readily accessible in person
• Highly sensitive information / interviewer not familiar with respondent
What are the characteristics of focus groups?
• Group of people (usually 7-10)
• Participants share some common characteristics
• Produce qualitative data - related to perceptions, attitudes of consumers about products, services, or opportunities (not intended to develop consensus or make decisions)
• Involve a focused discussion (carefully predetermined and sequenced)
What type of information is best gathered through focus groups?
• Produce qualitative data - related to perceptions, attitudes of consumers about products, services, or opportunities (not intended to develop consensus or make decisions)
Advantages of Focus Groups
• Place people in real-life social situations (where they are influenced by other's opinions)
• Allow moderator to probe (or dig deeper) for information.
• Have high face validity
• Have relatively low costs
• Provide speedy results
• Allow for the inclusion of more respondents (compared to interviews)
Limitations of Focus Groups
• Less control over the course of the discussion
• Data are more complex to analyze
• Moderating requires training
• Large variability between groups
• Requires more effort to assemble
• More tailored environment required
How can data collected through focus groups compliment data collected through more quantitative data collection strategies such as surveys?
• Focus group precedes quantitative measures
• Help identify thinking patterns and learn related vocabulary
• Triangulation
• Both qualitative and quantitative measures are used to address the same issue
• Focus group follows quantitative measures
• To help interpret data that comes from a questionnaire, or to develop action strategies for concepts identified by the survey.
What purpose do the Guiding Principles serve for evaluators?
• Promote ethical evaluation practice
• Foster continuing professional development
• Stimulate discussion within and outside evaluation
Evaluators conduct systematic, data-based inquiries
-Adhere to highest technical standards
-Explore strengths and shortcomings of evaluation questions and approaches
-Communicate approaches, methods and limitations accurately
Evaluators provide competent performance to stakeholders
-Possess appropriate skills and experience
-Demonstrate cultural competence
-Practice within limits of competence
-Continually improve competencies
Evaluators display honesty and integrity and attempt to ensure them throughout the entire evaluation process:
-Negotiate honestly with clients and stakeholders
-Disclose values, interests and conflicts of interest
-Represent accurately methods, data and findings
-Disclose source of request and financial support for evaluation
Evaluators respect security, dignity and self-worth of all stakeholders:
-Understand evaluation context
-Get informed consent and protect confidentiality
-Maximize benefits and minimize harm
-Foster social equity
-Respect differences among stakeholders
Evaluators take into account general and public interests:
-Include relevant stakeholders
-Balance client and stakeholder needs
-Examine assumptions and potential side effects
-Present results in understandable forms
Face Validity
The extent to which evaluation stakeholders agree the measure represents the construct
Content Validity
Similar to face validity, but relies on experts in the field to determine the level of representation.
Process Response Validity
The extent to which respondents using an instrument engage with and take it seriously.
Internal Structure Validity
The extent to which items on the same scale behave in the same way, suggesting they measure the same construct.
Concurrent Validity
The extent to which there are correlations among different measures of a construct.
Predictive Validity
The extent to which performance on one measure predicts performance on another
Convergent Validity
The extent to which there are correlations among constructs that are expected to be related
What practices are recommended for survey administration in the Tailored Design Method?
1. Respondent-friendly survey instrument
2. Total of 5 contacts; 4 by 1st class mail, one other special contact, pre-notice letter, survey mailing, thank you letter, replacement survey, final special contact
3. Return envelope with 1st class postage
4. Personalized correspondence
5. Token pre-paid financial incentives
Under what circumstances is each type of interview format best utilized?
Informal- responsive to individual and situational differences. Semi-structured- predetermined information to be collected and retains a fair amount of flexibility in individual interviews. Standardized-potential interviewer effects are reduced and data are often easier to analyze.