45 terms

Evaluation of Educational Programs


Terms in this set (...)

gathering data
assess means to scrutinize carefully, sit close and observe

Educational assessment is the process of documenting, usually in measurable terms, knowledge, skills, attitudes, and beliefs. Assessment can focus on the individual learner, the learning community (class, workshop, or other organized group of learners), the institution, or the educational system as a whole (also known as granularity).

The final purposes and assessment practices in education depend on the theoretical framework of the practitioners and researchers, their assumptions and beliefs about the nature of human mind, the origin of knowledge and the process of learning
Taken from Wikipedia

Jump to: navigation, search
Grading to determine a mark
interpreting data

Evaluation is a systematic determination of a subject's merit, worth and significance, using criteria governed by a set of standards. It can assist an organization to assess any aim, realizable concept/proposal, or any alternative, to help in decision-making; or to ascertain the degree of achievement or value in regard to the aim and objectives and results of any such action that has been completed. The primary purpose of evaluation, in addition to gaining insight into prior or existing initiatives, is to enable reflection and assist in the identification of future change.

Evaluation is often used to characterize and appraise subjects of interest in a wide range of human enterprises, including the arts, criminal justice, foundations, non-profit organizations, government, health care, and other human services.
Taken from Wikipedia
knowledge that, declarative knowledge, knowledge what
factual knowledge expressed in propositions
Ryle's (1949) definition of declarative knowledge in The Trouble with Learning Outcomes by Hussey & Smith
Knowledge how, procedural knowledge
"Learned skills and abilities (for example, teaching skills, the ability to judge the level at which to pitch a lecture, the suitability of teaching materials, how to judge performance at that level... The relationship between knowledge that and knowledge how is complex. Translating Knowledge how into knowledge that - into a set of statements - and that this is largely either fatuous or impossible" (Hussey &Smith, 2002, p. 230).
Research vs. Evaluation
Researchers seek to prove or disapprove a hypothesis, using the scientific method of examination. Finding a definitive and replicable answer is often the desirable result in research.
In evaluation, results establish valid and worth (to communities or individuals) of a particular intervention or service.
Program Evaluation: Principles and Practices
audit culture
"refers to contexts in which the techniques and values of accountancy have become a central organizing principle in the governance and management of human conduct - and the new kinds of relationships, habits and practices that this is creating" (Shore, 2008, p. 278)
" a condition shaped by the use of modern techniques and principles of financial audit, but in contexts far removed from the world of financial accountancy."
the "scientific management" movement pioneered in the late 19th century by the American engineer, F. W. Taylor. "Taylorist" techniques were designed to construct a mechanism for governing the conduct of industrial workers in order to improve efficiency and increase the capacity of management. But the managerial revolution it precipitated had implications that went way beyond factories or assembly lines. As the French philosopher Michael Foucault (1997, 1991) has shown, routines and disciplinary practices are the vehicles through which governments seek to instil new norms of conduct and behaviour into the populations over which they rule" Shore, 2008, p. 279,80)
Appreciative inquiry
alternative research approach "appreciative inquiry refers to both a search for knowledge and a theory of intentional collective action which are designed to help evolve the normative vision and will of a group, organization, o r society as a whole. It is an inquiry process that affirms our symbolic capacities of imagination and mind as well as our social capacity for conscious choice and cultural evolution" (Cooperrider and Suresh, 1987, p. 24). It has the potential to unite theory and practice and enliven action research.
action research
is problem-based, insufficiency-focused research method designed with a purpose to unite theory and practice and transform organizations and societies.
generative capacity
"capacity to challenge the guiding assumptions of the culture, to raise fundamental questions regrading contemporary social life, to foster reconsideration of that which is taken for granted and thereby furnish new alternatives for social actions" (as cited in Cooperrider and Suresh, 1987, p. 2,3 Green, 1978, p. 1346)"
According to Rick Ross:
is a process where members seek to understand each other's viewpoints and deeply held assumptions.
"dia" means through. l
"logos" means word
Dialogue leads to collective meaning making and shared understanding
holding off judgements
equality and the absence of coercive elements
skilled discussion
According to Rick Ross:
A way of talking that leads to decisions.
a continuum between a debate and a dialogue
evaluation capacity building (ECB)
technique and evaluation with a goal "to continuously create and sustain overall organizational processes that make quality evaluation and its uses routine" (as cited in Baron, 2011, p. 87 Baiserman, Compton,& Stockdill, 2002, p.14)
It is about learning how to think evaluatively and how to engage in sound evaluation practice" ) Preskill& Boyle, 2008, p.443)
internal evaluation
"evaluation conducted by a staff member or unit from within the organization being studied" (the evaluation Center, n.d.)
Moral concerns
what is right or just for all as we are connected simply as human beings
Ethical concerns
what is appropriate for us insofar as we are members of a specific group, collectively worth to unique history and tradition
what is individually or collectively considered valuable, worth striving for and essential to human happiness
the use of statistical methods to combine results of individual studies
standard deviation
is a statistic that tells you how tightly all the various examples are clustered around the mean in a set of data

When examples are tightly bunched together and the bell-shaped curve is steep, the standard deviation is small. When the examples are spread apart and the bell curve is relatively flat, that tells you you have a relatively large standard deviation. It can help point you in the right direction when asking why information is the way it is. The standard deviation can also help you evaluate the worth of all those so-called "studies" that seem to be released to the press every day.
learning outcomes
are statements of what students are expected to know, understand, and/or be able to demonstrate at the end of the indicated grade, course
Gagne's 1974, Ing's 1978 definition of learning outcomes as " the products of the learning process within the pupil that can be directly related to assessment"(as cited in Hussey & Smith, 2004, p. 223)
Hermeneutics is the art of interpreting. Although it began as a legal and theological methodology governing the application of civil law, canon law, and the interpretation of Scripture, it developed into a general theory of human understanding through the work of Friedrich Schleiermacher, Wilhelm Dilthey, Martin Heidegger, Hans-Georg Gadamer, Paul Ricoeur, and Jacques Derrida. Hermeneutics proved to be much bigger than theology or legal theory. The comprehension of any written text requires hermeneutics; reading a literary text is as much a hermeneutic act as interpreting law or Scripture.

Without collapsing critical thinking into relativism, hermeneutics recognizes the historicity of human understanding. Ideas are nested in historical, linguistic, and cultural horizons of meaning. A philosophical, theological, or literary problem can only be genuinely understood through a grasp of its origin.

Retrieved from http://groups.chass.utoronto.ca/iih/AboutHermeneutics.htm
Extended-term Mixed-method Evaluation Design (ETMM)
alternative research approach which is long-term, uses both systematic and analytic, formative and summative, qualitative and quantitative evaluation research
founded on scientific realism Chatterji (2004)
Heuristic evaluation
is a form of usability inspection where usability specialists judge whether each element of a user interface follows a list of established usability heuristics. Expert evaluation is similar, but does not use specific heuristics.
From Wikipedia

Heuristic ("find" or "discover") refers to experience-based techniques for problem solving, learning, and discovery that give a solution which is not guaranteed to be optimal. Where the exhaustive search is impractical, heuristic methods are used to speed up the process of finding a satisfactory solution via mental shortcuts to ease the cognitive load of making a decision. Examples of this method include using a rule of thumb, an educated guess, an intuitive judgment, stereotyping, or common sense.

In more precise terms, heuristics are strategies using readily
accessible, though loosely applicable, information to control problem solving in human beings and machines
academic research
is usually scientist-initiated, theory and conclusion-driven
evaluation research
is aimed at decision making and is usually motivated by a desire for social change
formative assessment
assessment for learning
"helps learners identify where they are and what they need to do next. Its primary purpose is to provide feedback that will promote student learning"
Taken from: Integrating CLB Assessment into your ESL Classroom
summative assessment
assessment of learning "after learning is supposed to have occurred to determine if it did"
Taken from: Integrating CLB Assessment into your ESL Classroom
logical positivism
school of philosophy in which causality is based on regular chains of contingent events (if x then y) predictability is a purpose (Chatterji, 2004, p. 17)
scientific realism
a more realistic philosophical approach in which causality can be established through "the dedicated study of environmental conditions, along with substantively guided causal interpretation and explanation" (Chatterji, 2004, p. 18).
Levels of analysis
The term "level of analysis" is a
social sciences term that points to the location, size, or scale of a research target.
The smallest unit of analysis in the social sciences is an individual in their social setting. At the micro-level, also referred to as the local level, the research population typically is an individual in their social setting or a small group of individuals in a particular social context. Examples of micro-level levels of analysis include, but are not limited to, the following.

Alien, stateless person, asylum seeker, refugee
Partnership, marriage
Taken from Wikipedia
In general, a meso-level analysis indicates a population size that falls between the micro- and macro-levels, such as a community or an organization. However, meso-level may also refer to analyses that are specifically designed to reveal connections between micro- and macro-levels. Sometimes referred to as mid-range, especially in sociology. Examples of meso-level units of analysis include, but are not limited to, the following.
Village, town, city
Formal organization

Taken from Wikipedia
Macro-level analyses generally trace the outcomes of interactions, such as economic or other resource transfer interactions over a large population. Also referred to as the global level. Examples of macro-level units of analysis include, but are not limited to, the following.


Taken from Wikipedia
quantitative research
In sociology, quantitative research refers to the systematic empirical investigation of social phenomena via statistical, mathematical or computational techniques. The objective of quantitative research is to develop and employ mathematical models, theories and/or hypotheses pertaining to phenomena. The process of measurement is central to quantitative research because it provides the fundamental connection between empirical observation and mathematical expression of quantitative relationships. Quantitative data is any data that is in numerical form such as statistics, percentages, etc. In layman's terms, this means that the quantitative researcher asks a specific, narrow question and collects a sample of
numerical data from participants to answer the question. The researcher analyzes the data with the help of statistics.

Taken from Wikipedia
qualitative research
Qualitative research is a method of inquiry employed in many different academic disciplines, traditionally in the social sciences, but also in market research and further contexts.Qualitative researchers aim to gather an in-depth understanding of human behavior and the reasons that govern such behavior. The qualitative method investigates the why and how of decision making, not just what, where, when. Hence, smaller but focused samples are more often needed than large samples.

In the conventional view, qualitative methods produce information only on the particular cases studied, and any more general conclusions are only propositions (informed assertions). Quantitative methods can then be used to seek empirical support for such research hypotheses.

Qualitative researchers face many choices related to data collection ranging from grounded theory practice, narratology, storytelling, classical ethnography, or shadowing. Qualitative methods are also loosely present in other methodological approaches, such as action research or actor-network theory. Forms of the data collected can include interviews and group
discussions, observation and reflection field notes, various texts, pictures, and other materials.

Taken from Wikipedia
establishing construct validity through convergence in findings from multiple studies (Chatterji, 2004 p. 18)

refers to the use of more than one approach to the
investigation of a research question in order to enhance confidence in the ensuing
findings. Since much social research is founded on the use of a single research
method and as such may suffer from limitations associated with that method or from
the specific application of it, triangulation offers the prospect of enhanced confidence.
Triangulation is one of the several rationales for
The term derives from surveying, where it refers to the use of a series of triangles to map out an area.

Taken from: http://www.referenceworld.com/sage/socialscience/triangulation.pdf
Types of Triangulation
Denzin (1970) extended the idea of triangulation beyond its conventional association
with research methods and designs. He distinguished four forms of triangulation:
Data triangulation
, which entails gathering data through several sampling
strategies, so that slices of data at different times and social situations, as well as
on a variety of people, are gathered.
Investigator triangulation
, which refers to the use of more than one researcher in
the field to gather and interpret data.
Theoretical triangulation
, which refers to the use of more than one theoretical
position in interpreting data.

Methodological triangulation
, which refers to the use of more than one method.

Taken from: http://www.referenceworld.com/sage/socialscience/triangulation.pdf
conscious evolution
enables us to collaborate actively with the evolutionary process can motivate action toward "conscious evolution" is enabled by self-reflective consciousness (Benathy, 1999, p. 163).
self-reflective consciousness
is a process by which individuals, groups, organizations, and societies contemplate and make presentations of their perceptions of the world, and their understanding of their place in the world, in their individual and collective minds (Benathy, 1999, p. 163)
evolutionary learning
enables us to cope with changes and complexity, renew our perspectives, and redesign our systems, often reorganizing them tat higher levels of complexity. It will help us to progress from unconscious adaptation to our environment to conscious innovation and coevolution with the environment an d the development of the ability to direct and manage change. (Benathy, 1999, p. 164)
means unrolling a process by which successive forms and content unfold creatively.
lifeworld according to Habermas
a term that Habermas adapts from Alfred Schutz. By the lifeworld Habermas means the shared common understandings, including values, that develop through face to face contacts over time in various social groups, from families to communities. The lifeworld carries all sorts of assumptions about who we are as people and what we value about ourselves: what we believe, what shocks and offends us, what we aspire to, what we desire, what we are willing to sacrifice to which ends, and so forth. Most of these assumptions are latent in Parsons's sense of latency. Habermas writes that to make lifeworld assumptions fully reflective—to speak of them explicitly—is already to destroy them. Their power is their "of course" or "taken for granted" quality. Questions about the
lifeworld—why do you believe such-and-such? —can only be answered (if at all) by some version of "because that's who I am and who we are".
Retrieved from:
systemsworld according to Habermas
his usage probably following Weber more than any other theorist. Systems are fully rationalized; George Ritzer's McDonaldization book is an excellent desciption of a fully rationalized system. The principles of rationalization—evident in McDonalds—are efficiency, calculability, predictability, and control. The point of such rationalization is to reduce the person to part of the "machinery" by which the system does what it does; individual scope of action and decision are minimized: "choices" are strictly limited. Ritzer points out how McDonalds "works" as a system by putting customers to work: the customer becomes part of the assembly line, picking up food, taking it to tables, clearing off the tables, etc. There is minimal possibility for customer and staff to talk to each other, much less to reach any common understandings; no place for "communicative action". Staff have no possibility of making decisions about how the restaurant will be run, and customers are expected to move on at regular intervals (Ritzer points out that seats are built so that people won't sit too long). Everyone involved has to act as the system directs them. The quantitative system (so many "served daily", as quickly as
possible, and what they are served is advertised for size, not quality) colonizes any lifeworld communication.

Retrieved from: http://people.ucalgary.ca/~frank/habermas.html
strategic competence
the ability to manage the integration and application of all other language competence components to the specific context and situation of language use;involves planning, assessing communication breakdown; and using effective devices to ensure the effectiveness of communication (CLB, 2010, p. 209)
sociolinguistic knowledge
the ability to produce and understand utterances appropriate to a given social context: politeness, register, dialect, culture, customs (CLB, 2010, p. 209)