Ed.D. candidates in the Educational Leadership Program have available three general options with regard to the form their dissertation may take.

Download PDF Version

Rationale:
In order to prepare all Ed.D. candidates in the Educational Leadership Program well for the various positions they may undertake (i.e.: administrator, researcher), we will offer our students three general optional forms of the dissertation project. Students who are aiming toward careers that will entail a considerable amount of research, as well as students who have an interest in researching a question of particular interest to them, should choose the first option, the “research dissertation”. However, students who are preparing themselves for careers as full time administrators may wish to choose from options two and three, the “program evaluation dissertation” and the “policy formulation dissertation.” These options enable our students to build the skill sets appropriate to their future professional positions.

There are important differences among these options which are detailed below, but it is important to emphasize the commonalties they share. Each option:
• Expects of the students a substantial commitment of time and effort and the production of an extended piece of writing;
• Entails an extended review of the body (ies) of literature relevant to the project; and
• Culminates in a production of a final written product.

The First Option: Research Dissertation
Dissertation research is customarily viewed as an enterprise designed to produce knowledge. The first form which we call a "research dissertation" is usually written from a disciplinary perspective, e.g. the perspective of economics, finance, philosophy, history, law or a social science. The research dissertation may seek to test or generate hypotheses, offer an explanation for phenomena or event, explore and advance theory, or advance an argument. Empirically based research dissertations, especially those rooted in a social science, often seek to establish generalizeable propositions. Research dissertations may use a qualitative, quantitative, or analytical methodology. Students undertaking the "research dissertation" are expected to have taken three appropriate courses in educational research methodology, including ED 506: Doctoral Research Methods.

The Second Option: Program Evaluation Dissertation
The Ed.D. student may also seek to write a dissertation which consists of an evaluation of a program or policy. The “program evaluation dissertation” identifies, clarifies, and applies defensible criteria to determine the worth of an educational program, policy, or product. The evaluation may either be a formal summative or a formal formative evaluation. The dissertation should yield a document of use to actual decision makers, as well as meeting the canons of a properly conducted evaluation. Students undertaking a "program evaluation dissertation" are required to have taken ED 506: Doctoral Research Methods; two additional research methodology courses; ED 520: Introduction to Program Evaluation; and ED 521: Advanced Program Evaluation.

Defining Program Evaluation
Program Evaluation is defined as “the identification, clarification, and application of defensible criteria to determine an object’s worth”
--Fitzpatrick, Sanders, and Worthen, 2003

In educational evaluation, the “object” might be a:
• Program:

o “our K-12 special education program”
o “our Reading Recovery program”
o “our after-school program” etc.

• Project:

o “our grant-funded technical assistance project for co-teachers”

• Process:

o “the transition from pre-school to Kindergarten”
o “curriculum mapping”
o “teacher practices in an inclusive classroom” etc.

• Product:

o “a new textbook series for ELA”
o “Reader Rabbit software” etc.

Informal vs. Formal Evaluation
Informal evaluation is the everyday, unsystematic evaluation used by people based on their experience, instinct, generalization, and reasoning to form judgments that lead to faulty or wise choices and decisions.

This kind of evaluation is what the superintendents see as the art of leadership and is probably covered more in the decision-making courses that precede the program evaluation course. The day-to-day leader must have great use of this facility to survive, but informal evaluation does not have the systematic rigor and care found in well planned and implemented formal program evaluations. These would not make good dissertations.

Formal evaluations are the structured and public evaluations of programs, projects, processes, and products using methods, general principles, and rules.

These evaluations, when done well, could have the scope and depth of a typical dissertation; however, the purposes of the conclusions may or may not result in a contribution to knowledge outside of the school district where the evaluation is conducted. These have the potential of being a viable alternative dissertation.

Formative vs. Summative Evaluation
Formative—the primary purpose is to provide program improvement.

The choice to evaluate a “K-12 special education program” is most likely brought about by a desire to improve the existing program. How do we make it “better”? This could mean many different things: “serve more children in general education,” “cost less,” “provide more advocacy,” “prevent dropouts,” or “more achievement on state tests.” Because of federal law, special education isn’t likely to be discontinued soon in any public school, so how do we make it better? Depending on the scope of the questions and the methods used, a formative evaluation could be a viable alternative dissertation.

Summative—the primary purpose is to provide information to serve decisions about program adoption, continuation or discontinuation, or expansion.

The choice to evaluate an “after-school program” may come about because there is a desire to serve a perceived need or needs among students. Should we adopt one or design one of our own? Or, we’ve had an after-school program now for five years; should we continue with it or is it time to close it? Or, should we open it up to both elementary and middle school students? Depending on the scope of the questions and the methods used, a summative evaluation could be a viable alternative dissertation.

Internal vs. External Evaluation
Internal—conducted by program employees.
External—conducted by an evaluator outside of the organization.

Advantages of internal and external evaluators:

There are limitations to both internal and external evaluations; however, if the confounds are properly acknowledged in the dissertation, both kinds could be viable alternatives to the standard dissertation.

Standards for Program Evaluation—Major categories
• Utility—the results will be useful to decision makers;
• Feasibility—the evaluation will be manageable within the constraints of the organization and will be able to produce timely information;
• Propriety—the evaluation is conducted ethically and responsibly with the recognition that the study is often conducted in a highly political environment; and
• Accuracy—the evaluation will reveal and convey technically adequate information.

A program evaluation that is used for an alternative dissertation should meet all of these standards.

Hierarchy of Evaluation Questions

Need—the identification of the problem that the program is addressing.
If the program contains a solution that does not address the needs of those it was designed for, the decision makers have misdiagnosed the problem. You can have a “good” program that misses the point. The evaluator is sometimes asked to conduct a needs assessment to determine the problems that the “customers” of the program have. Depending on the scope and depth of the evaluation, a needs assessment could be a viable alternative dissertation.

Theory—the program’s “blueprint” for addressing the “customers’” needs.
Program theories sometimes resemble causal chains that indicate a program’s intentions for addressing customer needs or a service delivery model. Program theory is about how the program is supposed to work. Many long-established and well-funded social programs, including educational programs, may lack well-articulated program theories. The evaluator may be asked to assist in the development of a new program to meet a newly found need, assess the current understanding of what an existing program’s theory is in the organization, further assess whether it is consistent at different levels of an organization, and compare and contrast different theories that exist in the same program. A superintendent may ask me to evaluate his “inclusion” program. I may have to ask a program theory question to see if inclusion and its intended results are consistently understood throughout the organization. A bad concept for inclusion may be undermining good intentions. Depending on the scope and depth of the evaluation, an investigation into program theory could be a viable alternative dissertation.

Implementation — the manner in which a program actually carries out the program blueprint or theory.
Implementation questions address process issues and the actual delivery of the program. Implementation questions are usually formative in nature. The evaluator is often asked to compare “how it is supposed to work” to “how it is actually working.” When programs are implemented improperly, leadership is often an issue. Examining the inner workings of school processes (e.g., curriculum, instruction, assessment, professional development, discipline, etc.) in a given school are often at the heart of educational evaluation. These are the major responsibilities of an instructional leader. An evaluator must be sure that enough implementation has occurred before he looks into effects. Depending on the scope and depth of the evaluation, a process evaluation could be a viable alternative dissertation.

Outcomes—measures or indicators of program impact.
Outcome questions concentrate on the actual effects, if any, that the program has on its customers. Outcome data are usually more summative, influencing the continuation or discontinuation of a program, or at least reports at the end of a cycle of program delivery. The evaluator often looks for intended and unintended effects. His job is often to see whether the program had realistic or unrealistic expectations. In the “real world,” outcome data must be measured often under tight deadlines, with a consideration of the level of true implementation and resource allotment. The reporting of outcomes doesn’t occur in a laboratory or a scholarly journal very often, but more likely in school board meetings, faculty meetings, and with other stakeholder groups. Decisions based on any aspect of a program evaluation, but especially the outcome data, can have both positive and negative repercussions throughout the program and the larger organization. Depending on the scope and depth of the evaluation, an outcome or impact evaluation could be a viable alternative dissertation.

Cost—the measure of resources used to carry out the program.
Where outcome indicators address “the bang,” cost indicators measure “the buck” and other expenditures of energy to “run” a program. The evaluator may be asked to decide if the benefits of the program are worth the cost of providing it. Money, time, and other material, immaterial, and human resources are considered. Will a child who is deaf and blind be best served in his home district or a school for the blind in a nearby city? What will each program deliver? How much will each alternative cost? Although cost questions are the last ones considered on this hierarchy, many requests for evaluation are spurred on by concerns about cost. An evaluator will often have to ask other questions in this hierarchy before he can answer the cost question. However, depending on the scope and depth of the evaluation, a cost analysis could be a viable alternative dissertation.

Other Considerations about Program Evaluation as a Dissertation
• An evaluation might include all, a part, or only one kind of question on this hierarchy;
• Evaluators get told rather than asked what they are going to look at;
• Quantitative and qualitative approaches are both valued in evaluation- many evaluations contain the use of mixed methods;
• The evaluator has many roles that he can choose to approach the evaluation. The concentration can be on educational objectives, management concerns, the needs and rights of the consumers, the wisdom of experts in the field, or the perceptions of participants in the program;
• The highly political nature of conducting an evaluation can result in ethical dilemmas for the evaluator. An evaluation can go places because of a change in administration that the evaluator can’t predict at proposal.

Dissertation Requirements
The program evaluation dissertation must address a significant program that involves a sizeable budget expenditure and affects a substantial number of people.

Dissertation Elements
The “program evaluation dissertation” must contain each of the following elements:
• Problem identification and specification;
• A discussion of alternative approaches to program evaluation considered for the
dissertation;
• A detailed description of the program’s history, purpose, operations;
• An extended analytical literature review;
• A discussion of data sources and methods of collecting data;
• A discussion of methods of data analysis;
• An analysis of the data;
• A discussion of the results; and
• A discussion of implications.

The Third Option: Policy Formulation Dissertation
The “decision making dissertation” asks students to engage in a project which leads to a decision regarding a policy recommendation addressing a problem faced by an educational organization. The decision making dissertation is "forward" looking in that the recommendation should address future actions which the organization should undertake. Decision-making typically entails reliance on evaluation data that may have been produced by a program evaluation. In addition, it seeks to evaluate the likely consequences of action that has not yet been undertaken. The decision-making dissertation is expected to use and properly execute perspectives, "tools", and principles for reaching reasoned decisions which are recognized in the fields of public policy formation and decision making. Students undertaking the "decision-making dissertation" are required to have taken ED 506 Doctoral Research Methods; two additional research methodology courses; ED 520 Introduction to Program Evaluation; and EDU 515: Decision-Making for Educational Leaders I; and EDU 516: Decision-Marking of Educational Leaders II.

Goals of the Policy Formulation Dissertation
• To teach students to be critical appraisers of educational research and to teach students the advantages and limitations of using research for decision making;
• To develop students who are skilled producers of "knowledge" i.e. information, useful for decision making;
• To develop students who have a demonstrated capacity to make an argument (normative, empirical, prudential) in support of a decision;
• To encourage students to be reflective practitioners and practitioners who are capable of using reason to solve problems;
• To provide students the opportunity to demonstrate their mastery of leadership and managerial skills (presumably) taught in a course [the idea here is analogous to the final recital or performance of a music student]. This goal would clearly need further specification regarding the leadership and managerial skills we’d want to see demonstrated; and
• To permit students to undertake a project which has relevance to their employers.

Dissertation Requirements
The problem must:
• Effect a large number of people and/or involve a significant budget expenditure;
• Reflect an issue that involves a choice regarding possible change in direction; and
• Have a component of risk and/or uncertainty which should be assessed and discussed. This discussion should reflect an understanding of probability and risk and techniques of assessment.

Dissertation Elements
The following elements should be addressed or included:
• Identification and analysis of the problem;
• The political dimension of the problem needs to be outlined and assessed. If necessary, a political strategy should be discussed;
• A statement about the general context of the problem and how it arose;
• A statement regarding the legal and ethical background relevant to the problem;
• Must involve a component of problem framing discussion or alternative ways that the problem could have been framed, why various alternatives were rejected, and the reason one used was chosen- reflecting on “frameworks” that were used and rejected in formulating the problem;
• Must include a review, synthesis and analysis of relevant research;
• Must include a statement of goals and objectives and a discussion explaining, analyzing and justifying the goals and objectives. Any political, legal or other considerations that affected the formulation of the educational goals should be discussed;
• Generation of criteria for decision (what was used and what was rejected, description of the analysis that went into this, justification of the criteria used);
• Must include a discussion regarding the search for alternatives including constraints affecting the search for alternatives (how conducted, justification of the search process, specification of alternatives);
• Generation of the results of analysis of alternatives in light of objectives (techniques for analysis, justification of ranking schema, data sources explained and commented upon, risk assessments and how they were accomplished, how risk was used in assessing the alternatives, political assessment of the alternatives – how various stakeholders would rank the alternatives and their “power”);
• Discuss the political obstacles regarding getting the final decision accepted and implemented;
• Involve a final presentation of 40 minutes; and
• Must include a concluding essay which reflects on the experience, lessons learned, and point toward needed continuing professional development. In this section, students draw on the kind of materials taken up in Decision Making II.