12 Accountability and Critical Thinking in K-12 Education

In The Educated Mind: How Cognitive Tools Shape Our Understanding, Kieran Egan (1997) concludes with this perspective:

Evolution has not equipped us ideally for the educational tasks required by advanced literate societies. We are equipped intellectually for the condition of small nonliterate social groups sharing unquestioned ideologies and images of the cosmos. Our preparation for such groups is only too evident despite our educational assaults on our young, and helps to explain why we have such difficulty and pain in expanding our understanding into and through adulthood. We have to adapt our undifferentiated learning capacity to deal with much more complex and flexible learning than it has been evolutionarily shaped to handle. We cannot tinker with the “hardware” supplied to us by evolution, so we have to adapt the “software” of educational programs in order to subvert the natural constraints on our intellectual flexibility. (278)

Policy-developers work with policy-“makers” (elected officials) in ministries of education. They influence the “software” of K-12 education programs by helping to develop policies regarding assessment, accountability, curriculum development and implementation, and teacher and principal leadership development.

Policy-developers want to create educated citizens as graduates of their K-12 public learning systems. Policy-makers believe that parents, community members, business leaders, and citizens, as well as student leaders and educators, want to develop young people who are truth-seeking, open and curious, self-confident in their critical thinking skills, and thoughtfully mature in their judgments (Facione, Facione, and Giancarlo-Gittens 2000, 23). There is broad agreement that these are the dispositions needed for contemporary citizenship as well as for productive, civil communities and knowledge economies. In Canada and the United States, thinking skills and dispositions are crucial for students participating in modern multicultural and multilinguistic societies where diversity of background, language, culture, and orientation is a way of life.

But policy-makers face many challenges when they try to translate these — and other lofty thoughts — into action. One of the crucial challenges is described by Levin (2001b):

Governments are particularly susceptible to issues that take on public salience through the media. As most people get their information about public events from the mass media, an issue that is played up in the media often becomes something that a government must respond to, even if the issue was no part of the government’s policy or plan. Media coverage is itself motivated by a number of considerations, but long-term importance to public welfare is not necessarily one of them. Indeed, novelty is an important requisite for the media in order to sustain reader or viewer interest, so that governments are likely to be faced with an ever-changing array of issues supposedly requiring immediate attention. (5)

Despite these pressures, Levin (2001b) is optimistic about policy-making, suggesting that the situation has improved because of “the changing nature of the political process… Three particularly important, developments concern the growing importance of public debate, growing importance of research and evidence, and the growing understanding of the importance of implementation and adaptation” (8-9).

According to Levin, we can make better public education policies by focusing on that which matters. We should avoid fads and pay attention to central issues that are well researched and sustained over time. We should share strong public-policy ideas through think tanks and the productive use of the media (to make sure that evidence and ideas get into the public arena). We should build links with users, politicians, civil servants, community organizations, professional organizations, and foundations, at all stages of idea development. And we should ensure ongoing discussion about what research should be done, how it should be done, and its outcomes and conclusions.

Some hardheaded realism on the part of researchers and analysts is required, including a willingness to understand and accept the realities of government. If we are to take seriously the constraints and requirements of political action, we improve our chance to bring the increasing knowledge about better schooling to bear on policy. (Levin 2001b, 11)

If this hardheaded approach is kept in mind, policy-developers and critical thinking researchers are well positioned to successfully apply expanding knowledge about critical thinking theory and evidence-based thinking to all aspects of Canadian learning systems. Policy-developers see critical thinking as a cornerstone of our elementary and secondary learning programs and of our development of an educated citizenry. They want to build learning systems that develop thoughtful democratic citizens. Critical thinking theorists and practitioners have created workable strategies for developing thinking and a useful research base from which developers can draw. But, in the process, one must heed Levin’s points and recognize that the current era is one of citizen scrutiny, media desire for novelty, and taxpayer, citizen, and political demands for accountability.

The situation might be summarized as one which requires a reconciling of competing points of view. Policy-developers want to create an educated citizenry. They want to give the public information that demonstrates that the K-12 system is working well. The public wants to be sure that they are getting good value for their education tax dollars; that their young people are attending productive and caring public schools. Academics and teachers want to develop thoughtful, democratic learners who are able to think critically, creatively, and imaginatively.

The tension between these compelling but sometimes competing desires — accountability for investment, assurance of productive and caring schools, and development of critical thinkers — is played out in a North American landscape inundated by large-scale and increasingly high-stakes testing programs. In the minds of the public (and often the media), testing programs equate with accountability. Increasingly, some policy-developers fear this trend, believing that the testing drive is leading policy-makers away from a focus on a high-quality thinking curriculum and towards a narrower definition of excellence and accountability.

In an era of high accountability and a focus on test performance as a system measure of educational success, both policy-developers and educators must struggle to find ways for practitioners at the school level to strengthen their focus on thinking. If there is agreement that critical thinking is necessary for democratic societies and for the individuals who live in them, then the question becomes how education policy-developers can ensure that critical thinking is at the centre of all their work? Even more challenging is this question: How can policies, once developed, contribute in a culture characterized by intense media and public scrutiny, a national mood of searching for certainty and security, and a national demand for transparency in decision-making and accountability for the expenditure of public funds?

Such questions must be asked and answered in a manner which recognizes that the needs of all learners must be addressed. Accountability must mean accountability to all segments of society. Therefore, one must ask, “Are all the learners in the system acquiring the literacy, mathematical problem-solving and citizenship skills and commitments they need to engage in lifelong learning, thinking, and civic participation?” and, “Are all the learners in the system, regardless of geographic location, family background, gender, orientation, language or culture, acquiring the dispositions and skills of critical thinking at the highest possible levels?”

Accountability and Intelligent Assessment Practices:
The British Columbia Approach

To answer questions of accountability and questions of assessment, one must recognize that they are deeply intertwined. The province of British Columbia’s key policy-developers have adopted an approach to accountability which operates through the following five connected initiatives:

1. Some large-scale assessment for purposes of system accountability.
2. A focus on intelligent classroom assessment.
3. An inquiry-based school and district review process that values evidence and critical thinking.
4. A reduction in the number of mandated curriculum outcomes and the development of a thinking-focused curriculum.
5. A focus on school improvement leadership that connects motivated teacher and principal leaders in an inquiry-based, active research community.

These five initiatives have been shaped by a distinction among three different kinds of assessment described by Earl and Katz (2006) in Rethinking Classroom Assessment with Purpose in Mind: Assessment for Learning, Assessment as Learning and Assessment of Learning. An overall approach to evaluation and accountability must place a priority on all three forms of assessment: large-scale assessments of learning to ensure that all learners are obtaining the levels of foundational learning success they need to participate in society; classroom-based assessments for learning with clear intellectual standards that help students and their teachers to see what constitutes clear, critical thinking and performance in the discipline being studied; and learner-based assessments as learning whereby individual learners are helped to reflect on their thinking processes in order to become more proficient and self-aware in their critical thinking.

Large-scale assessment

British Columbia policy-developers accept the importance of some large-scale assessments of learning. Provincial assessment leaders work collaboratively with academics and teams of teachers to ensure that the assessments used at Grades 4, 7, 10, 11, and 12 reflect the importance of thinking critically. At the same time, they are realistic about the constraints of time and format in capturing the richness and depth of student thinking. Consequently, the data provided by provincial assessments form some, but not the only or most important, pieces of the assessment puzzle.

With such limitations in mind, policy-developers and policy-makers agree that the results from large-scale assessment are useful in focusing attention on areas of system failure. In British Columbia such results have highlighted the failure to ensure that aboriginal learners succeed at high levels. Years of individual professional judgment and individualized classroom assessment have not drawn sufficient public attention to the significant problems of the province’s aboriginal learners. One positive result of the evidence provided by provincial assessments in literacy and numeracy has been a growing collective demand for focused attention at all levels — province, district, school, and classroom — on the improvement of learning for aboriginal students and their families.

Classroom assessment

Creating the right classroom environment for developing young people with critical thinking dispositions requires the sustained effort of teachers and principals working together in thoughtful teams. Daily assessment, close observation, careful design of learning opportunities, and regular and thoughtful “close-to-the-action” descriptive feedback are school conditions needed to create strong thinkers. These are the for and as learning assessment practices that British Columbia has adopted in setting overall policy directions for accountability, assessment, and school improvement.

The emphasis on assessment for and as learning must be thoughtfully linked to students’ self-assessments in classroom and school practice in order to help learners self-assess and avoid the difficulties Kruger and Dunning (1999) point to in their article “Unskilled and Unaware of It: How Difficulties in Recognizing One’s Own Incompetence Lead to Inflated Self-Assessments.” If the dangers for critical thinking inherent in inappropriately high self-assessments are to be avoided, it is crucial that high standards of thinking performance be taught explicitly. Once high intellectual standards have become a regular feature of the classroom environment, the metacognitive skill involved in thoughtful self-assessment can be applied from a strong base of understanding. As students develop as critical thinkers using the strengths of assessment for and as learning, they will become self-critical, which is one of the most important traits of the critical thinker (because it is the key to improvement).

Humanity needs all the metacognitive power that can be mustered and brought to bear on the decision-making and thinking tasks involved in addressing a host of serious global problems. Policy-developers in British Columbia believe that the daily use, in the classroom and out-of-school settings, of high intellectual standards built into criteria in thoughtfully constructed assessment for learning tools can play a significant role in keeping critical thinking at the centre of the learning enterprise.

The Accountability Model:
Encouraging Thinking Through Inquiry

Policy-developers and educational practitioners in British Columbia are in substantial agreement on the importance of all three forms of assessment in creating a framework for the development of thoughtful learners. In creating models of assessment, they have also addressed the question of how an accountability policy (designed to help government and the citizenry know what they are getting for their public investment) can be combined with an inquiry model of education designed to highlight the importance of critical thinking and evidence-based research for schools.

The argument for an inquiry-based approach is straightforward. There is a desire for educators to engage in the development of critical thinking for themselves and for their students. Assessing schools and districts, therefore, needs to be done in the context of a model that publicly values both inquiry and critical thinking as ways of assessing intelligent learning systems. A belief in the centrality of critical thinking, in the importance of evidence-based decision-making, and in the value of both quantitative and qualitative inquiry has produced an approach to accountability that is based on a district and school review. A “ten points of inquiry” model is used to assess effectiveness in focusing on the continuous improvement of learning (see Appendix)

One key component of the “ten points of inquiry” model is its insistence that the evidence proffered for learning gains must include thoughtful, classroom-based assessment information. In this way, the review process actively discourages over-reliance on large-scale testing measures as the most important indicator of learning results. To ensure a broad appreciation of all the issues of assessment, a critical thinking disposition frames the development of the training of review team members. This model was developed as a genuine effort to point policy, political, and educational leaders at the provincial, district, and school levels in the direction of thinking critically about testing as a means of improving public education.

Right-sized assessment

In developing policies like those in British Columbia, policy-developers face a number of challenges. Developers of accountability systems, including those based on inquiry, need to be vigilant about issues of system flexibility. Analysis of global practices done by the BC network leaders found that politicians and policy-makers were unlikely to back away from the existing assessment approach after their jurisdictions made a sizeable investment in extensive amounts of annual, large-scale, standardized testing. Politicians, policy-developers, parents, communities, and educators are interested in trends-over-time data and this shared interest acts as an impediment to changes to the testing regime, even when these changes would make the testing more effective (see Kaser and Halbert 2004).

As new and better forms of large-scale and in-class assessment are created, it becomes difficult to add them into an already developed program of large-scale testing. This is all the more so because think tanks and the media enjoy using testing data to create lists of winners and losers among schools and districts. Despite their limitations, there is a public appetite for such rankings and politicians are pressed to act within the time frame of their electoral mandate. Critical thinking theorists, as many contributors to this volume have expressed, want time to develop stronger assessment measures that truly capture the development of thinking dispositions and skills, but this need and desire must compete for the interest of the public and policy-makers.

Policy-developers must keep in mind the public’s appetite for information. It can create a very challenging context for policy-developers who want to ensure their system is an open-source one, where there is room in the overall assessment and evaluation systems for a critical thinking perspective as knowledge about cultivating thinking deepens. In circumstances like this, if one is to work persuasively it is important to keep any large-scale assessment program “right-sized.” Such an approach provides enough large-scale assessment information to allow thoughtful decisions about the allocation of resources (to create high measures of equality and quality of outcomes), while ensuring at the same time that standardized testing is minimized so that there is enough time and energy for classroom assessment, and so that resistance to new assessment initiatives might be avoided.

In British Columbia, large-scale assessment is criteria-referenced and is intended to measure curriculum-based reading, writing, and mathematical problem-solving. Such assessment occurs in Grades 4, 7, 10, 11, and 12. Viewpoints will vary on the right amount of large-scale assessment, but the British Columbia program of testing, when judged by world standards, is relatively light.

Right-sized “stakes”

A second challenge for policy-makers is the need to keep the “stakes” for students and schools within a moderate range. Without enough consequences for individuals and schools, the public will not be confident that students are learning the key skills for participation in society. But governments must ensure that the consequences are not so heavy that they cause a learning system distortion by forcing an emphasis on test performances rather than “close-to-the-learner” thinking assessments.

The information that is provided from large-scale assessments must yield publicly a direction for the intelligent allocation of resources. In Canada, the policy support for allocation of resources to learners exhibiting vulnerability has been assisted by a community of researchers who have examined before-school indicators of child and family health and learning. Politicians of all stripes and at all levels of government have been informed of the resulting body of evidence, which shows that an early investment in supporting learners to assist them out of their vulnerability and into learning confidence is a wise investment for both social and economic policy reasons. People working in school communities know first-hand that it is easier to teach thinking in classrooms when vulnerability issues have been addressed proactively and in a preventative manner. As a result of this growing and evidence-based understanding, investments are increasing in the development of high levels of early success — in oral language, in listening to understand, in writing, in quantitative understanding, in reading critically, and in social responsibility. This investment is a tangible recognition that the development of the intellect in preschool and primary years is a critically important part of the development of young thinkers and of successful critical thinking learning in the intermediate and secondary years.

Classroom-level assessment

Classroom-level assessment poses a third challenge for policy-developers in the evaluation arena. Systems that have attempted to “close learning gaps” in low-performing jurisdictions through high levels of system-wide testing and inspection have shown mixed learning results. Some systems with policies of high testing loads have also found disturbing side effects in the form of teacher discouragement leading to difficulties in teacher retention and recruitment (as Giancarlo-Gittens argues in this volume). A policy environment that reduces intellectual capital in schools is unlikely to sustain thinking for students.

The international evidence on school improvement suggests that policy-developers who want a healthier thinking culture should develop an intense and systemic focus on the principles of informative assessment — assessment for learning and assessment as learning. Fortunately, new and more powerful forms of assessing thinking are being developed. These forms build on the assessment insights of Black and William (1998) in the United Kingdom, Earl and Katz. (2006) in Canada, and Stiggins (2002) in the United States.

As new areas of research knowledge and imaginative education develop, they provide powerful new teaching forms for educators. In such a context, it is critically important that assessment systems be flexible enough to capture the important new learning they create. This flexibility can be accomplished in two important ways: by making the large-scale system light enough so that there is room for emerging knowledge, and by making local system assessment at the classroom and school levels robust enough to capture the new learning in rigorous and compelling ways.

A broader challenge is to reinforce moves in this direction by ensuring that the larger community — the province, state, or country — responds quickly to intelligent, evidence-based work at the international level. This can prevent less-informed systems from becoming “laminated” or “hardwired” into the “brain” of a computerized information system that is integrated with the education system. Instead, the overall assessment system needs to have the qualities of a healthy and sustainable ecological system — with a lot of diversity and natural experimentation. Hardwired, large-scale, high-stakes assessments are problematic because they can work to lessen, or even prevent, diversity.

Overall, a too dominant test- and technology-dependent approach may diminish thinking capacity rather than build it. The “one best test” approach needs to be replaced with a powerful open-source assessment community with high standards of thinking at every level. British Columbia continues to explore the testing-assessment balance through open debate, review, and ongoing inquiry. The evidence from district review recommendations and follow-through as well as network growth in size and impact suggests that districts are beginning to move towards greater assessment balance. It will be important to sustain this encouraging direction.

Local review and assessment

If the need for an open-source, high-standards-of-thinking approach stands up to scrutiny, then it follows that policy-developers need to encourage the building of powerful models of local assessment as forms of distributed thinking in all parts of the learning system (schools, colleges, universities, institutes, districts, regions, and professional networks). These local implementation models can be seen as a series of natural experiments that can be critically evaluated. Technology can be used to bring the strongest models to professional and public awareness through video presentations and video journals that engage learners, teachers, and principals.

In British Columbia, standards of intellectual performance using criteria for high levels of thinking, writing, reading, math problem-solving, physical health decision-making, artistic inquiry, and citizenship/ social responsibility are being developed by and with local teachers. The inquiry-based implementation of these standards is making its way into province-wide communities of professional practice. Case and his associates have been instrumental in building a thinking community though their work with “TC2” — a “Critical Thinking Cooperative” which includes educators from several school districts, faculties of education, and professional teacher associations.

A number of other local groups are also at work within the province. Active inquiry communities studying early and later literacy success have been established in every district. The Network of Performance-Based Schools is a geographically distributed group of schools committed to critical thinking, inquiry, research, and the publication of their findings. They have consciously been developed as a “third space” where teams of educators can think about making classroom assessment as thoughtful, reliable, and valid as possible. Like the other networked groups, they share the conviction that thinking criteria (such as the scoring guides embedded in the British Columbia performance standards for writing, mathematical problem-solving, and social responsibility, or in the Case and colleagues critical thinking scoring guides) must be shared with learners, their families, and their communities.

Curriculum Design and Implementation

There are many ways in which policies shaping provincial, regional, and national curriculum can play a key role in the development of critical thinking by all students. Many jurisdictions are seriously considering the importance of redesigning curriculum by reducing the number of outcomes in every grade and in every discipline. A “thoughtful outcomes” reduction process allows teachers and learners to shift the emphasis from a focus on covering a large number of knowledge outcomes toward more time for, and a greater emphasis on, thinking more deeply about key ideas and important questions. An “unstuffed” curriculum is important to young thinkers and their teachers.

The critical thinking curriculum must emphasize the development of dispositions as well as skills. As Facione (2000) points out,

[s]kill and disposition are two separate things in people. Employers and educators prize both (Facione, Facione, & Giancarlo, 1996). A developmental perspective suggests that skills and dispositions are mutually reinforcing; and, hence, should be explicitly taught and modeled together (Kitchener & King, 1995 [sic]). Common sense tells us that a strong overall disposition toward critical thinking is integral to insuring the use of critical thinking skills outside the narrow instructional setting. Motivational theory (Lewin, 1935) provides the theoretical grounds for the assumption that the disposition to value and utilize critical thinking would impel an individual to achieve mastery over critical thinking skills, being motivated to close the gap between what is valued and what is attained. (2000, 32-3)

A curriculum is more likely to build dispositions and skills if it emphasizes key questions, rich tasks, and assessment indicators that are built into it in each discipline. Implementation practices that actively encourage dialogue and debate through lesson study and the shared assessment of student work samples — including the use of photography and video clips to capture student dispositions, skills, and metacognitive language — form important aspects of a culture of critical thinking. Ensuring that every curriculum and assessment document is examined from a critical thinking perspective is vital if critical thinking standards are to become a way of life-long learning.

Effective curriculum development must be backed by staff development that values and models critical thinking. How staff development is conducted must demonstrate thinking dispositions and skills. If a learning system is to have a thinking disposition, then its staff-development models must value thinking. High levels of thinking cannot be expected of teachers who are restricted to scripted, directive staff development. Thoughtful staff development can be strengthened by linking professional school-level communities of practice with university-based researchers and educators.

Leadership as Distributed Critical Thinking

The leadership development of educators can contribute to a learning environment conducive to thinking skills and dispositions. One important aspect of this work is the selection, development, and supporting of new teacher and principal leaders. If the goal is thoughtful students who are disposed to using reasons and evidence and who can demonstrate maturity of judgment, then there is a need for teacher and principal leaders who can work cooperatively to turn their schools into thinking communities. Preparation of and ongoing support for school leaders need to be designed with this in mind. Leader teams working at the classroom and school levels need to be supported through reasonable investments of human and fiscal resources.

The model for these programs needs to be built on a strategy that applies research evidence to practice. As Spillane, Diamond, and Jita (2003) point out, most teachers will have to be introduced to instructional reforms and supported at the school level. Two challenges arise in such a context. The first is making sure that reforms are not seen only in a few pilot schools of enthusiasts for instructional innovation. The second is the challenge of ensuring substance and depth of thinking. Even with these challenges, all reforms must be enacted in ways consistent with the spirit and dispositions of critical thinking.

The research of Spillane and his colleagues (2003) examines how leadership thinking and acting are distributed across the learning community. In their model, a distributed practice of leadership is “stretched” over multiple leaders, knowledge sources, and activities over time, creating a leadership group which has “cognitive properties that exceed those of any one member” (5). This suggests that we need to understand leadership practice at the collective rather than just the individual level. An effective commitment to critical thinking education will need to have distributed critical thinking skills and dispositions.

To the extent that leadership extends beyond the mind of the individual, it is even more important that those in the leadership group demonstrate the highest levels of critical thinking. They must be able to distinguish among competing claims for attention and bring accuracy to their interpretation of evidence; identify a rationale for action; evaluate major alternative viewpoints; draw warranted and reasoned conclusions; justify key results and strategies; and fair-mindedly follow evidence and reasons where they lead. In demonstrating these traits, the leadership group must be able to exhibit mature professional judgment in a work environment characterized by intense time pressures and immediacy of action, as well as in a political environment characterized by a powerful desire for short-term, quickly achieved results.

Policy-developers need to ensure that the distributed leadership capacity, characterized by a thinking disposition and a powerful set of thinking tools, is supported through the development of a sustained leadership learning program. International research suggests that such a program can best be developed through multiple partnerships — with pre-service, mentoring, and in-service programs that are linked and of the highest quality. Small group contact, technology-enhanced links, and ongoing study and practice are the best ways to develop thinking leadership with the highest standards of intellectual practice in action.

The development of a good leadership program requires cooperation from practitioners, universities, policy-developers, and governments. In British Columbia, the evidence suggests that this cooperative institutional teamwork is more likely to develop with a policy-based blend of incentives for connection and negative consequences for isolation. Both pressure and support are needed — support for communities of practice across institutional boundaries, and pressure for change for institutions that prefer to stay within their traditional territorial boundaries.

One example of this positive interdependency is found in the United Kingdom, through links between the National College for Leadership, the Higher Education Institutions, and the school-based Networked Learning Communities. International school improvement researchers suggest that other examples can be found, including in Finland and Taiwan, where there are university-practice communities which share a thinking-based teaching/ learning worldview characterized by frequent and ongoing collaboration, and the study and refinement of practice. Strong evidence suggests that this is the kind of big system culture that needs to be developed if learning and thinking skills are to be instilled in all students.

The next step in British Columbia policy development is the study of successful examples of leadership development from the international learning community and the attempt to use those examples to shape a contextually appropriate teacher/principal leadership program. In keeping with the commitment to inquiry and critical thinking, it is critically important that policy-developers and policy-makers study the international evidence of thoughtful performance in cultures with high equality and quality outcomes and then bring the evidence to bear on their own decision-making.

Conclusion

I have tried to sketch a policy-developer’s view of the issues that confront critical thinking initiatives in K-12 education. In British Columbia, the policy development community is committed to:

• an interlocking set of policies that create a culture of intelligent classroom-based assessment of thinking in core disciplines;
• an accountability approach that relies on inquiry and critical examination of a range of evidence and improvement practices;
• an implementation approach that encourages the formation of face-to-face and virtual extended learning communities characterized by the qualities of critical thinking and a more focused and thoughtful curriculum; and
• a concentrated focus on developing and supporting critical thinking leadership, as the most promising way to create an environment for learners that encourages each one to think critically.

The province’s aspiration is to have both its learners and its democracy become the beneficiaries of a systemic orientation to critical thinking as a way of life in schools.

References

Bailin, S., R. Case, J. Coombs, and L. Daniels. 1999. Common misconceptions of critical thinking. Journal of Curriculum Studies 31(3): 269-83.

Black, P., and D. William. 1998. Inside the black box: Raising standards through classroom assessment. Phi Delta Kappan 80(2): 139-49. Online. Available from http://wvvw.pdkintl.org/kappan/kbla9810.htm.

Case, R., and L. Daniels. 1994-2004. Critical challenges across the curriculum. Vancouver, BC: Pacific Educational Press.

Earl, L. 2004. Assessment as learning: Using classroom assessment to maximize student learning. Thousand Oaks, CA: Corwin Press.

Earl, L., and S. Katz. 2006. Rethinking classroom assessment with purpose in mind: Assessment for learning, assessment as learning and assessment of learning. Western Northern Canadian Protocol. Online. Available from http://wwvv.wncp.ca/.

Egan, K. 1997. The educated mind: How cognitive tools shape our understanding. Chicago, IL: University of Chicago Press.

Ennis, R. 2000. An outline of goals for a critical thinking curriculum and its assessment. Online. Available at http://www.criticalthinking.net/goals.html.

Facione, P., and N. Facione. 1994. Holistic critical thinking scoring rubric. Millbrae, CA: California Academic Press.

Facione, P., N. Facione, and C. Giancarlo-Gittens. 2000. The disposition toward critical thinking: Its character, measurement, and relationship to critical thinking skill. Journal of Informal Logic 20(1): 61-84.

Facione, P., Facione, N., and C. Giancarlo. 1996. The motivation to think in working and learning. In Defining expectations for student learning, ed. E. Jones. San Francisco, CA: Jossey-Bass.

Hautamaki, J. 2002. Assessing learning-to-learn: A framework. Helsinki: Centre for Educational Assessment.

Huber, G. 2004. Preparing school leaders for the 21st century: An international comparison of development programs in 15 countries. London: Routledge Falmer.

Kaser, L., and J. Halbert. 2004. Networks of inquiry for school and district improvement. Paper presented at the International Congress of School Improvement and Effectiveness, Rotterdam, The Netherlands.

King, P., and K. Kitchener. 1994. Developing reflective judgment. San Francisco, CA: Jossey-Bass.

Kruger, J., and D. Dunning. 1999. Unskilled and unaware of it: How difficulties in recognizing one’s own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology 77(6): 1121-34.

Levin, B. 2001a. Reforming education: From origins to outcomes. London, UK: Routledge Falmer.

Levin, B. 200 lb. Governments and school improvement. International Electronic Journal for Leadership in Learning 5(9): 1-13.

Lewin, K. 1935. A dynamic theory of personality: Selected papers. Translated by D. K. Adams and K. E. Zener. New York: McGraw Hill.

Mintzberg, H. 2004. Managers, not MBAs: A hard look at the soft practice of managing and management development. San Francisco, CA: Berrett-Koehler Publishers.

Reynolds, D., B. Creemers, S. Stringfield, C. Teddlie, and G. Schaffer. 2002. World class schools: International perspectives on school effectiveness. London, UK: Routledge Falmer.

Spillane, J., J. Diamond, and J. Jita. 2003. Leading instruction: The distribution of leadership for instruction. Journal of Curriculum Studies 36(1): 1-14.

Stiggins, R. 2002. Assessment crisis: The absence of assessment for learning. Phi Delta Kappan 83(10): 758-65. Online. Available at http://www. pdlcintl.org/kappan/k0206sti.htm.

Willms, J., ed. 2002. Vulnerable children: Findings from Canada’s national longitudinal survey of children and youth. Edmonton: University of Alberta Press.

Appendix
British Columbia Ministry of Education District Review

The ten points of inquiry for use at the school and district levels are:

1. What one or two important goals have you set for improving learning?
2. What rationale have you used to set them — from evidence sources and your own critical thinking?
3. What evidence have you used and will you use to guide your improvement work?
4. What set of strategies are you using — strategies based on blending research evidence, emerging thoughtful practice, and innovative thinking?
5. What organizational structures are you changing to make your improvement work more powerful?
6. How are you making sure your work is coherent?
7. How are you informing your community about your work?
8. How are families and parents involved?
9. What are you doing to share and develop leadership at all levels?
10. What important learning gains are you making?