Pennsylvania State University
200 University Drive
Schuylkill Haven, PA 17972
(570) 385-6267
stonej@psu.edu
The need for a technologically literate workforce is a defined public problem in the United States. Consequently, federal and state governments have been actively funding policies designed to build technological literacy or so-called "21st Century Skills" among K-12 students. The lack of a uniform definition for technological literacy, an absence of student assessment, and the decentralized nature of the K-12 system have made it difficult to determine if these policies and programs (federal and state-level) have been effective. The study discussed in this article uses a mixed-methods approach to examine the implementation of one state-level technological literacy policy: Pennsylvania's Classrooms for the Future (CFF) program.
Public education represents the largest public enterprise in the United States, both in terms of public investment and bureaucracy. Given the complex, loosely coupled, and diverse nature of the educational system, investigations of educational policy implementation require a consideration of street-level characteristics and environmental context. Efforts to build technological literacy in public schools require understanding and support from so-called "street-level bureaucrats"; i.e., the teachers and administrators responsible for policy implementation. Existing implementation research suggests that successful implementation requires street-level attitudes, perceptions, and behaviors commensurate with policy goals.
Pennsylvania's CFF program has involved over $150 million in state expenditures, but little is known about how the policy is actually being implemented. The purpose of this study is to begin filling in this information gap by analyzing how CFF is implemented in one Pennsylvania Intermediate Unit (IU). This study focuses on the implementing agents - school district teachers and administrators. The attitudes, behavior, and perceptions of street-level implementers are recognized as being of fundamental importance to policy implementation. Given that the CFF program intends to make a dramatic shift in public education practices, consideration of street-level context and implementer characteristics is critical to understanding CFF implementation. The fundamental question addressed by this study is as follows: How are state-level technological literacy policies implemented in context?
This educational policy implementation study investigates the local-level policy context, outputs (activities) and implementer characteristics (attitudes, perceptions, and behaviors) as they apply to the CFF program. This study uses a modified "bottom-up"-style approach to policy implementation research by investigating a specific educational policy through the lens of street-level bureaucrats (teachers) and public managers (school and district administrators) with considerations of environmental context. The information obtained from this study is expected to provide insight into: (1) how a state-level technological literacy policy is implemented in context, and (2) how the attitudes, behavior, and perceptions of street-level implementers and public managers affect the implementation of technological literacy policies.
First, the article examines the conceptual and practical background of technological literacy and related policies in the United States, with a special focus on Pennsylvania and the CFF program. The article then proceeds to describe a research design used for investigating the implementation of CFF in a set of eight school districts. Finally, the results of this study are reported and discussed.
There is no single definition of technological literacy. The definitions are said to be constantly evolving due to rapid technological change (Patterson, 2005). A commonly held misconception is that technological literacy equates to simply the ability to use computers or other information technologies proficiently (Weber, 2005). While the use and application of computers and information technology is a critical component, technological literacy is an educational outcome that involves more than just computer-based abilities. Technological literacy includes "the ability to use, manage, understand, and assess technology" (Emeagwali, 2004) and is said to involve three interrelated dimensions: knowledge, ways of thinking/acting, and capabilities. These dimensions are intended to permit individuals to make informed and intelligent decisions about technology and the world in general, thus enabling them to be productive members of modern society (Pearson & Young, 2002).
The traditional staples of literacy - reading, writing, and arithmetic skills - are necessarily foundations for technological literacy, as is the ability to proficiently use information and computer technologies. Technological literacy can be considered a new form of literacy that reflects the complexity of the Information Age and the global economy. The aptitudes that make an individual technologically literate are often called "21st Century Skills." These skills are based on the ability to use, manage, interpret, validate, and synthesize information. In the 21st century, information is increasingly consumed, synthesized, and disseminated via electronic means. The proficient use of information and computer technologies is therefore a necessity for both academic and professional success. Technological literacy sees information technologies as only one element used to solve real-world problems. Modern problems may be simple or complex; span disciplinary, geographic, and cultural boundaries; and necessitate collaboration, creativity, and innovation. Technical skills must therefore be complemented with traditional "soft skills" such as creativity, cultural awareness, and leadership in order to effectively solve real-world problems. The modern conception of technological literacy therefore stresses technical, cognitive, and interpersonal skills rather than simply the memorization of knowledge. A list of these "21st Century Skills" can be found in Table 1.
Fundamental to the concept of technological literacy is the ability to think critically and solve problems. Such skills are important as computerization has raised the demand for these non-routine abilities (The Partnership for 21st Century Skills, 2008). Real-world problems are often open-ended, abstract, multidisciplinary, and require innovative solutions. A technologically literate individual would have the ability to create and/or identify innovative solutions to these problems. The problem-solving process would be enabled by information and computer technologies and would necessarily consider the multiple stakeholders, potential effects, and the issues (social, political, cultural, economic, and technical) that impact the problem. Solutions would necessarily be communicated using multiple modes (e.g., oral, written) and multiple media. The modern workplace is said to place a premium on these "portable" skills, as well as the ability to apply critical judgment to a situational analysis (Wallis & Steptoe, 2006).
Technological literacy can be thought of as an amalgamation of other theoretical and applied notions of modern literacy. The ability to use, assess, and distill a large amount of information is a critical component of the problem-solving process. The term information fluency is often used to describe this component of technological literacy. Information fluency represents the intersection of information literacy, computer literacy, and critical thinking (O'Hanlon, 2002). Information literacy focuses on the acquisition, generation, evaluation, and utilization of information. Information literacy encompasses a basic knowledge of computers, including the functional use of application software and Internet search capabilities (Wen & Shih, 2006; Murray et al., 2007). Computer literacy has evolved from the original emphasis on computer applications to a focus on the use of information technologies with critical understanding (Hoffman & Vance, 2005). Another term, media literacy, has also intersected with information fluency. Media literacy involves the ability to evaluate and critically assess information products for bias and accuracy (Minkel, 2002). Traditional sources of information have been replaced with more visual, multimedia sources, often with little or no editorial control. Media literacy emphasizes the social construction of information and advocates the skills necessary to identify the relationships between information, power, and populations (Kellner & Share, 2007).
A final aspect of technological literacy involves personal and workplace productivity skills. This includes so-called emotional intelligence and interpersonal skills, as well as time management and the ability to adapt to changing priorities and circumstances (Wallis & Steptoe, 2006). The emphasis on teamwork and the geographic dispersion of many industries means that problem-solving is often collaborative and multicultural in nature. Coordination, communication, and leadership skills are therefore essential (Wallis & Steptoe, 2006; The Partnership for 21st Century Skills, 2008). Technologically literate individuals must be comfortable in team situations, knowledgeable about other cultures, and be able to communicate in multiple media and languages. Technological literacy also emphasizes the need for lifelong learning; the rapidly changing nature of technology, industries, and geopolitical realities means that adaptation is an essential skill in the modern workplace.
A number of studies (e.g., O'Hanlon, 2002; Stone & Madigan, 2007; Hilberg & Meiselwitz, 2008) have raised concerns that K-12 graduates reach the university level with insufficient technological literacy skills. Technological literacy has become an implicit expectation that public schools, colleges, and universities have for their students. Today's recent K-12 graduates are viewed as digital natives, proficient with information and computer technologies and having the personalities, learning styles, and communications commensurate with a connected society (Prensky, 2001; Hoffman & Vance, 2005). Research confirms that information and computer technologies are integral to the daily lives of many of these students (O'Hanlon, 2002; Hilberg & Meiselwitz, 2008).
However, recent evidence suggests that while information and computer technology plays a large part in students' lives, technology usage does not necessarily translate into being technologically literate. A study by the Educational Testing Service suggests that students use computers heavily for communication tasks (e.g., e-mail, social networking, instant messaging) but lack the problem-solving and advanced software skills necessary to use computers in a technologically literate manner and, thus, achieve academic and professional success (Murray et al., 2007; Hilberg & Meiselwitz, 2008). Organization for Economic Co-operation and Development (OECD) assessments of 15- year-olds across 40 countries through the Programme for International Student Assessment (PISA) in 2006 showed U.S. students as scoring below the OECD average in both science and mathematics. These assessments measure the inquiry-based skills considered fundamental to technological literacy (Organization for Economic Co-operation and Development, 2007).
The technological literacy of K-12 and collegiate graduates is of fundamental importance to U.S. national security, economic and otherwise. For example, the Bureau of Labor Statistics (2007) reports that Science, Technology, Engineering, and Mathematics (STEM) careers will experience some of the largest salary and employment growth over the next 10 years. If the U.S. cannot produce STEM graduates with the necessary skills to enter this workforce, the U.S. national economy will suffer. Recent statistics have indicated that the number of STEM degrees granted in the United States has been in steady decline and is not keeping pace with the growth of STEM-related jobs (National Science Board, 2006, Still, 2009). These factors, coupled with comparatively poor STEM literacy among K-12 students, raises concerns that the U.S. is losing its competitive edge and its economic security (Stake & Mares, 2001). The global economy means that American workers are competing for jobs with workers in countries that invest heavily in technology education (e.g., India, China). OECD data suggests that performance on the PISA assessments is correlated with the level of GDP growth, further underscoring the importance of the "21st Century Skills" found in technological literacy. To produce the knowledge workers of tomorrow, schools must produce technologically literate students today.
Public policies at both the federal and state levels have begun to address the issue of how to build technological literacy among K-12 students. The federal government invests heavily in public education, both in terms of funding and policy construction. The U.S. Department of Education, the National Science Foundation, and other federal agencies fund, evaluate, direct, and otherwise guide public STEM education to the tune of approximately $3 billion per year. The Federal No Child Left Behind Act of 2001 (NCLB), though primarily designed to enact standards-based reform in public education, mandates technological literacy for eighth grade students. The NCLB also initiated the Enhancing Education through Technology program to provide a federal source of funding for K-12 technology, though the program was discontinued in 2011 (Devaney, 2011). The recent National Education Technology Plan has also reaffirmed the need for more integrated use of educational technology in K-12 (U.S. Department of Education, 2010). Efforts by policymakers to build technological literacy into the curriculum have been complicated by a several factors - the lack of a uniform definition for technological literacy, the lack of systematic assessment data, the myriad state standards and local-level curricula, and a dearth of policy leadership at the federal level.
The individual states have been the primary policy actors in building technological literacy among K-12 students. States have been working to share practices, to form a common agreement on what technological literacy actually means, and to devise the best methods of assessment. Despite the federal mandate of technologically literate eighth grade students, evidence supporting progress in this area is almost nonexistent. The overall lack of assessment has two main causes. First, the specific reporting requirements for this mandate have not been defined by the U.S. Department of Education, leaving many states hesitant to invest in assessment tools for technological literacy. This is especially true as NCLB has already imposed substantial data collection, reporting, and assessment burdens on the states for other subjects. Second, the lack of a federal definition of technological literacy has led to a myriad of competing standards for technological literacy (e.g., the International Society for Technology in Education, International Technology Education Association).
One method by which states and local school districts have attempted to build technological literacy into the K-12 curriculum is through so-called "one-to-one" programs. These programs provide a laptop to every student in the hopes of encouraging teachers to build instructional processes more appropriate for teaching 21st Century Skills. These programs use technology to encourage fundamental changes in traditional teacher and student roles. The shift from traditional pedagogical methods to more collaborative, inquiry-based approaches creates a partnership of sorts in the student-teacher relationship. Students assume more responsibility for their learning and work in teams to solve complex, real-world problems. Students benefit through increases in collaborative learning, individualized instruction, and interdisciplinary experiences, as well as increasing self-esteem, respect, and self-confidence. Teachers see their roles shift to facilitator, coordinator, and instructional designer. Expected benefits for teachers are the acquisition of new technological and pedagogical skills and an improved classroom climate (McGhee & Kozma, 2003; Fairman, 2004). Other benefits are thought to be a decrease in disciplinary problems and an increase in academic achievement (Bebell, 2005). Many of these programs include funding for both equipment and teacher training.
Transforming educational practice and learning outcomes is inherently complex, but there is anecdotal evidence that one-to-one programs can have positive impacts. One example of an apparently successful one-to-one program can be found in the state of Maine, which implemented the Maine Learning Technology Initiative (MLTI) in 2002. This pioneering one-to-one program targeted all seventh- and eighth-grade students in the state. As of 2007 the program had reached all Maine middle schools, with 37,000 students participating (Clark, 2007). The MLTI program was approved by the state legislature in 2002 and was renewed in 2006. Formal evaluations have been overwhelmingly positive, citing the aforementioned benefits of one-to-one programs (Fairman, 2004; Garthwait & Weller, 2004).
Pennsylvania's CFF program was designed to increase the ability of Pennsylvania's K-12 graduates to compete in the global economy. The CFF program had two key components. First, the state offered complete funding for the purchase of laptops and software to eligible high schools and technical schools across Pennsylvania. Second, the state provided teachers with the professional development, training, and technical support necessary to fully take advantage of technology within the classroom using modern instructional "best practices." The CFF program places an emphasis on the aforementioned change in instructional processes and learning methodologies, as well as a change in classroom roles. The goal of the CFF program was to create "environments for deeper cognitive development through inquiry, real and relevant project-based learning, and differentiated instruction" and to enable Pennsylvania students to acquire the 21st century skills they need (Pennsylvania Department of Education, 2009).
Implementation of the CFF program began in 2006-2007. The program was implemented in 447 school districts as of 2009. More than 140,000 laptops were purchased via the program, resulting in an estimated participation of 500,000 high school students as of 2009. All eligible schools must teach courses in "core" subject areas (Math, Science, Language Arts, and Social Studies) and be accountable to the state via NCLB requirements. Eligible schools were funded through a competitive application process. Schools that participate in CFF are provided with funding for laptops, software, teacher training and professional development, and supporting equipment. Funding is also provided for an on-site "coach" to assist teachers in the use of equipment and software. The Pennsylvania Department of Education also provides Web-based resources to enhance curriculum development and instruction (Wagner, 2008).
Actual state expenditures for the CFF program totaled $155 million by the end of the 2008-2009 school year. The staggered implementation of the program resulted in only 30 districts and one vocational school being fully funded as of 2008. The other 417 participating districts were at various stages of funding and implementation (Wagner, 2008). The 2009-2010 Pennsylvania state budget did not provide any additional funds for CFF, though additional funding was available in some circumstances through the American Reinvestment & Recovery Act. Preliminary evaluation reports from the Pennsylvania State University noted the successes of the program (e.g., Peck et al., 2009). High school teachers involved in the program were found to be more energized and had a greater appreciation for the value of technology in their instruction. Lecture time was reduced as students engaged in more group-based, problem-based learning activities. Students were found to be more engaged and teachers were excited by the program (Wagner, 2008).
The enormity of the K-12 system, the de-centralized nature of the public education system, and sometimes competing educational priorities (i.e., NCLB) make technological literacy programs a particularly complex educational policy issue. The slow development of federal guidance and funding support has led individual states to build their own technological literacy policies and programs. While states often share experiences and best practices, the diversity of state policies complicates any attempt at any generalized evaluation of their effectiveness. The diversity of educational experiences and socioeconomic conditions across school districts further complicates these efforts. More specific, context-dependent information is needed to assess the effectiveness of state policies related to technological literacy. Studies that evaluate the implementation of these policies must take into account street-level attitudes, perceptions, understandings, and behavior as well as the context in which implementation occurs.
The CFF program intends to make a dramatic shift in public education practices by changing how students are taught and how they learn. However, capacity to implement a policy is about more than just the provision of tangible resources. Implementers must see the policy as valuable enough to force a break from established practices. Policy implementation is said to begin with the first actions and decisions of implementers; as a result, their behavior, preferences, and understandings has great bearing on understanding the implementation process (Goggin, 1986). Policy "meaning" is created in context, based on the skills, knowledge, values, and biases of implementers, as well as the environment in which implementation takes place (Yanow, 1996). Policies often contain compromise language that is intentionally vague and sometimes conflicting, leading implementers to exercise discretion (Hill & Hupe, 2002; Hill, 2003). Recognizing the understandings, perceptions, and cognitive abilities of implementers - street-level bureaucrats - is therefore essential to understanding the implementation process.
Implementing agent discretion has been recognized as an impetus for policy success or failure (Riccucci, 2005). Many studies of street-level implementation use implementer attitudes and perceptions as proxies for behavior (Kelly, 1994; Hill & Hupe, 2002; Riccucci, 2005). Consideration of implementer understandings, resources, and motives is considered important for explaining what unfolds in actual implementation (Lester & Goggin, 1998; O'Toole, 2000). The bottom-up perspective of implementation research recognizes that street-level personnel exercise discretion in implementation, and this discretion ultimately is a factor in the eventual "success" of the implementation. Consequently, analysis relies heavily on the goals, perceptions, understandings, and activities of implementation personnel (Sabatier, 1986; Matland, 1995; Riccucci, 2005).
Scholars such as Ingram (1990) and Matland (1995) moved the study of policy implementation towards contingency theories. Contingency theories recognize the importance of the environmental context in determining an appropriate implementation strategy (Chackerian & Mavima, 2001; deLeon, 1999). Contingency perspectives argue that both top-down and bottom-up perspectives can be useful, depending on the scenario (Chackerian & Mavima, 2001). The public education system in the United States is difficult to change, given the high levels of teacher autonomy and the loose coupling of schools and districts (McDermott, 2006). Pennsylvania's funding system for schools - which focuses on property taxes as the primary funding source - means that educational capacity varies significantly across Pennsylvania. As a result, the environmental context is an important factor in CFF implementation.
This educational policy implementation study employs a modified bottom-up, mixed-methods approach to investigate how Pennsylvania's CFF program is implemented in a set of eight (8) rural school districts. The purpose of this study is to better understand how one state-level technological literacy policy is implemented at the street level. By combining the bottom-up focus on street-level implementers with contingency-style concerns of implementation context, this study provides insight into how the CFF policy is implemented. Such information can have valuable implications for understanding what makes these policies and programs successful (or not) and can inform subsequent educational policy efforts in this domain.
There are three top-level research questions this study will address. The first research question is: How are K-12 Teachers Using Technology to Enhance Learning? This question is intended to illuminate the on-the-ground outputs (activities) of districts designed to build technological literacy. The second research question is: How are K-12 Teachers Adapting Their Practices Through the Use of CFF Resources? The activities of K-12 teachers (the street-level implementers) represent the actual implementation of CFF. The CFF program is designed to facilitate a transformation in pedagogical practices; if teachers are not making this transformation, the policy will not be successful. The third research question is: What Contextual Challenges Exist in CFF Implementation? Local-level context is the setting in which implementation occurs - consequently, it colors many of the activities, perceptions, attitudes, and behaviors of implementers. This includes personal, environmental, and organizational challenges, as well as the actions of administrators to encourage the implementation process.
The eight rural school districts participating in this study exist within a single Pennsylvania Intermediate Unit. These districts include 13 middle and high schools. All districts are CFF-eligible and all have participated in the state CFF program for at least one year. The study involves two distinct groups of participants in each district. The first group is the set of teachers at the individual schools ("teachers"). Teachers are ultimately responsible for using the CFF equipment and resources, and for being innovators in the implementation of CFF-style curriculum content. The attitudes, perceptions, and behaviors of the teachers will be a significant factor in whether or not CFF is successful. The second group is the combination of district superintendents, curriculum coordinators (district-level and/or school-level), and school principals ("administrators"). These agents represent the managers in this context. In terms of CFF, the leadership actions of administrators - tangible and intangible - should directly affect implementation of the CFF policy by teachers.
Data collection involved both quantitative and qualitative methods. Custom survey instruments were delivered to both teachers and administrators. These surveys were developed in consultation with the existing literature, as well as input from school administrators, teachers, and faculty colleagues. The set of specific survey questions was broken into three subsets. The first subset targeted technological literacy efforts within the school and/or district (not specific to CFF), including questions about technological literacy practices. The second subset of questions was directed at practices specific to the CFF program. The third subset of questions included individual demography questions. Questions necessarily varied by group but many of the questions were shared between surveys. Survey participants were recruited by e-mail. Each survey was delivered to participants via the SurveyMonkey online survey tool (http://www.surveymonkey.com/). The survey responses were then coded and analyzed using SPSS statistical software. Besides frequency analysis, three statistical analysis methods were utilized. The Chi-Square test was used to detect significant correlations between categorical variables. The Cramer's V test was used to verify the Chi-Square results and to measure the strength of significant Chi-Square associations. Finally, Pearson's r was used to measure the strength and direction of significant Chi-Square associations between ordinal variables.
A series of four (4) focus group sessions was also held to gain a deeper understanding of how technological literacy and CFF is being implemented in the participating districts. These sessions involved only teachers, were held within four different school districts, and involved teachers from a diverse set of disciplines and levels of CFF involvement. Each focus group session was recorded and transcribed. Focus group questions were broken into three primary subsets. The first subset of questions was targeted at technological literacy efforts within the school and/or district (CFF-specific and otherwise). The second subset of questions was directed at teachers' perceptions of CFF impacts on their teaching practices. The third subset of questions gathered information on the personal, organizational, and/or environmental factors that challenged teachers' implementation of CFF.
Focus group participants were selected by two methods. First, teachers were recruited via e-mail. Second, this researcher worked with district administrators to ensure an adequate number of participants in each focus group. The four districts chosen from the overall sample included the district with the highest amount of CFF funding, the district with the largest enrollment, the district with the highest per-pupil expenditures and revenues, and the district with the highest AYP percentages of students assessed as "advanced or proficient" for reading and mathematics among the sample districts. The choice of districts for the focus groups was based solely on teacher participation. Qualitative data from the focus groups were coded and analyzed using MaxQDA 2007 software.
A third aspect of data collection involved solicitation and review of sample project assignments provided by teachers in the participating districts. These materials were solicited prior to the focus group sessions. These classroom materials were expected to provide insight into the actual classroom activities of teachers and therefore complement the self-reported data from the surveys and the qualitative data from the focus groups. These materials were both paper-based and electronic (sent via e-mail).
The following sections describe survey, focus group, and documentation results designed to elicit the on-the-ground outputs (activities) of districts intending to build technological literacy, primarily through the CFF program. The activities of K-12 teachers (the street-level implementers) and administrators (the public managers) represent the actual implementation of CFF. By gaining this knowledge, it was expected that a clearer picture of the implementation behavior of street-level personnel would be obtained.
A total of 151 teacher survey responses were recorded for a response rate of 26.2% (N=574.7 FTE). The number of survey responses varied by question. The respondents were primarily female (68.4%, N=114) and White/Caucasian (95.6%, N=113). Most respondents had received a Masters Degree (57.6%, N=115). The teaching disciplines of respondents were also varied. Teachers from the four "core" subjects represented over half of the respondents (N=113). In terms of specific subjects, 23.0% teach Language Arts, 15.9% teach Science, 8.0% teach Mathematics, and 7.1% teach Social Studies. Prominent other disciplines included Special Education (15.9%), Physical Education/Health (5.6%), and Art, Business, Technology Education, and Foreign Languages (3.5% each).
Ninety-nine of the 151 teacher respondents were eligible to answer the CFF-related questions. Specifically, only those teachers who reported themselves as teaching in grades 9-12 were given access to the CFF-related questions. This restriction was due to the nature of the CFF program, which specifically targets Mathematics, Social Studies, Language Arts, and Science teachers in grades 9-12. Of these 99 teachers, 45 (45.5%) reported themselves as CFF coaches or users of CFF equipment. Teachers in grades 7-8 were not given access to the CFF-related questions. Teachers in grades 7-8 were given access to technological literacy questions that were not CFF-specific, as it is likely that these concepts impact their pedagogy.
Teachers were asked a series of survey questions about technology use for classroom instruction. These questions focused on three things: access to technology in the classroom, the frequency of teachers' technology use in the classroom, and the frequency of student technology use in the classroom. All questions involved a set of 16 specific technologies, ranging from traditional office software to more modern Web 2.0 technologies and interactive Smartboards. A description of these 16 technologies is provided in Table 2.
Teachers reported having access to a wide variety of technologies for classroom instruction. Traditional Microsoft Office-style tools - word processing (91.6%, N=119), spreadsheets (73.1%, N=119), and presentation software (73.1%, N=119) - were the most frequently cited accessible technologies, followed by Smartboards (71.4%, N=119), student laptops (64.7%, N=119), and streaming video (63.0%, N=119). Web 2.0 technologies such as social networking Web sites (5.9%, N=119); RSS (10.1%, N=119); Podcasting (30.3%, N=119); and blogs (33.6%, N=119) were cited infrequently. This lack of access was to be expected, as all school districts in the study make many Web 2.0 sites (e.g., Facebook, MySpace, YouTube) inaccessible to faculty, staff, and students. Some Web 2.0 technologies like Wikis (42.9%, N=119) and streaming video (63.0%, N=119) are frequently provided as internally or externally controlled options in lieu of more public outlets. Multimedia authoring software was also rarely cited (26.1%, N=119), despite modern perceptions of students as "visual learners."
Teachers were then asked to indicate the frequency of their use of these 16 technologies for classroom instruction. The frequency was measured using a six-level scale (Not Used, Daily, Weekly, Monthly, Quarterly, or Unknown). The results suggest a general lack of technology use by teachers in the classroom. Not Used was the most popular response for 13 of the 16 technologies. Not surprisingly, Web 2.0 technologies like Blogs, Social Networking, and Podcasts/Podcasting were among the most frequently cited as Not Used. Three other technologies - presentation software, word processing software, and Smartboards - were most frequently cited as being used on a Daily basis.
Responses from CFF coaches and teachers alone were slightly more diverse. Not Used was the most frequent response for 10 of the 16 technologies, but CFF coaches and teachers most often reported Daily use for Smartboards (73.3%, N=45) and for word processing, presentation, and database software. Student laptops, the core of the CFF program, were most often reported as being used Weekly by CFF coaches and teachers (31.8%, N=44), with only 15.9% reporting Daily use. The frequent use of Smartboards by all groups is not surprising, as Smartboards are one of the more currently popular educational technologies.
Two additional questions were posed to uncover how technology and technological literacy are integrated into classroom practice. Teachers were asked how often they assigned problems, projects, or assignments that required their students to use the 15 component skills of technological literacy listed in Table 1. Teachers were given a five-level scale (Never, Daily, Weekly, Monthly, and Quarterly) from which to choose. Nine of the 15 skills were most often reported as necessary for Daily projects, problems, or assignments. The other six component skills - inquiry-based learning skills, collaborative skills, basic computer skills, leadership and coordination skills, cultural awareness, and multidisciplinary thinking - were most often reported as being necessary for Weekly projects, problems, or assignments. The components most frequently reported for Daily assignments included several of the so-called "soft skills" - time management skills (74.8%, N=115), interpersonal skills (69.8%, N=116), communication skills (66.4%, N=116), and lifelong learning (63.5%, N=115) - along with critical thinking skills (57.8%, N=116) and problem-solving skills (60.0%, N=115). The results for CFF coaches and teachers were similar; nine of the 15 component skills were most often reported as necessary for Daily projects, problems, or assignments. The other six component skills - inquiry-based learning skills, creativity and innovation, collaborative skills, basic computer skills, leadership and coordination skills, and multidisciplinary thinking - were most often reported as being necessary for Weekly projects, problems, or assignments.
When asked how often students use technology to build technological literacy skills in their classes, the teacher responses were more uniform. Teachers were given a six-level scale (Never, Daily, Weekly, Monthly, Quarterly, and Unknown) from which to choose. Teachers most often reported that students use technology in their classes Weekly to build 14 of the 15 component skills. The lone exception - lifelong learning skills - was reported equally between two categories (Daily and Monthly). The component skills most frequently reported being used via Weekly student technology use were critical thinking skills (43.5%, N=115), creativity and innovation (39.1%, N=115), problem-solving and inquiry-based learning skills (both 37.4%, N=115), and basic computer skills (36.8%, N=114). CFF coaches and teachers most often reported that students use technology Weekly to build all 15 component skills.
Chi-Square and Cramer's V analysis was performed to identify significant correlations between the frequency of assigned problems, projects, or assignments that required students to use technological literacy skills and how often students use technology to build these skills. The results (Table 3) showed significant correlations at the (p ≤ 0.01) level for 10 of the 15 component skills, and at the (p ≤ 0.05) level for three other component skills. These results suggest that efforts to build technological literacy skills often involve the actual use of technology, though earlier results suggest that the set of technologies used may not be diverse.
Administrators were also asked a series of similar questions to elicit information on teachers' access to technology in the classroom, the frequency of teachers' technology use in the classroom, and the frequency of student technology use in the classroom. All questions involved the set of 16 specific technologies found in the teacher survey.
Administrators reported that teachers in their school/district have access to a wide variety of technologies for classroom instruction. Word processing software and Smartboards or similar devices were the most frequently selected technologies (100.0% for both, N=21), followed by student laptops (95.2%) and spreadsheets and presentation software (85.7% for both, N=21). Web 2.0 technologies such as social networking Web sites (9.5%, N=21); Real Simple Syndication (9.5%, N=21); Podcasting (42.9%, N=21) were cited infrequently. This lack of access was to be expected, given the aforementioned "blocking" of many Web 2.0 sites. Some Web 2.0 technologies - Wikis (66.7%, N=21), blogs (71.4%, N=21), and streaming video (81.0%, N=21) - were frequent selections. Wikis and streaming video are frequently provided as in-house, controlled options in lieu of more public outlets. As with the teacher survey, multimedia authoring software was cited infrequently (33.3%, N=21).
Administrators were then asked to indicate the frequency of teachers' use of these 16 technologies for classroom instruction. The frequency was measured using a six-level scale (Not Used, Daily, Weekly, Monthly, Quarterly, or Unknown). The results show a general lack of knowledge on how frequently technology is used by teachers in the classroom, but a more positive perception on classroom technology use than teachers. Unknown was the most frequent selection for 10 of the 16 technologies. Despite the aforementioned restrictions on Web 2.0 technologies, Not Used was the most popular response for only one Web 2.0 technology (Social Networking, 68.4%, N=19). The office productivity software - presentations, word processing, and spreadsheets - were most frequently cited as being used on a daily basis. Common CFF technologies - Smartboards (81.0%, N=21) and student laptops (52.4%, N=21) - were also cited most frequently as being used on a daily basis.
Administrators were next asked how often students use technology to build technological literacy skills in class. The administrator responses suggest a greater confidence or knowledge that technology is being used in the classroom, despite earlier responses. Administrators were given a six-level scale (Never, Daily, Weekly, Monthly, Quarterly, and Unknown) from which to choose. Administrators most often reported that students use technology in their classes daily to build communication skills (42.9%, N=21), critical thinking skills (42.9%, N=21), problem-solving skills (47.6%, N=21), and basic computer skills (61.9%, N=21). The use of technology to build creativity and innovation skills was equally split between Daily and Weekly (38.1%, N=21 each), and the use of technology to build students' ability to use, interpret, validate, and synthesize information was equally split between Daily and Unknown (33.3%, N=21). Not surprisingly, Unknown was the most frequent selection (either alone or tied) for nine of the 15 component skills.
As a companion to the earlier questions regarding technology use, teachers in grades 9-12 were asked to identify how frequently they use the CFF resources within their school/district. The responses indicated that 42.0% (N=100) do not use the available CFF resources, 35.0% use them on a daily basis, 11.0% use them weekly, and 5% use the CFF resources monthly. For CFF coaches and teachers alone (N=45), 4.4% reported non-use of the CFF resources or monthly use, 64.4% reported daily use, and 24.44% reported weekly use. These results suggest heavy integration of CFF resources by CFF teachers, as can reasonably be expected.
In terms of classroom pedagogy, teachers overwhelmingly agreed/strongly agreed that technology is an integral part of their day-to-day instruction (74.8%, N=127), in contrast to the prior results suggesting limited classroom technology use. Teachers also frequently agreed/strongly agreed that their classes were structured around active or inquiry-based learning, both overall (74.8%, N=127) and for CFF coaches and teachers alone (77.8%, N=45). The responses for these two questions were highly correlated for all teachers (X2=165.487, df=16, p ≤ 0.01; Cramer's V=0.573, p ≤ 0.01), suggesting that the use of technology and active/inquiry-based learning methods go hand-in-hand in the participating districts.
CFF stresses the need for more active and inquiry-based pedagogy; surprisingly, CFF coaches and teachers were most often Neutral (Undecided) when asked whether CFF has encouraged them to use more inquiry-based teaching methods (47.5%, N=97). This may be due to the prior use of such methods by CFF faculty; inquiry-based learning has been a popular movement for years in Science and other educational disciplines. This result may also reflect the type of teacher most likely to be proactive in their use of CFF resources: a willingness to explore new pedagogical possibilities like CFF may reflect a prior willingness to explore active/inquiry-based methods. The responses to the two questions on active/inquiry-based learning methods - whether teachers' classes were structured that way and whether CFF encouraged these methods - were significantly related for CFF coaches and users (X2=36.292, df=1, p ≤ 0.01). The strength and direction ofthe relationship (Pearson's r = 0.468, p ≤ 0.01) suggests, at least for CFF coaches and teachers, that CFF is having a significant, positive impact on pedagogy.
Administrator Survey: How are Teachers Adapting Their Practices via CFF Resources?Administrators were asked to provide an approximate number of teachers who utilized the CFF equipment, both within their school and/or their district. The most frequently selected estimates were 6-10 CFF teachers at their school (45.0%, N=20) and 11-20 CFF teachers within their district (35.3%, N=17). Two administrators (10.0%, N=20) estimated that 41 or more teachers used the CFF equipment in their school or district.
Administrators were also asked about the frequency at which teachers utilize the CFF equipment in their school and district. The majority of respondents indicated daily use of the CFF equipment, both within their school (60.0%, N=20) and their district (57.9%, N=19). It should be noted that a significant portion of respondents indicated they did not know how often the equipment was used (15.0% in their school, N=20; 26.3% in their district, N=19). Administrators overwhelmingly agreed/strongly agreed that participation in CFF has encouraged teachers to use more inquiry-based teaching methods (85.7%, N=21), though the teacher results suggest that this may be more perception than reality.
Focus Group Results
The following sections discuss the relevant focus group responses. Two of the focus groups involved primarily CFF-related teachers, while two of the groups were more diverse (either entirely non-CFF teachers or a mix). The four focus group sessions were comprised of 22 total participants.
Teachers' Use of Technology and CFF Resources
The two focus groups involving primarily non-CFF personnel were asked the question How do you specifically use technology in the classroom to enhance student learning? While this question was not specifically posed to the other two CFF-heavy focus groups, all participants were very forthcoming about their use of technology in the classroom. The responses indicate a fairly diverse set of technology uses and student projects in the classroom, more so than is indicated by the survey data. While some teachers indicated minimal technology use (e.g., "I really do not use it that often"), many others spoke excitedly about their use of technology to engage students and provide "fun" demonstrations, in-class activities, and student assignments.
The examples of classroom technology use given by focus group participants were fairly diverse. Smartboards were the most frequently mentioned technology across disciplines. Smartboards are often used in tandem with interactive exercises, discipline-specific software, and especially PowerPoint presentations. Web searching was also frequently mentioned as a component of classroom technology use by students. In one case, the use of custom-built, internally-controlled Wikis was mentioned as an alternative to other forms of out-of-class interactivity. The following descriptions provide examples of technology-related class activities discussed by the focus groups:
Other uses of technology mentioned in the focus groups included online learning games, computer-aided drafting software, and streaming audio.
Besides the aforementioned uses of Smartboards and other CFF-specific resources, focus group participants pointed out another vital CFF resource - the CFF "coach" at each school. The role of the CFF coach is to help other CFF teachers get up and running with a variety of CFF tools and resources, as well as to act in a mentoring role for those faculty looking to explore their options. Several of the focus group participants noted the important contribution of the CFF coach in revealing the possibilities afforded by the CFF program. The use of student laptops was infrequently mentioned, though the implication was that the computer-intensive work in the aforementioned projects was performed using CFF laptops.
CFF Impact on Teaching Styles and Practices
The focus groups involving CFF personnel were asked the question How has the CFF program changed your teaching methods or style? The focus group responses suggest a more dramatic change in practices and styles than the teacher survey results indicated. Participants noted how the CFF technologies caused them to re-think their methods; for example, one participant noted, "Sometimes I think it's taken me back to the drawing board…where I need to re-think everything." Others noted how the CFF technologies had given them a sense of freedom ("it got me away from just standing in front of the room") and opened up new possibilities for instruction and interactivity ("it opened up a definite potential for other things to do").
A few of the younger faculty noted that they had never experienced a classroom without the CFF program and, thus, considered the technologies and methods advocated by CFF to be part of the basic K-12 experience. These teachers see their role as primarily a facilitator rather than a rote lecturer. The reliance on CFF technologies and tools is ingrained: "I teach with my Smartboard everyday so like if it doesn't work I don't know what I would do." While some mathematics teachers noted that importance of the Smartboard ("the Smartboard is the best thing that has impacted teaching as far as Math would go"), other CFF tools were seen as providing a change of pace for mathematics students rather than a transformative experience. Other CFF faculty noted the importance of the Smartboards as well as the importance of the student laptops. The overall perspective from the focus group participants is that the CFF program has impacted these teachers' methods, styles, and procedures. One participant summed it up best by saying:
"The way I would describe it is as a mechanic trying to fix a car with an adjustable wrench - you can do it, but it's pretty tough. But giving those computers to me was like giving me a set of tools. Now I can work on the engine and get it going the way I wanted to."Administrator Encouragement
Each focus group was asked about their perceptions of administrator support and encouragement. For those focus groups involving CFF personnel, the question was phrased How has administration been supportive or non-supportive of CFF in your school? For focus groups devoid of CFF personnel, the question was phrased Do you feel administrators encourage you to use technology? The focus group responses correlate with the teacher survey results. In all cases, teachers responded that administration was supportive of the CFF program and/or their use of technology. Examples included administrative initiative in getting the CFF grant process started and allocating time for interested teachers to get the requisite CFF training. The use of special in-service programs for CFF personnel was also noted. There was no mention of administrative verification of technology use. For non-CFF teachers, the methods by which administration encourages technology use were somewhat perfunctory. One method in particular stands out: "It used to be that we had to write our lessons plans, now we have to type them on the computer."
It was obvious during the focus group sessions that the level of participants' technology knowledge and their frequency of technology use impacted their responses. More than one respondent indicated that students seemed to know more about modern technology than the teachers, even going so far as to use students for classroom IT support. As implementing agents, teachers must be able to recognize if they lack the requisite skills for implementing policy. Teachers must then seek out sources for building those skills. Administrators must both encourage teachers and direct them to appropriate learning resources. The focus group results suggest that a minimal amount of training is provided. For tech-savvy personnel, this is not a problem. However, for those teachers with fewer technology skills, the lack of understanding of technology use - and the resultant need for training and/or learning resources - acts as a barrier to successful implementation of CFF and related programs.
Contextual Challenges to ImplementationAll of the focus groups were asked the question What personal, organizational, or environmental challenges do you face in implementing CFF or any other technology program? The most common "personal challenge" cited was a lack of time to fully grasp the technologies and the program. The general mood seemed to be that teachers are limited, either contractually or personally, to a strict eight-hour workday. Teachers overwhelming agreed that other commitments - class time, paperwork, keeping current in their fields - limited their ability to explore the new possibilities afforded by CFF. One participant put it succinctly:
"The teachers are already committed to their time in the classroom. They already have to keep current with what they have to do, and then you're offering them new things and while they're enthusiastic - they just don't have the time."
Things like district curriculum and state standardized test preparation were said to take away time for teachers to explore new approaches like CFF. Participants pointed out that many of the CFF personnel at their school used their own time to complete the requisite CFF training, and indicated that teachers' willingness to use their own time factored into their selection as CFF coaches and users by administrators. Enthusiasm among CFF personnel was high, but participants lamented the lack of time for collaboration, sharing of ideas, and follow-up training. Lunch was commonly cited as a prime time for CFF teachers to discuss experiences and share best practices.
Technical difficulties were also a commonly cited environmental challenge. One focus group remarked about the problems getting a wireless network set up in their school; the delay in getting this connectivity limited the use of CFF equipment when it first arrived. These problems were exacerbated by the advanced age of the participating high schools and their inherently wireless-unfriendly building materials. One participant pointed out that teachers need to be in a wireless-enabled classroom in order to use CFF student laptop carts, and some of the classrooms that interested teachers use are not wireless capable. Laptop connectivity issues also tended to be intermittent in addition to other technical glitches ("We have 30 on the cart, but 30 will never work on a good day").
Sluggish network logons were also cited by more than one focus group. The slow login procedure was said to inhibit the use of CFF laptops and other computers ("You could have a 40-minute period and the first 15 minutes are devoted to getting everyone up and running"). Limited classroom space was also mentioned as inhibiting the use of laptop carts ("It's just not worth it - you gotta move desks, I had to move the teacher desk just to have a place to put it so you can open it up so the kids could get the laptops out"). Problems with logins, network connectivity, and network outages were the most frequently mentioned environmental challenges for the use of CFF resources and technology in general.
Despite the survey findings of robust classroom technology access, limited computer access was another challenge cited by the focus groups. More than one focus group complained about the small number of computer labs in their high school. Technology-specific departments - usually business - mentioned having technology classrooms, but general-purpose technology classrooms seemed to be sparse. In one high school, only one general purpose lab exists (in the library) and scheduling is done months in advance. This lab is continually booked. While each department has a cart of student laptops that can be checked out, this too presents issues of contention. When asked if the laptops are heavily used, one participant answered "when they work," highlighting the aforementioned technical issues.
Few organizational challenges were mentioned. One participant pointed out how the philosophy emphasized by CFF training materials - the idea of learning authentically, via pared-down curricula - contrasted greatly with state and district regulations and curriculum guidelines. The Pennsylvania state-level standardized exams and other commitments were viewed as a box in which teachers must work, limiting full use of the CFF philosophy and resources. Differences in academic requirements between districts were also shown to be a limiting factor on technology-centric course offerings. One participant pointed out that his district uses a seven-period day, as opposed to the common eight-period day. This loss of one period reduced the ability of students to take elective courses. As a result, this district focuses on mandatory subject areas and does not have the technology offerings found in other districts.
Document ReviewsSample projects and assignments were solicited from teachers in grades 9-12. These examples were intended to permit a more accurate examination of how technology is used for student learning. These examples were analyzed to further investigate how technological literacy skills were being built in the participating districts. Despite the best efforts of the investigator and the generous support of district administrators, only 15 assignments were received from five districts. The 15 assignments came from the CFF disciplines of Science (6), Language Arts (5), Mathematics (3), and Social Studies (1), though examples were solicited from teachers in all disciplines. The example assignments were coded based on their content, their use of technology, and their pedagogical goals (explicit or implicit).
The example assignments provided by participating teachers represented a mix of traditional "observe-and-report" assignments combined with more creative, critical-thinking based projects. Communication skills, critical thinking, collaboration skills, and creativity and innovation appeared to be the technological literacy skills most emphasized among the sample assignments. Written communications were required for 12 of the 15 example assignments. Examples of these assignments included Chemistry research reports, the production of "newspaper reports" for historical events, and video-driven question sheets. Oral presentations were required on three of the example assignments, and one assignment required only visual communication (via an electronic propaganda poster).
While six of the example assignments were structured around simple rote observation and/or memorization, four assignments emphasized critical analysis of source material or classroom experiences. Creativity was an essential part of four assignments, as was collaborative teamwork among students. Examples of assignments emphasizing creativity included the construction of a photo montage describing the themes of Romeo and Juliet and the use of PowerPoint to discuss one of Chaucer's tales.
In terms of specific technologies, PowerPoint was the primary deliverable (3 assignments), though one assignment did explicitly require the use of a blog and a Wiki. Multimedia presentations and/or content were a common theme, occurring in 12 example assignments. Six assignments required the use of external Web sites. These sites were used as vehicles for streaming video (2 assignments), mathematical simulations (3), or as multimedia data sources (1). The use of student laptops was explicitly mentioned on only one assignment.
The example assignments provided by participating teachers parallel the results found in the survey and focus group analyses. Technology use by teachers seems to be fairly narrow, focusing on the use of PowerPoint and external Web sites. The use of multimedia and external Web sites seem to be simply a vehicle for providing visual flair to otherwise traditional, observe-and-report assignments. Multimedia did, however, provide avenues for creativity in a few of the example assignments. Technological literacy skills are a core piece of these assignments but the emphasis is too narrow, primarily focusing on written skills. While the size of the example assignments sample is too small to make generalizations, the results suggest that innovative applications of technology to build technological literacy skills via student assignments is sporadic rather than widespread.
The CFF program began implementation in 2006. Based on the perceived success of other states in implementing similar programs, policymakers saw the widespread provision of computers and other classroom technologies - coupled with a standardized set of training resources - as key to keeping Pennsylvania competitive by building the technological literacy skills of K-12 students. The policy was a victim of the 2009 state budget battle, though most schools in Pennsylvania were able to participate before the cessation of funding, and some continue using the CFF equipment and resources even today. The findings in this implementation study suggest that CFF faces a number of implementation challenges.
The survey and focus group results indicate that teachers perceive their classes as using a limited set of technologies to build a wide variety of skills. Despite widespread access to multiple technologies, teachers report that classroom technology use is predominantly limited to traditional Office-style software and interactive Smartboards. More modern Web 2.0 tools are rarely used due to in-house restrictions on Internet content. Student laptops, the core technology provided by CFF, are used somewhat sparingly. Despite this, enthusiasm for the CFF program among the focus group participants was evident, and both technology and active/inquiry-based methods were widely considered to be an integral part of K-12 pedagogy.
Despite the focus on a limited set of technologies, teachers report assigning problems, projects, or assignments that exercise all technological literacy skills on at least a weekly basis. This perceived breadth of coverage was not supported by the teacher-provided example assignments. The results suggest a general focus on Technology Literacy skills rather than technology itself, although the level of that focus is questionable. This finding raises questions about the scope of CFF implementation and suggests potential problems with implementation within the participating districts. CFF is designed to provide the tools necessary to make a "transformative" change in educational practice. These results suggest that the intended "transformation" may not be occurring within these districts. The finding of sparse technology usage raises questions about whether old pedagogical methods are simply being modified to include a limited set of new technologies (i.e., "old wine in new bottles").
The success or failure of a policy is a product of many factors, including the individual behavior of the implementing agents. Without shared understandings of the policy and its implementation between public managers (i.e., administrators) and street-level bureaucrats (i.e., teachers), one or both groups may exercise considerable discretion during implementation. The study results indicate that there exist substantive differences between administrators and teachers in their perceptions of behaviors related to CFF implementation. The dissonance between administrators and teachers inevitably results in implementer discretion, limiting the possibilities for systematic assessment and, ultimately, impacting the chances of programmatic success in these districts.
Administrators had a largely incomplete view of classroom technology access, use, and integration, CFF-specific and otherwise. The results suggest administrators in these districts have minimal knowledge about how different technologies are used to build students' technological literacy skills and the frequency with which technology is used in the classroom. Administrators also reported a more optimistic outlook about the effect of the CFF program on teachers' use of inquiry-based teaching methods than teachers. The findings indicate that administrators may have insufficient knowledge of how the CFF program is being implemented, calling into question the ability to accurately assess the success of the policy. Administrators need more detailed understanding of how CFF is being implemented; without this understanding, teachers may exercise considerable discretion in policy implementation.
One potential reason for teachers' limited classroom technology use and integration may be a lack of preparation. Focus group participants reported that their biggest challenges were time and training (to more fully comprehend CFF, its concepts, and its components) and technical and logistic difficulties (insufficient equipment or rooms, technical glitches). As implementing agents, teachers who lack technology and policy-specific learning resources are limited in their ability to successfully implement policy. As public managers, school administrators must be proactive in helping teachers seek out the resources they need to adequately implement the CFF policy. Teachers did report that administrators actively encourage CFF participation through multiple methods, but it seems as though consistent and timely teacher training is a shortfall of programmatic and administrative efforts.
These results are compelling but are limited by the focus on a small set of rural school districts. According to the U.S. Census Bureau, approximately 27 percent of Pennsylvanians live in areas considered rural, though the majority of Pennsylvania counties (48 of 67) are classified as rural. The inherent limitation of this research is a lack of generalization; however, the results of this study suggest that further research is needed to determine if the problems and challenges found within the participating districts are common across the state and across socio-economic school profiles.
The purpose of this study was to analyze how Pennsylvania's CFF program is being implemented in a small set of rural school districts. The results suggest that the CFF implementation may not be triggering the transformation in educational practice called for by the policy. Teachers perceive building a wide variety of student skills with few technologies, calling into question whether new tools (e.g., student laptops) are simply being made to fit old methods. The two groups of CFF implementers in this study - teachers and administrators - exhibit some dissonance in their perceived implementation of the CFF policy. This dissonance increases the likelihood of implementer discretion and, thus, challenges the policy in its ability to achieve its goals within the study districts. The potential for implementer discretion is exacerbated by inadequate training resources, technical and logistical difficulties, and competition from other priorities. The state-level fiscal death of the CFF policy means that in order to be successful in the long-term, local-level training and investment efforts will become crucial to implementation efforts. The implementation study findings suggest an initial level of enthusiasm for CFF in the participating districts, but considerable challenges to its long-term chances of success exist.