North Carolina has worked diligently to make digital teaching and learning a priority from the top, down to districts and schools. Efforts to “de-silo” curriculum, instructional technology and assessment throughout the state over the past ten years have gotten the attention of national agendas such as Race to the Top funds, positioning this state as an early adopter and leader in technology-integrated instruction. The state’s NCTest online assessment delivery platform, launched in 2005, is currently expanding to offer all high-stakes and many non-high stakes assessments online through a new platform called TestNav in 2014-2015. This expansion has been successful largely because of the state’s history of taking a proactive stance of (a) anticipating and reducing potential logistical barriers before they arose, and (b) viewing assessment as one of many uses of instructional technologies.
History and Background
Over the past decade, the North Carolina Department of Public Instruction (NCDPI, or DPI) has been leading a digital reform effort across North Carolina.
In May 2007, the State Board of Education convened a Blue Ribbon Commission on Testing and Accountability to begin the process of assisting the Board in charting a course for realizing five goals and strategies aligned to DPI’s mission. DPI’s efforts to address the Commission’s recommendations took place in stages over five years (2008-2012) via a comprehensive initiative called ACRE—the Accountability and Curriculum Reform Effort. Through ACRE, North Carolina’s education leaders are among the first in the nation to simultaneously address learning standards, student assessments, and school accountability. For example, ACRE is one of very few statewide initiatives in the country that supports schools in creating grade-level plans aligned to the new CCSS. Where most other states are unable to fund this kind of deep curriculum work, North Carolina has been able to do so with reduced budget pressures. ACRE invests simultaneously in infrastructure, professionals, and devices, in part with Race to the Top (RttT) dollars. This approach helps the state reach 2524 public schools across 100 county-wide and 15 city-wide districts, plus 107 charter schools, serving nearly 1.5 million students across the state.
Strong federal commitments to new instructional standards and teacher capacity development around CCSS instruction called for an updated assessment and data sharing system within the state. Where the prior system was more accountability-oriented, this new system needed to increase the “flexibility and responsiveness, and balance” demanded by new and forthcoming information needs. North Carolina’s affiliation with the Smarter Balanced Assessment Consortium (SBAC) underscored the state’s commitment to a fully online assessment system by the 2014-2015 school year.
In an effort to expand participation in online testing, North Carolina’s RttT proposal included various provisions for supporting LEAs and charter schools in the transition to online assessments, principally by publishing a best practices guide. This comprehensive document: (a) features case studies of schools that use school-wide online assessments, (b) addresses issues of scheduling, financial planning, and technical requirements in order to move to online assessment, and (c) provides specific steps taken to plan for and administer online assessments.
- The ACRE (Accountability and Curriculum Reform Effort) report from Spring 2010 describes statewide readiness for online assessment in the areas of personnel, hardware, and connectivity.
- Feb 2011 Report Addendum
- NCDPI’s Online assessment update from Feb 2012 includes a brief history of the move to online testing in NC.
Milestones to Online Assessment
Before statewide launch
- 2005 – First fully operational online test: Online Test of Computer Skills.
- Spring 2006 – End-of-Course (EOC) Test of Physics was field tested as the first online EOC assessment.
- 2006-2007 – EOC online tests in Algebra I and II, Biology, Civics, English I, Geometry and US History fully available.
- 2007-2008 – Fully available EOC tests expanded to include Physical Science, and the entirely online (i.e. no paper-and-pencil version) Physics exam.
- 2008 – NCDPI implements the Accountability and Curriculum Reform Effort, ACRE, a five-year investment in the coordination of technology services related to devices, professionals, and infrastructure.
- 2009-2011 – NCDPI conducted assessments of personnel and infrastructure readiness to support online assessment, and reported findings to the State Board of Education. NCDPI believed that evaluating readiness to deliver online assessments at the LEA and charter school level is an important early step in the process of moving to administering End-of-Grade (EOGs) and End-of-Course (EOCs) assessments in an online only format. To that end, NCDPI solicited responses from the LEAs and charter schools, with response rates of 100% and 58% respectively. A key finding was that LEAs were further along in their readiness than charter schools, and that all schools would benefit from a more comprehensive approach to supporting schools in their move online.
- 2010 – Conclusions from a study comparing NC’s online EOC and paper-and-pencil assessment programs that was conducted by the Council of Chief State School Officers indicated that scores from paper and online assessments are comparable in North Carolina.
- July 2011 – State publishes online assessments best practices guide, revised prior to the 2012–13 school year.
- 2011-2012 – Current version of online test system (NCTest version 2.5; math, science and ELA) was field tested. At the end of the 2011–12 school year, about 19 percent of all EOC assessments administered in North Carolina were administered in the online format, as opposed to the traditional paper-and-pencil format.
Year of statewide launch: 2012-2013
- 2012-2013 – Current version of summative assessments, aligned to CCSS (for ELA and Mathematics), and NC Essential Standards (for Science), became fully operational and administered.
After full launch
- Summer 2013 – NCDPI launches a new online system for benchmark and classroom assessments. The new assessment system will feature formative tools and interim assessment builders for designing non-high-stakes assessments to help teachers adjust instruction based on student needs. Formative assessments, used to improve instruction, will be delivered through a statewide Instructional Improvement System (IIS), known as Home Base.
- 2014-2015 – NCDPI plans to deliver Smarter Balanced online assessments through TestNav, rendering NCTest obsolete. These new assessments will include computer-adaptive testing for Grades 3-8 Math and ELA and one high school assessment in ELA and Mathematics (with Performance Tasks in 9 & 10). Few, if any, high-stakes assessments will be delivered in paper-and-pencil format.
Evolution of the Infrastructure
With state-funded School Connectivity Initiative dollars, North Carolina was one of the first states to provide no cost scalable Internet access to all public schools statewide. The Center for Urban Affairs and Community Service (CUACS) commissioned a state-level capacity/stress test conducted by INTG in 2011-12 that proved NCTest’s ability to support 250,000 concurrent users. Their report describes the architecture and methodology necessary to implement the test, and concluded that the anticipated network and bandwidth footprints per student testing session are similar to many other Internet applications. Specifically:
- 250,000 concurrent users will create 10% load on the NCTest infrastructure
- NCTest start up phase places the largest load on the network
- NCTest start up period should be spread out over a longer period of time (30 minutes)
With statewide network capacity for testing in place, NCDPI provided a specific and clear message to schools: they were required to meet all technical requirements well enough in advance of live testing to make necessary changes, if they wanted to ensure students received a valid and reliable online assessment.
For most, the first steps were purchasing and reconfiguring their infrastructure to include more devices – laptops – set up as computer labs, and adding wireless networks to their existing network connectivity. NCDPI advised that until the student‐to‐computing device ratio was nearly one‐to‐one, full implementation of online 21st Century assessments would be difficult, especially for larger schools. NCDPI recommended focusing on using state, federal, commercial, and grant resources to fund ubiquitous hardware access for students in grades 3 through 12. NCDPI personnel suggested that LEAs consider using any available RttT funds to purchase hardware and make investments in school connectivity.
Initially, there was some resistance from school-based technology personnel who were frustrated with waiting so long for clarity on the system requirements needed for supporting instruction to make wise hardware purchases. Today, many districts are still saying they don’t have enough devices to support online testing, with the largest districts struggling the most with full compliance.
Testing with technology takes up only approximately 5 percent of instructional time, leaving the technology available for instruction the remaining 95 percent of the time. Critical to local hardware purchases and upgrades in North Carolina was the need to address a multitude of ways that end users were expecting computers to perform, from emailing to streaming content, that extend well beyond testing demands. One challenge is selecting hardware features today to support future needs that are currently unknown or difficult to project. Yet the importance of doing so is clear. In Neill Kimrey, NCDPI’s Director of Digital Teaching and Learning’s view, “If we are testing with technology, we have to be teaching with technology.”
Making the Transition
One benefit of North Carolina’s long and involved history with online assessment is that many educators are currently past the ‘naysayer’ stage. They have been involved in the process long enough to understand how and why online assessment is important, and that this enormous change affects everyone across the entire state. Educators have also observed many groups—LEAs, legislature, governor’s office, state BOE, NCDPI—wanting to work together to enact change in a thoughtful and prepared way by following a common language and framework. NCDPI has deployed clear, consistent and timely messages about how to be prepared for online assessment through communication channels dedicated to this purpose. They have also committed resources to capacity building around technology-enhanced lessons and efforts to use the data made available by the new assessments.
Establishing a common language and vision across several levels that is oriented toward reducing barriers to technology-supported instruction. NCDPI reports an excellent relationship with the state legislature and governor’s office, working closely to make digital teaching and learning a priority across the state. Together, they have teamed up to reduce and remove typical barriers to deeper technology integration through:
- Legislation. In the fall of 2012, the legislature convened a study committee that interviewed national leaders and convened four meetings with various national, state and local leaders to determine how to best integrate technology in instruction. In the legislative session of 2013, the legislature introduced four bills that are essentially in direct response to joint recommendations from NCDPI and the committee:
Reducing barriers to affordable, accessible technology and digital content. NC was one of the first states to provide no cost scalable Internet access to all education agencies including firewall, web security, etc., and to host educational data in the cloud allowing LEAs to spend dollars otherwise allocated to Internet access and servers on more classroom technology. Cloud computing reduces the number of servers and server maintenance previously required by schools. With scalable bandwidth and cloud based applications, local education agency technicians can focus more on the end-user experience. North Carolina is also moving toward state-level purchasing of devices and digital content, allowing schools to reallocate textbook funds for digital content.
Sending clear, timely messages about expectations and readiness. NCDPI has made it a priority to keep schools informed about how they can prepare for the move to online assessment, and what they can expect on test day, by: conducting thorough field tests, creating a network of regional testing liaisons, and providing tutorials and handouts that break down key information into manageable chunks.
- Conduct thorough field test(s). NCDPI hosted extensive field tests prior to online launch to test new item types and for schools to test capacity. What can be easily forgotten during the busy planning stages of a move to online assessment but wasn’t overlooked in NC is the lasting impression that a field test can have on teachers and students. Educators across the state worked very hard to fully simulate testing conditions when possible during the field tests to fully convey expectations and instill confidence in this comprehensive and complex change to collecting high-stakes data.
- Create a network of regional liaisons. to support LEAs and charter schools as they make the transition to online only assessments. NC’s Regional Accountability Coordinators are available to answer individual questions for schools or LEAs and host inter‐LEA conversations about best practices and other online assessment concerns as they arise.
- Provide resources and handouts that break down key information into manageable chunks. NCDPI publishes a two-page overview on testing for schools: what is being tested, timeline, and links to resources including the testing schedule/calendar. NCDPI recommends schools identify personnel that can be trained to serve in the essential roles necessary for full online-only assessment. NCDPI also recommends the following if schools have never administered online assessments:
- Learn about your network infrastructure.
- Check the technical requirements and make sure computers and browsers can run NCTest.
- Check content filtering and traffic shaping to make sure it is open for traffic from key servers
- Use the Online Assessment Tutorial and the Online Released Forms to simulate a testing event.
- Follow NCDPI’s recommendation of starting with a very limited number of online test offerings, which gives schools a chance to practice and prepare for a more widespread rollout.
Committing resources to capacity building—it’s more than just moving the old test online. NCDPI views online assessment as a teaching and learning with technology initiative, not exclusively an online assessment initiative. This focus prompted a discussion about approaching technology as a core component of instructional programming, rather as its own stand-alone programming. At the same time, instructional staff was reluctant to deeply reform their teaching to include more technology just because of online testing. NCDPI invested in a teacher professional learning community initiative that calls on the data from online assessments and utilizes technology tools to add traction and meaning to the instructional shift.
Recognizing that much of what gets students prepared for online tests also prepares teachers. Different groups evolve in their readiness for online assessment at different rates. In the case of one school in North Carolina, for example, teachers of one tested subject area worried that students would be distracted by the logistics of interacting with the online interface and initially resisted the move to online testing. These teachers also knew the paper and pencil EOG high stakes tests well, and were very skilled at teaching students how to fill out bubble sheets, etc. They were generally less confident in how to apply those concepts to online testing, and didn’t want student scores to reflect poorly for this reason.
NC DPI provides an Assessment Tutorial that is available to the public prior to the testing window to familiarize test administrators and students with the functional components of the system. As of 2012-2013, all students are required to complete the tutorial before test day on a computer that will be used for testing. The NCTest Administration page allows local staff to create test sessions, complete accommodation requests, reset, resume, and finalize exams, and provides a link for help.
- Professional learning community survey report (August 2012)
What Districts Did
Districts and schools across NC have varied in their rates of adopting online assessments since the tests were first available in 2005. Some have joined the effort very recently, coinciding with the state’s timeline for readiness, while others have been involved in various facets of online testing for many years. Many schools remain very concerned about transitioning to entirely online assessments, even in North Carolina where strong leadership and resources have been committed to these changes over the past several years.
District 1: Mooresville Graded School District
Mooresville Graded School District in Mooresville, NC, has been recognized nationally as a leader in applying digital learning for catalyzing transformational change in education. Dr. Mark Edwards, Scholastic Administrator’s Superintendent of the Year for 2013, is the visionary leader of Mooresville Graded School District and author of “Every Child, Every Day.” In his book, Dr. Edwards describes how having a good working relationship among the curriculum, assessment, and technology functions reduces isolation and redundancy, thereby improving technology integration school-wide. Over the past five years, he has recommitted the district to using technology for instruction and assessment and teacher development because “it’s best for kids.”
Infrastructure and Connectivity. Mooresville began by designing their technology infrastructure and upgrades with many redundant nodes of wireless and other access points to assure system reliability. The district recently upgraded to SETDA’s current wireless recommendations, which Chief Technical Officer Scott Smith says are, “in place now and SETDA’s recommendations are right on target.” Lowe’s Home Improvement (headquartered in Mooresville) contributed $250,000 to help fund the in-building wireless infrastructure. Dr. Smith says it is important to think about “the connections within, between and among buildings as well as back to central office and outside the network” during the design stages.
The purchase of computers was a natural next step because the culture shift toward more and better technology use was already underway through Dr. Edwards’ comprehensive digital conversion. The devices that were purchased met a natural need for more ubiquitous use by putting Macbook Air computers into the hands of every teacher and every student in grades 4-12, that could be taken home every day. Further, all K-2nd graders have a laptop for use at school, and K-2nd graders have shared laptop carts used substantially throughout the day. Mooresville leases the devices for three years, which costs a bit more up front but provides known cost savings over the long term as opposed to buying computers outright and having to keep up with maintenance and upgrade costs over time. The district takes advantage of the statewide policy of reallocating textbook money for digital content. At a cost of less than $70 per student for the year for over 25 digital media content providers, Mooresville can reallocate resources to sustain the system with an overall cost savings relative to before the technology was in place.
Field Testing. Mooresville GSD participated in the early online field tests, involving approximately 4000 students. To achieve success with field testing, leaders at Mooresville prepared a thorough strategic plan for just this effort that stated goals, objectives, actions and timelines. From a rollout standpoint, Chief Technical Officer Smith was very concerned that the pilot continue to progress forward, rather than having it fail and people falling back to more costly and less efficient paper and pencil methods. Even at a tech savvy district such as Mooresville, the biggest challenges were with technology integration and hardware early in the field test. During the second year, this went better because people were more familiar with the process, but also because they had received training in how to use the technology for purposes other than assessment and were more familiar with the technology overall.
Guided Teacher Readiness. Dr. Smith described the district’s efforts to mitigate any teacher struggles with the overall expectation to use more technology as taking a practice stance. In this orientation, teachers do regular needs assessments and have ample time during the year for ongoing professional development.
- Needs assessment. Not all teachers are ready to jump into deeper technology integration. At Mooresville, they use a needs assessment rubric that maps the ways in which various stakeholder groups are prepared to use technology. Another rubric provides examples at each proficiency level, which guides professional development planning.
- Monthly teacher professional development on technology integration. Teachers at Mooresville spend one half day each month during the school year involved in professional development on how to use technology for quality instruction, not just assessment. The district secured buy-in from local parents for this unconventional commitment to educator development by following the logic of a medical professional development model. Administrators posed the question, “Wouldn’t you want your doctor to be up to date on their practices? Then teachers, also professionals, need this too.” This district also offers learning experiences over the summer, which 90% of teachers voluntarily attend.
District 2: Davie County Schools
Whereas Mooresville CSD jumped into digital instruction and assessment with two feet, Davie County Schools in Mocksville, NC took a more gradual approach, which has been very good for them. Teachers have been very focused on the CCSS curriculum transition and other widespread changes, with the move to online assessment following behind. Erin Foil, Director of Accountability and Student Information for the district, reflected, “While a lot of change at one time is really hard, it can also keep things in perspective. With so much change elsewhere, the change with online assessments feels like just another change that’s happening, rather than [one that’s] being singled out.”
Built a solid foundation before scaling up. Davie County Schools started administering online assessments in 2009 with Algebra I and Biology. The district followed NCDPI’s recommendation of starting with a very limited number of online test offerings, which gave them a chance to practice and prepare for a more widespread rollout. The only other available online test at that time would have been in a subject area where the teachers were not yet comfortable with the exam. The district is currently testing online in the elementary grades with a relatively small group that is already pushing the limits of the existing hardware infrastructure. Ms. Foil and the technology director in her district are working closely to plan for hardware and computer lab expansion to support the upcoming expanded online testing demands.
Davie County Schools also participated in the state’s early field testing, which Ms. Foil remarks, “went really well”. Field testing provided the usual benefits of infrastructure testing and end user readiness, but also helped to train several new testing coordinators who benefitted from the practice they got by administering field tests prior to “live” tests.
Clearly defined communication channels for questions and discussion about changes. Davie County Schools relies heavily on NCDPI’s Regional Accountability Coordinators (RACs) and a searchable bulletin board to keep them up to date on issues related to testing. RACs interact directly with Ms. Foil, who oversees the school test coordinator. The school test coordinator develops and executes the district’s testing plan (e.g. which tests need a separate room, separate proctor, etc.) based on the framework provided to them by the RAC via Foil. Ms. Foil appreciates the communication structure because, “it is very manageable, and everyone knows where to go to get their questions answered.”
Recognized that much of what gets students prepared for online tests also prepares teachers. Ms. Foil approached the situation involving teachers who were reluctant to test online with the same patience and clear messaging adopted by NCDPI. These teachers spent extra time with Ms. Foil discussing what the transition involved, and also walked through a few testing simulations and online tutorials. New administration at the high school further encouraged the online testing agenda.
- Testing simulations. Ms. Foil found online tutorials and testing simulations that provided good examples of online testing scenarios for teachers and students to practice with. This district pays for ClassScape, which is a service that provides online test mockups that teachers and students can explore prior to taking the high stakes tests online. This helped to alleviate some apprehension and nervousness about what to expect on the tests.
- NCDPI’s Online. Assessment Tutorial is a requirement for all students to complete in a school setting at least one day prior to testing that helps testers understand what to expect and how to fill out an online test.
- Let people move at their own pace, if possible. Initially, Foil honored the reluctant teachers’ wish not to test online. Teachers eventually understood the decision was above her head and accepted having to administer the online tests. It was making sure their needs were heard and tended to that mattered most.
Had detailed instructions written down. One of the most difficult challenges was dealing with the ramifications of rapid, sometimes day-to-day, changes to the system and process. As Ms. Foil put it, “we are often building and flying at the same time, presenting opportunities for bottlenecks.” She found it helpful to have step by step instructions available to ensure things go smoothly, particularly on and prior to test day.
School Technology Tips on Test Day
- Allow more time to stagger computer logins (begin setup earlier).
- Restrict video streaming or other online activities.
- Have extra computers available.
- Have a plan for moving students with extended time.
- Distribute tips for interruptions, items not displaying properly, items appearing slowly.
Create an Online Assessment Tutorial
- Who? Every student participating in online EOC or EOG assessments
- Where? At least one time at school
- When? Before test day
- How? Using video or script
About NCTest Online
North Carolina has administered assessments online since the first operational version of the online Computer Skills Test was administered to all grade 8 students in 2005. The next generation of online assessments will allow the state to incorporate technology-enhanced items and implement computer adaptive testing, which will allow for a more accurate assessment of student learning. Support for moving to online assessments includes working with LEAs/Charter Schools to ensure that schools have the logistical and technical knowledge and skills to move rapidly to an online testing environment and a Best Practices Guide for Online Assessments (July 2011).
NCDPI articulates four pillars that guide their transition to online testing:
1. A comprehensive and balanced assessment system that includes:
- Formative assessments (e.g. learning activities, feedback protocols, questioning techniques, discussion activities)
- Summative assessments (e.g. selected response, multiple-choice, high-stakes testing)
- Performance assessments
2. Updated EOG and EOCs (online, innovative item types, computer adaptive): All current End-of-Course (EOC) assessments have been available online since 2007-08, and as of the completion of the 2012-13 school year, 61% of the EOCs were administered online. 2012-13 was the first year that any End-of-Grade (EOG) assessments were available online and 68% of the EOGs (for Science in grades 5 and 8) were administered online. For 2014-2015, the plan is to have all EOCs/EOGs developed specifically for an online environment with limited use of an alternate paper and pencil version.
Click here for an overview of Michigan’s 2013-2014 statewide assessments.
As of 2012-2013, all LEAs were expected to participate in online EOC testing. The following assessments were developed specifically for an online environment, some with technology-enhanced items.
- EOG Science at Grades 5 and 8
- NCEXTEND2 Science at Grades 5 and 8
- NCEXTEND2 Mathematics and ELA Grades 3-8
- EOC English I
- EOC Algebra I
- EOC Biology
- NCEXTEND2 English I
- NCEXTEND2 Algebra I
- NCEXTEND2 Biology
3. Tools to build common benchmark assessments and a variety of formative assessment tools.
4. Data tools to improve instruction
The Instructional Improvement System (IIS) is an online repository of shared tools and applications to help educators improve instruction. A key feature is integrating data from multiple sources including student course grades and central records.
The state has worked diligently to make digital teaching and learning a priority from the top, down to districts and schools. Efforts to “de-silo” curriculum, instructional technology and assessment throughout the state over the past ten years have gotten the attention of national agendas such as Race to the Top funds, positioning this state as an early adopter and leader in technology-integrated instruction.
With NC Education’s cloud-based model fully operational, the “face” of the shift from NCTest to SMARTER Balanced will be as inconspicuous as having a new link within the existing portal. Behind the scenes, schools are continuing to ready their hardware and connectivity infrastructures, as well as their people capacity, to fully deliver online SMARTER Balanced assessments and use the data they generate in 2014-2015.