Virginia has over a dozen years under its belt administering various iterations of online assessments. They began in the late 1990’s building a robust infrastructure in the public school divisions (or districts) and have continued to invest nearly $60M annually to support and maintain the infrastructure for both instruction and assessment needs. The first online administration of Virginia’s Standards of Learning (SOL) tests occurred in fall 2001, and following that, online assessments were gradually phased in over five years, first with end-of-course tests in high schools by 2004, followed by middle school and elementary school assessments by 2006. During the phase-in, each school division could determine its own timeline for implementing online testing while Virginia continued to provide financial, assessment, and technological support to assist the transition away from paper-and-pencil tests.
The positive online testing experiences among early adopting school divisions served as one of the strongest endorsements for other school divisions to begin implementing online tests. Participating divisions were significantly reducing the number of boxes, and even pallets, of paper test booklets and answer documents they received, counted, and distributed to schools for each test administration only then to be handled, counted, and received again in reverse prior to shipping them off to be scored. Students seemed to prefer the online tests, and the online test results were available almost immediately while schools waited days and even weeks for paper-and-pencil test results to be available. Virginia passed legislation in 2010 requiring schools to administer all end-of-course SOL tests online by spring 2011except in cases where a student had a documented need for a paper-and-pencil test (e.g., Braille test). In spring 2012, middle school students were required to test online and in spring 2013, the same requirement extended to Virginia’s elementary school students. Currently, Virginia’s 132 school divisions administer over 3.2 million online assessments each school year and have administered over 180,000 online tests in a day and over 780,000 in a week during peak online testing periods.
History and Background
The Standards of Learning (SOL) Technology Initiative was passed as legislation in 2000 by the Virginia General Assembly and laid the foundation for online testing in the state. The Initiative’s goal was to equip schools statewide for improved instruction, remediation, and assessment through four priorities:
- Providing at least one computer for every five students
- Establishing Internet-ready local area networks
- Installing high-speed Internet access for every school with adequate bandwidth to meet demands for both instruction and assessment
- Establishing a statewide online assessment system
The Virginia General Assembly authorized the Virginia Public School Authority (VPSA) to sell Technology Equipment Notes, a form of municipal bonds, to generate nearly $60 million in proceeds annually to fund the Initiative. As part of each school division’s agreement to receive the annual technology funds, school divisions have contributed about $11.6 million annually as a required local match to the General Assembly’s Technology Grants. Funds were directed toward school division infrastructure and workstations first at the high schools, followed by middle schools, and then elementary schools as the state continued its phase-in of online testing each year.
As Virginia’s SOL Technology Initiative went live in 2000, the state was administering about 2.5 million paper-and-pencil tests each year across grades 3, 5 and 8, with end-of-course (EOC) tests administered mostly in high schools. One of the Initiative’s early activities called for the Virginia Department of Education (VDOE) to select an online assessment vendor that could demonstrate its success with administering online assessments in Virginia. VDOE officials knew their investment in infrastructure and successful pilot testing were necessary first steps for assuring a successful eventual conversion from paper-and-pencil tests to fully online test administrations.
VDOE gradually phased in online tests, retaining paper-and-pencil tests and post-equating scores from each mode separately to account for any differences between the testing modalities. Interest in online testing spread quickly after early adopters observed students’ preference to test online during the first online test administrations in 2001. (Figure 1.) By 2004, all but one of Virginia’s school divisions was administering online tests.
The number of different SOL tests offered online in Virginia has increased each year from 2001 through the spring of 2006. (Figure 2.) VDOE began with its summative EOC tests online in the high schools, and gradually phased in the online versions of the SOL tests for the middle and elementary grades over the next five years. While VDOE was transitioning its SOL tests to an online format, the school divisions were continuing to upgrade their infrastructure and equipment. By spring 2009 when the Virginia legislature expected all schools to be capable of testing online, all 132 school divisions were voluntarily administering some number of tests online. Between fall 2010 and spring 2013, the General Assembly phased in a requirement for all SOL tests to be administered online except to those students with a documented need for a paper-and-pencil test.
VDOE has approached the Technology Initiative’s four priorities with a project management structure that continues to guide school divisions today.
Milestones to Online Assessment
Before statewide launch
- 1999 – VDOE received $50M in proceeds from the sale of Technology Equipment Notes, or municipal bonds to implement online assessments.
- Winter 1999 – The Commonwealth of Virginia launched the SOL Technology Initiative that outlined four priorities for delivering a 21st Century education statewide and prioritized funding to implement the Initiative.
- Fall 2000 – Initial field-testing among three vendors across nine high schools with varying levels of technical capacity resulted in the selection of one online test vendor, Pearson, in 2000.
- Winter 2001 – VDOE introduced preliminary infrastructure architectural guidelines for high schools, followed by final guidelines in July 2001.
- 2001 – The first operational administration of an online test in Virginia took place during the fall of 2001. 1700 tests were administered online and scored.
- Summer 2004 – Online assessment certification procedures were revised to expand divisions’ usability, flexibility, and capacity to test online.
- 2004 – All divisions – primarily high school but some middle school as well – participated, at least to some degree, in online assessment.
Year of statewide launch: 2005
- Fall 2005– All 132 divisions participated in online SOL testing, even though some still also administered paper-and-pencil tests.
After full launch
- 2006 – All tests were available online for the spring test administration except for the writing test. Schools could go completely online at this time if they wanted, but online testing was still voluntary.
- Fall 2008 – The number of paper-and-pencil tests administered to students continued to decline as schools opted to administer more of their tests online.
- 2010 – Legislation passed in the General Assembly that required the phase-in of mandatory online SOL testing except for students with a documented need to complete a paper-and-pencil test (types of documented needs include Braille, large print, or seizure disorders that prevent students from using computers). Also in 2010, VDOE received a federal grant to expand its data warehouse to include data from higher education and the workforce.
- Spring 2011 – New online item types called technology-enhanced items (TEI) were introduced and field-tested in online mathematics tests, allowing for assessment with question types other than multiple choice.
- Spring 2012 – VDOE administered its new online mathematics tests that included TEI and conducted its first online English Writing test as an online census field-test at grades 5, 8, and EOC. Technology-enhanced items were field tested in reading, science and writing tests. Also in 2012, VDOE participated in research with students responding to different item types, including open-ended items, with various devices such as tablets, tablets with external keyboards, laptops, and desktops.
- Spring 2013 – VDOE continues to expand its data warehouse to include information from preschools, institutions of higher education, and the workforce. Virginia’s first operational online English Writing test was administered at grades 5, 8, and at EOC.
Evolution of the Infrastructure
In Virginia, the SOL Technology Initiative provided direct guidelines and funding for infrastructure development to support online testing through School Readiness Certification. In order to test online, schools had to first certify their network was capable of handling a simultaneous test load by completing testing with a diagnostic simulator. Initially co-designed with test developer Pearson, the simulator grew into a certification process that demonstrated division network capacity to handle online testing. As the number of individual schools administering online tests increased in each school division, it became logistically difficult to test load at all locations simultaneously. Pearson, in collaboration with VDOE, developed a bandwidth estimator that allowed division technology staff to map their network with existing bandwidth metrics and calculate network demand based on the numbers of concurrent testers.
The SOL Technology Initiative funding provided each school division with $50,000 for the division, plus another $26,000 per school, as long as the division provided a 20 percent local match. The state portion of these funds could be spent on capital purchases such as power upgrades, wiring, computers and wireless routers. All of the 20 percent local match portion, along with only five percent of the state allotment, could be used for “consumables” such as software and training.
Divisions have used SOL Technology Initiative funding to achieve and maintain School Readiness Certification. As of 2012, all high schools, 99% of Virginia’s 133 middle schools and 98% of 131 elementary schools were ready to fully administer online tests, having achieved the certification. Due to high numbers of currently certified divisions, the process for becoming certified has been substantially abbreviated from its original format, currently involving a single signature page Excel file called the School Readiness Workbook.
Today, bandwidth and network capacity are virtually non-issues, as all school divisions in Virginia are currently online with adequate broadband and many are moving to wireless connectivity. VDOE is now turning its attention to managing the information generated by online assessments. While there is no statewide student information system per se, all school divisions submit student test scores to a data warehouse that began tracking students by unique identifiers in 2005. VDOE recently received a $6M federal grant to further develop their longitudinal data system so it includes more information about students than assessment scores alone. In 2010, VDOE received a second federal grant to add data from higher education and the workforce to the system.
Meanwhile, divisions are craving assessments that can be launched on touch screen devices, since test software currently only runs on desktop and laptop computers with certain specific operating systems. VDOE is currently doing research on what it means to assess students on a touchscreen, including pilot testing touchscreen-compatible assessments during 2012-2013 and 2013-2014, and addressing security and policy issues around the shift to smaller, more mobile devices such as tablets. The Department is planning a gradual phase in of tablet-compatible assessments during 2014-2015 and beyond that may look much like the original phase in of computer-based assessments, with various tests becoming available on tablets and mobile devices in waves rather than all at once.
Due to technological problems with Pearson, the testing contractor in 2007, a small overall percentage of students struggled with testing on two particular days. This situation caused VDOE and Pearson to come up with a two-part solution for better tolerating Internet loss during testing.
- Implementing changes that would prevent error messages and blue screens from displaying and distracting students when testing during loss of Internet connectivity.
- Supporting continued testing during a loss of connectivity by using cached test questions and saving student responses locally that could be uploaded later when Internet connectivity was restored.
Making the Transition
Adequately funding technology and online assessment. A major factor in Virginia’s successful transition to online testing was the financial resources applied to the effort each year through the SOL Technology Initiative with associated directives requiring those funds to be used toward achieving school readiness. The Virginia General Assembly has funded several aspects of launching online assessments ranging from infrastructure installations to hardware upgrades to studies that compare scores from online to paper-and-pencil tests.
Setting a realistic pace for change. VDOE has viewed the transition to online assessment as a decade-long effort that spanned virtually every aspect of schooling. The Department began by transitioning high school assessments online, which were followed by middle and elementary online assessments. Online tests were added each year through 2006, and while the Department strongly encouraged the transition to online testing, the paper-and-pencil test versions remained available to schools to ease the transition. This option of administering paper-and-pencil tests enabled each school division to determine its own plan for completing the transition. While some school divisions transitioned to online tests by grade level, others transitioned by subject area, or even by school location. This option continued until legislation was passed in 2010 setting the stage for required online testing at all levels by spring 2013 except for those students with a documented need for a paper-pencil test. Whereas other states found the dual administration modalities to be cumbersome, Virginia found that providing each school division with flexibility to determine its own transition plan provided more effective plans and positive experiences for all involved.
Differentiating testing configurations in secondary schools from elementary schools. VDOE discovered that testing configurations in secondary schools were different than elementary schools because of how computers were deployed. Computers in high schools were more typically set up in labs, which facilitated testing entire classes of students. By contrast, elementary schools tended to have a handful of computers in each classroom and limited computer lab space, so the ability to use carts with class sets of laptops for testing was more important to this group. Conveniently, wireless technologies were becoming more prevalent at the same time many Virginia elementary schools were beginning to revise their infrastructure to support online testing. Mobile carts with laptops were being equipped with wireless access points for connecting the laptops to the Internet both for instructional use and online SOL testing.
Delaying the move to online for writing assessments. VDOE handled online writing tests separately from other subjects because of concerns with students composing on the computer. The Department conducted a field test of the online SOL Writing test in spring 2012 with students in grades 5, 8, and 11. Teachers and students reacted positively, with the majority of 5th graders indicating they preferred the online writing test. The issues related to keyboarding skills that were anticipated for this group of younger writers in particular were not observed during the field test. In general, this approach helped generate buy-in for the overall move to online tests.
Post-equating online scores separately from paper-and-pencil scores. Due to the high stakes nature of Virginia’s assessment program, the VDOE invested in conducting comparability studies to check for differences between online and paper-and-pencil tests when administered to high and middle schoolers. Studies found only very small differences between testing modalities for high school reading, science, history, and math tests that sometimes favored the online modality and sometimes paper-and-pencil. The Department continued to post-equate online tests separately from paper-and-pencil tests until not enough paper tests were being administered in each test for the calculation to be possible.
Setting up a process for information flow with roles designated for overseeing online testing. In addition to infrastructure development, VDOE relied on the existing positions of Division Director of Testing (DDOT) and School Test Coordinator (STC) who together were responsible for assuring the paper-and-pencil assessments were deployed properly and became responsible for understanding the assessment aspects of online testing. The VDOE required that each division identify a Project Manager (PM) from among existing division-level technology staff who could lead the technical aspects of the SOL Technology Initiative and collaborate with the DDOT. Most often, this was the division’s director of technology. The Department facilitated and strongly encouraged regular communication and planning meetings between the DDOT and PM. The DDOT and PM serve as the point of contact between the school division and VDOE and have division-wide responsibility for implementing online tests within the division; the analogous STC serves as the point of contact between the DDOT and the school. STCs, often guidance counselors or assistant principals in the schools, are also responsible for ensuring that all assessment procedures required for online tests are implemented within the school and that the security of test materials is maintained. The PM usually has individuals with technical experience in each school or assigned to a group of schools to serve in a similar role as the STC but with regards to the technical procedures required in the school for online testing. Test examiners and proctors in each of the state’s 132 school divisions, positions held by professional staff with other responsibilities such as guidance counselor or classroom teacher, are responsible for administering online tests according to procedures on test day.
The gradual transition to online testing revealed a range of enthusiasm for the change. Approximately 1700 online tests were administered in the early-adopting divisions during the fall of 2001, which increased precipitously as word spread across the state about the benefits of testing online, and as more schools and divisions achieved School Readiness Certification.
Any criticism and discomfort with moving online largely came from teachers and adults who were initially concerned about the possibility of their students not understanding or being anxious about the new test format and ultimately scoring lower on online tests than paper-and-pencil tests. As post-equated tests revealed no significant differences between testing modalities and teachers observed students’ strong preference for testing online, any concerns about the move to online waned. Teachers also recognized how well prepared the infrastructure was to support testing, which underscored the promise for a smooth transition.
During the years between 2001 and 2004 when most districts began testing online in some way, VDOE continued to emphasize the benefits of online testing including a quicker return of test results which allowed school divisions to revise their test schedules and continue instructing students longer into the school year before stopping to administer the high stakes tests. A memo from the Superintendent of Public Instruction in 2008 further communicated educational technology guidelines to school divisions, including updated information about school certification for readiness to test online. This memo was part of the continued, unified message sent about the state’s ongoing prioritization of resources toward improving technology capacity statewide.
What Districts Did
Mary Wills, Supervisor of Assessment and Testing and DDOT for the moderately sized Fauquier County Schools in Warrenton, Virginia recalled how teachers in her division had to see for themselves how well online testing was working. For them, the transition to online testing required a culture shift with good communication at the heart of the change. Then, once they began to shift, the overwhelming interest in moving online generated a rapid need for devices. “It was a very short time period from ‘yeah I’ll try it’ to ‘everyone online’.” Two of Mary’s most successful strategies have been to:
- Require all district technology purchases to fit the minimum requirements for online testing, and to visibly mark computers that are not usable for testing. The current SOL tests do not run on tablets, for example, so tablet purchases must be made through grants and other non-Technology Initiative sources. Mary makes sure all tablets and any computer that cannot be used for testing have a visible sticker placed on the front so they are clearly understood to not support testing.
- Keep a spare class set of laptops set aside, since they often end up needing to be used. For example, Mary often finds those laptops come in handy to create private testing areas for accommodations such as read aloud test administrations.
Wills and her colleague from another school division Mark Blankenship, Bedford County’s Division Test Coordinator, agree that student enthusiasm and aptitude for testing online were both very high as online testing began in their districts. As Blankenship recalled, “the kids would beg to take the tests online. And they would have no problem logging on to take them.” Teachers came to appreciate that scores were returned quickly and were supported with graphical displays that gave them more meaningful information than before.
The pair also agrees that good planning and teamwork can overcome the setbacks from inevitable power outages, loss of Internet connectivity, and even human error. Wills and Blankenship also remind that the most proactive teams include a variety of people in different roles such as building level guidance counselors, technology personnel, and even VDOE staff.
Mary calls the level of support she has received from VDOE during the transition to online testing “generous” and “personal”. She praised the Department for wisely prioritizing infrastructure before testing. This personal contact she has with the Department provides a welcome complement to the modules, practice items, implementation manuals, available from the Department’s website that parents and the public can use to get informed about online testing.
Virginia’s online tests are designed to assess whether students meet specific minimum expectations for learning and achievement expressed in the Virginia Standards of Learning. Virginia teachers, school administrators and content specialists participate in the development of Virginia’s online assessments by serving on committees that review test items and forms to ensure they measure student knowledge accurately and fairly.
Test blueprints guide developers as they write test questions and construct the assessments. They are available for: English, Mathematics, Science, History/Social Science. The online version of the English Writing tests for grades 5, 8, and EOC levels required significant changes in how the test was developed and administered. The Division of Student Assessment and School Improvement began the initial planning for this eventual transition during the 2008-2009 school year.
VDOE also releases sample sets of online test questions that are representative of the content and skills addressed on prior tests of: English, Mathematics, Science, and History/Social Science. The Department provides additional practice tools available for the public and parents to use such as Practice Items and Practice Item Guides, Writing Practice Tool, Reading Demonstration Items, and Electronic Practice Assessment Tools (ePAT).
In its first iterations, Virginia’s online assessments were simply computer-based versions of their paper-and-pencil predecessors. Now, the Department has phased in some technology-enhanced items, fill in the blank, and drag and drop items that depart from the standard multiple-choice format. These items are scored in the same way as their paper-and-pencil predecessors with no partial credit, even though students might have to complete more complex tasks such as marking more than one answer or constructing a graph. Virginia’s current online mathematics, reading, writing, and science tests, at all levels, now include up to 20 percent technology-enhanced items along with the multiple-choice items that all assess the recently revised and more rigorous Standards of Learning. At present, Virginia’s online assessments are only administered on a laptop or computer, with anything other than a Windows or Mac workstation (e.g. chromebook, linux-based machine) being unsupported.
As with other assessment transitions in Virginia, VDOE is investigating computer adaptive testing through pilot tests in mathematics in a handful of middle schools during the fall of 2013 and in reading during the spring of 2014. The Department expects improved test security, savings of time and resources, and better reporting to result from the addition of computer adaptive testing. Computer adaptive testing and the introduction of additional device types such as tablets are future changes Virginia is working to implement for their state assessments.
The Department administered all of its tests online totaling approximately 3.2M tests in 2012-2013; 2.6M were given in the spring of 2013. Up to 180,000 online tests were administered on a given testing day during the spring. VDOE has planned for this kind of volume by providing ample resources on their website to support online testing such as test implementation manuals, sample questions and test simulators.
Click here for an overview of Michigan’s 2013-2014 statewide assessments.