Idaho’s rural setting fueled early investments in technology hardware and infrastructure with the promise of greater connectivity among districts. The state’s rollout of the Idaho Standards Achievement Tests (ISAT) in 2003 made Idaho an early adopter of statewide online assessment. While nearly all districts currently test online, some districts have lagged in securing the technical and/or human capacity to do so, resulting in a lengthy period of offering both online and paper-and-pencil tests. Statewide efforts toward online assessment readiness have focused on bolstering building-level capacity for online testing to match the robust infrastructure and reduce the effort spent issuing both types of exams.
History and Background
Idaho is a local control state comprised of more than 700 schools across 115 school districts, and nearly 50 charter schools, many of which are small and/or geographically remote. The promise of greater connectivity for such a rural populace spawned early investments in technology funded by federal, state, and grant dollars. For example, the Idaho Education Network (IEN) was instrumental in bringing high-speed broadband access to every school in the state. Despite most schools having the necessary hardware and bandwidth to support online testing, not all districts had the corresponding technical or human capacity to fully implement online assessments when they were first introduced statewide in 2003. What followed was gradual roll out of online assessment, including a lengthy period during which the state administered both paper-and-pencil and online exams. Officials at the State Department of Education (SDE) felt the dual testing modalities blunted the benefits of both systems.
The move to full online assessment in 2003 followed the recommendation of a ten-member committee comprised of business leaders and educators who held meetings across the state to find out what schools and communities wanted from statewide assessments. Alex Macdonald, Director of Instructional Technology for SDE, recalls, “When we moved the tests online, [SDE] wanted to really drive home collecting a lot of good data, and be able to provide that data back to our schools to make instructional and programmatic changes, and not have to wait months for results.” Idaho’s practice of listening to their constituents continued with the state’s choice to host adaptive assessments that could measure both proficiency on standards and student growth.
With the obvious first steps – infrastructure and direction – underway, SDE quickly realized the onus was on local district personnel to manage the functionality of online testing and management. Many of Idaho’s small and remote districts needed help increasing their level of technology skill down to the school and test proctor levels in order to successfully test online. Like the infrastructure and leadership support they put into the process early on, SDE responded with a robust plan for assessment professional development including specific goals and outcomes.
Milestones to Online Assessment
Before statewide launch
- Late 1990s—State and federal legislation sparked standards-based reform in Idaho, creating the new Idaho Achievement Standards intended to expand expectations and measures of student learning. The state also issued grants to districts to build their capacity for technology integration (see Idaho Technology Initiative Status Report 1994 – 2002).
- 2001—A state-convened selection committee took initial steps to develop a standards-based, computer-based system that measured individual student achievement. This prompted schools and districts to weigh in on what they wanted from future assessments.
- 2002—Reauthorization of the Elementary and Secondary Education Act helped to carve out mandates for the new system: that tests (a) are aligned with state standards, (b) provide diagnostics for individual students, and (c) permit class data to be broken down by subgroups such as race and income. Idaho further resolved to provide results to teachers in a timely manner.
- Pilot testing for an online, computer-based version of the ISAT was launched in the fall.
Year of statewide launch: 2003 – 2004
- 2003—The computer-based ISAT, fully online, was offered statewide.
After full launch
- 2008—Idaho’s Superintendent of Public Instruction launched a three-year initiative that addressed the state’s gap in a centralized longitudinal data system.
- 2010—SDE launched the Idaho System for Educational Excellence (ISEE), a longitudinal data system that collects student test scores, enrollment, attendance, and other data from districts across the state.
- 2011—Idaho launched ISEE Phase II longitudinal data system and the accompanying Schoolnet, which provides Idaho’s content standards, sample assessment items, and digital resources to Idaho’s teachers and school administrators through a software program. A $21 million grant from the J.A. and Kathryn Albertson Foundation ensured ISEE Phase II would be available to every classroom, providing teachers and administrators with real-time information to assess individual student and classroom learning.
- 2012—Idaho worked to align its testing system with the Common Core-compliant Smarter Balanced assessments, and selected districts to pilot the new assessment items.
- 2014-2015 (projected)—Full implementation of Smarter Balanced assessments.
Evolution of the Infrastructure
During the late 1990’s, the state issued grants to districts to build their capacity for technology integration (see Idaho Technology Initiative Status Report 1994 – 2002). This largely included the installation of computers and hardware, and expansion of bandwidth and wireless access. Although there were some early problems with computer speed and bandwidth at the time of Idaho’s piloting of the online ISAT in Fall 2002, it was estimated that schools in 75% of districts took the online tests, while others completed the same exams via paper/pencil. As the state prepares for Smarter Balanced assessments, all districts are being instructed to review the Technology Readiness Tool (TRT). Since the TRT’s minimum standards are at or below the state’s current requirements for the ISATs, the state anticipates about 98% of their districts already have the technology infrastructure in place to host the new online tests. As of 2010, Idaho has been operating on an average statewide ratio of 3:1 students-to-computers.
Recent political efforts have reflected the shift in priorities from devices to pedagogy. In 2010, Idaho governor C.L. Otter urged lawmakers to come together in funding the $60 million Idaho Education Network (IEN) with dollars from ARRA, e-rate, and grants. Otter called the IEN “one of the most promising opportunities for education and communication since the telegraph.” IEN is working to bring high-speed broadband access to every school in the state and is well on its way, having already met its goal of extending broadband connectivity to every Idaho high school one year ahead of schedule. This will both optimize online assessment items and help to manage complex data systems in the cloud. These efforts are foundations to linking Idaho’s rural school districts to progressive, 21st Century digital learning resources
One of Idaho’s larger technical concerns moving forward appears to be test security. Personnel are constantly making ongoing improvements on this front (e.g., updating their current ISAT to be secure on a Windows-based platform). State officials are working with districts to install a secure browser system on their devices, further reducing risk. While many schools have their own device policies and BYOD (bring your own device) provisions, statewide, SDE’s policy is that summative assessments can only be administered on secure devices. Macdonald foresees more centralized state- and district-level control over device selection when the Smarter Balanced assessments are released because of the need to assure test security.
While the onus to manage the logistics and functionality of online tests has largely been passed down to local districts, Macdonald stresses the importance of documenting current best practices and troubleshooting techniques at the local level that can be shared with districts statewide: “What we will try to do is produce a tool kit after the pilot period Spring 2013, so if a district has over 15,000 students for example, here are some practices and procedures that worked during our pilot program. I think that really helps to be able to address any concerns or any issues that are out there.”
Making the Transition
Involving educators in the change process. Through a contract with the Smarter Balanced Assessment Consortium, 125 schools were selected at random by SBAC across many districts to participate in the Smarter Balanced assessment pilot, through which SDE will garner best practices and feedback. Interest in the new assessment items was high—the state had to turn away districts for the pilot. This is indicative of the state’s history of involving educators at different levels in the ideation and vetting of change initiatives. When Idaho’s previous overhaul of its assessment system began over ten years ago, teachers, principals and community members were approached from the very beginning—and it was their feedback that prompted the state to zero in on computer-based testing. Teachers had wanted test results that were linked to standards, could give them relevant information about their students, and could measure individual student growth. The state moved forward accordingly.
The SDE provides districts the option to complete a survey, through a partnership with Idaho State University, that provides an inventory of what technologies they’re using and what things teachers and tech specialists need to learn about in order to utilize the technologies. This feedback is then shared with district leaders so they can narrow the scope of subsequent professional development. Macdonald notes this tailored approach is key, particularly for Idaho’s local-control, largely rural districts.
The SDE has a series of activities and resources for harnessing educator feedback and best practices, many of which are combined with incentives that translate to additional dollars for classroom budgets. Examples include an assessment best practices recognition program, a classroom assessment users group on Edmodo, a best practices survey, and student focus groups on the topic of assessment.
Shifting the focus from devices to pedagogy. SDE personnel currently feel “locked and loaded” with readiness for Smarter Balanced assessments. This is not to say the state has not faced its setbacks. In 2011, the governor and state superintendent had proposed a suite of education reform packages called Students Come First. The reform package included some key provisions (e.g., a 1:1 laptop program for high schools) that were dovetailed into the assessment, digital content, and online learning components. These laws were rejected by voter referendum in November 2012, forcing the SDE to move forward without these initiatives.
Despite the setback at the polls, this cloud’s silver lining was SDE’s shift in developmental focus from devices to the pedagogy—“because I think that’s really what got lost in translation,” says Macdonald. “It’s not the laptop, it’s what the laptop can do for the classroom, for the students…. In all honesty, the early focus was too much on the technology (i.e., connecting to projectors, trouble-shooting access issues) and not on the educational aspects…we really lost that conversational piece, the actual educational outcomes…”
Building capacity through professional development. SDE is currently completing the design of professional development modules that encourage an integrated approach of the agency’s three forward-facing initiatives – Idaho Core Standards, online assessment and digital content. Specifically, the modules are designed to help administrators understand the ties between the initiatives and how to use tools in their districts to meet these objectives. The state’s assessment and content area teams are working together to incorporate all learning objectives into the modules. While the state leads a top-down, train-the-trainer approach to professional development rollout, districts are simultaneously empowered to create professional development models that suit their unique local needs with “mini-grants” that cover the cost of developing capacity at the building level and among teachers.
Projecting a unified message. During rollout of the initial online system in the early 2000s, the state faced many communication challenges. From 2003-2005, just one member of the Office of the State Board of Education was tasked with fielding questions from every school district regarding the newly online ISATs. The State Board hosted regional workshops, and at one point, officials held regional meetings twice a year in an effort to communicate all the changes.
SDE hosted regional workshops, and at one point, officials held regional meetings twice a year in an effort to communicate all the changes. State personnel recall that communications from various departments were not always aligned in the early days. Since then, SDE has made a concerted effort to send a unified message about the pedagogical importance of online testing, Idaho Core Standards, and digital content.
Heterogenous district connectivity and the need to develop local capacity to administer the exams necessitated a gradual rollout of online testing. Early on, SDE recognized the need to provide ongoing, real time support to districts prior to and during the testing cycle. This involved securing the local personnel to complete important tasks related to testing (see box), and also to address technical issues as they come up. Since adding specialized personnel can be challenging for small and remote schools in particular, SDE recommends that districts embed the need for assessment training and a “helpline” into their strategic plans. This proactive, long-range orientation paves the way for a sustainable local online assessment effort, even if implementing the plan can take years in some locations.
Macdonald, who had been working at the district level in the early 2000s when online testing rolled out, foresees navigating the logistics of test administration as the biggest challenge for new states adopting online assessments, since such considerations as the scope of test sessions they need to run and the distribution of passwords to students each require several steps.
For districts with test administration personnel and processes in place, SDE underscores the importance of preparing teachers and students for the new assessments. In addition to training teachers and students on how to take the exams, SDE has articulated goals and plans for teachers to receive professional development on assessment. Like many states, SDE supports a train-the-trainer approach to local capacity building. SDE also recommends that students have ample opportunity to ‘practice’ with the testing platform, particularly the audio components, prior to live testing.
Full online management of Testing Process involves:
- Data uploads
- Editing exiting data
- Adding/Deleting Students
- Printing tickets
- Monitor/managing testing
- Reactivations, invalidations
- Managing permissions and passwords
- Manage reporting
- Connectivity issues
What Districts Did
Bolstered professional development opportunities for teachers in the areas of online testing and using assessment data. Empowered by Idaho’s local control model, districts have created and implemented their own vision for how they want to approach online testing. Districts not only have the flexibility to invest in infrastructures as they see fit, but also to define how they build the local capacity that is critical for statewide online assessment. For example, one of Idaho’s largest school districts, Coeur d’Alene (CDA), has implemented a differentiated professional development model that decentralizes efforts among two tiers of teacher leaders:
- The first level of teacher leadership through which assessment professional development and test preparedness are delivered within the district is called the Super 8. The Super 8 comprises eight of the district’s top teachers who were selected and first convened with district leaders in June 2012. This group received professional development based on Rick Stiggins’ Assessment Training Institute, and currently meets monthly to assist the district office with designing materials for the second tier of teacher leaders, the Top 40.
- The Top 40 is the second tier of teacher leaders, comprised of 40 of the district’s “best and the brightest” in their system. These teachers span grades K-12 and cover a variety of content areas. The Top 40 are trained at the district level to deliver assessment and test readiness professional development to their colleagues at the building level.
By contrast, the very small Highland Joint School District’s approach to capacity building has been less hierarchical. Because they are contained within a single building, educators there proceed more organically. The district’s combined Superintendent and Elementary Principal, Dr. Cindy Orr, described their change process as “slow and methodical.” For example, when they implemented a one-to-one computer program, teachers received iPads several months before students received theirs. This gave teachers a chance to get comfortable with the technology, develop a positive experience with the change process, and then move on to the next step. They adopted the same step-by-step approach with the new Smarter Balanced testing: “That way, when we started to see hiccups, people didn’t panic and say this is a horrible idea, but [instead] ‘this is working well; we just need to fix this.’ We can handle the hiccups, because their confidence is strong enough to survive the hiccups.”
Prioritized the equitable distribution of computers. Jean Bengfort, Director of Technology, and Mike Nelson, Assessment Director, described the number of logistical considerations they have had to implement since the onset of online testing at CDA. Since they do not have devices that are dedicated solely to testing, CDA blocks off existing media centers and utilizes mobile labs for testing. When the labs were first purchased, there were no specific tools available for inventorying their equipment and capacities. Nelson and his colleagues have created a matrix that aligns student testing cohorts and schedules with lab and technology availability across the district’s schools. Currently, CDA is experimenting with a shift from the traditional/lab testing approach to “hub” format where no computer sits empty. This limits the impact on instruction and maximizes the number of students that can be tested at a given time.
The Highland Joint School District is taking a different tack. For the eight years Dr. Orr has been administering the ISAT online in Idaho schools, including two years at Highland Joint School District, students have typically been seated in computer labs as a class and adhered to a specific testing schedule. Currently, two tests are administered per day—one class each in the morning and afternoon—in each of the building’s two computer labs throughout the five-week ISAT testing period. Rather than using existing computers in a “hub” configuration as CDA is doing, Dr. Orr and her colleagues are incorporating one-to-one devices that will allow students to take their time with the assessments and not feel rushed before the next class is ushered in. This is particularly important for HJSD’s mixed SES student population:
I think we’re really going to see the haves and have-nots start to filter through, because kids that have had computers at home since birth and parents who have a lot of technology there, they type very well…Kids who don’t have that experience at home and never see that technology until they come to school don’t have that…That’s part of the reason we went to the iPads in K-5, to get them onto technology, and we’re really looking at improving our keyboarding at a younger age…so that doesn’t hold them back.
About ISAT Online
The state contracted with Portland, Oregon-based non-profit group Northwest Evaluation Association (NWEA)—long-time experts in computer-based testing but newcomers to statewide assessment systems —to design the first online ISAT tests. A bank of more than 20,000 multiple choice questions in math, reading and ELA were developed and tailored to Idaho’s standards in 2001-2002. Online tests were distributed in fall and spring each year for grades 2-10, with a pilot launch in Fall 2002. Rollout was accompanied by workshops run by the Office of the State Board of Education (in collaboration with district testing and technology directors—who were often the same person), covering: (a) the basics of logging into the system, (b) understanding how the test works, and (c) administering online tests with students.
Today, through Computerized Assessments and Learning’s (CAL) digital assessment system, the summative, high-stakes ISATs are administered online in reading, language usage, and mathematics for grades 3-8 and 10, and in science for students in grades 5, 7, and 10, via a computerized format. Tests are multiple-choice only and not adaptive.
CAL’s ISAT test delivery engine is installed on each computer, which creates an ISAT shortcut icon on the desktop. Double-clicking this icon will open the test application, which also offers access to student tutorials and practice tests. Students receive preliminary scores immediately while teachers receive preliminary class summaries within 24 hours, and schools/districts receive data reports within 72 hours. More than 95 percent of the ID student population completes tests via CAL. Accommodations are available for students who need to access the test in alternate formats (i.e., braille, audio, large print, paper and pencil; see the 2012-13 Testing Coordinator’s Guide for more).
A decade after launching its ISAT online testing pilot, the state is now administering their first adaptive online assessments as part of the SMARTER Balanced pilot. The new tests have fewer multiple choice questions and deeper assessment items than the ISAT (see the SDE’s presentation comparing current ISAT design with the SBAC tests).
Students across the state have largely been taking ISAT exams in classroom rotations through computer labs in 1.5-hour intervals. To meet the challenge of increasingly varied test times, some ID districts such as Coeur d’Alene have been experimenting with the “hub approach,” whereby new students are ushered onto each testing computer as it becomes available, so no device sits empty.
SDE feels ready for the upcoming Smarter Balanced assessments. Idaho’s history of listening to its people and building local capacity will undoubtedly continue through the Smarter Balanced assessment transition in 2014-2015. As Mike Nelson from CDA said,
“There is that feeling in our district that we must maintain the promises that we have in our local community; in other words, maintaining a highly effective school system. I think part of it is just riding the wave of change. In Idaho, we have been very used to change over the past couple of years. We have several large initiatives move in, move out…I think that Idaho teachers are pretty resilient and I think that they heed the change, but hopefully we’ve built enough capacity in what the Common Core is, what it stands for, and how the curriculum will change in our district to thrive during the transition.”
Click here for an overview of Michigan’s 2013-2014 statewide assessments.