Digitally Based Assessments

Schools are increasingly using technology in the classroom to teach and assess students. NAEP assessments are moving forward to align with school practices.

In addition, this move will allow NAEP to:

• Use tools available in digital platforms to measure content in new ways;
• Use assistive technology to provide enhanced accommodations for students with special needs;
• Collect new types of data that deepen our understanding of what students know and can do, including how they engage with new technologies to approach problem solving;
• Make the assessments potentially more engaging to the students; and,
• Align with the delivery mode of many other large-scale assessments.

Across the program, NAEP has introduced a variety of new tasks in the assessments, such as scenario-based tasks and interactive computer tasks. These tasks ask students to engage with real-world problems or situations and to work through a series of questions to demonstrate what they know and are able to do in ways that more accurately reflect what is happening in today’s classrooms.

NAEP’s digitally based assessment tasks reflect the growing use of technology in education. Some tasks may include multimedia, such as audio and video. Others may allow the use of embedded technological features (such as an equation editor) to form a response.

These new assessment characteristics are expected to allow NAEP to capture information about students’ problem-solving processes and the strategies they use when responding to assessment questions. For example, while a paper-based assessment would only yield the final responses in the test booklet, digitally based assessments can capture information about students’ use of the tools and whether students change their answer. Thus, digitally based assessments expand what NAEP can learn about the performance of both successful and less successful students.

The transition to digitally based assessments will allow NAEP to make various universal design features available to all students, and to use assistive technology to offer accommodations for students with special needs. In a digital environment, what was previously an accommodation in paper-based testing becomes a seamless part of universal design. That means that features like adjusting font size, having test items read aloud in English (text-to-speech), changing the appearance of the testing interface to have a higher contrast, and using a highlighter tool, are available to all students during the test administration.

In addition to these universal design features, NAEP continues to offer accommodations required in students’ Individualized Education Program (IEP) and 504 plans. Some accommodations are available in the testing system (such as additional time or bilingual forms of the questions in mathematics), while others are provided by the test administrator or the school (such as breaks during testing or sign language interpretation of the test).

NAEP began conducting studies to support the transition to digitally based assessments (DBA) in 2001. Initial research was conducted during 2001 through 2003. These investigations included a math online study, a writing online study, and a problem solving in technology-rich environments study. From 2008 through 2015, NCES incorporated technology in a variety of assessments to have a broader coverage of new frameworks. These assessments include science interactive computer tasks, writing, and technology and engineering literacy (TEL) — all of which require students to interact with digitally based platforms. Additionally, in 2015, NAEP began conducting digitally based pilot studies to gather data to support the transition to DBA. Over the next several years, all NAEP subjects will transition to DBA. The graphic below highlights milestones in NAEP’s exploration of digitally based assessments.

A team of approximately 3 - 4 NAEP field staff set up and administer the assessments. NAEP provides all necessary equipment including student tablets with an attached keyboard, stylus, earbuds, an administrator tablet, and a router that provides a closed wireless network for the devices to communicate. Schools are asked to provide space, desks or tables, and access to electrical outlets. Schools do not need to provide internet access. Two sequential sessions are conducted at each sampled school, with approximately 25 students in each session. The graphic below illustrates the setup and flow of data during the administration.

NAEP assessments are designed to test the knowledge and skills that are relevant to the content area being tested and not to test students’ proficiency with technology. NAEP conducted cognitive laboratories and play testing with students during the development of the digitally based platform. To support students’ interaction with the digital assessment, NAEP offers tutorials before the assessment that allow test-takers to become familiar with the system, its tools, and its functionality. These tutorials provide demonstrations of the assessment platform and tools and allows students to have hands-on experience before the assessment begins. The tutorials are available to the public online, allowing participants to become familiar with the digital platform. Additionally, there are help screens built into the system that students can access at any time during the assessment. Finally, NAEP is conducting research to investigate the relationship between student performance and computer access and familiarity.

Some students in each state and urban district took the paper-and-pencil forms of the assessment in 2017 in order to investigate potential differences between digitally based and paper-and-pencil administrations within the same year.

The 2017 transition to digitally based assessments in reading and mathematics at grades 4 and 8 will lengthen the period between administration and reporting results that states and districts have previously experienced. NCES plans to extend the typical six-month window for the reporting of 2017 NAEP mathematics and reading results. The extension of the reporting timeline is necessary to allow for the additional analyses required because of the administration of both digitally based and paper-and-pencil assessments. Results are expected to be released in early 2018. Future grade 4 and 8 mathematics and reading results should be released under the traditional six-month reporting timeframe.

NCES designed the 2017 assessment to ensure the move from a paper-based assessment to a digitally based assessment did not compromise our ability to report on student progress. As reported by other large-scale assessments, student performance generally differs when the assessment mode changes. NCES examined this potential mode effect in NAEP with a set of rigorous analyses.

In 2017, random samples of students took either the paper or digital version of the reading or mathematics assessment in each state and participating urban district. NCES then compared the performance of all students and student groups (e.g., race/ethnicity, gender) nationally, as well as for states and large urban districts. These analyses will determine whether specific states, districts, or student groups in the nation are impacted differently given the NAEP transition to digitally based assessments.

The impact of the transition to digitally based assessments on the NAEP frameworks varies by subject. For example, the technology and engineering literacy (TEL) framework was originally written for digital administration; both the science and the writing frameworks have been changed to address the transition to digitally based assessment; and, the frameworks for reading, mathematics, and the social sciences have not changed. However, the item types used during NAEP paper-and-pencil administrations were limited to two formats: multiple-choice and constructed response (short or extended). Digitally based administrations allow for the inclusion of other types of selected-response items, such as multiple-choice questions where more than one correct answer can be selected; items where students drag and drop elements to complete a matching or ordering task; questions where students select an area of an image (such as a map or graph) rather than selecting a traditional text option; and, some additional interactive tools that have been deemed appropriate for assessing the content knowledge, skills, and abilities described in the frameworks.

Federal law dictates complete privacy for all test-takers and their families. Under the National Assessment of Educational Progress Authorization Act (Public Law 107-279 III, section 303), the Commissioner of the National Center for Education Statistics (NCES) is charged with ensuring that NAEP assessments do not ask test-takers about their personal or family beliefs or make their personal identity information publicly available.