1 Introduction

1.1 Objectives of this Volume of the Voting Systems Standards

Volume II, Voting System Qualification Testing Standards, is a complementary document to Volume I, Voting System Performance Standards. While Section 9 of Volume I provides an overview of the qualification testing process performed by the Independent Test Authorities (ITAs), Volume II provides specific detail about the process that is necessary for ITAs, vendors, and election officials participating in the qualification process. The Standards envision a diverse set of users for Volume II, including:

¨ Vendors: Voting system vendors will use Volume II to guide the design, construction, documentation, internal testing, and maintenance of voting systems to ensure conformance with the Standards. Vendors will also use Volume II to help define the obligations of organizations that support the vendor’s system, such as suppliers, testers, and consultants.

¨ Independent Testing Authorities: Testing authorities certified to qualify systems will use Volume II to guide the testing of voting systems and preparation of test reports. Laboratories and other parties interested in becoming ITAs can use Volume II to understand the requirements and obligations placed on the ITAs involved in the process.

¨ Election officials: Voting officials in many jurisdictions will use Volume II to guide system certification, procurement and acceptance requirements and processes, which may include additional requirements and adjustments to those requirements included in the Standards.

1.2 General Contents of Volume II

To support these primary users of the Standards, Volume II provides:

a. A discussion of the general sequencing of tests performed by the ITAs: Volume II identifies the tests where sequencing is important and provides such required sequences. Volume II also indicates other tests that may be conducted in parallel.

b. A detailed description of the information required to be submitted by voting system vendors in the Technical Data Package (TDP): The TDP consists of a comprehensive set of documentation that contain system design specifications, operating procedures, system testing information, facility and resource requirements for system operations, system maintenance instructions for jurisdictions, and vendor practices for quality assurance and configuration management that underlie the development and update of the system. The TDP focuses predominantly on the required documentation contents, providing flexibility to vendors to determine the best format for meeting the content requirements.

c. Delineation of specific system tests to be conducted by the ITAs: Volume II identifies specific tests that are to be conducted relating to system components and to the integrated system as a whole. Tests are defined for system functionality, hardware, software, telecommunications and security that address the performance standards delineated in Volume I.

d. Delineation of specific examinations of other information provided by the vendor: Volume II identifies the criteria to be used by the ITAs in conducting examinations of the information submitted in the TDP. These criteria address the documentation provided in the TDP, including documentation of the system and related operational procedures as well as vendor practices for quality assurance and configuration management.

e. Description of process for handling failures: A system may fail to pass one or more of the tests and examinations performed by the ITAs. Volume II describes the practices to be used by the ITAs when the system or its documentation fails a test or examination, including the nature and depth of re-testing required for corrections submitted by the vendor.

f. Outline of Qualification Test Report. Volume II provides an outline of the report issued by the ITAs at the conclusion of testing, providing the specific requirements for this report.

1.3 Qualification Testing Focus

Qualification tests focus on multiple aspects of the voting system and the process for development and maintenance. Although multiple ITAs may conduct qualification testing, with each ITA conducting tests in its areas of expertise, the focus of their combined activities remains the same. Overall, qualification testing focuses on:

a. The functional capabilities of the system to support specific election activities performed by system users, including election officials and voters, as defined in Volume I, Section 2 of the Standards;

b. The performance capabilities of the system that ensure accuracy, integrity, and reliability of system operations and the election activities that rely on them, as defined in Volume I, Sections 3, 4, 5 and 6 of the Standards;

c. The system development and maintenance processes and related quality assurance activities performed by the vendor to ensure system quality, as addressed in Volume I, Section 7 of the Standards;

d. The configuration management activities used to control the development and modification of the system and its individual components, and maintain accurate information about the version and status of the system and its components throughout the system life cycle, as addressed in Volume I, Section 8 of the Standards; and

e. The documentation developed and maintained by the vendor to support system development, testing, installation, maintenance and operation, as addressed by the TDP described in Volume II, Section 2.

1.4 Qualification Testing Sequence

The overall qualification test process progresses through several stages involving pre-testing, testing, and post-testing activities as described in Volume I, Section 9 of the Standards. Whereas Volume I describes the flow of the overall process, Volume II focuses on the details of activities conducted by the ITA and activities conducted by the vendor to facilitate testing and respond to errors, anomalies, and other findings of concern during the test process.

Qualification testing involves a series of physical tests and other examinations that are conducted in a particular sequence. This sequence is intended to maximize overall testing effectiveness, as well as conduct testing in as efficient a manner as possible. The ITA follows the general sequence of activities indicated below. Note that test errors and anomalies are communicated to the vendor throughout the process.

a. Initial examination of the system and TDP provided by the vendor to ensure that all components and documentation needed to conduct testing have been submitted, and to help determine the scope and level of effort of testing needed;

b. Development of a detailed system test plan that reflects the scope and complexity of the system, and the status of system qualification (i.e., initial qualification or re-qualification);

c. Operational testing of hardware components, including environmental tests, to ensure that operational performance requirements are achieved;

d. Functional and performance testing of hardware components;

e. Examination of the vendor’s Quality Assurance Program and Configuration Management Plan;

f. Code review for selected software components;

g. Functional and performance testing of software components;

h. System installation testing and testing of related documentation for system installation and diagnostic testing;

i. Functional and performance testing of the integrated system, including testing of the full scope of system functionality, performance tests for telecommunications and security; and examination and testing of the System Operations Manual;

j. Examination of the System Maintenance Manual;

k. Witnessing of a system ‘build’ conducted by the vendor to conclusively establish the system version and components being tested; and

l. Preparation of the Qualification Test Report.

1.5 Evolution of Testing

An ITA will conduct extensive tests on a voting system to evaluate it against the requirements of the Standards. Taking advantage of the experience gained in examining other voting systems, ITAs will design tests specifically for the system design and documentation provided by the vendor. Additionally, new threats may be identified by the information technology professional community over time that are not directly addressed by the Standards or the system. As new threats to a voting system is discovered during the system’s operation, or during the operation of other computer-based systems that use technologies comparable to those of another voting system, ITAs shall expand the tests used for system security to address the threats that are applicable to a particular design of voting system.

1.6 Outline of Contents

Volume II of the Voting Systems Standards is organized as follows:.

¨ Section 2 describes the requirements for the Technical Data Package;

¨ Section 3 describes functionality testing;

¨ Sections 4 and 5 describe specific testing standards for hardware and software;

¨ Section 6 describes standards for testing the fully integrated system, including telecommunications and security capabilities, and the documentation used to operate the system;

¨ Section 7 describes the standards for examining the documentation of vendor practices for quality assurance and configuration management;

¨ Appendix A provides an outline for the Qualification Test Plan;

¨ Appendix B provides an outline for the Qualification Test Report; and

¨ Appendix C describes the guiding principles used to design the voting system qualification testing process performed by ITAs.

Volume II, Section 2

Table of Contents

2 Technical Data Package

2.1 Introduction

2.1.1 Content and Format.............................................................................................................

2.1.1.1 Required Content for Initial Qualification....................................................................

2.1.1.2 Required Content for System Changes and Re-qualification....................................

2.1.1.3 Format........................................................................................................................

2.1.2 Other Uses for Documentation...........................................................................................

2.1.3 Protection of Proprietary Information.................................................................................

2.2 System Overview

2.2.1 System Description..............................................................................................................

2.2.2 System Performance...........................................................................................................

2.3 System Functionality Description........................................................................................................

2.4 System Hardware Specification..........................................................................................................

2.4.1 System Hardware Characteristics.....................................................................................

2.4.2 Design and Construction.....................................................................................................

2.5 Software Design and Specification.....................................................................................................

2.4.3 Purpose and Scope.............................................................................................................

2.4.4 Applicable Documents........................................................................................................

2.4.5 Software Overview...............................................................................................................

2.4.6 Software Standards and Conventions.............................................................................

2.4.7 Software Operating Environment.....................................................................................

2.4.7.1 Hardware Environment and Constraints..................................................................

2.4.7.2 Software Environment..............................................................................................

2.4.8 Software Functional Specification...................................................................................

2.4.8.1 Configurations and Operating Modes......................................................................

2.4.8.2 Software Functions..................................................................................................

2.4.9 Programming Specifications............................................................................................

2.4.9.1 Programming Specifications Overview...................................................................

2.4.9.2 Programming Specifications Details.......................................................................

2.4.10 System Database............................................................................................................

2.4.11 Interfaces

2.4.11.1 Interface Identification.............................................................................................

2.4.11.2 Interface Description..............................................................................................

2.4.12 Appendices......................................................................................................................

2.5 System Security Specification...........................................................................................................

2.5.1 Penetration Analysis..........................................................................................................

2.5.2 Access Control Policy.......................................................................................................

2.5.3 Access Control Measures.................................................................................................

2.5.4 Equipment and Data Security...........................................................................................

2.5.5 Software Installation...........................................................................................................

2.5.6 Telecommunications and Data Transmission Security.................................................

2.5.7 Other Elements of an Effective Security Program..........................................................

2.6 System Test and Verification Specification.......................................................................................

2.6.1 Development Test Specifications....................................................................................

2.6.2 Qualification Test Specifications......................................................................................

2.7 System Operations Procedures........................................................................................................

2.6.3 Introduction

2.6.4 Operational Environment...................................................................................................

2.6.5 System Installation and Test Specification......................................................................

2.6.6 Operational Features.........................................................................................................

2.6.7 Operating Procedures.......................................................................................................

2.6.8 Operations Support...........................................................................................................

2.6.9 Appendices

2.7 System Maintenance Procedures.....................................................................................................

2.7.1 Introduction

2.7.2 Maintenance Procedures..................................................................................................

2.7.2.1 Preventive Maintenance Procedures.......................................................................

2.7.2.2 Corrective Maintenance Procedures.......................................................................

2.7.3 Maintenance Equipment...................................................................................................

2.7.4 Parts and Materials............................................................................................................

2.7.4.1 Common Standards.................................................................................................

2.7.4.2 Paper-Based Systems.............................................................................................

2.7.5 Maintenance Facilities and Support................................................................................

2.7.6 Appendices

2.8 Personnel Deployment and Training Requirements.........................................................................

2.8.1 Personnel

2.8.2 Training

2.9 Configuration Management Plan.......................................................................................................

2.9.1 Configuration Management Policy...................................................................................

2.9.2 Configuration Identification...............................................................................................

2.9.3 Baseline, Promotion, and Demotion Procedures..........................................................

2.9.4 Configuration Control Procedures...................................................................................

2.9.5 Release Process...............................................................................................................

2.9.6 Configuration Audits..........................................................................................................

2.9.7 Configuration Management Resources...........................................................................

2.10 Quality Assurance Program............................................................................................................

2.10.1 Quality Assurance Policy................................................................................................

2.10.2 Parts & Materials Special Tests and Examinations....................................................

2.10.3 Quality Conformance Inspections..................................................................................

2.10.4 Documentation.................................................................................................................

2.11 System Change Notes....................................................................................................................

2 Technical Data Package

2.1 Introduction

This section contains a description of vendor documentation relating to the voting system that shall be submitted with the system as a precondition of qualification testing. These items are necessary to define the product and its method of operation; to provide vendor technical and test data supporting the vendor's claims of the system's functional capabilities and performance levels; and to document instructions and procedures governing system operation and field maintenance. Other items relevant to the system evaluation shall be submitted along with this documentation (such as disks, tapes, source code, object code, and sample output report formats).

Both formal documentation and notes of the vendor's system development process shall be submitted for qualification tests. Documentation outlining system development permits assessment of the vendor's systematic efforts to test the system and correct defects. Inspection of this process also enables the design of a more precise qualification test plan. If the vendor's developmental test data is incomplete, the test agency shall design and conduct the appropriate tests.

2.1.1 Content and Format

The content of the Technical Data Package (TDP) is intended to collect clear, complete descriptions of the following information about the system:

¨ Overall system design, including subsystems, modules and the interfaces among them;

¨ Specific functional capabilities provided;

¨ Performance and design specifications;

¨ Design constraints, applicable standards, and compatibility requirements;

¨ Personnel, equipment, and facility requirements for system operation, maintenance, and logistical support;

¨ Vendor practices for assuring system quality during the system’s development and subsequent maintenance; and

¨ Vendor practices for managing the configuration of the system during development and for modifications to the system throughout its life cycle.

The vendor shall list all documents controlling the design, construction, operation, and maintenance of the system. Documents shall be listed in order of precedence

2.1.1.1 Required Content for Initial Qualification

At a minimum, the TDP shall contain the following documentation:

a. System configuration overview;

b. System functionality description;

c. System hardware specifications;

d. Software design and specifications;

e. System test and verification specifications;

f. System security specifications;

g. User/system operations procedures;

h. System maintenance procedures;

i. Personnel deployment and training requirements;

j. Configuration management plan; and

k. Quality assurance program; and

l. System change notes.

Systems in existence at the time the revised standards are promulgated may not have all required developmental documentation. When they are subject to evaluation as a result of system modification, vendors shall provide what information they can.

Vendors may also submit other information relevant to the evaluation of the system, such as documentation of tests performed by other independent test authorities and records of the system's performance history, if any.

2.1.1.2 Required Content for System Changes and Re-qualification

For systems seeking re-qualification, vendors shall submit System Change Notes as described in Section 2.11, as well as current versions of all documents that have been updated to reflect system changes.

2.1.1.3 Format

The formats presented are general in nature; specific format details are of the vendor’s choosing. Other items submitted by the vendor, such as documentation of tests conducted by other test authorities, performance history, failure analysis, and corrective action may be provided in a format of the vendor's choosing.

The TDP shall include a detailed table of contents for the required documents, an abstract of each document and listing each of the informational sections and appendices presented within each. A cross-index shall be provided indicating the portions of the documents that are responsive to documentation requirements for any item presented using the vendor's format.

2.1.2 Other Uses for Documentation

Although all of this documentation is required for qualification testing, some of these same items shall also be required during the state certification process and, possibly, local level acceptance testing. It is recommended that the technical documentation required for certification and acceptance testing be deposited in escrow.

2.1.3 Protection of Proprietary Information

The vendor shall identify all documents, or portions of documents, containing proprietary information not approved for public release. Any person or test agency receiving these documents shall agree to use the information contained therein solely for the purpose of analyzing and testing the system, and shall refrain from otherwise using the proprietary information or disclosing it to any other person or agency without the prior written consent of the vendor.

2.2 System Overview

In the system overview, the vendor shall provide information that enables the test authority identify the functional and physical components of the system, how they are structured, and the interfaces between them.

2.2.1 System Description

The system description shall include paragraphs, drawings and diagrams that present:

a. A description of the functional components (or subsystems) as defined by the vendor (e.g., environment, election management and control, vote recording, vote conversion, reporting, and their interconnection);

b. A description of the operational environment of the system that provides an overview of the hardware, software and communications structure;

c. A theory of operation that explains each system function, and how the function is achieved in the design;

d. Descriptions of the functional and physical interfaces between subsystems and components;

e. Identification of all COTS hardware and software products and communications services used in the development and/or operation of the voting system, identifying the name, vendor and version used for each such component, including:

1) Operating systems;

2) Database software;

3) Communications routers;

4) Modem drivers; and

5) Dial-up networking software.

f. Interfaces among internal components, and interfaces with external systems. For components that interface with other components for which multiple products may be used, the identification of:

1) file specifications, data objects, or other means used for information exchange; and

2) the public standard used for such file specifications, data objects, or other means.

g. Benchmark directory listings for all software (including firmware elements) and associated documentation included in the vendor’s release as they would normally be installed upon setup and installation.

2.2.2 System Performance

The vendor shall provide system performance information that includes:

the expected values and acceptable ranges of performance attributes for each.

The vendor shall provide descriptions of the following:

a. For all operating modes and functions, their performance characteristics in terms of expected and maximum speed, throughput capacity, maximum volume, and processing frequency;

b. Quality attributes such as reliability, maintainability, availability, usability, and portability;

c. Provisions for safety, security, privacy, and continuity of operation; and

d. Design constraints, applicable standards, and compatibility requirements.