View Our Work

V&V Handbook

Nexight Group implemented a V&V assessment process that is considered a best practice in the federal government. We started by developing an NSF V&V Handbook. The handbook provided the NSF measurement community with a reference document that describes the V&V process, the V&V criteria on which measures are assessed, and the requirements for meeting the criteria. The five criteria are:

1. Complete: The results reported for the measure represent the entire population of elements within the measure’s specified scope.
2. Consistent: Data are collected, analyzed, and reported using clearly defined and uniform procedures across people and time periods.
3. Accurate: Data collection processes and procedures maximize data quality and minimize the potential for errors in the collection, recording, and reporting of performance information.
4. Timely: Data are collected and updated at regular intervals, such that reporting deadlines are met, and performance information is current enough to inform decision-making.
5. Valid: Data can be used to draw useful and meaningful conclusions about the agency’s performance that inform internal and external agency decisions.

Assessment Report

Nexight Group conducted annual V&V assessments on NSF’s publicly reported measures (12-20 measures per year) for six reporting cycles. Each year we discussed the nature and use of each measure with the measure owner, obtained supporting documentation, and determined the completeness and reliability of the data based on the Handbook’s five assessment criteria. We examined data files for each measure and confirmed whether reported data was correct. We prepared 3rd and 4th Quarter reports identifying each measure’s strengths and weaknesses relative to each criterion and, where appropriate, provided recommendations for improving data quality. Our reports contained a statement attesting to the extent to which the data are complete and reliable. The V&V reports serve multiple audiences:

• Measure Owners: Helped build measurement capacity by explaining how the V&V criteria in particular, and measurement principles in general, apply to their measure. Measure-specific recommendations in the report helped measure owners improve their measures.
• Leadership: Provided an overview of the quality of performance information across the agency and general recommendations for improving agency data quality
• External Stakeholders: Provided assurance that complete and reliable performance data is available for agency decision-making

Building Measurement Capacity

Nexight’s approach has been to build internal measurement capacity within NSF as we conduct the V&V assessments; this has allowed us to help NSF build measurement capacity through the following products and services:

• Performance Measure Description Form (PMDF): Identifies the scope of each measure, the process used to collect data, potential sources of error, and efforts to mitigate error; serves as a high-level standard operating procedure (SOP) for each measure and is an effective practice for ensuring consistency and reliability, especially when staff turnover occurs.
• Case Studies: Illustrate an important measurement principle relative to meeting the V&V criteria. Each focuses on a fictitious performance measure that encounters a common pitfall or challenge; describes what went wrong, why it’s a problem, and the program’s solution; and concludes with a key take-away that highlights an important principle of data quality (most effective when used in conjunction with a consultation).
• Guidance Documents: Focus on a particular aspect of data quality; include examples and worksheets to help measure owners apply the guidance and meet the V&V criteria (most effective when used in conjunction with a consultation)
• Consultations: Allow programs to bounce ideas off us and allow us to suggest how to enhance the quality of measures before they undergo the V&V assessment.

Lessons Learned Report

At the end of each assessment cycle, we held a lessons-learned session with measure owners, getting their views on what worked and what could be improved, which we documented in a lessons-learned report. More importantly, we used those results to improve all aspects of the process.