Evaluation

Environmental Training Program Strategy

Project Description

Booz Allen provides strategic and evaluation support to the U.S. Postal Service (USPS) Safety and Environmental Performance Management (SEPM) department’s training program.  Our tasks are aligned with the program’s goals and objectives, as outlined in its Programming, Planning, and Budget System:  monitor regulatory and policy requirements and business need; enhance awareness of environmental activities through training opportunities and products; maximize training resources by identifying and implementing the most efficient training mechanisms; and enhance stakeholder satisfaction by measuring effectiveness.

Develop Training Strategy Report

To evaluate the current program Booz Allen developed survey and interview protocols for eight stakeholder groups with the following objectives: identify how training needs are being met; describe the preferred curriculum to meet needs (i.e., identify gaps); describe the most effective methods for communicating training opportunities; identify opportunities to enhance support provided by the National Center for Employee Development (NCED) to meet needs; and evaluate technology, infrastructure, or organizational barriers to training.  In FY05, we gathered and analyzed input from four stakeholder groups and developed a draft strategy report for the program.  This task addresses goals 1 and 4.

Develop Regulatory Matrix

Working with an integrated team (of environmental engineers, attorneys, and training professionals), we conducted a review of laws, regulations, and USPS policy and determined the applicability of training requirements to USPS tasks.  We reviewed documentation about existing training requirements and past analysis and described training needs. This task also addresses goal 1.

Evaluate USPS NCED Training

Booz Allen investigated the current effectiveness of the NCED environmental curriculum in supporting existing regulatory training requirements.  We developed evaluation forms for each course and distributed the training materials to two USPS subject matter experts (SME) and one Booz Allen SME.  We evaluated the responses and presented recommendations for course revisions to the Environmental Training Work Group. This task addresses the program’s goal 3.

Review Environmental Portion of the Orientation for New Employees (ONE) Program

Booz Allen reviewed the current environmental awareness video and gathered input from the USPS NCED on how it is used.  We solicited input from several SEPM staff and provided recommendations on dissemination and content to the work group.  We also developed an outline for a 1-hour awareness training course to supplement the ONE program. This task addresses the program’s goal 2.

Develop Environmental Training Outreach and Awareness Plan

We reviewed existing communication documents and gathered data from stakeholders on current communication practices and needs.  We analyzed the data gathered and wrote a draft Training Program Awareness Plan. This task also addresses goal 2.

Intended or Achieved Results

As described in the tasks above, we have achieved or will achieve the following results.

• Training Strategy Report:  We developed a draft Training Strategy Report in September 2005 and are developing a final report that is due in July 2006.  This final report will outline a recommended national curriculum based on the regulatory matrix and a cost-benefit analysis of different delivery options.  This will provide consistent guidance across the USPS for staff to meet regulatory requirements.

• Regulatory Matrix: A final matrix of training requirements that apply to USPS personnel was completed and approved by the USPS General Counsel.  It lists 12 training requirements that include hazardous and universal waste handling procedures, procedures for reporting spills and applying pesticides, and energy conservation requirements. This provides authoritative guidance across the USPS for addressing regulatory requirements.

• USPS NCED Training Evaluation: We developed an evaluation report that provided recommendations for 10 existing courses that ranged from minor revisions to completely rewriting the courses.  This evaluation will improve existing training and demonstrate to stakeholders a thorough and impartial review, increasing NCED’s credibility in the field.

• ONE Program: We submitted a report with recommendations for ensuring that environmental awareness training is included in the next update of the ONE program and an outline for 1-hour awareness training to meet the immediate need for environmental awareness training throughout the organization.

• Environmental Training Outreach and Awareness Plan: A draft plan was developed describing recommended communication tools and activities to address the needs of the various stakeholders.

Non-response Follow-up Training Evaluation

Project Description

In the years leading up to a national decennial census, the Census Bureau conducts various field tests including one for the Non-response Follow-up (NRFU) operation, which is a door-to-door enumeration of households. The Census Bureau is conducting a test of NRFU between March and July 2006. A temporary labor force is hired to execute NRFU operations. Recruits, which are managed from a Local Census Office (LCO), perform the roles of field office supervisor (FOS), crew leader (CL), crew leader assistant (CLA), and enumerator. Their training employs verbatim lecture, drill and practice, and role play methods and uses a cascading model in which each level trains the next. The Census Bureau contracted Booz Allen to observe and assess the training conducted for CLs, CLAs, and enumerators.

The purpose of the training observation and assessment is to answer the Census Bureau’s question: Given our operational needs and constraints, how can we improve the composition and delivery of the enumerator, CL, and CLA training?  To answer this question, Booz Allen is performing three tasks.

Task 1.  Observation and evaluation of CL, CLA, and enumerator training delivery

Booz Allen analysts applied their expertise in training systems and adult learning to observe training delivery. They viewed the training classes as user tests conducted at the end of the development phase of a training endeavor. Analysts used general requirements of a training endeavor (as would be included in a training plan) to frame their data collection via document review, interviews with subject matter experts, and observation protocols. Analysts documented their observations, assessment, and recommendations in a training analysis report submitted to the Census Bureau.

Task 2.  On-the-job evaluation via field observation of the operation

Our analysts will observe enumerator trainees’ on-the-job performance at the beginning as well as toward the end of the 8-week field operations to compare differences in performance. Analysts will also observe the daily debriefing sessions that CLs and CLAs lead. In addition, as they conduct the field observation and assessment of enumerator, CL, and CLA on-the-job performance, analysts will aim to isolate job task performance factors and the probable role of training in performance improvement. Conceivably, there may be observed gaps in job performance that may not be closed or addressed by training.
Following these observations, the analysts will draft a Field Observations Report that will be based on a review and analysis ofdata captured on the protocols.  It will also provide an on-the-job performance assessment. We will carefully map all evaluation factors to learning objectives, which will ensure focus on institutional performance, even if the Census Bureau is not interested in a formal, performance-based evaluation at this time.

Task 3. Comprehensive review and evaluation of training documentation and materials

Booz Allen analysts will build upon the findings of the training and field observations to develop a full picture of whether training materials, both content and design, are meeting learner and Census Bureau requirements.  Analysts will follow a seven-step process for conducting the evaluation of training materials. These steps are identifying evaluation objectives, identifying stakeholders’ needs, developing an evaluation strategy, designing an evaluation instrument, conducting the evaluation, analyzing results, and developing a report.

Analysts will evaluate the training materials using approaches and instruments approved by the Census Bureau. They will collect data from related materials (e.g., summary analysis of pertinent comments from students) and any key personnel (e.g., materials developers, instructors, trainees). Booz Allen analysts have expertise in a broad variety of data analysis techniques, including descriptive and inferential statistics.  Analysts will use the techniques that are most suitable to the type of data collected.  They will prepare a reader-friendly report that addresses findings, conclusions, and recommendations within the context of the evaluation objectives developed at the beginning of the process.

Intended or Achieved Results

This project directly impacts the Census Bureau’s Strategic Objective 1.1:  produce accurate, timely, and relevant information about the United States—its population, economy, and governments. Booz Allen has received positive verbal feedback on the value of its contributions to date.

Booz Allen will prepare three evaluation reports that include “easy to implement” recommendations for the 2008 census dress rehearsal and “longer term development” recommendations for the 2010 census and beyond. In May, Booz Allen submitted an initial draft training observations report that outlines recommendations for the 2008 census dress rehearsal, the 2010 census, and beyond.  All recommendations are directly aligned with the Census Bureau’s Strategic Objective 4.5:  improve the timeliness, accuracy, and relevance of our surveys and censuses through new applications of technologies and methodologies and share them globally.