Survey Assessment Services

Social Security Administration (SSA) Integrated Quality Program (IQP)

Project Description

The Social Security Administrations (SSA) disability determination program consists of both Federal and State operating components, each with their own unique tools, training and processes in the adjudication of disability claims.  This challenges SSA’s overall ability to understand the various Quality Assurance (QA) processes that are unique to each respective operating component.  This also hinders SSA’s ability to develop solutions (i.e., technology, processes, and people) that are flexible enough to be used by any operating component to improve the overall quality of the disability determination program.

SSA required an evaluation of the current disability programs quality assurance process and development of specific recommendations for quality improvements that can be leveraged at each operating component.  As part of this evaluation effort SSA tasked Booz Allen to conduct an employee survey to gauge the perceptions of quality at SSA.

Intended or Achieved Results

Booz Allen developed and conducted, to 4,800 employees, a comprehensive web based quality survey questionnaire, designed to capture perceptions of SSA’s Formal QA processes and general view of quality at SSA.  Booz Allen worked with SSA to identify a stratified random sample that represented each operating component as well as the views of both management and staff. 

Based on an analysis of the survey results it was determined that SSA has a high level of commitment to quality at all levels and that quality is viewed a key mechanism to ensure the agency’s continued success.  However this analysis also showed that current working definitions and measures of quality are not aligned with the Commissioner’s vision of a multi dimensional view of quality encompassing (Accuracy, Timeliness, Cost Productivity and Service) and do not support performance improvements.  As a result of the quality survey questionnaire several key themes were identified:

• The survey garnered ~3,000 respondents from population of ~4,800; the response rate was 65%
• Respondents believe that the current approach to quality does not fully capture multiple dimensions of quality
• Respondents believe that the formal QA processes do little to ensure timeliness or correctness of case adjudication
• Respondents believe that the present quality approach does not help employees do their jobs better
• Respondents believe that SSA’s current quality process provides insufficient information for data-driven decision making
• Respondents believe that both end-of-line and in-line reviews can help improve quality
• Respondents do not feel there is adequate communication with other organizational units to discuss quality

Booz Allen used the results and analysis of the survey as a key element in the formulation of both short term and long term recommendations on how to improve QA in the disability determination process.

SAFECOM National Interoperability Baseline Survey Support

Project Description

In January 2005, the Department of Homeland Security (DHS) hired Booz Allen Hamilton to conduct a landmark National Interoperability Communications Baseline Study.  Commissioned as part of the ongoing efforts of the SAFECOM Program within DHS, the study represents the first comprehensive effort to survey public safety first responder agencies across Law Enforcement, Fire, and Emergency Medical Services (EMS) disciplines in all 50 states and the District of Columbia.  The goal of the Nationwide Interoperable Communications Baseline Study is to create a national and statistically valid snapshot of the capacity for and use of interoperability.  The study plan engaged the public safety community to develop a descriptive and measurable definition of interoperable communications and a robust survey assessment methodology.

Intended or Achieved Results

The Booz Allen team used the existing Interoperability Continuum as a starting point for the survey constructs, ultimately developing a more descriptive and measurable definition of interoperability for the public safety community.  We conducted a facilitated subject matter expert meeting and utilized best practices for survey design to create a survey instrument that provided respondents with clear, unambiguous, and complete questions and response scales. For each question, a response scale was developed that included four progressive stages: Early, Moderate, Full, and Advanced.  The stages were used to ensure that the questions’ customized response scales were consistent in measuring interoperability approaches.  To capture differences in interoperability with different parties, most topics were measured on three interoperability “levels”: interoperability with other disciplines, interoperability with other jurisdictions, and interoperability between levels of government. 

The team recommended a stratified random sampling approach, with the strata determined by a geographic factor.  First responder public safety agencies were sampled at the state level (to ensure a national sample that had good geographic distribution), with a confidence level of 95% and a confidence interval of +7.5 points.  The total number of participants nationwide was 22,416 first responder organizations.

We employed a customized Booz Allen survey tool using a ColdFusion web application server with a Microsoft SQL Server relational database over a 2-month period to collect the data.  An extensive outreach campaign was designed and conducted to alert the first responder community to the survey, its purpose, and its timeline.  Participants received an introduction letter signed by DHS Secretary Chertoff, followed by two reminders.  Finally, a call center engaged to contact non-respondents and urge them to take the survey. 

The Booz team developed a thorough descriptive and inferential analysis plan for the survey data, that included a preliminary determination of whether a weighting scheme was necessary and a missing values analysis.  The final data set contained 6,819 agencies, thus yielding a 99-percent confidence level and +1.46 point confidence interval (based on the national population of first responders).  We used a variety of descriptive and inferential statistics to understand the results, identify differences between demographic groups, and find relationships among the variables:  missing values analysis, frequency distributions, cross tabs with chi square and residuals, logistical regression.

Many significant findings that can be generalized to the entire national population of first responders were written into the final report.  This work will become the basis for policy and budgetary decisions by SAFECOM and DHS.  The Booz Allen Baseline team received an award from the Undersecretary of DHS for their work on this project.

U.S. Army Reintegration Study - Soldier Satisfaction Survey

Project Description

As the Army continues to increase the number of Soldiers deploying and redeploying in support of Operation Enduring Freedom and Operation Iraqi Freedom there is increased focus on meeting Soldier and family needs during deployment.  OASA M&RA has taken on a number of initiatives that focus exclusively on ensuring that Active and Reserve Component Soldiers and families receive the transitional services due to them during the deployment cycle.  The Soldier Satisfaction Survey is an effort mandated by the Army’s Delivery of Transitional Services Action Plan and approved by the Secretary of the Army.  The goal of this effort is to measure Soldier and family satisfaction with key programs and services provided during the deployment cycle, so that the Army can continue to improve these services and their delivery.

Intended or Achieved Results

Booz Allen worked closely with the client to specify the goals and objectives of the survey, and then began narrowing the scope of the questions to be asked.  The deployment cycle has over 113 activities and services that are provided in 7 different phases.  The survey will cover topics that are linked to problem areas already identified or topics of particular interest to Headquarters Department of the Army.  Some of these include suicide prevention, health assessment, substance abuse, sexual assault prevention, marital assessment, and family reintegration.  Using these hot topics and standard customer service constructs, Booz Allen developed a set of questions with a Likert opinion scale (e.g., strongly agree-strongly disagree).  The survey also includes an open ended question that will allow Soldiers to include any other feedback they may have regarding the phases of the deployment cycle.  A similar survey will measure family satisfaction with the services they receive.

The survey will be administered electronically to all Soldiers who are preparing to deploy and preparing to return home within 90-days of the survey launch date.  The telephonic survey will be administered to the families of these Soldiers during the same period of time in late Spring/Summer of 2007.

The survey data will be analyzed to determine levels of satisfaction with the services provided in each of the phases of the deployment cycle.  Booz Allen will determine whether there are differences between Active Component and Reserve Component Soldier or family satisfaction, and whether there are geographic differences that might indicate differences in the way the services are being provided.  We anticipate completing the report of this project in late 2007.

FDA OSS/OITSS Operational Effectiveness Assessment

Project Description

The Food and Drug Administration (FDA) contracted with Booz Allen to provide broad implementation support as it transitions to a shared services model for the delivery of administrative services (i.e., buildings/facilities services, travel services, finance services, EEO/diversity services, contracts and grants services, computer services, and telephone services).  As part of this effort, the FDA contracted with Booz Allen to conduct an operational effectiveness assessment one year after stand up of the shared services organization.  The overall objectives of the assessment were to assess employee and customer perceptions of service delivery, capture baseline data for future trend analyses, and derive recommendations of tangible near-term and long-term improvement strategies.

Intended or Achieved Results

Booz Allen began by establishing an evaluation model to ensure that the assessment was correctly targeted to produce useful, meaningful results and to plan which issues should be explored using which data collection methodologies.  The resultant evaluation model included five dimensions:  communication; customer service delivery and quality; tools/resources; structure and processes; and outcomes.

Booz Allen then used a multi-method approach to collect data in the areas identified in the evaluation model using the following methods:

• Leadership interviews – Sixteen interviews were conducted, the results of which were used to help shape the direction of the assessment by gathering input about goals and priorities for the assessment
• A web-based survey – A web-based survey was administered to assess overall, as well as service-specific, customer perceptions of the shared services organization, the contact center interface, and service provider perceptions of working in the shared services organization; using branching technology, a single survey collected data from two key audiences -- customers and service providers
• Focus groups with employees in particular service delivery or customer roles – Eight focus groups were conducted to follow-up on the general questions asked in the survey and to delve deeper into some of the successes and challenges
• Benchmark interviews with selected private- and public-sector organizations – Five benchmark interviews were conducted to gather lessons learned on the effective delivery of shared services, with particular consideration for that which can be applied to FDA

After using the appropriate techniques to analyze the qualitative data (e.g., thematic analysis) and quantitative data (e.g., descriptive and inferential statistics), Booz Allen summarized the results in a final evaluation report that identified major findings, conclusions, and recommendations targeted to address key issues.  Booz Allen also prepared a high-level action plan, which matrixed recommendations with potential benefits, timeframes, and priority levels, to serve as a starting point for FDA leadership to begin their action planning efforts.