In order to obtain the most accurate, detailed, and useful data for this study, the research team promised anonymity to all interviewees. Therefore, neither the names of the agencies nor the people interviewed can be disclosed. There are a small number of interviewees whom we have asked to release us from this promise, and who agreed to do so.
We realized that a focus on just two or three agencies and leaders would provide a tremendous depth of information about them (just as an individual program evaluation would), but would not give confidence that conclusions from those few leaders and agencies apply across government. Similarly, a survey of hundreds of agencies could give a high confidence that the conclusions will apply to many agencies, but the conclusions could only be based on a shallow understanding of those hundreds of leaders and agencies. So, a series of detailed case studies was selected to give a balance between depth and breadth. (An exception, for example, was that an author of this study, Steve Kelman, received three nominations for success in leading the Office of Federal Procurement Policy. Kelman was not considered as a potential subject for this study in order to assure the independence of our analysis. Only one nominee was not included in this study because he declined to participate.)
Leaders and the agencies they led were selected based on nominations solicited from two groups of independent experts: (1) fellows of the National Academy of Public Administration (NAPA), a congressionally chartered honorary association for distinguished scholars and practitioners studying or working in government, and (2) principals of the Council for Excellence in Government (CEG), a private non-profit organization open to people who have previously served in senior federal government positions but are not currently serving (most CEG principals currently work in academia/think tanks or in business). Nominations were solicited from all members of NAPA and all members of CEG. We asked them to “think about leaders (at the subcabinet or perhaps cabinet level) during the Bill Clinton and George W. Bush administrations l) with a clear, ambitious vision of what they wanted to accomplish during their tenure at their agency and 2) where implementing the vision required some significant changes in the agency's traditional behavior.” Nominators were asked to “limit your nominations to executives and strategies that were pursued long enough to give it a chance to succeed or run into problems.”
The request was made for nomination of “up to four executives and associated strategies—up to two where, in the nominator’s view, the executives were relatively successful in implementing the strategy, and up to two where, in the nominator’s view, the executives ran into significant difficulties.” To be a potential candidate for inclusion in the research, a successful leader must have received at least three nominations.
As a robustness check of the nominations, we asked a defined group of people familiar with each agency whom we interviewed to gather data for this research (some insiders, and some outside observers) to rate the nominee's success in achieving his or her goals on a 1-10 scale. Results of this second verification process closely mirrored initial nominations.
Since relatively few leaders were classified as failures, an additional source of comparisons to the successes was added. These were “counterparts” to the successful leaders—those executives who held the same positions, appointed at the same time in the lifecycle of an administration, in the administration other than the one of the successful leader. So if the success was President Bush’s first appointee, the counterpart would be President Clinton’s first appointee to the position. The idea was to control for outside factors influencing success as much as possible. As can be seen, the criterion for selecting the counterparts was a negative one: they were not nominated through our procedure as people who had tried to execute a strong, transformational vision.
In this study, the shorthand term “success” refers to a certain kind of success—successful execution of a strong vision requiring significant organizational change, as opposed to general organizational success. To distinguish the leaders whose practices would be considered “successful,” we simply relied upon the nominations received from independent experts: (1) fellows of the National Academy of Public Administration, a congressionally chartered honorary association for distinguished scholars and practitioners studying or working in government, and (2) principals of the Council for Excellence in Government. We had originally hoped to use objective criteria (e.g., achievement of specific performance targets set in advance, success in obtaining funding or authorization from Congress). We found, however, that these sorts of objective criteria were extremely difficult or impossible to apply consistently across widely varied agencies. For example, very few leaders made performance commitments early in their tenure. Similarly, some leaders and agencies did not require either funding or legislative action in order to achieve their goals—thus it is not appropriate to insist that they are only successful if they receive that approval. Eventually, we felt that the definition of success varied so widely that the subjective perceptual judgments of independent experts was the most appropriate determiner of success.
In a word, no. Our study found a correlation—between a number of leadership methods and the success of those who used them—along with an inverse correlation—between those same methods and unsuccessful / less ambitious leaders who did not use them. Thus our research does not prove that, if a leaders adopts the techniques we identify as associated with success, the leader will achieve his or her goals.
Never the less, this study was designed to avoid the pitfalls of typical “best practices” research that draws conclusions based only on successful cases. This type of methodology presents the problem of selection based on the dependent variable. For example, if one evaluates only successes and finds that they engaged in A, B, and C, one cannot conclude that A, B, and C are even correlated to their success, since other leaders (about which one has no information) may have done A, B, and C as well.
Central to our research design, therefore, was creation of a control group alongside the successful executives, so we can compare behaviors of successes with those of others. The control group includes leaders who attempted to achieve an ambitious vision requiring some significant changes in their agency's traditional behavior, but ran into significant difficulties, as well as “counterpart” leaders – meaning leaders of the same agency from the previous or subsequent administration who did not attempt major organizational change. This methodology of comparing successes to a control group establishes much clearer association between the leadership methods and the success.
Leadership techniques were selected based on an extensive review of management and leadership literature. The research team selected thirty-five initial hypotheses as likely factors in successful organizational change and tested each hypothesis with the successful leaders and the control group.
As we conducted data gathering efforts for this research, we noticed a number of additional commonalities across the successful leaders and agencies, and consequently, we added an nine more hypotheses.
No. The intent of this study was not to compare the performance of political parties and presidential administrations. We studied successes and failures from both the Bush and Clinton administrations and the lessons learned are applicable regardless of administration.
Our intention is to use the results of this study to support public leaders in achieving their visions of change. To do this, we plan to publish the results in professional and academic journals, host conferences to share the results, and offer the results to executive education programs that prepare candidates for the Senior Executive Service. The Harvard Kennedy School has also made tentative plans to develop a teaching case that will draw upon the lessons of this research to train future government leaders. Of course we will also ensure that Booz Allen professionals are prepared to help our clients take advantage of the lessons from this research.
For a modest fee (well within the discretion of most agency executives), you may request that Booz Allen conduct a two-day diagnostic review of your goals, strategies, and plans to change. We will review your approach, comparing it to those used by the successful executives from the cases we studied in our “What it Takes to Change Government” research. We will also conduct a summary capability maturity diagnostic in up to four agreed-upon areas, such as strategic planning process, performance management, and stakeholder engagement. Click here to Request a diagnostic
Information about this research is available in a variety of formats, including a brief (two page) information sheet; a reprint of our article in the July 2009 issue of Government Executive Magazine (available following publication), and an working paper that has been submitted for consideration by a peer-reviewed academic journal. In addition to those sources of information about this research, other resources include:
Yes. We would be happy to discuss with you how these methods can help your agency better achieve its mission; <<click here to learn more>>
