The following is a summary of the CDA Institute roundtable “How Cost Estimating can Improve the Defence Procurement Process” held in Ottawa on 15 April 2015. These roundtable discussions are held under the Chatham House Rule. This summary reflects Analyst Melissa Hawco‘s perception of the discussion. The CDA Institute thanks Lockheed Martin Canada for its generous sponsorship of the 2015/16 Roundtable Discussion Series.
The event began with an introductory discussion on the topic of cost estimating in the defence procurement process in Canada. By definition, the process of cost estimating collects and analyzes historical data in order to apply quantitative models and techniques that can predict the costs of future projects. Significantly, despite the best efforts of these professionals, cost does not equal price. During the estimating process, cost estimators must analyze historical data from identical or similar projects, and then adjust for a variety of factors, such as new technologies and inflation. Of course, there are cases in which relevant historical data is difficult or impossible to find. Innovative projects, such as NASA’s mission to Mars, is an example. In such cases, any possibly relevant data may be used to predict mission expenses.
Costs are predicted by employing a number of standard methodologies, including analogies (subjective comparisons), parametric (statistical patterns from multiple systems), build-up (“bottom-up” compositions), and lastly, expert opinions. For defence procurement, seasoned cost estimators recognize a strong correlation between cost and weight. Fortunately, weight and other basic physical parameters are often the first pieces of available data from the engineering crew of any new project, therefore allowing an early cost estimate from a very limited amount of information.
The discrepancies between cost and price in defence projects is often created during the execution stage. For example, during the Afghanistan mission, cost estimators were provided with certain figures regarding mission length and intensity that resulted in forecasts for the cost of that mission. However, the Afghanistan mission ultimately extended in length and increased in intensity, thus rendering the previous estimates inappropriate for the actual project prices.
Moreover, cost estimating is not a practice that produces a single correct answer for policy makers. Rather, estimators are able to produce a range of possible correct answers based on historical data, project adjustments, and risk analysis. This process is anchored not by certainties, but by probabilities. Knowledgeable professionals describe the process of their predictions in terms of ‘completeness,’ ‘reasonableness,’ ‘credibility,’ and ‘analytic defensibility.’ For policy makers and contractors, these descriptors and range of probable possibilities for costs are not wholly compatible with the precise nature of budgeting or the approval decisions being made.
Finally, communication is key to help bridge the gap between estimators and policy makers. It is notable that there are no undergraduate and only a very few graduate courses specifically dedicated to cost estimating. As a result, the community of cost estimators is comprised of a wide variety of former accountants, mathematicians, economists, computer scientists, logisticians, and engineers. During communications between cost estimators and policy makers and project managers, it is important to ask and prepare for the right questions. It is advisable to engage in a discourse about the underlying historical data used, assumptions and labour rates, data sources, risk analyses and the many unavoidable uncertainties of predicting costs for defence procurement.
The following powerpoint presentation slides have been posted with permission from the speaker.
Also discussed were the significant problems still embedded in the Canadian defence procurement system and the notable underspending of the allotted defence budget. Historically, DND’s main challenge has been insufficient funds, but in recent years the real challenge is the inability to spend between a fifth and quarter of DND’s capital budget. According to charted data from the past few decades and predictions for 2014/15, the current level of underspending in DND is both unique and does not appear to be improving. A major procurement backlog now exists, while for current equipment the metaphor of rust-out is transforming into a reality, such as in the case of naval ships being removed from service due to deteriorating hulls. Significant losses in purchasing power and de-scoping project requirements in order to fit within budgets also constitute major issues with the procurement process.
There is a mismatch between DND’s current workload and their available workforce. While defence projects are increasing in scale, time, and complexity, the workforce has decreased in number and capacity training. In fact, the workload in terms of the ratio of Vote 5 / ADM Material workforce has near doubled, and other departments are facing comparable struggles. The important training and job experience required for defence procurement positions in DND is often curtailed by the swift ‘revolving door’ rotation of the workforce. Moreover, the lack of trust and clear communication internally and externally with departments is producing enough friction to significantly slow down the procurement process.
In order to address issues of trust and communication in departments, and improve the current pessimistic perception of the defence procurement process, new governance structures have been imposed to modify behavior and institute double or triple checks into the procurement system. It is essential to improve capacity training in the defence procurement process through relevant education in master’s programs for project management and slowing down job rotation to attain more substantial job training and experience. There should also be a thorough rationalization of the capital plan on how much money realistically there is to spend, a renewal of the Canada First Defence Strategy, and a complete prioritization from the Capital Investment Program Plan Review
Common costing practices that are standard across the government for contingencies, cost escalation, and operating and supporting costs will facilitate predictability and easy communication between departments. The use of an external review system for budgeting in the defence procurement process would not prove amiss either. Finally, sharing a bit of good news with the public and across various departments will help restore trust and revitalize defence procurement.