Current press releases and speeches

 

Report of the Review of the Safety and Functionality of HMNZS Canterbury

Contents | Previous | Next

Part 1 - The Acquisition of HMNZS Canterbury

The constraints of selecting a Commercial Ro-Ro design as the basis for MRV

  1. From the outset of the project, there was insufficient appreciation of the constraints to the ship’s operations imposed by the selection of a commercial Ro-Ro design as the basis of the design for the MRV. This has been at the root of differences of opinion between the Tenix, MoD and NZDF and the subsequent shortfalls in performance.
    1. The Tenix proposal for the MRV is based on the Ro-Ro Ferry Ben my Chree. Tenix placed a sub-contract for design and manufacture of the MRV with Merwede B.V. in the Netherlands who also manufactured the Ben my Chree.
    2. The Ben my Chree is a ‘short/fat’ ship that operates across the Irish Sea where conditions are akin to coastal waters, where the seas are generally short crested as compared with deep oceans. Even a cursory examination of her design and operating profile should have raised questions over her suitability, once modified, for long operational patrols in the Southern Oceans. It is axiomatic that the hull form of a ship designed for short sea crossings may not be ideally suited as a solution to fulfil the full Functional Performance Specification1 for the MRV.
    3. While the MoD’s technical evaluation of the Tenix proposal for the MRV identified a number of concerns associated with the design, it did not recognise that the behaviour of the ship in a seaway was already predetermined. The technical evaluation team judged the Tenix proposal to be “medium risk” and thus we find it surprising that the team did not visit the Ben my Chree and improve its understanding of the design being offered.
    4. Following the evaluation but pre-contract, the Project Team sought to clarify aspects of the Tenix design and, as part of this process, commissioned a high level assessment of the Tenix MRV proposal (undertaken by BMT Defence Services Ltd (BMT); dated April 2004.) This assessment identified high level risks for which BMT recommended that the MoD request further information. These high-level risks encompassed sea keeping, stability, aviation operations and aircraft handling, powering and electrical generation, structural integrity and aspects of design that impact on sea keeping issues and operational effectiveness. Many of these have subsequently come to be issues of substance.
    5. The treatment of the BMT report by the Project Team was somewhat selective since some of the key issues were not put to Tenix and, therefore, not investigated. Tenix’s responses to the issues put to them, led to the recommendation that Tenix should be confirmed as the preferred bidder for the MRV.
    6. Concern about sea keeping dominated the acquisition of the MRV after contract award. In February 2005, a matter of months after the contract had been placed, serious concerns about the dynamic behaviour of the MRV in high sea states began to emerge. The contract between Tenix and the MoD was suspended on 24 March 05 and then lifted on 17 May 05 following acceptance by the MoD of Tenix’s remedial plans – including that for additional data on sea keeping, in particular revised computer simulation of the MRV’s performance (known as the AMOG results). Preliminary information from AMOG was available in June although the formal submission was in October 05. Subsequently an agreement was made between the MoD, Tenix and Merwede to collect more evidence by undertaking tank testing using a scaled model.
    7. The tank testing, undertaken by the Maritime Research Institute Netherlands (MARIN), took place in November 05. The results were published in February 06. The MARIN tests did not produce a materially different result to the AMOG simulations although they did highlight the risk of water ingress into the alcoves and the potential risk of sea boat damage. There then followed a protracted round of correspondence between the Project Director, Tenix, a consultant Naval Architect and equipment suppliers to understand the ramifications of the tank test results and to consider what should then be done. With delivery and acceptance planned for 31 May 07, the Project Director wrote to the Navy, as the future operating authority, in April 07 to “close out” the MARIN issues. We are surprised that the Naval Staff were not more closely associated with the discussions on the results of the MARIN tests given the complexity of the issues and the expertise that they could have provided.
  2. Under the terms of the contract Tenix agreed to deliver a sea keeping performance for the MRV, and its ship to shore transfer system, considerably in excess of that specified in the FPS. The company, in its response to the RFP, submitted an analysis of sea keeping performance based upon frequency domain strip theory. While such analysis is reasonably accurate in predicting sea keeping performance up to sea state 6, it may not be relied on as an indicator of performance at higher sea states. The ability of the contractor to deliver such enhanced performance levels, could only have been challenged by the project team had it been equipped or had access to a high level of domain knowledge. In the event, the scale of the limitations of the Tenix design in relation to the contract, while recognised by the RNZN at the time of contract suspension, only became clear with the completion of a more sophisticated analysis using time domain strip theory – undertaken by AMOG - and confirmed by the MARIN tank tests.
  3. The MARIN tests may have deepened understanding of how the MRV would perform in higher sea states but they could do nothing to change that performance. The performance of the ship was pre-determined by the design offered by Tenix and the ship was effectively completed while the MARIN issues were being extensively debated. We doubt whether the MARIN tests were really necessary since they merely confirmed the AMOG results. Had action been taken to address sea keeping at the time of the AMOG simulations, greater progress would have been made in addressing performance issues that currently confront HMNZS CANTERBURY.
  4. We conclude that insufficient technical understanding was brought to bear from the outset of the programme. That this was not recognised suggests that there was insufficient technical expertise in the project team to fully comprehend some key design issues and, therefore, the ability to articulate concerns and seek guidance. Furthermore, there was an absence outside the project of technical governance on key issues such as sea keeping, or, as identified later, the challenges of safety in the context of meeting SOLAS2 and other statutory requirements, and the limitations of Lloyd’s Register’s certification on a commercial Ro-Ro design incorporating significant military features.

A Programme managed to achieve the earliest delivery, characterised by wishful thinking

  1. All programmes should be managed to achieve their performance, cost and time objectives. While we understand the desire, emanating from very senior levels, to reduce the time from the frigate Canterbury paying off and to the delivery of the MRV, we have found a relentless determination to deliver to time, across both the MoD and NZDF, despite evidence of likely performance shortfalls e.g.
    1. At the outset of Project Protector, the stated intention was to employ a sequential approach to design, before freezing the design that would be produced. This is not what happened - production got ahead of design from an early stage. Steel was cut in mid-April 05 - probably because of production pressures in Merwede - during the period of contract suspension. The decision to lift the contract suspension in May 05, despite unresolved issues on the MRV’s sea keeping characteristics, was critical in enabling production to continue. With hindsight, contract suspension should have been the time to extract firm performance guarantees from Tenix or to negotiate relaxations.
    2. The MARIN Tank tests were undertaken in November 05 and the report issued in February 06. The protracted resolution of the issues arising from the MARIN test results, lasting some 15 months were completed only a matter of a few weeks before the acceptance of the ship. This meant that little or nothing could be done to address these issues during manufacture. The MoD’s agreement to conduct the MARIN tests effectively let Tenix off the hook of addressing sea keeping performance before ship construction was too far advanced
    3. The ship set sail from Rotterdam in August 06 before planned acceptance had taken place of all essential seagoing functions and the material configuration, functionality and condition of the ship.
    4. Incidents, noted in the Canterbury Engineering Master Log, during the journey from the Netherlands to Australia were warning signs of sea keeping difficulties to come e.g. bow slamming (12 Sept 06) and the swamping of the starboard RHIB with green water and the opening of the starboard alcove door (22 Sept 06).

    The Ship Acceptance Process

    1. The conduct of the acceptance of the ship off contract illustrates vividly the drive to deliver HMNZS CANTERBURY and get her into service at the earliest opportunity. There was, however, no explicit consideration of the risks of acceptance with defects, as opposed to tolerating delay to rectify defects before delivery. The discussion of acceptance at the Capability Development Executive (CDE) was initially not recorded in minutes; there was no discussion at the Integrated Capability Management Committee (ICMC), and there was little recorded discussion at the Executive Capability Board (ECB). 3
    2. At the time of acceptance there were a number of deficiencies outstanding of varying importance. Advice from project staff to the Project Manager and Director made clear that the technical issues outstanding would present a significant challenge to the RNZN if the MRV were to be accepted on 31 May 2007 as planned. The incompleteness of the verification and acceptance (V&A) process immediately before planned acceptance was specifically highlighted. Indeed, only a very small percentage of the deliverables required by the V&A plan had been formally agreed by the project team at acceptance.
    3. The Project Director, in his advice to Sec Def, recommended acceptance citing the need to have the ship available for ship’s personnel and expressing the concern that delay could lead to demoralisation and people leaving the service. He went on, however, to acknowledge that while individual defects are minor in sum their culmination of defects could be considered major. He also commented that knowing what he did at the time of writing he would have delayed acceptance by 8 weeks. These apparent drawbacks of acceptance were not developed further; we are not aware that they were subsequently questioned.
    4. Initially, Naval Staff advice drew attention to the considerable deficiencies in the contractual deliverables provided by Tenix, the need for a rectification plan, weighed in the balance the risks of acceptance, and concluded in favour of acceptance. This advice was withdrawn and substituted by a shorter document that played down or removed reference to the drawbacks of acceptance, thus placing relatively greater emphasis on the adverse impacts to the Navy if acceptance were to be delayed. The imperative reported by the Chief of the Navy to CDF and Sec Def was of the need to get HMNZS CANTERBURY into service.
    5. Having decided to proceed with acceptance, more should have been done to protect the Crown’s position.
    6. The underpinning document is the Report on Material State at Completion (RMSC). This document lists areas to be rectified by the contractor and also includes over 40 MoD Outstanding Actions and Issues of Concern. We do not feel that expressing “concerns” in the RMSC will have been sufficient to protect the interests of the Crown.
    7. Furthermore, in view of the potential for debate about the RMSC and the status of the concerns listed therein, we would have expected commercial/legal advice to have been taken on the form of acceptance and legal risks of accepting the vessel on the date envisaged.
  2. We conclude that, on a number of occasions, the MoD could have taken a more robust line with Tenix to enforce the contract. We suspect that this was not done because of a lack of understanding of the technical issues, caused in part by under-staffing in the project team and poor governance outside it. The approach taken by those involved in key decisions was, it seems to us, “to crack on come what may” and to hope perhaps that something might turn up. We note that many of the problems identified at an early stage of the programme manifested themselves after acceptance and are now the subject of an extensive Warranty claim that has been submitted to Tenix.

Recommendation:

An under-estimated Project Management task; a team under-sized and under-skilled

  1. Project Protector is not a simple acquisition project. While not at the cutting edge of technology, it involves the procurement of three vessel types, modification of commercial designs (which need to be understood), commercial and military certification and a contractor and sub-contractors across the globe. The complexity and challenges of the programme have been under-estimated in all respects.
  2. The approach to setting the contingency at the outset of the programme and to certification are symptomatic of the under-estimation of the project management task.
    1. Setting the contingency - All programmes require a contingency which should be established after an analysis of risk. The sum eventually assigned as contingency may not be as much as analysis suggests but decisions by senior management should be taken in the light of an objective assessment. It is symptomatic of project management for Protector that such an analysis did not take place - the programme contingency was simply the difference between the estimated programme costs and the sum that the Cabinet authorised ($500M NZ).
    2. Certification, or more accurately Lloyd’s Register Rules and Regulations for the Classification of Ships, and other statutory rules were assumed to be straightforward. It would be a task for Lloyd’s Register but the process for seeking waivers, in particular for the impact of military features, was not thought through at the time of contract signature. The granting of waivers by the Maritime Safety Authority (now New Zealand Maritime) did not get off the ground and a Naval Authority had to be quickly established to act as the Flag State Authority. The certification process led to dysfunctional behaviour and friction between parts of the Navy and the Acquisition branch. The Naval Authority, which was line managed by the Navy, should have been regarded as an independent safety authority. Its role, however, was difficult to separate from the Navy’s role in providing advice to the project team - the Naval Authority was regarded by the MoD as a risk to the programme.
  3. A fundamental deficiency, and possibly the underlying cause of some of the problems cited above, has been the staffing of the project team. The project team has lacked the size and range of skills to manage a project of the complexity of HMNZS CANTERBURY.
  4. Our expectations for a project team are as follows:

Expectations for a project team

What good looks like:

An empowered Project leader

A culture of debate and discussion in the team

Embedded technical, financial and commercial/legal skills

Risk Management and risk mitigation

Staff in sufficient numbers to manage the programme
  1. We found the project team to be lacking in a number of these areas.
    1. Staff have been over-stretched at all levels over a long period of time. The team comprises a Project Director, with other responsibilities, based in Wellington, and a Project Manager and five seconded naval staff in Melbourne. At the critical time of acceptance, the Project team was way behind on Verification and Acceptance because of shortage of manpower. Anecdotally, there have been areas of acute stress in the project team.
    2. The team has lacked the range of technical skills to understand the programme. It needed an embedded Naval Architect particularly at the design stage of the programme. While we note that the current Project Director engaged consultants to perform this role, this was regrettably too late in the programme.
    3. The project team contains neither financial nor commercial experts. As far as we can tell, financial control of Project Protector overall has been adequate although we have not sought to interrogate the financials for the MRV.
    4. We note the absence of embedded commercial or legal expertise to guide the Project Director. Calling on external legal advice is only a partial solution as summoning such legal advice is largely discretionary. We were surprised to find that only 5.8 man weeks of legal advice was sought on Project Protector during the period since the contract was placed and July 08 i.e. in almost four years during which time some very substantial decisions have been taken on the contract. For the Crown’s position to be protected, and for the Project Director and Project Manager to be appropriately advised, commercial experts would need to be a party to key contract correspondence and to attend key project meetings.
    5. Risk assessments were undertaken jointly by the project team and Tenix and on five occasions by an external risk management consultant, approximately six-monthly. While a risk process has been followed, risk management and risk mitigation have not been embedded in the way that the programme has been managed. The absence of formal risk assessments at the time of acceptance or of the sea keeping issues, are cases in point.
    6. At the time of our review, we found the culture of the project team to be more akin to that of a military command rather than a team managing a significant contract and sum of taxpayers’ money. At times, junior staff seemed not to have been aware of what, if any, attention had been paid to their advice. We also detected personal antipathies. Project management in acquisition often involves taking decisions where the right course is difficult to select and the consequences of which are enduring. We would like to have found evidence of more open debate within the Project Protector team.
  2. We were told that that one of the reasons for the Project Team being kept small was because it was felt that reliance could be placed on both commercial certification and the fixed price contract with Tenix. This was, perhaps, over-simplistic. It should have been evident at the outset to anyone with a knowledge of naval construction that certification would be complicated given that the MRV is a military vessel, albeit one based on a commercial design. Nor too can reliance be placed on a fixed price contract for sorting out all problems. This pre-supposes that the contractor himself has adequately understood all the risks in the programme and is able and willing to bear them. In short, a project team needs to have the range of skills (either embedded within it or access to those skills) to enable it to be an intelligent customer.

Recommendations:

The Need for Good Governance

  1. Better governance above the Project Team may have helped both the understanding and the conduct of the MRV acquisition. There have been, however, significant shortcomings in the governance of the HMNZS CANTERBURY acquisition, exacerbated by some strained relationships between the MoD and NZDF. Reporting of project performance has been generally inadequate.
  2. While the approach to governance depends to a degree on what is being governed, there are certain basics for a complex capital programme. We would expect to see the following:
Expectations for the project team

What good looks like:

Review periodically by the project leader’s superior, supported by subject matter experts as necessary, of progress against the project plan, project finances and review of risks and their mitigation.

Clear written reports to enable such review to take place.

Competence on the part of the reviewers in the fields of project management, domain technical knowledge, finance and commercial issues.

Active stakeholder management

Timely escalation of key issues to senior management for resolution.
  1. In addressing governance, we have considered first the role of acquisition line management and secondly the NZDF/MoD governance structure under the Capability Management Framework 2004 (CMF2004) which was in place for the period of HMNZS CANTERBURY’s acquisition.

Acquisition Line Management

  1. We understand that Project Director Protector was not subject to routine formal review by Dep Sec (A). The Project Director did, however, provide written reports to Dep Sec (A) following a prescribed format. Dep Sec (A) in turn submitted reports to Sec Def. The latter varied considerably in quality and breadth. The Project Protector briefs submitted by Dep Sec (A) to Sec Def from August 04 are sparse until July 05. There is little information on schedule and risks. There is, surprisingly in our view, no mention at all of contract suspension in March 05. Later briefs include schedule risk information but there is no discussion of risk mitigation.
  2. Financial information is provided in Dep Sec (A)’s reports on Project Protector but there is little discussion in these reports of forecasts or explanation of variances. We also note that the financial information is for Project Protector overall. There has been no specific monitoring of HMNZS CANTERBURY’s costs against the notional $100M USD ceiling set by the Government.
  3. While we are very conscious of the taut staffing of the MoD’s acquisition community, we have not identified within it the degree of challenge and scrutiny required for a programme of the complexity as the MRV. We observe, in addition, that the reports provided to Sec Def were exclusive to him and not formally disseminated further.

CMF 2004

  1. CMF2004 describes a hierarchy of committees:
    1. The Capability Development Executive (CDE) which enables interest groups to meet, discuss and solve emerging issues within the acquisition phase of a project.
    2. The Integrated Capability Management Committee (ICMC) which was charged, among other things, to “monitor project and programme progress against agreed dimensions, risk mitigation and communications”. CMF 2004 also states that “The ICMC reports to ECB on all capability issues except for acquisition (for which Dep Sec A is solely responsible to Sec Def).” At first sight, there could be confusion about reconciling “monitoring progress” with line management responsibilities. We return to this later.
    3. The Executive Capability Board (ECB) which was and is the key governance body.
  2. We have considered how these committees discharged their responsibilities for the MRV acquisition.
    1. CDE - we have reviewed the minutes of CDE meetings for Protector between 1 Jul 04-10. Dec 07. CDE’s have discussed issues of the moment but they have not reviewed progress against plan, risks etc. Minutes of 21 May 07 meeting, a critical meeting that has been cited as supporting the decision to accept the MRV, are draft and incomplete. While the intention to accept the ship is noted, there is no evidence of a debate on acceptance (and its risks in view of the defects on the ship at the time).
    2. ICMC - For the period 4 Oct 05 - 1 Apr 08, the ICMC received no written reports on Project Protector – indeed it received almost no written reports at all from the Acquisition Division. Review of the ICMC minutes shows some discussion of sea keeping issues but there appears to have been no debate on the lifting of the contract suspension (May 05) or of the risks inherent in the acceptance process (May 07). We note that in October 2004 VCDF commented that the ICMC was not receiving adequate reporting. While the ICMC received oral briefing from the then Dep Sec (A), it is difficult to see how it could scrutinise and challenge without written reports as the basis of its work.
    3. ECB - The ECB discussed the MRV on many occasions. What we have not detected, however, either from review of the minutes or interviews with those present, is evidence of the ECB being able to dig deeply into difficult issues. Overall, ECB discussion can be best characterised by the Acquisition Division seeking to demonstrate that all issues were being adequately addressed and the Navy seeking to ensure that vessels were accepted into service as soon as possible. If there were doubters in the ECB, their voices were not recorded or heard.
  3. Many we spoke to commented that the CMF has not worked particularly well for Protector, in part because of poor relationships in places between the MoD and NZDF. It has been suggested that the atmosphere started to deteriorate with the reluctance of the Acquisition Division to explain its selection decision to other Government Agencies, exacerbated by leaks early in the programme (probably from outside the acquisition organisation). Subsequently, relationships soured with the creation of the Naval Authority which seemed to lack independence from the Navy and which, in conducting its new role, undertook work that seemed to cut across the responsibilities of the project team.
  4. CMF 2004 will not have helped matters by giving confusing messages on the role of the ICMC. In our view, however, there does not need to be a contradiction between the Dep Sec (A) being solely responsible to Sec Def and the desirability of the acquisition community reporting and discussing project progress and risks to the ICMC. We would regard this as good stakeholder management and an important part of wider NZDF/MoD governance.
  5. Whatever the causes, however, we conclude that the basics of good project governance were fundamentally lacking – there was no effective review or challenge to either the Project Director or to Dep Sec (A) and patchy stakeholder management.
  6. Improvements to governance should now be made for the remainder of Project Protector and, in view of many of the comments made to us, on the conduct of other acquisition programmes too. At root, there are two imperatives to be balanced. First, the contracting authority (Acquisition Division) should be the only channel of communication with the contractor, sub-contractor and any agencies interested in the conduct of the project. If the NZDF with in-service issues in mind needs to involve other companies in assessments of safety, they should inform the Project Team before so doing. The quid pro quo is that the Acquisition Division must see it as part of its task to have full regard for the in-service concerns of the NZDF and to make it part of their task to communicate effectively with stakeholders. The ICMC should ensure that this is done. We also recommend that the ICMC is tasked formally to review the progress of projects in the acquisition phase, noting that Dep Sec (A) remains responsible to Sec Def.
  7. The Naval Authority may not be appropriately placed within the Navy. Some of the friction that occurred with the Acquisition Division stemmed from a perception that the Navy could not split the role of Naval Authority from its interest in getting a ship that met their concept of the capability. As the purpose of the Naval Authority is the provision of independent advice on material safety, it should perhaps be situated outside the RNZN. Consideration should be given to placing the Naval Authority under the Sec Def.
  8. If the ICMC and ECB have had a specific shortcoming, it has been in their absence of challenge in what they were being told. Ideally both committees should contain some individuals who are independent with adequate project management, financial and commercial knowledge to question the judgements of those responsible for the project. In view of the taut staffing of the MoD/NZDF, it may not be possible to assemble all the requisite expertise from within those organisations. A pragmatic course would be for the ECB to include one or more non-executive directors drawn from outside the MoD/NZDF to provide independent challenge.

Recommendations:

Roles and Responsibilities: a need for greater clarity

  1. We noted a number of cases in which roles and responsibilities were not as clear as they should be in particular at the interface between acquisition and introduction into service. Roles and responsibilities need to be clarified in a number of areas, notably for safety during the acquisition phase and design responsibility throughout the project’s life; and improvements could also be made to the ship trials and acceptance process.
  2. Trials and acceptance - Under the MRV’s contractual arrangements, some capabilities (about 10%) were planned to be finally evaluated “off contract” and some fall outside the warranty period. This arrangement has two distinct disadvantages – the contractor may not be held accountable for performance of the ship and its systems, and the project team’s expertise may no longer be available to support the programme having discharged its responsibility. An alternative arrangement would be to plan for an additional period of ship’s trials with the purpose of system proving before the formal acceptance off contract. We suggest that consideration is given to such an approach for future ship projects.
  3. Safety - We had expected to find a clear statement in the terms of reference of acquisition staff that, among other things, they are responsible for supplying equipment that is safe to use. We take the view that, notwithstanding other performance issues, safety should be paramount.
  4. The Protector Project Director’s terms of reference state that he is responsible for “the delivery of the Protector Fleet of Vessels and supporting Deliverables to specification...”. While “to specification” may imply “safe”, it would be clearer to spell out explicitly the need for equipment to be safe and fit for purpose in ToRs of all Project Managers and Directors.
  5. Design responsibility - We also found that responsibility for design throughout the life of HMNZS CANTERBURY had not been defined. We had expected to find a clear statement in the contract and elsewhere specifying the Design Authority (DA). We would expect the Design Authority to be the lead technical authority responsible for providing advice on the coherence of the system at all stages of its life.
  6. Understanding and defining the DA role becomes particularly important for maintaining configuration control of equipment once in service. Usually the DA would be the body that designed and built the equipment. In the absence of a DA for HMNZS CANTERBURY, the design authority de facto now rests with the Navy.
  7. Deliverables under design such as documentation and initial outfit of spares were significantly deficient as high lighted in the Report on Material State on Acceptance. We were advised that the situation had improved significantly during the following 12 months. A number of issues remain outstanding and the provision of low value spares from the original Eastern European suppliers make for a long supply chain. As the Project Protector team has no enduring responsibility for the ship, even routine warranty items frequently fall to the ship to raise. Under the MRV contract neither Tenix nor equipment suppliers were contracted for any enduring technical support either at the equipment level or for any subsequent design integration task.
  8. For future acquisition programmes, consideration should be given at the contracting stage to the likely need for the supplier to have a continuing design role for which he can be held accountable. For Project Protector, we recommend that consideration is given to entering into one or more arrangements for design support of the Protector Fleet to maintain the long-term sustainability of the supplier base.

Recommendations:

  1. The Functional Performance Specification is essentially the military requirement that the contract seeks to deliver.
  2. SOLAS = Safety of Life At Sea
  3. The respective roles of these bodies is described in paragraph 19.

Top | Previous | Next

Page contents Accessibility (list of Access Keys) Sitemap Homepage About us Defence Policy Acquisition Activities Reports & Publications Links Contact Us Search box New Zealand Government websites homepage