Agile Cyber Procurement and Beyond

In February 2020, EXA hosted the Canadian Defence Procurement Today and Tomorrow – Succeeding in the Era of Cybersecurity symposium. This sold-out event included representatives from industry,government, and academia.

In April 2021, EXA hosted the Agile Procurement in the Era of Cyber online event. We had senior presenters from industry and government in Canada and the USA. Over 100 people from the Canadian government, industry, and NGOs attended the conference.

Trends in Cyber Procurement

Canada is innovating its procurement processes to varying degrees of success. Canadian government departments are demonstrating a better understanding of their own requirements, and they are migrating towards outcomes-based goals.

Treasury Board policies and funding envelopes and PSPC regulations remain deeply rooted in the legacy procurement of tangible things. Bidders face cyber-based RFPs that require contractors to design, build, and deliver a complete cyber solution. The technical requirements and the bidder's firm price remain fixed over the 10-year lifetime of the program in an accelerated Moore's Law era when information technology doubles its capacity every year. These cyber programs are doomed to suffer technological obsolescence on the first day they go live, and their support costs will prove to be much higher than what clients have budgeted.

In terms of support, Canadian cyber procurements have shown a predominance for commercial off-the-shelf (COTS) products without a clear understanding of the resulting logistics, risks, and costs. Software COTS products encounter obsolescence at a remarkably high rate compared to hardware-based systems. Security represents the most significant risk to obsolescence in software COTS products. Think of how many times your operating system and applications have upgraded your PC's software with security patches in the last year.

Beyond security patches, all software products undergo upgrades and replacements. As OEM software products become obsolete, operators of some integrated systems elect to keep the outdated software product because the replacement software version is incompatible with the system’s other integrated software products. The cost up updating, re-engineering, and re-integrating the software components proves too costly for the product support budget.  When this occurs, the unsupported software may degrade in performance, and it becomes cyber-vulnerable without regular security updates.  

Operators do not like cyber-vulnerable products, so they often upgrade the software, and then develop (or hire a contractor to develop) customized interfaces that "glue" otherwise incompatible COTS software components together by translating signals between the new software and the legacy products. Eventually, an in-service COTS-based system will depend heavily upon a haphazard patchwork of custom middleware, utilities, and scripts as individual COTS products successively become obsolete. The client's COTS system devolves into an assortment of custom patches.

Custom patches themselves grow obsolete and require support. EXA has seen paralysis set in after the people who implemented the patches move on, and no one has the knowledge, skills, ortools to support the system. In the end, the client forsakes the benefits they sought from a COTS based solution.

Cyber Procurement in Canada

Contracting policy and regulations are the single largest obstacles to a collaborative, cyclical, adaptive procurement system essential for cyber-based procurements. The Canadian Government views industry "collaboration" through contracting vehicles like Invitations to Qualify, which could be best described as down-selecting, as in meet my requirements or you are out.

Limited cyclical development does occur. Canadian RFPs misname this process as agile procurement. In this process, Canada requires the contractor to develop a prototype, but it is a prototype in name only. In truth, it is a proof of concept. If you can prove the 'prototype' meets all the RFP specifications, you are eligible to proceed to a full system contract. Otherwise, Canada awards the next bidder in line a chance to develop its prototype. The real issue here is that, unlike actual prototype development, EXA has not seen RFPs that permit changes to the technical specifications or resulting price, even if the prototype demonstrates the technical specifications will not meet the client's needs. This rigid process stems from inflexible procurement rules.

Consider two procurements that each take ten years from capability deficiency identification to final operating capability (FOC). The first procurement process (today's process) defines the deliverables early in the cycle – within the year of the procurement cycle. At FOC, the contractor delivers the system to specifications that are at least eight years out of date. Cyber years are like dog years – eight-year-old specifications might as well be 56 years old. Add the client's changes they wish they could have implemented due to operational priorities that evolved in the intervening eight years. In the end, the client takes delivery of a fully compliant but not especially useful system.

Now consider a new procurement process that also requires ten years from capability deficiency identification to FOC. Early in this process, the client defines outcomes, not the deliverables (deliverables define the artifacts and services the contractor must develop and produce under contract, whereas outcomes define what the client can achieve with the deliverables). The client also specifies the requirements for the absolute basic, most elemental system the client could accept as a preliminary system. We could call this preliminary system the prototype, but industry calls it the minimum viable product.

The contractor builds the minimum viable product, and then the contractor successively updates the product through several iterative development cycles. At the end of each cycle, the client and contractor collaboratively determine:

• The outcomes the client can achieve with the current product;

• The total remaining work the contractor must do to deliver the final product;

• The outcomes the contractor should implement in the next development cycle;

• The bugs the contractor must address in the next development cycle;

• The risks the project must address in the next development cycle, and

• The obsolescent issues the contractor must rectify in the next development cycle.

Instead of defining every blessed requirement at the beginning of the procurement cycle, the client and the contractor collaboratively refine the requirements at each development cycle. Instead of defining fixed deliverables at the start of the procurement process, the client and contractor negotiate outcomes at every cycle. When the contractor delivers the system at FOC, the client has a fully functioning system that reflects the client's operational needs that evolved during the system's development. More than that, the delivered system has virtually no obsolete components because the contractor addressed obsolescence at every cycle.

Agile Procurement

The second process, described above, is called agile procurement. It does not resemble programs EXA has observed when Canada says it is implementing agile procurement.

Let's be clear. The transition to true agile cyber procurement is neither easy nor straightforward. It is an enterprise-wide transformation that impacts clients, stakeholders, and suppliers.

Every epic journey has a first step. The first step is to recognize Canada’s outdated procurement processes are rooted in decades old doctrine that it developed before cybersecurity and agile procurement were terms in our vocabulary. Elements of a more robust cyber procurement solution include the following:

• Canada must abandon its fixed-price, fixed-solution, cyber procurement methods because they do not serve the government clients' needs.

• From the top, starting with Cabinet and Treasury Board, Canada must build flexibility into its procurement processes. Government leaders must accept you cannot predict or dictate the cost and schedule of a cyber system development.

• Canada must commit to respectful communications with industry. Issuing a draft RFP and then going silent for eight months, with no communication with industry, followed by the surprise. release of the final RFP, does not engage industry as a collaborating entity.

• Canada must recognize Agile Procurement is an enterprise-wide endeavour. Procurement alone does not fix the problem. Implementing effective agile procurement requires:

1. Political governance and oversight better aligned to agile procurement;
2. Let processes and procedures accommodate agile procurement;
3. Realigning contract requirements from deliverables to outcomes;
4. Collaborative mindset and communications during procurement and contracting;
5. Breaking large RFPs into smaller, linked, sequential contracts;
6. Implementing contracts in successive development cycles;
7. Viewing and managing costs and risks at the full life cycle, noting that in agile procurement, the line dividing the acquisition and support phases is vague; and
8. Developing a pool of trusted suppliers the government can rely on for rapid responses to complex, urgent issues.

Cyber threats change so rapidly that Canada's existing procurement system may be worse than outdated. It gives the illusion of accomplishing progress, but by the time a contractor delivers a compliant system, whose technical requirements and costs were rigidly defined ten years earlier, that cyber system is feeble and obsolete next to the cyber threats of the day. Only through responsive, effective agile procurement and development can Canada put the tools it needs into place to protect and support its assets. No doubt, the implementation of agile cyber procurement will be costly, but if you think cybersecurity is expensive, consider the cost of an enterprise-wide crippling attack.

Beyond Procurement

Agile procurement is an essential component to cybersecurity, but it is only one piece in a larger solution. In cybersecurity, you are only as strong as your weakest link. Cyber threat vectors come from unexpected, unpredictable,and often trusted sources.  Entities under cyberattack that have no direct relation to your organization (Canada Revenue Agency, Colonial Pipeline, JBS Meat Packing) can still impact a community's economy and operations, and they can damage your business.

Things will get worse before they get better. The tempo, severity, and impact of cyber-attacks will continue to rise as state-sponsored actors, including actors with tacit state approval, strike with increasingly sophisticated and devastating payloads.  Zero-day attacks (attacks that exploit a previously unknown vulnerability) are increasing in frequency and severity. Networks of interconnected networks have become so vast, ubiquitous, and deeply interwoven, and the technologies supporting these networks (and the connected systems that rely on them) have grown so complex and change so rapidly, it is impossible to predict where the next cyberthreat will come from and how severely it will impact your organization.

Not only are the frequency, randomness, and severity of attacks increasing, but the global reach of individual attacks is also expanding. Recent attacks upon formerly secure cyber supply chain entities multiply the number of targets an attack can infect by orders of magnitude. For example, in the recent Solar Wind and Kaseya attacks, the infiltration of a single host allowed the attacker to copy and deliver its payload to targets around the world by exploiting the hosts’ own trusted software distribution networks.

All boats rise with the tide. Beyond government procurement, part of the solution, and part of the government's responsibility, involves ensuring all entities that touch a supply chain have robust and effective cyber resilience. The US DoD recognizes that imperative with its Cybersecurity Maturity Model certification program, already in practice today.

Agile cyber procurement is just one important part of the larger solution. Government and industry must develop an immunity against cyber attacks through collaborative measures. The extensive interdependencies within the ecosystem prevent any one organization from securing itself from all cyber threat vectors. Every business, government department, agency, and NGO must undertake a collaborative, distributed, cyber-resilience capability development that detects threats when cyber events occur, warns others of the threats, and then collaboratively contain the breach.

When it comes to cybersecurity, we are all in this together.