Thoughts on SWAP

I just finished reading the SWAP study. The thrilling congressionally-mandated “Software Acquisition and Practices” study conducted by the Defense Innovation Board. 292 pages of discussion about the US government’s software acquisition practices.

It’s actually quite a bit funnier than you’d expect. Admittedly, it helps to be in on the jokes.

These are the same folks that brought us: Detecting Agile BS. A document which is unexpectedly funny, for a government report, and accurate.

If you’re interested in the topic, the SWAP study deserves a read. The Extended Abstract and Executive Summaries will probably get you most of the way, and don’t take long.

There’s a bunch of other great stuff in there, and the report kicks off by explaining why anyone should care about software development in the military, and why we know we’re screwed up on it.

Some oft the recommendations that piqued my interest, though, as someone interested in how the DoD manages software development talent:

  • Fundamental theme two of three: Digital talent matters because software is made by and for people. We need organic military and civilian software capabilities.
  • Line of effort two of four: We must create digital infrastructure to enable “rapid deployment, scaling, testing, and optimization of software”.
    • Services need to build this, and we need to incentivize it’s use – even by contractors…
    • We need fully-automatable approaches for test and eval.
    • We need approval to use these things across services once one service has approved them.
  • Line of effort three of four: Software development must be a high-visibility, high-priority career track. All services need units with mil and civ personnel that develop and deploy software using DevSecOps practices.

I think those recommendations are right on, and have been advocating for each of them for years. I’m glad to see a report to Congress I can cite now.

Another bit of the report I loved was the comparison between DoD’s management of medical/legal professionals and software developers. Individuals practicing those skills are managed very differently from the rest of the force because their skills are vital to the military, difficult to attain and maintain, and highly sought in the private sector. Software development skills tick all those same boxes, yet, “software developers, designers, and managers in the Services must practice their skills intermittently and often without support as they endure frequent rotations into other roles.”

A great quote I’ve not heard so succinctly before:

Speed increases security. Conventional wisdom in DoD says that programs must move slowly because moving quickly would threaten security. Often, the opposite is true.

As soon as anyone knows about a bug they can start to accomplish the other activities required to exploit it and close the kill-chain. Thus, the longer that bug is exploitable on a system, the more likely it is to be exploited. Speedily fixing it shuts down the adversary’s kill chain.

The paper goes on to explain that if we can, “deploy software faster without sacrificing [ability] to test and validate software”, we’ll have more secure systems. And this sounds like a no-brainer, of course. And folks who haven’t heard of DevSecOps will think this is an impossible pipe dream involving Unicorns. But those folks should read The Phoenix Project, or any of the success stories surrounding DevOps/DevSecOps.

Our software development must be a continuous flow in many ways. Software is never “finished”, although we may choose to stop developing it. Many of the “steps” (dev, test, integrate, test, deploy…) should all happen at the same time, with developed code flowing through those later stages automatically. Money must flow continuously for these activities, and should not be discretized between activities. The report explains all these things.

Two more recommendations from the report that I’ve long-advocated…

“Require program managers to stay with a project to its end.”

I learned this in my introduction to acquisitions course. This has long been one of the ways the private sector avoids project disaster. We teach this bit of knowledge to our acquisition workforce, then require/incentivize them to move frequently.

“Shift from certification of executables, to certification of code, to certification of the development, integration, and deployment toolchain, with the goal of enabling rapid fielding of mission-critical code at high levels of information assurance.”

Certification generally involves a staffing exercise, often up to very high levels of leadership. Those folks are not individuals who understand why or if something should be certified, but through the staffing process those individuals/teams (hopefully) buy-in on the certification. Then the leadership reviews that buy-in and signs, sealing the whole deal. It makes a lot of sense, honestly, and has the added benefit of pinning responsibility on a person who must take responsibility.

But it’s extremely laborious. And the most obvious thing to certify, the executable, gets replaced (and therefore re-certified) every time there’s an update… So updates are stove-piped and slowed due to human-process reasons.

A way to avoid that issue while still providing the same buy-in guarantees is to certify a process instead of a product. We do this all the time in other parts of the military, notably when a regulation mandates that individuals holding specific roles accomplish specific tasks/reviews/etc. We can do it with software by certifying automated systems and processes too. Then those things can operate as quickly as they can execute (much more quickly than human processes), and produce outputs that are certified by extension.

This gets to the heart of that digital infrastructure line of effort. But – instead of keeping it in-house we’d open it up for organic and private-sector developers to employ, and we’d share it across services.

Totally awesome ideas.

Providing Cloud Services in the Air Force

I was thinking this morning about how I might manage an Air Force unit that provides networked server management services. For some reason. I realized that, while I know a bit about some of the technology used to provide cloud services, and manage a server farm – or at least what’s used by some cloud providers – I don’t know much about how they organize their business. I started to wonder if someone from Rackspace, or AWS, or DigitalOcean had written a book about their management practices, or company organization.

Searching, I found some Rackspace SEC filings that seemed interesting. I’m just gonna put some notes here.

The services they provide are:

  • Dedicated Hosting: this looks like an option where a business gets full access to a computer on the Internet, and then does whatever they need with that server. They’re in charge of managing it themselves, generally, but have support staff to help out along the way.
  • Managed Hosting: this option has Rackspace providing, “a dedicated team of experts who provide comprehensive design, engineering, management, and monitoring expertise”. This is specifically for customers who “lack the technical expertise to support [these services] in-house”. So, it seems like the customer would work with Rackspace to determine what’s needed, then Rackspace would do the build-out and support.
  • Email Hosting: pretty obvious what this is. I’m not that interested in this, so I’m going to ignore it.
  • Cloud Hosting: this “includes tools for customers to develop, manage, and deliver new web-based services”. And, it targets “customers that do not have in-house IT expertise to support the OS layer of their IT systems”, yet want to focus on the app-side. This looks like it encompasses everything from VPS to “server-less” tech, although this was written in 2008 so back then it may have just meant VPS…
  • Platform Hosting: this was still “in-development” at the time, but seems to be colocation services +, where companies can move/buy-in infrastructure at Rackspace, take lots of responsibility in management and administration, take advantage of Rackspace support and facilities, and probably take easy advantage of other Rackspace capabilities.

I can order these in terms of required customer technical interaction/expertise, increasing:

  1. Managed Hosting
  2. Cloud Hosting
  3. Dedicated Hosting
  4. Platform Hosting

And in terms of price, for similar service, I suspect the list is the same but reversed, with Platform Hosting costing the least. However, each of these plans is really geared toward very different types of services… Platform Hosting customers would probably require a much greater amount of service than Managed Hosting users, and thus would pay a much greater amount. Cloud Hosting customers generally run the gamut, some generally requiring very little service and some generally requiring a great deal, but many also require the ability to rapidly scale from small to large resources.

Rackspace’s customers numbered 29k, with 36k servers, and 32k cloud hosting domains. With that level of requirement, they had the following sales and marketing team:

  • Direct sales: 180 folks working leads and such.
  • Channel sales: 850 partners that, I presume, do customized IT services and like to use Rackspace for their customers.
  • Marketing: No specific team numbers here, but the mission is clear.

They outline their support team structure, which was about 700 Rackers on teams of 12 to 20. One on the team was an account manager acting as a customer’s single point of contact, and at least some on each team were “technical specialists to meet ongoing customer needs.”

Rackspace had R&D efforts geared at deploying new tech to meet emerging trends, developing internal-use tools, and developing sales/support processes. These efforts also integrated management and ops personnel, but otherwise only involved 86 personnel.

Regarding applicability to the Air Force, I suspect all of these service types, sales units, support activities and R&D activities are relevant to a unit involved in managing and providing networked server services to other units.

Regarding the service types, there’s a huge push to enable innovation within units. A service provider (SP) enabling cheap/free easy small hosting of specific services, perhaps only on internal networks, could be a huge step for enabling that innovation. The clients of this would be similar to low-level non-complex cloud hosting clients. They might agree to potentially low-availability services, and other service limits. With backend technology permitting these services to be given a very low priority, we might host such services in only the server time not required by higher priority customers. This capability might be funded by innovation funds. Mid and higher-level cloud hosting customers in the AF would probably result largely from innovation successes, and would be paid for by customers directly.

Managed hosting is useful when customers need a set of IT services that they can define in English, with a set of documents, but that they cannot or will not build themselves. After working with a customer to define what’s required, internal technical experts would build out the services. This service would, at times, require huge amounts of manpower capable of interfacing with the customers then designing and building out capability.

Dedicated hosting and platform hosting would primarily let AF customers take advantage of existing servers and networking capability, saving cost through economy of scale and centralization. Those customers would want nearly complete control over the devices or services they’ve deployed, and with that they’d take responsibility for management and security, but we might be responsible for physical requirement satisfaction (power, network, cooling…) and auditing.

Sales and marketing is useful as a function, although the end-motive is very different, and perhaps no people would be dedicated to this purpose. “Direct sales” functions would correspond to networking with other units who require services like what we provide, and making sure they’re aware of our capabilities. “Channel sales” functions correspond to educating leadership, even across-service, and working with other similar service providers to ensure the military as a whole is maximizing capability and minimizing cost. “Marketing” looks like outreach and more networking at every opportunity. Do folks realize that there is a place they can go to make their network-based innovations real? That’s the job of a marketing function.

The benefit of support teams is fairly obvious, the customer-focused rep seems like a fantastic organizational strategy, but a backup or other way for work to continue when someone’s out of the office is important. Small, cross-functional teams, are probably also a rewarding way for everyone to operate. The R&D function is something I’m personally very interested in, and keeping such a function working is as vital as it is difficult.

Ok. Thoughts on paper.