Thoughts on SWAP

I just finished reading the SWAP study. The thrilling congressionally-mandated “Software Acquisition and Practices” study conducted by the Defense Innovation Board. 292 pages of discussion about the US government’s software acquisition practices.

It’s actually quite a bit funnier than you’d expect. Admittedly, it helps to be in on the jokes.

These are the same folks that brought us: Detecting Agile BS. A document which is unexpectedly funny, for a government report, and accurate.

If you’re interested in the topic, the SWAP study deserves a read. The Extended Abstract and Executive Summaries will probably get you most of the way, and don’t take long.

There’s a bunch of other great stuff in there, and the report kicks off by explaining why anyone should care about software development in the military, and why we know we’re screwed up on it.

Some oft the recommendations that piqued my interest, though, as someone interested in how the DoD manages software development talent:

  • Fundamental theme two of three: Digital talent matters because software is made by and for people. We need organic military and civilian software capabilities.
  • Line of effort two of four: We must create digital infrastructure to enable “rapid deployment, scaling, testing, and optimization of software”.
    • Services need to build this, and we need to incentivize it’s use – even by contractors…
    • We need fully-automatable approaches for test and eval.
    • We need approval to use these things across services once one service has approved them.
  • Line of effort three of four: Software development must be a high-visibility, high-priority career track. All services need units with mil and civ personnel that develop and deploy software using DevSecOps practices.

I think those recommendations are right on, and have been advocating for each of them for years. I’m glad to see a report to Congress I can cite now.

Another bit of the report I loved was the comparison between DoD’s management of medical/legal professionals and software developers. Individuals practicing those skills are managed very differently from the rest of the force because their skills are vital to the military, difficult to attain and maintain, and highly sought in the private sector. Software development skills tick all those same boxes, yet, “software developers, designers, and managers in the Services must practice their skills intermittently and often without support as they endure frequent rotations into other roles.”

A great quote I’ve not heard so succinctly before:

Speed increases security. Conventional wisdom in DoD says that programs must move slowly because moving quickly would threaten security. Often, the opposite is true.

As soon as anyone knows about a bug they can start to accomplish the other activities required to exploit it and close the kill-chain. Thus, the longer that bug is exploitable on a system, the more likely it is to be exploited. Speedily fixing it shuts down the adversary’s kill chain.

The paper goes on to explain that if we can, “deploy software faster without sacrificing [ability] to test and validate software”, we’ll have more secure systems. And this sounds like a no-brainer, of course. And folks who haven’t heard of DevSecOps will think this is an impossible pipe dream involving Unicorns. But those folks should read The Phoenix Project, or any of the success stories surrounding DevOps/DevSecOps.

Our software development must be a continuous flow in many ways. Software is never “finished”, although we may choose to stop developing it. Many of the “steps” (dev, test, integrate, test, deploy…) should all happen at the same time, with developed code flowing through those later stages automatically. Money must flow continuously for these activities, and should not be discretized between activities. The report explains all these things.

Two more recommendations from the report that I’ve long-advocated…

“Require program managers to stay with a project to its end.”

I learned this in my introduction to acquisitions course. This has long been one of the ways the private sector avoids project disaster. We teach this bit of knowledge to our acquisition workforce, then require/incentivize them to move frequently.

“Shift from certification of executables, to certification of code, to certification of the development, integration, and deployment toolchain, with the goal of enabling rapid fielding of mission-critical code at high levels of information assurance.”

Certification generally involves a staffing exercise, often up to very high levels of leadership. Those folks are not individuals who understand why or if something should be certified, but through the staffing process those individuals/teams (hopefully) buy-in on the certification. Then the leadership reviews that buy-in and signs, sealing the whole deal. It makes a lot of sense, honestly, and has the added benefit of pinning responsibility on a person who must take responsibility.

But it’s extremely laborious. And the most obvious thing to certify, the executable, gets replaced (and therefore re-certified) every time there’s an update… So updates are stove-piped and slowed due to human-process reasons.

A way to avoid that issue while still providing the same buy-in guarantees is to certify a process instead of a product. We do this all the time in other parts of the military, notably when a regulation mandates that individuals holding specific roles accomplish specific tasks/reviews/etc. We can do it with software by certifying automated systems and processes too. Then those things can operate as quickly as they can execute (much more quickly than human processes), and produce outputs that are certified by extension.

This gets to the heart of that digital infrastructure line of effort. But – instead of keeping it in-house we’d open it up for organic and private-sector developers to employ, and we’d share it across services.

Totally awesome ideas.

Providing Cloud Services in the Air Force

I was thinking this morning about how I might manage an Air Force unit that provides networked server management services. For some reason. I realized that, while I know a bit about some of the technology used to provide cloud services, and manage a server farm – or at least what’s used by some cloud providers – I don’t know much about how they organize their business. I started to wonder if someone from Rackspace, or AWS, or DigitalOcean had written a book about their management practices, or company organization.

Searching, I found some Rackspace SEC filings that seemed interesting. I’m just gonna put some notes here.

The services they provide are:

  • Dedicated Hosting: this looks like an option where a business gets full access to a computer on the Internet, and then does whatever they need with that server. They’re in charge of managing it themselves, generally, but have support staff to help out along the way.
  • Managed Hosting: this option has Rackspace providing, “a dedicated team of experts who provide comprehensive design, engineering, management, and monitoring expertise”. This is specifically for customers who “lack the technical expertise to support [these services] in-house”. So, it seems like the customer would work with Rackspace to determine what’s needed, then Rackspace would do the build-out and support.
  • Email Hosting: pretty obvious what this is. I’m not that interested in this, so I’m going to ignore it.
  • Cloud Hosting: this “includes tools for customers to develop, manage, and deliver new web-based services”. And, it targets “customers that do not have in-house IT expertise to support the OS layer of their IT systems”, yet want to focus on the app-side. This looks like it encompasses everything from VPS to “server-less” tech, although this was written in 2008 so back then it may have just meant VPS…
  • Platform Hosting: this was still “in-development” at the time, but seems to be colocation services +, where companies can move/buy-in infrastructure at Rackspace, take lots of responsibility in management and administration, take advantage of Rackspace support and facilities, and probably take easy advantage of other Rackspace capabilities.

I can order these in terms of required customer technical interaction/expertise, increasing:

  1. Managed Hosting
  2. Cloud Hosting
  3. Dedicated Hosting
  4. Platform Hosting

And in terms of price, for similar service, I suspect the list is the same but reversed, with Platform Hosting costing the least. However, each of these plans is really geared toward very different types of services… Platform Hosting customers would probably require a much greater amount of service than Managed Hosting users, and thus would pay a much greater amount. Cloud Hosting customers generally run the gamut, some generally requiring very little service and some generally requiring a great deal, but many also require the ability to rapidly scale from small to large resources.

Rackspace’s customers numbered 29k, with 36k servers, and 32k cloud hosting domains. With that level of requirement, they had the following sales and marketing team:

  • Direct sales: 180 folks working leads and such.
  • Channel sales: 850 partners that, I presume, do customized IT services and like to use Rackspace for their customers.
  • Marketing: No specific team numbers here, but the mission is clear.

They outline their support team structure, which was about 700 Rackers on teams of 12 to 20. One on the team was an account manager acting as a customer’s single point of contact, and at least some on each team were “technical specialists to meet ongoing customer needs.”

Rackspace had R&D efforts geared at deploying new tech to meet emerging trends, developing internal-use tools, and developing sales/support processes. These efforts also integrated management and ops personnel, but otherwise only involved 86 personnel.

Regarding applicability to the Air Force, I suspect all of these service types, sales units, support activities and R&D activities are relevant to a unit involved in managing and providing networked server services to other units.

Regarding the service types, there’s a huge push to enable innovation within units. A service provider (SP) enabling cheap/free easy small hosting of specific services, perhaps only on internal networks, could be a huge step for enabling that innovation. The clients of this would be similar to low-level non-complex cloud hosting clients. They might agree to potentially low-availability services, and other service limits. With backend technology permitting these services to be given a very low priority, we might host such services in only the server time not required by higher priority customers. This capability might be funded by innovation funds. Mid and higher-level cloud hosting customers in the AF would probably result largely from innovation successes, and would be paid for by customers directly.

Managed hosting is useful when customers need a set of IT services that they can define in English, with a set of documents, but that they cannot or will not build themselves. After working with a customer to define what’s required, internal technical experts would build out the services. This service would, at times, require huge amounts of manpower capable of interfacing with the customers then designing and building out capability.

Dedicated hosting and platform hosting would primarily let AF customers take advantage of existing servers and networking capability, saving cost through economy of scale and centralization. Those customers would want nearly complete control over the devices or services they’ve deployed, and with that they’d take responsibility for management and security, but we might be responsible for physical requirement satisfaction (power, network, cooling…) and auditing.

Sales and marketing is useful as a function, although the end-motive is very different, and perhaps no people would be dedicated to this purpose. “Direct sales” functions would correspond to networking with other units who require services like what we provide, and making sure they’re aware of our capabilities. “Channel sales” functions correspond to educating leadership, even across-service, and working with other similar service providers to ensure the military as a whole is maximizing capability and minimizing cost. “Marketing” looks like outreach and more networking at every opportunity. Do folks realize that there is a place they can go to make their network-based innovations real? That’s the job of a marketing function.

The benefit of support teams is fairly obvious, the customer-focused rep seems like a fantastic organizational strategy, but a backup or other way for work to continue when someone’s out of the office is important. Small, cross-functional teams, are probably also a rewarding way for everyone to operate. The R&D function is something I’m personally very interested in, and keeping such a function working is as vital as it is difficult.

Ok. Thoughts on paper.

Smokin a Turkey

Alright! It’s Christmas! I meant to do this a week ago, but the turkey was still frozen, so we’re doing it today. If it turns out terrible we’ll be eating Chinese for dinner.

I got a pre-brined 12 lbs turkey for less than $1 per pound, put olive oil on the skin, then sprinkled on the rub I use on pork butt because it’s similar to what this person recommends anyway. We’ll smoke it at 225℉, and it should take about 6 hours at that temperature. We’ve got temperature probes in the turkey, so we should know when it’s done.

1020: bird on the smoker, smoke holes half open.

1620: it’s chugging along nicely. The places where I originally put the temperature probes are over-temp now, by a little, but the base of the breast is still at 161℉. I moved a probe to that spot. We have sides cooking too, so I bumped the temp to 275℉ to get it done a little quicker.

1700: all done! It was juicy, delicious, and we have a bunch leftover. The rub was a fine choice, although it’s not adding all that much to the turkey. Adding more flavor through a brine or rub could be good next time.

Brisket! Sourdough!

15.5 pound brisket, $3.78 per pound, Sam’s in Idaho Falls, 4 Tbsp salt, 4 Tbsp pepper, 2 Tbsp garlic powder. Gonna put them on at 225℉ to start.

1810, 9 Nov: brisket on the smoker.

1830: sourdough is fermenting.

2200: flat is at 160℉, point at 142℉.

0210, 10 Nov: flat is 166℉, point 164℉. Crutched it!

0730: flat is 186℉, point 187℉. Turned temperature up to 250℉.

0845: put the rest of the ingredients in the bread. I’ve continued doing the 1 tsp regular yeast along with the sourdough starter, and doubling sourdough starter over the Josie Baker recommendation.

0930: flat is 201℉, point 200℉. Almost there!

1000: both are at 202℉. Just one degree more!

1045: sourdough starting bulk rise.

1110: brisket at 203℉. Taking it off.

Cleaned up, sliced up, gotta cool a bit…

1400: sourdough in the pan for final rise.

1712: sourdough in the oven.

Now brisket sandwiches will be almost from scratch… Everything except cheese and BBQ sauce

CybatiWorks PI – Running on QEMU

CybatiWorks is an educational and research tool for learning about Industrial Control System cyber components. I haven’t used it much, but it looks like it’ll simulate a PLC controlling a process, and it’ll do it on a Raspberry PI, GPIO-connected hardware, and a controlling HMI (Human-Machine Interface) desktop. You can buy the hardware pre-setup, then use it in a course.

The person who runs the company is Matthew Luallen, and he’s quite responsive over email. I’ve been trying to look into the system a bit, and CybatiWorks offers the RasPI image for free through their “Community” program. Unfortunately that’s run by Google+, and is now a broken link. Emailing the responsive founder, however, will get you a link to the necessary image.

Now that I had the RasPI image though, I needed to run it, and didn’t have a PI handy. It was time for QEMU. This gentleman had a great start, and following his instructions allowed me to investigate the system partially, but that methodology gets you only 256MB RAM total. I needed more to start up all the services in the image, so I could see them work together.

QEMU’s documentation had a way forward – use the “virt” machine instead of versatile.., but this necessitated building a new kernel. Something I learned during this process – kernels built for one ARM machine don’t seem to work well on others. I’m not 100% why, I’ve definitely seen lots of binaries work interoperably, but kernels seem to be very specific (at least with QEMU).

The RasPI image comes with a kernel, f3n3s7ra’s page recommended a kernel… Unfortunately the QEMU documentation recommends installing a Debian image to get the kernel and initrd. That took several hours – now that I extracted them I’ve got them available for download via the links in the previous sentence (these came from the Debian project on 1 Nov 2019).

Once you’ve got initrd, vmlinuz, and CybatiWorksPI.img extracted from the email Matthew can send you, the command below will startup QEMU with a working network stack and kick you to a shell as root. You may have to switch the window view over to “serial0”.

sudo qemu-system-arm -M virt -m 1024 -kernel vmlinuz-3.16.0-6-armmp-lpae -initrd initrd.img-3.16.0-6-armmp-lpae -drive if=none,file=CybatiWorksPI.img,format=raw,id=hd -device virtio-blk-device,drive=hd -netdev tap,id=ethdev -device virtio-net-device,netdev=ethdev -no-reboot -append "root=/dev/vda2 rootfstype=ext4 rw init=/bin/bash"

You won’t get the typical startup sequence via systemd, and I haven’t been able to get that working yet, but you can do something similar with the command below (from the QEMU command line). This’ll kick off runlevel 3 startup scripts.

cd /etc/rc3.d
for i in S*; do ./$i restart; done

Now an ifconfig should reveal that eth0 is up and at Back on your host computer “sudo ip add add dev tap0” will configure tap0 to communicate with the QEMU box. You should now be able to ping from your host.

The default services now should be:
TCP 22 – SSH
TCP 80 – lighttpd
TCP 2812 – monit
TCP 7777 – RexWSTCP
TCP 8000 – WebIOPi
TCP 43981 – RexCore

If you want to run the HMI VM Matthew will send you, don’t set your host to, so the VM can take that address. After starting the VM up, you may have to configure subnets more intelligently, and IP forwarding on your host (so the different network devices in your host can communicate).