Thoughts on SWAP

I just finished reading the SWAP study. The thrilling congressionally-mandated “Software Acquisition and Practices” study conducted by the Defense Innovation Board. 292 pages of discussion about the US government’s software acquisition practices.

It’s actually quite a bit funnier than you’d expect. Admittedly, it helps to be in on the jokes.

These are the same folks that brought us: Detecting Agile BS. A document which is unexpectedly funny, for a government report, and accurate.

If you’re interested in the topic, the SWAP study deserves a read. The Extended Abstract and Executive Summaries will probably get you most of the way, and don’t take long.

There’s a bunch of other great stuff in there, and the report kicks off by explaining why anyone should care about software development in the military, and why we know we’re screwed up on it.

Some oft the recommendations that piqued my interest, though, as someone interested in how the DoD manages software development talent:

  • Fundamental theme two of three: Digital talent matters because software is made by and for people. We need organic military and civilian software capabilities.
  • Line of effort two of four: We must create digital infrastructure to enable “rapid deployment, scaling, testing, and optimization of software”.
    • Services need to build this, and we need to incentivize it’s use – even by contractors…
    • We need fully-automatable approaches for test and eval.
    • We need approval to use these things across services once one service has approved them.
  • Line of effort three of four: Software development must be a high-visibility, high-priority career track. All services need units with mil and civ personnel that develop and deploy software using DevSecOps practices.

I think those recommendations are right on, and have been advocating for each of them for years. I’m glad to see a report to Congress I can cite now.

Another bit of the report I loved was the comparison between DoD’s management of medical/legal professionals and software developers. Individuals practicing those skills are managed very differently from the rest of the force because their skills are vital to the military, difficult to attain and maintain, and highly sought in the private sector. Software development skills tick all those same boxes, yet, “software developers, designers, and managers in the Services must practice their skills intermittently and often without support as they endure frequent rotations into other roles.”

A great quote I’ve not heard so succinctly before:

Speed increases security. Conventional wisdom in DoD says that programs must move slowly because moving quickly would threaten security. Often, the opposite is true.

As soon as anyone knows about a bug they can start to accomplish the other activities required to exploit it and close the kill-chain. Thus, the longer that bug is exploitable on a system, the more likely it is to be exploited. Speedily fixing it shuts down the adversary’s kill chain.

The paper goes on to explain that if we can, “deploy software faster without sacrificing [ability] to test and validate software”, we’ll have more secure systems. And this sounds like a no-brainer, of course. And folks who haven’t heard of DevSecOps will think this is an impossible pipe dream involving Unicorns. But those folks should read The Phoenix Project, or any of the success stories surrounding DevOps/DevSecOps.

Our software development must be a continuous flow in many ways. Software is never “finished”, although we may choose to stop developing it. Many of the “steps” (dev, test, integrate, test, deploy…) should all happen at the same time, with developed code flowing through those later stages automatically. Money must flow continuously for these activities, and should not be discretized between activities. The report explains all these things.

Two more recommendations from the report that I’ve long-advocated…

“Require program managers to stay with a project to its end.”

I learned this in my introduction to acquisitions course. This has long been one of the ways the private sector avoids project disaster. We teach this bit of knowledge to our acquisition workforce, then require/incentivize them to move frequently.

“Shift from certification of executables, to certification of code, to certification of the development, integration, and deployment toolchain, with the goal of enabling rapid fielding of mission-critical code at high levels of information assurance.”

Certification generally involves a staffing exercise, often up to very high levels of leadership. Those folks are not individuals who understand why or if something should be certified, but through the staffing process those individuals/teams (hopefully) buy-in on the certification. Then the leadership reviews that buy-in and signs, sealing the whole deal. It makes a lot of sense, honestly, and has the added benefit of pinning responsibility on a person who must take responsibility.

But it’s extremely laborious. And the most obvious thing to certify, the executable, gets replaced (and therefore re-certified) every time there’s an update… So updates are stove-piped and slowed due to human-process reasons.

A way to avoid that issue while still providing the same buy-in guarantees is to certify a process instead of a product. We do this all the time in other parts of the military, notably when a regulation mandates that individuals holding specific roles accomplish specific tasks/reviews/etc. We can do it with software by certifying automated systems and processes too. Then those things can operate as quickly as they can execute (much more quickly than human processes), and produce outputs that are certified by extension.

This gets to the heart of that digital infrastructure line of effort. But – instead of keeping it in-house we’d open it up for organic and private-sector developers to employ, and we’d share it across services.

Totally awesome ideas.

Providing Cloud Services in the Air Force

I was thinking this morning about how I might manage an Air Force unit that provides networked server management services. For some reason. I realized that, while I know a bit about some of the technology used to provide cloud services, and manage a server farm – or at least what’s used by some cloud providers – I don’t know much about how they organize their business. I started to wonder if someone from Rackspace, or AWS, or DigitalOcean had written a book about their management practices, or company organization.

Searching, I found some Rackspace SEC filings that seemed interesting. I’m just gonna put some notes here.

The services they provide are:

  • Dedicated Hosting: this looks like an option where a business gets full access to a computer on the Internet, and then does whatever they need with that server. They’re in charge of managing it themselves, generally, but have support staff to help out along the way.
  • Managed Hosting: this option has Rackspace providing, “a dedicated team of experts who provide comprehensive design, engineering, management, and monitoring expertise”. This is specifically for customers who “lack the technical expertise to support [these services] in-house”. So, it seems like the customer would work with Rackspace to determine what’s needed, then Rackspace would do the build-out and support.
  • Email Hosting: pretty obvious what this is. I’m not that interested in this, so I’m going to ignore it.
  • Cloud Hosting: this “includes tools for customers to develop, manage, and deliver new web-based services”. And, it targets “customers that do not have in-house IT expertise to support the OS layer of their IT systems”, yet want to focus on the app-side. This looks like it encompasses everything from VPS to “server-less” tech, although this was written in 2008 so back then it may have just meant VPS…
  • Platform Hosting: this was still “in-development” at the time, but seems to be colocation services +, where companies can move/buy-in infrastructure at Rackspace, take lots of responsibility in management and administration, take advantage of Rackspace support and facilities, and probably take easy advantage of other Rackspace capabilities.

I can order these in terms of required customer technical interaction/expertise, increasing:

  1. Managed Hosting
  2. Cloud Hosting
  3. Dedicated Hosting
  4. Platform Hosting

And in terms of price, for similar service, I suspect the list is the same but reversed, with Platform Hosting costing the least. However, each of these plans is really geared toward very different types of services… Platform Hosting customers would probably require a much greater amount of service than Managed Hosting users, and thus would pay a much greater amount. Cloud Hosting customers generally run the gamut, some generally requiring very little service and some generally requiring a great deal, but many also require the ability to rapidly scale from small to large resources.

Rackspace’s customers numbered 29k, with 36k servers, and 32k cloud hosting domains. With that level of requirement, they had the following sales and marketing team:

  • Direct sales: 180 folks working leads and such.
  • Channel sales: 850 partners that, I presume, do customized IT services and like to use Rackspace for their customers.
  • Marketing: No specific team numbers here, but the mission is clear.

They outline their support team structure, which was about 700 Rackers on teams of 12 to 20. One on the team was an account manager acting as a customer’s single point of contact, and at least some on each team were “technical specialists to meet ongoing customer needs.”

Rackspace had R&D efforts geared at deploying new tech to meet emerging trends, developing internal-use tools, and developing sales/support processes. These efforts also integrated management and ops personnel, but otherwise only involved 86 personnel.

Regarding applicability to the Air Force, I suspect all of these service types, sales units, support activities and R&D activities are relevant to a unit involved in managing and providing networked server services to other units.

Regarding the service types, there’s a huge push to enable innovation within units. A service provider (SP) enabling cheap/free easy small hosting of specific services, perhaps only on internal networks, could be a huge step for enabling that innovation. The clients of this would be similar to low-level non-complex cloud hosting clients. They might agree to potentially low-availability services, and other service limits. With backend technology permitting these services to be given a very low priority, we might host such services in only the server time not required by higher priority customers. This capability might be funded by innovation funds. Mid and higher-level cloud hosting customers in the AF would probably result largely from innovation successes, and would be paid for by customers directly.

Managed hosting is useful when customers need a set of IT services that they can define in English, with a set of documents, but that they cannot or will not build themselves. After working with a customer to define what’s required, internal technical experts would build out the services. This service would, at times, require huge amounts of manpower capable of interfacing with the customers then designing and building out capability.

Dedicated hosting and platform hosting would primarily let AF customers take advantage of existing servers and networking capability, saving cost through economy of scale and centralization. Those customers would want nearly complete control over the devices or services they’ve deployed, and with that they’d take responsibility for management and security, but we might be responsible for physical requirement satisfaction (power, network, cooling…) and auditing.

Sales and marketing is useful as a function, although the end-motive is very different, and perhaps no people would be dedicated to this purpose. “Direct sales” functions would correspond to networking with other units who require services like what we provide, and making sure they’re aware of our capabilities. “Channel sales” functions correspond to educating leadership, even across-service, and working with other similar service providers to ensure the military as a whole is maximizing capability and minimizing cost. “Marketing” looks like outreach and more networking at every opportunity. Do folks realize that there is a place they can go to make their network-based innovations real? That’s the job of a marketing function.

The benefit of support teams is fairly obvious, the customer-focused rep seems like a fantastic organizational strategy, but a backup or other way for work to continue when someone’s out of the office is important. Small, cross-functional teams, are probably also a rewarding way for everyone to operate. The R&D function is something I’m personally very interested in, and keeping such a function working is as vital as it is difficult.

Ok. Thoughts on paper.

CybatiWorks PI – Running on QEMU

CybatiWorks is an educational and research tool for learning about Industrial Control System cyber components. I haven’t used it much, but it looks like it’ll simulate a PLC controlling a process, and it’ll do it on a Raspberry PI, GPIO-connected hardware, and a controlling HMI (Human-Machine Interface) desktop. You can buy the hardware pre-setup, then use it in a course.

The person who runs the company is Matthew Luallen, and he’s quite responsive over email. I’ve been trying to look into the system a bit, and CybatiWorks offers the RasPI image for free through their “Community” program. Unfortunately that’s run by Google+, and is now a broken link. Emailing the responsive founder, however, will get you a link to the necessary image.

Now that I had the RasPI image though, I needed to run it, and didn’t have a PI handy. It was time for QEMU. This gentleman had a great start, and following his instructions allowed me to investigate the system partially, but that methodology gets you only 256MB RAM total. I needed more to start up all the services in the image, so I could see them work together.

QEMU’s documentation had a way forward – use the “virt” machine instead of versatile.., but this necessitated building a new kernel. Something I learned during this process – kernels built for one ARM machine don’t seem to work well on others. I’m not 100% why, I’ve definitely seen lots of binaries work interoperably, but kernels seem to be very specific (at least with QEMU).

The RasPI image comes with a kernel, f3n3s7ra’s page recommended a kernel… Unfortunately the QEMU documentation recommends installing a Debian image to get the kernel and initrd. That took several hours – now that I extracted them I’ve got them available for download via the links in the previous sentence (these came from the Debian project on 1 Nov 2019).

Once you’ve got initrd, vmlinuz, and CybatiWorksPI.img extracted from the email Matthew can send you, the command below will startup QEMU with a working network stack and kick you to a shell as root. You may have to switch the window view over to “serial0”.

sudo qemu-system-arm -M virt -m 1024 -kernel vmlinuz-3.16.0-6-armmp-lpae -initrd initrd.img-3.16.0-6-armmp-lpae -drive if=none,file=CybatiWorksPI.img,format=raw,id=hd -device virtio-blk-device,drive=hd -netdev tap,id=ethdev -device virtio-net-device,netdev=ethdev -no-reboot -append "root=/dev/vda2 rootfstype=ext4 rw init=/bin/bash"

You won’t get the typical startup sequence via systemd, and I haven’t been able to get that working yet, but you can do something similar with the command below (from the QEMU command line). This’ll kick off runlevel 3 startup scripts.

cd /etc/rc3.d
for i in S*; do ./$i restart; done

Now an ifconfig should reveal that eth0 is up and at Back on your host computer “sudo ip add add dev tap0” will configure tap0 to communicate with the QEMU box. You should now be able to ping from your host.

The default services now should be:
TCP 22 – SSH
TCP 80 – lighttpd
TCP 2812 – monit
TCP 7777 – RexWSTCP
TCP 8000 – WebIOPi
TCP 43981 – RexCore

If you want to run the HMI VM Matthew will send you, don’t set your host to, so the VM can take that address. After starting the VM up, you may have to configure subnets more intelligently, and IP forwarding on your host (so the different network devices in your host can communicate).

Blockchain Use in Software Code Signing & Malware C2

I’ve done some small research about blockchain recently, and just want to put my thoughts down on paperblog so I can stop thinking them. Most of this is rehashing information I’ve read, but the “signed code verification” piece towards the end is an idea of mine that I’ve not read about elsewhere.

Blockchain is a hot term these days. It’s a popular management buzzword, and as such it can get thrown about as a cure for just about all that ails you. All businesses need to store data, and blockchain is known as a data-store, so everyone wants to make sure you’ve considered their (probably expensive) blockchain solution for data storage…

Blockchain is good at solving a couple problems though.

  • It can provide a publicly-verifiable record of data’s existence at a point in time. At any point in the future, anybody with access to the blockchain can prove that a certain piece of data existed at the point it was stored on the blockchain. If you’ve got a document that has been digitally signed, you can store the hash of that document on the blockchain. Later blockchain links will all chain back to your document hash, and their presence will prove that your document predated their transactions. Because blockchain additions (is most implementations) occur on fixed schedules, it will be possible to reconstruct exactly when your document must have been added to the chain.
  • Publicly-verifiable records of data’s existence can prove transactions have occurred, or contracts have been signed. This is how a blockchain can act as a ledger. Transactions on the blockchain can represent physical-world transactions, if the participants assign them that meaning, enabling tracking of the transfer of real-world entities between blockchain participants. On a public blockchain these transfers are transparent to everyone involved at all points in the future. They cannot be reverted by any individual except by a future transaction.
  • Transparency means that no trusted third-party needs to exist. Transactions can occur more easily between non-trusting parties without an escrow.
  • Public blockchains get stored by many parties, each having an economic incentive to participate. That means any data stored on them is stored in many places, providing data replication and the potential for access from disparate parts of the world. To store a small amount of data, little economic incentive is required. To store larger amounts of data, much greater incentive is required.

These solutions are enabled by some prerequisites.

  • Blockchains require lots of computational power. Specifically, they require more than your adversaries. To prove data’s existence at a point in time, links must be added to the chain periodically. When adding links to the chain, the other participants in the blockchain must agree on their addition. If there are malicious participants in the blockchain, they may decide to disagree about a set of new links, and instead agree on a different set of links. If the malicious participants form a majority, other participants in the chain will be influenced to believe the new malicious links are legitimate and ignore the others. Participants make decisions based upon cryptographic data that’s passed around, and computing that cryptographic data requires computational power. Therefore, to prevent a malicious takeover of a blockchain, you must have more computational power than any malicious adversaries can muster. In a public blockchain, you get the benefit of all the disparate participants’ computational power. Your adversaries must then overpower the public, instead of just you.
  • Blockchains require a computer network connecting participants. When designing a blockchain for use in low-Internet access areas you really restrict your ability to use existing public blockchains. To add data to a blockchain you must submit your transaction to the other participants, and enough of them must get it for your data to have any chance of being added. If you stop using public blockchains, or use small ones, perhaps because you’re in a remote area, you open yourself up to attacks based on computational power.
  • Public blockchains require economic incentive. Computational power costs money – for hardware, network access, and electricity. Participants in a blockchain require computational power. Thus, participants need a monetary incentive to participate. You pay for data additions to Bitcoin’s blockchain by supplying a small amount of bitcoin that is automatically paid to the individual who adds your data to the blockchain. The amount of data added by each transaction is small, so larger chunks of data require multiple transactions and more bitcoin.

Lots of proposed uses for blockchain have limited applicability due to these requirements.

Potential Use Case: Malware Command and Control (C2)

Malware C2 is an interesting use for blockchain, though. Malware running on end-points often needs to reach out to its creators for further instruction. “Steal files”, “learn about the local network”, “propagate to a nearby computer”, “record keystrokes”, or “delete yourself” are all things malware might want to do, but only when commanded by a remote attacker. Often malware reaches out to one destination for these commands. This is the simplest C2 implementation, where one or a few hard-coded server name or IP address provide the C2 to the malware.

Network defenders can try to detect and block this behavior by redirecting those C2 servers to alternate locations, pretending to be those C2 servers, or by taking over the C2 servers with the permission of law enforcement or the server’s rightful owner.

Blockchain’s distributed nature makes this much more difficult. Given a good-enough implementation, it could be difficult or impossible for defenders to block access to a copy of the blockchain. Because there’s no central server it’s not possible for a defender to take over the blockchain, either. Once a C2 command is added to the blockchain, it is impractical to remove it, too.

We’ve seen a few uses of it this method already. Omer Zohar built and demonstrated this use-case in early 2018. He used Ethereum and its “smart contracts” system to implement encrypted C2 of a nearly unlimited number of malware endpoints. The result was a system that is extremely difficult to block or subvert. As Ethereum increases in popularity the system will be increasingly hard to block. His major limitation was operational cost. Each message to and from an endpoint cost a small amount of Ether, translating to (at the time) about $39 per year per malware instance.

Anonymity is a major benefit of such a system. Many consider blockchain participation to be completely anonymous, however participation often requires money, and that money must enter the system from some point. That money typically comes from a traceable source, however malware authors also steal or “mine” cryptocurrency. Such activity would provide a less-traceable source, and make the system nearly entirely anonymous.

Potential Use Case: Code Signing Transparency

Another potential use for blockchain is in software signature verification. Microsoft’s Windows and Apple’s OS X use software signatures to verify that software was produced by an entity (a company or individual). Software producers compile their code into binaries, then digitally sign them before sending those binaries out into the wild.

This provides end users the assurance that a specific company made the binary. An example is if someone emails you a version of Microsoft’s Notepad. Before executing it you would want to make sure it was actually from Microsoft, otherwise a malicious actor could have modified Notepad to include malicious software. If Microsoft has signed this copy of Notepad, you can verify that signature and prove that Microsoft created it. You would know then that the version of Notepad was safe to execute. Windows and OS X now make it more difficult to execute software that’s not digitally signed by some vendor.

Stuxnet is a piece of malware that abused software signatures. It included components that were signed by legitimate, trusted software vendors. This made those components more likely to be trusted and less likely to be detected. Other malware has abused software signatures, but none has been as high-profile as Stuxnet.

Blockchain’s distributed, transparent, nearly-immutable nature can help solve this problem.

Software signatures can currently be verified by a client who trusts a root certificate, and can the follow that root certificate trust through a set of other certificates to the software signature in question. One certificate signs the next, which signs the next… Good verification requires checking a revocation list too, so when invalid signatures are found in-the-wild they can be cancelled.

If an additional step for verification required signatures to be found in a public database of software signatures, all malicious code signers would have to publish their signatures to this public database too. Companies could check the database to determine if someone is signing code on their behalf.

In the event of Stuxnet, JMicron Technology and Realtek Semiconductor would have been able to check the public database regularly. They would have seen a software signature they did not issue, and they could have placed it on the revocation list immediately. They could have then taken action to prevent further signatures in their name.

A blockchain can act as this public database. The result would be widely distributed, and it would be practically impossible to modify or remove signature entries after they were added. As an added benefit, it would become more obvious to observers when a company holds compromised certificates that should no-longer be trusted, and that company’s security practices could become (rightly) suspect. Because blockchain provides an irrefutable timestamp when data is added, signature attack timelines will also become more transparent to security researchers.

Every valid software signature would incur a small cost to be added to the blockchain. Additionally, the signature verification process would become more complex and require Internet access. However, software and hardware vendors could implement API endpoints that handle the blockchain portion of verification, simplifying lookup code for the endpoints they sell. The result would still provide transparency for all signature creators and verifiers.

I haven’t seen this solution proposed elsewhere, however Kim, Kwon and Dumitras recommend that code signing tools log all transactions they complete [Kim, 2017]. Tools like “signtool.exe” in Windows would log “the hash value of program code and the certificates” to Microsoft, then third parties could “periodically audit the log and identify code signing abuse”. This is great, however it doesn’t require software to verify that signatures are present in that log during signature verification. Without that, any attacker that subverts the signature reporting process, by preventing reporting to Microsoft, gets their software signed without reporting it.


Some discussion of blockchain benefits and requirements:

Paper about blockchain potential in the military:

Doowon Kim, Bum Jun Kwon, and Tudor Dumitraş. 2017. Certified Malware: Measuring Breaches of Trust in the Windows Code-Signing PKI. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (CCS ’17). ACM, New York, NY, USA, 1435-1448. DOI: URL:

Pitching Success – CO-STAR

Idaho National Labs has a program right now called “CO-STAR”.  Their researchers do great work, but as with any research group they are constantly advocating for funding, and researchers are constantly advocating among themselves for time.

Everybody spends time advocating for something.  “Pitching” something.  You want your boss to consider a smarter way of working, one that you’ve come up with?  You pitch it to her.  You want someone to use an open-source project you’ve created? You pitch it to them.

Some folks are naturals at this.

Nerds (researchers, open-source software creators, me) are often not good at this.  At least this is my experience.

I don’t like to brag about my stuff.  I have a much easier time bragging about someone else’s work.  I think a lot of folks have this specific problem with pitching – but of course there are a number of other sticking points folks experience.

The CO-STAR system is a way to break out of that a bit, because it is a process for performing a pitch that one can simply follow.  Moxie and a persuasive style may help a pitch, and CO-STAR can’t help those, but it does set out the requirements your target will need to evaluate your pitch logically.  It’s a set of steps you can follow in a discussion – and it’s even ordered!  In an actual pitch a more-experienced speaker may choose to skip some of the steps that are already understood by all parties, but one could simply progress through the steps in order and have success. This link is a description of the system in short.

First, you need an idea you want to pitch and a target to pitch to.  Perhaps you want your boss to let you adopt a new way of washing the dishes, maybe you want your company’s training division to send you to a conference, maybe you want a separate company to use your widget. 

Next, use a couple sentences to describe and answer each of the following.  These are quotes with only small clarification on my part, and these come from that PDF linked above, which is copyright 2017 Enterprise Development Group, Inc.

Your Hook: Begin with a compelling question, fact, or statement that will generate (in the mind of your target) curiosity related to your idea.

Customer: Who is the customer for your idea?  What needs do they have that might be met by your idea?

Opportunity: What is the opportunity for them?

Solution: What is your solution to their problem?

Team: Who needs to be on your team to solve this?

Advantage: What is the competitive advantage your solution provides over other solutions?

Results: What results will you achieve?

The Ask: Conclude with a specific request regarding the next steps your target should take.

Don’t spend too much time on the solution.  Most people get too excited about the solution because, after all, the solution was their big idea!  But your pitch target probably cares more about the other things like opportunity, advantage, and most of all results.

Remember that when considering your advantage one “other solution” is for your target to keep doing whatever they are currently doing.  Even if that’s “nothing”.  Make sure you consider the advantages of your solution over “doing nothing”.

Regarding the ask and next steps maybe you want a meeting with the target where you can discuss a way-forward in more depth, or maybe you just want to ask for the time and money to do your idea (in the event of conference attendance, for example, that’s all you might need).

The CO-STAR system is pretty simple, and anyone who has pitched before has used parts of it.  One benefit is that it lays steps out that, when followed in-order, clearly explain the value proposition you want to make.  It systematizes and simplifies a pitch.

There are a number of non-traditional pitches I’ve made recently… Websites, emails about opportunities, persuasive papers… All benefit from answering the questions in the CO-STAR model.