CybatiWorks PI – Running on QEMU

CybatiWorks is an educational and research tool for learning about Industrial Control System cyber components. I haven’t used it much, but it looks like it’ll simulate a PLC controlling a process, and it’ll do it on a Raspberry PI, GPIO-connected hardware, and a controlling HMI (Human-Machine Interface) desktop. You can buy the hardware pre-setup, then use it in a course.

The person who runs the company is Matthew Luallen, and he’s quite responsive over email. I’ve been trying to look into the system a bit, and CybatiWorks offers the RasPI image for free through their “Community” program. Unfortunately that’s run by Google+, and is now a broken link. Emailing the responsive founder, however, will get you a link to the necessary image.

Now that I had the RasPI image though, I needed to run it, and didn’t have a PI handy. It was time for QEMU. This gentleman had a great start, and following his instructions allowed me to investigate the system partially, but that methodology gets you only 256MB RAM total. I needed more to start up all the services in the image, so I could see them work together.

QEMU’s documentation had a way forward – use the “virt” machine instead of versatile.., but this necessitated building a new kernel. Something I learned during this process – kernels built for one ARM machine don’t seem to work well on others. I’m not 100% why, I’ve definitely seen lots of binaries work interoperably, but kernels seem to be very specific (at least with QEMU).

The RasPI image comes with a kernel, f3n3s7ra’s page recommended a kernel… Unfortunately the QEMU documentation recommends installing a Debian image to get the kernel and initrd. That took several hours – now that I extracted them I’ve got them available for download via the links in the previous sentence (these came from the Debian project on 1 Nov 2019).

Once you’ve got initrd, vmlinuz, and CybatiWorksPI.img extracted from the email Matthew can send you, the command below will startup QEMU with a working network stack and kick you to a shell as root. You may have to switch the window view over to “serial0”.

sudo qemu-system-arm -M virt -m 1024 -kernel vmlinuz-3.16.0-6-armmp-lpae -initrd initrd.img-3.16.0-6-armmp-lpae -drive if=none,file=CybatiWorksPI.img,format=raw,id=hd -device virtio-blk-device,drive=hd -netdev tap,id=ethdev -device virtio-net-device,netdev=ethdev -no-reboot -append "root=/dev/vda2 rootfstype=ext4 rw init=/bin/bash"

You won’t get the typical startup sequence via systemd, and I haven’t been able to get that working yet, but you can do something similar with the command below (from the QEMU command line). This’ll kick off runlevel 3 startup scripts.

cd /etc/rc3.d
for i in S*; do ./$i restart; done

Now an ifconfig should reveal that eth0 is up and at 172.16.192.30/24. Back on your host computer “sudo ip add add 172.16.192.10/24 dev tap0” will configure tap0 to communicate with the QEMU box. You should now be able to ping 172.16.192.30 from your host.

The default services now should be:
TCP 22 – SSH
TCP 80 – lighttpd
TCP 2812 – monit
TCP 7777 – RexWSTCP
TCP 8000 – WebIOPi
TCP 43981 – RexCore

If you want to run the HMI VM Matthew will send you, don’t set your host to 172.16.192.30, so the VM can take that address. After starting the VM up, you may have to configure subnets more intelligently, and IP forwarding on your host (so the different network devices in your host can communicate).

Blockchain Use in Software Code Signing & Malware C2

I’ve done some small research about blockchain recently, and just want to put my thoughts down on paperblog so I can stop thinking them. Most of this is rehashing information I’ve read, but the “signed code verification” piece towards the end is an idea of mine that I’ve not read about elsewhere.

Blockchain is a hot term these days. It’s a popular management buzzword, and as such it can get thrown about as a cure for just about all that ails you. All businesses need to store data, and blockchain is known as a data-store, so everyone wants to make sure you’ve considered their (probably expensive) blockchain solution for data storage…

Blockchain is good at solving a couple problems though.

  • It can provide a publicly-verifiable record of data’s existence at a point in time. At any point in the future, anybody with access to the blockchain can prove that a certain piece of data existed at the point it was stored on the blockchain. If you’ve got a document that has been digitally signed, you can store the hash of that document on the blockchain. Later blockchain links will all chain back to your document hash, and their presence will prove that your document predated their transactions. Because blockchain additions (is most implementations) occur on fixed schedules, it will be possible to reconstruct exactly when your document must have been added to the chain.
  • Publicly-verifiable records of data’s existence can prove transactions have occurred, or contracts have been signed. This is how a blockchain can act as a ledger. Transactions on the blockchain can represent physical-world transactions, if the participants assign them that meaning, enabling tracking of the transfer of real-world entities between blockchain participants. On a public blockchain these transfers are transparent to everyone involved at all points in the future. They cannot be reverted by any individual except by a future transaction.
  • Transparency means that no trusted third-party needs to exist. Transactions can occur more easily between non-trusting parties without an escrow.
  • Public blockchains get stored by many parties, each having an economic incentive to participate. That means any data stored on them is stored in many places, providing data replication and the potential for access from disparate parts of the world. To store a small amount of data, little economic incentive is required. To store larger amounts of data, much greater incentive is required.

These solutions are enabled by some prerequisites.

  • Blockchains require lots of computational power. Specifically, they require more than your adversaries. To prove data’s existence at a point in time, links must be added to the chain periodically. When adding links to the chain, the other participants in the blockchain must agree on their addition. If there are malicious participants in the blockchain, they may decide to disagree about a set of new links, and instead agree on a different set of links. If the malicious participants form a majority, other participants in the chain will be influenced to believe the new malicious links are legitimate and ignore the others. Participants make decisions based upon cryptographic data that’s passed around, and computing that cryptographic data requires computational power. Therefore, to prevent a malicious takeover of a blockchain, you must have more computational power than any malicious adversaries can muster. In a public blockchain, you get the benefit of all the disparate participants’ computational power. Your adversaries must then overpower the public, instead of just you.
  • Blockchains require a computer network connecting participants. When designing a blockchain for use in low-Internet access areas you really restrict your ability to use existing public blockchains. To add data to a blockchain you must submit your transaction to the other participants, and enough of them must get it for your data to have any chance of being added. If you stop using public blockchains, or use small ones, perhaps because you’re in a remote area, you open yourself up to attacks based on computational power.
  • Public blockchains require economic incentive. Computational power costs money – for hardware, network access, and electricity. Participants in a blockchain require computational power. Thus, participants need a monetary incentive to participate. You pay for data additions to Bitcoin’s blockchain by supplying a small amount of bitcoin that is automatically paid to the individual who adds your data to the blockchain. The amount of data added by each transaction is small, so larger chunks of data require multiple transactions and more bitcoin.

Lots of proposed uses for blockchain have limited applicability due to these requirements.

Potential Use Case: Malware Command and Control (C2)

Malware C2 is an interesting use for blockchain, though. Malware running on end-points often needs to reach out to its creators for further instruction. “Steal files”, “learn about the local network”, “propagate to a nearby computer”, “record keystrokes”, or “delete yourself” are all things malware might want to do, but only when commanded by a remote attacker. Often malware reaches out to one destination for these commands. This is the simplest C2 implementation, where one or a few hard-coded server name or IP address provide the C2 to the malware.

Network defenders can try to detect and block this behavior by redirecting those C2 servers to alternate locations, pretending to be those C2 servers, or by taking over the C2 servers with the permission of law enforcement or the server’s rightful owner.

Blockchain’s distributed nature makes this much more difficult. Given a good-enough implementation, it could be difficult or impossible for defenders to block access to a copy of the blockchain. Because there’s no central server it’s not possible for a defender to take over the blockchain, either. Once a C2 command is added to the blockchain, it is impractical to remove it, too.

We’ve seen a few uses of it this method already. Omer Zohar built and demonstrated this use-case in early 2018. He used Ethereum and its “smart contracts” system to implement encrypted C2 of a nearly unlimited number of malware endpoints. The result was a system that is extremely difficult to block or subvert. As Ethereum increases in popularity the system will be increasingly hard to block. His major limitation was operational cost. Each message to and from an endpoint cost a small amount of Ether, translating to (at the time) about $39 per year per malware instance.

Anonymity is a major benefit of such a system. Many consider blockchain participation to be completely anonymous, however participation often requires money, and that money must enter the system from some point. That money typically comes from a traceable source, however malware authors also steal or “mine” cryptocurrency. Such activity would provide a less-traceable source, and make the system nearly entirely anonymous.

Potential Use Case: Code Signing Transparency

Another potential use for blockchain is in software signature verification. Microsoft’s Windows and Apple’s OS X use software signatures to verify that software was produced by an entity (a company or individual). Software producers compile their code into binaries, then digitally sign them before sending those binaries out into the wild.

This provides end users the assurance that a specific company made the binary. An example is if someone emails you a version of Microsoft’s Notepad. Before executing it you would want to make sure it was actually from Microsoft, otherwise a malicious actor could have modified Notepad to include malicious software. If Microsoft has signed this copy of Notepad, you can verify that signature and prove that Microsoft created it. You would know then that the version of Notepad was safe to execute. Windows and OS X now make it more difficult to execute software that’s not digitally signed by some vendor.

Stuxnet is a piece of malware that abused software signatures. It included components that were signed by legitimate, trusted software vendors. This made those components more likely to be trusted and less likely to be detected. Other malware has abused software signatures, but none has been as high-profile as Stuxnet.

Blockchain’s distributed, transparent, nearly-immutable nature can help solve this problem.

Software signatures can currently be verified by a client who trusts a root certificate, and can the follow that root certificate trust through a set of other certificates to the software signature in question. One certificate signs the next, which signs the next… Good verification requires checking a revocation list too, so when invalid signatures are found in-the-wild they can be cancelled.

If an additional step for verification required signatures to be found in a public database of software signatures, all malicious code signers would have to publish their signatures to this public database too. Companies could check the database to determine if someone is signing code on their behalf.

In the event of Stuxnet, JMicron Technology and Realtek Semiconductor would have been able to check the public database regularly. They would have seen a software signature they did not issue, and they could have placed it on the revocation list immediately. They could have then taken action to prevent further signatures in their name.

A blockchain can act as this public database. The result would be widely distributed, and it would be practically impossible to modify or remove signature entries after they were added. As an added benefit, it would become more obvious to observers when a company holds compromised certificates that should no-longer be trusted, and that company’s security practices could become (rightly) suspect. Because blockchain provides an irrefutable timestamp when data is added, signature attack timelines will also become more transparent to security researchers.

Every valid software signature would incur a small cost to be added to the blockchain. Additionally, the signature verification process would become more complex and require Internet access. However, software and hardware vendors could implement API endpoints that handle the blockchain portion of verification, simplifying lookup code for the endpoints they sell. The result would still provide transparency for all signature creators and verifiers.

I haven’t seen this solution proposed elsewhere, however Kim, Kwon and Dumitras recommend that code signing tools log all transactions they complete [Kim, 2017]. Tools like “signtool.exe” in Windows would log “the hash value of program code and the certificates” to Microsoft, then third parties could “periodically audit the log and identify code signing abuse”. This is great, however it doesn’t require software to verify that signatures are present in that log during signature verification. Without that, any attacker that subverts the signature reporting process, by preventing reporting to Microsoft, gets their software signed without reporting it.

References:

Some discussion of blockchain benefits and requirements: https://blog.todotnet.com/2019/03/solving-real-world-problems-with-distributed-ledger-technology/

Paper about blockchain potential in the military: https://www.jcs.mil/Portals/36/Documents/Doctrine/Education/jpme_papers/barnas_n.pdf?ver=2017-12-29-142140-393

Doowon Kim, Bum Jun Kwon, and Tudor Dumitraş. 2017. Certified Malware: Measuring Breaches of Trust in the Windows Code-Signing PKI. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security (CCS ’17). ACM, New York, NY, USA, 1435-1448. DOI: https://doi.org/10.1145/3133956.3133958. URL: http://users.umiacs.umd.edu/~tdumitra/papers/CCS-2017.pdf

Pitching Success – CO-STAR

Idaho National Labs has a program right now called “CO-STAR”.  Their researchers do great work, but as with any research group they are constantly advocating for funding, and researchers are constantly advocating among themselves for time.

Everybody spends time advocating for something.  “Pitching” something.  You want your boss to consider a smarter way of working, one that you’ve come up with?  You pitch it to her.  You want someone to use an open-source project you’ve created? You pitch it to them.

Some folks are naturals at this.

Nerds (researchers, open-source software creators, me) are often not good at this.  At least this is my experience.

I don’t like to brag about my stuff.  I have a much easier time bragging about someone else’s work.  I think a lot of folks have this specific problem with pitching – but of course there are a number of other sticking points folks experience.

The CO-STAR system is a way to break out of that a bit, because it is a process for performing a pitch that one can simply follow.  Moxie and a persuasive style may help a pitch, and CO-STAR can’t help those, but it does set out the requirements your target will need to evaluate your pitch logically.  It’s a set of steps you can follow in a discussion – and it’s even ordered!  In an actual pitch a more-experienced speaker may choose to skip some of the steps that are already understood by all parties, but one could simply progress through the steps in order and have success. This link is a description of the system in short.

First, you need an idea you want to pitch and a target to pitch to.  Perhaps you want your boss to let you adopt a new way of washing the dishes, maybe you want your company’s training division to send you to a conference, maybe you want a separate company to use your widget. 

Next, use a couple sentences to describe and answer each of the following.  These are quotes with only small clarification on my part, and these come from that PDF linked above, which is copyright 2017 Enterprise Development Group, Inc.

Your Hook: Begin with a compelling question, fact, or statement that will generate (in the mind of your target) curiosity related to your idea.

Customer: Who is the customer for your idea?  What needs do they have that might be met by your idea?

Opportunity: What is the opportunity for them?

Solution: What is your solution to their problem?

Team: Who needs to be on your team to solve this?

Advantage: What is the competitive advantage your solution provides over other solutions?

Results: What results will you achieve?

The Ask: Conclude with a specific request regarding the next steps your target should take.

Don’t spend too much time on the solution.  Most people get too excited about the solution because, after all, the solution was their big idea!  But your pitch target probably cares more about the other things like opportunity, advantage, and most of all results.

Remember that when considering your advantage one “other solution” is for your target to keep doing whatever they are currently doing.  Even if that’s “nothing”.  Make sure you consider the advantages of your solution over “doing nothing”.

Regarding the ask and next steps maybe you want a meeting with the target where you can discuss a way-forward in more depth, or maybe you just want to ask for the time and money to do your idea (in the event of conference attendance, for example, that’s all you might need).

The CO-STAR system is pretty simple, and anyone who has pitched before has used parts of it.  One benefit is that it lays steps out that, when followed in-order, clearly explain the value proposition you want to make.  It systematizes and simplifies a pitch.

There are a number of non-traditional pitches I’ve made recently… Websites, emails about opportunities, persuasive papers… All benefit from answering the questions in the CO-STAR model.

Meet Differently

A recent episode of the Freakonomics podcast covered meetings. Two or more people gathering to accomplish the business of business, as they defined it. It gave me some things I hope I remember the next time I’m organizing a recurring staff meeting…

  1. Organize an agenda around questions-to-answer
  2. Hold smaller meetings
  3. Keep track and time

Meetings need agendas, just about every book I’ve read which touches on meetings agrees on this point. The agenda needs to be communicated to participants in advance, so folks can come ready to accomplish it. Folks need to know what to expect. Without an agenda meetings usually devolve into chaos, although sometimes it takes a couple aimless meetings in a row to hit this point. Staff meetings often don’t have a pre-published agenda, but they generally do have a set of topics they proceed through in a set order, and that becomes the known-agenda for the group.

Generally my agendas are lists of topics – that’s not the most effective way to manage a meeting though. In the podcast Steven Rogelberg says, “if you can’t come up with any questions, you shouldn’t have a meeting”. A meeting isn’t for distributing announcements – there are other, better, less time-intrusive means for that. A meeting is for collaboration on problems, which can be stated as questions.

Organize an agenda around questions-to-answer

This also makes it easy to shake out the folks that actually need to attend the meeting. Getting everyone to attend is a waste of their time. Getting just the folks who have inputs on question answers is the minimum group for attendance. If others have input they can attend too after seeing the meeting agenda.

One question I’ve needed to answer in the past is, “what work from my team am I going to present to my bosses this week?” I’ve often taken it on myself to answer this question, but it occurs to me that the transparency I’d provide by bring this question to the workers, getting their input, and providing my feedback, could be excellent.

Smaller meetings, like shorter meetings, make better use of my folks’ time. When nugging the group of attendees down based on inputs required, I’m building a smaller meeting. I can also build smaller meetings by holding more meetings but with targeted groups.

For instance, when I led tactics development I could have met with the team leads individually to gather their inputs for the week. What’s their status update, what are their blockers, what are their accomplishments… This would have given me personal time with each lead (valuable), time to track their updates, time for them to provide me feedback they might not in a larger group, and time for me to provide them more private feedback. Then, our larger group meeting might have asked the questions, “which of these accomplishments are most valuable to raise up”, “which blockers can be solved in the group”, and “which blockers are most valuable to raise up?”

I actually did that on occasion, thinking back. Not in such an organized way, but as a result of intuition. There were reasons at the time I didn’t fully embrace the method (partly because it wasn’t my job but I felt like I still had to do it) so I did it only about 90%. A mistake.

Anyway.

Hold smaller meetings

If a meeting has questions for the full group, and also some for only a subset, after the full group part is complete let the folks unconcerned for the rest leave. Open the door, take a 30 second break.

For small staff meetings, to improve group openness, risk-taking, and personal connection, consider taking 10 minutes at the beginning to do “a rose and a thorn.” Everyone tells what they see as the best part and worst part of their week/week-to-come.

Keep track and time

Start on time, end on time. Set the meeting for the amount of time the questions will take, not some arbitrary value like “an hour”. If there’s pre-reading required, consider scheduling-in time at the beginning for folks to complete it.

Take minutes, and if the group decides work needs to be done and follow-ups need to occur then assign those tasks to someone. Write that assignment down. Discuss when you’ll expect the result, then come back to it later. Put all that in the minutes and send that out to the participants.

Announce “last call” at the end of the meeting, or as we do in the Air Force ask for any “reattacks”. Maybe plan for the last ten minutes for this, and set an alarm that shouts “LAST CALL” or “REATTACKS!” at that time. That’d at least be fun.

Folks are going to complain about meetings. We all do – even when they’re useful. At the least I can try to make them a good use of my folks’ time.

Consider How Well-Defined a Problem is When Building Teams

To improve innovation on a team, consider building a team differently based on the problem you’re facing. There are many ways of categorizing a problem your team must solve, but one is along the axis of well-definedness.

Well-defined problems – here, you know what you’re trying to solve, you know what end-state your audience will find acceptable, maybe you’ve seen similar problems solved elsewhere or you even know some current acceptable solutions.

Poorly-defined problems – these might be more general problems, things you’ve never encountered before, problems nobody else is even thinking about yet. Maybe your boss doesn’t know the full shape of things, or maybe the full shape of things is not yet knowable.

Research shows that for well-defined problems, having a less experience-diverse team can produce more innovative solutions. Diverse is a loaded word these days, but in the case of the research diversity was measured in terms of experience relevant to the problem. I’m calling that experience-diversity. So – for well-defined problems, having a team comprised of individuals already versed in the problem set produced more innovative solutions.

For poorly-defined problems, having a more experience-diverse team can produce more innovative solutions. Diversity, again, along the lines of folks versed in the problem. This means – if you don’t have a clearly defined problem, get a bunch of folks with a variety of expertise. This will help you find all sorts of solutions, some of which may be obviously appropriate and good, and others which you will need to take more risk on. The whole definition of the poorly-defined problem, though, indicates that you don’t really know what you want until you see it.

You need a team that’ll produce the iPod, for poorly-defined problems. And that takes experience-diversity.

Consider How Well-Defined a Problem is When Building Teams