Blog

Why Is It So Hard To Build Safe Software?

Asking aircraft designers about airplane safety: Hairbun: Nothing is ever foolproof, but modern airliners are incredibly resilient. Flying is the safest way to travel. Asking building engineers about elevator safety: Cueball: Elevators are protected by multiple tried-and-tested failsafe mechanisms. They're nearly incapable of falling. Asking software engineers about computerized voting: Megan: That's terrifying. Ponytail: Wait, really? Megan: Don't trust voting software and don't listen to anyone who tells you it's safe. Ponytail: Why? Megan: I don't quite know how to put this, but our entire field is bad at what we do, and if you rely on us, everyone will die. Ponytail: They say they've fixed it with something called "blockchain." Megan: AAAAA!!! Cueball: Whatever they sold you, don't touch it. Megan: Bury it in the desert. Cueball: Wear gloves.
XKCD #2030: “Voting Software”; used under the terms of its Creative Commons Attribution-NonCommercial 2.5 License.

Or, “Robert Graham is dead wrong”.

This XKCD comic on voting software security has been going around my computer security Twitter feed today, and a lot of folks have Takes on it.

It gets at something fundamental. What is it that makes software safety so hard?

A couple years ago, at the March 2016 STAMP Workshop in Cambridge, Massachusetts I gave a talk titled “Safety Thinking in Cloud Software: Challenges and Opportunities” where I tried to answer that. (As always, I talk about work here but don’t speak on behalf of any former employer.) What follows is based on my notes for that talk.

Continue reading “Why Is It So Hard To Build Safe Software?”

Why I Won’t Work For Facebook

I just sent an unintentionally blistering response to a Facebook recruiter. Having invested the time in writing it, I remembered that I have a very disused blog, and perhaps people reading here would find it useful, either as fodder for your own such messages, or as a snapshot of my concerns regarding Facebook and fascism in America in 2018. If either of these apply to you, enjoy.

Continue reading “Why I Won’t Work For Facebook”

A Proposal For Some Fucking Software Liability

…It’s not a Modest Proposal, because that was originally meant in satire, however it’s been corrupted in these latter and debased days, and I’m quite serious here.

(I am less of an expert on this than other things I blog about, although more knowledgeable than some; no warranty expressed or implied, &c.)

Today, basically all software comes with a blanket waiver of liability. The owners and coders of it do not express or imply any warranty, etc, blah blah blah, if it kills you, you pay your own funeral bills, and also you’re dead. And this leads us to situations where we have insulin pumps running consumer-grade software which hit its end of life four years ago at the ripe old age (for a piece of consumer-grade software) of fifteen.

The NSA hoarding vulnerabilities angle on this is a red herring, and I wish people would drop it. As nice as it would be for the US government to invest more than they do today in defense of software, there’s always going to be an interest in offense against software, and if it’s not the NSA’s vulnerability stockpile getting breached, it’s the Bad Guys’®, however we define them today. Even with some kind of MLAT for software vulnerabilities, the Bad Guys® do not sign or abide by those treaties, and unlike building nuclear weapons, exploiting software at this scale is still the province of bored and clever CS undergrads. We must proceed from the assumption that big tranches of vulnerabilities in our software and exploits for those vulnerabilities exist and might get exposed all at once.

There’s been much speculation, basically all of it (including this) ill-informed about the various legal frameworks already in place, both for software and other more established engineering products, about what effect some nebulously expressed change in the liability laws or case-law around software would have on the industry and practice of software production.

And my modest contribution is this:

All software does not need a fucking warranty. It’s fine that your shitty Javascript framework is shitty, and you shouldn’t be rung up on charges of criminal negligence if a shitty and obvious bug in your shitty Javascript framework kills somebody because your Javascript framework got used in a medical radiation device.

The people who should be rung up on charges of criminal negligence are the people who decided to integrate your shitty Javascript framework into their shitty medical radiation device. Consumer software is different than safety-critical software, and everything about using one for the other is wrong.

There are many different lines within the software ecosystem you can draw, and probably we will need to draw all of them, but safety-critical versus consumer (and then industrial control, and god knows what else) are some important fucking distinctions.

If requiring this kind of liability of the people who make medical devices causes them to prefer to use upstream Javascript framework providers who are also willing to take on this kind of liability, then, well, bully for everybody.

The other obvious players in this are the insurance industry, who have so far entirely punted on insuring software against this liability, probably because there’s no money in it, probably because nobody is going to get sued, probably because there are no laws requiring that somebody who integrates a shitty Javascript framework into a medical radiation device and kills half a dozen people do some jail time, yet, which is a real fucking shame, because purely from a Hammurabian moral perspective they probably should have hot sand driven under their fingernails.

I don’t know what about the economics of medical devices today causes them to be such a shitshow that this liability regime isn’t in place already, although I assume it’s much like the shitshow of other electronic devices (eg. Android phones), where it’s a commodity market without a way of valuing security, and integrators cobble together whatever shit they can to check the feature boxes the marketing and sales departments want and keep their customers buying new shit fast enough to keep the company from going bankrupt, but not fast enough to give them margins such that they can afford to build not-shitty medical devices, because it’s apparently unreasonable to expect that these companies and the people working for them should value not killing other people who have no choice but to submit themselves to the tender ministrations of the healthcare industrial complex.

Possibly a liability system for safety-critical devices would cause them to rethink their shitty life choices, and, more importantly, realign their market so that they could act on what the goodness in me compels me to assume is the non-shittiness within their hearts.

This anyway is my best explanation for the health insurance industry of today, who have for most of my life been rapacious bastards who will put you on the streets for pre-existing conditions including depression, which is only the natural state of all beings confronted with the enormity of the problem of evil in the world, and who are now championing not going back to the bad old days, because there is a bit of humanity left in their Grinch hearts after all. (And also regulation like this is actually better for business, but shh, don’t tell the capitalists that, it confuses and frightens them.)

And obviously we need legal frameworks such that medical devices can get certified on one version of your shitty but liability-insured Javascript framework and reasonably accept and deploy security patches to same and remain (slightly-less-)shitty and also liability insured without a godwaful and too-expensive recertification process, which is apparently part of the problem here as of today, although obviously any such certification system might also quite reasonably be concerned that the security patches not introduce yet other bugs, and balancing that will be an interesting trick.

Reliable sources (the guy who runs the certification company) inform me that we can do this for airplane avionics software, and it’s only (what I presume is) the lack of a (regulated-to-be-)level playing field in the medical device industry which makes this hard today, so it seems plausible that some medical, legal, and technical folks inspired by aviation and other safety-critical industries could sit down and create some proposed legislation which Congress could adopt with minimal editorial oversight which would result in a better medical device industry, fewer hospitals crippled by ransomware attacks, lower insurance premiums, and fewer fucking dead people.

It’s not like people aren’t working on this: (link to I Am The Cavalry) (link to Cyber-ITL) (link to Engineering a Safer World).  Somehow this work hasn’t made the requisite impact yet, and maybe WannaCry will open people up to it, and maybe it won’t, but a mob of people with torches and pitchforks at their legislators’ offices asking “what are you doing about medical device cybersecurity” won’t hurt.

Because any sober and fundamentally good-hearted person can see that it’s past fucking time we fixed this.

Just How Many People Live in Boston’s Millennium Tower, Anyway?

Inspired-slash-frustrated by a conversation on Twitter a few weeks ago, which I am too tired now to find, I went and dug into just how many people actually live, more-or-less full-time, in Boston’s Millennium Tower.  I was reminded of this by a conversation today, and, since the question seems to keep coming up, I might as well post my back-of-the-envelope analysis here.

A very light sketch of the background: Millennium Tower is a 60-story, 442-unit super-swank luxury condo development in the heart of downtown Boston; the tallest to date in the city, and by far the most expensive. This makes it a very visible target for the ire of folks who are rightly concerned about rising housing prices in Boston, with housing prices up 50% over just the last five years and rents quickly following.

It’s been widely reported that many of the buyers at Millennium Tower are non-local, even international, and so while there may be 442 units in the building, the question is, are there 442 units’ worth of people actually living there? (Millennium Tower replaced a Filene’s department store, so it’s not like houses or smaller apartments were torn down and people were put out of housing by its construction, but you can rightly ask the same question of developments where they were.)

 

First an existence proof: you can in fact find properties in Millennium Tower available for rent. I found these by searching Boston Craigslist for “Millennium Tower”: you’ll pay somewhere between $2500/mo (two units available at that price) and $4600/mo for an 800 sq. ft., 1-bedroom apartment. That $2500/mo figure is comparable to what I’ve heard for 1-bedrooms in Harvard and Porter Squares, which surprised me. Considering where and what Millennium Tower gets you, from a tenant perspective that $2500 is actually quite a steal, assuming it’s legit, which I have no reason to doubt. So that’s not just an existence proof but a really positive one. Rents are priced to move.

Now the question: just how many units in Millennium Tower are either owned or rented by people who live there full time? That’s a hard question to answer, so I’m going to answer the related and somewhat easier question of how many units are either owned by people as their principal residence, or rented (and I’m going to assume the rental tenants live there full time). That’s probably not entirely true, but I think it’ll let us start to put some bounds on our uncertainty.

To do that, I turn to the Bates Real Estate report on Millennium Tower from last month, which I encourage you to check out if you’re interested, as it’s the best source of this data I’ve found. (It’s free, you just need to give an email address.)

Bates looks at the first 425 sales (of the 442 total). Of those 425 sales, 20% (85) were declared a “homestead” by their buyers, which means a number of things under Massachusetts law, but relevantly for our purposes means they are claiming it as their principal residence. So that gives us a floor on the number of units where people live full-time. This is low relative to other comparable properties in Boston (the report cites 79% of the units at The Seville declared a homestead).

Now for rentals: as of this report, “more than 100” residences had found tenants. (I count 106 lines, but the report doesn’t make it clear that they correspond 1-to-1 with rentals, so I’ll take the more conservative number for now.) That’s another 23% of the units which have people likely living in them full-time.

So together this gives us a rough guess, that, as of March, about 43% of the units had people living in them full-time, and that that’s much less than other comparable, but more-established, properties in Boston.

The report makes a big deal of how fast the units sold (90% from October 2014 to September 2015, 11 months) relative to other comparable properties, so while it does seem that there’s a disproportionately high number of non-resident owners, some of what may be going on here is that it takes time to find tenants, and perhaps also the Boston market for luxury housing isn’t what the buyers may have hoped.

Those two units listed for $2500/mo are asking below what the lowest rent the report found circa March ($3500), so at least some buyers are in fact responding to market pressures and lowering rents to try to find tenants. Mr. Bates, if you read this, I’d love to read an update on Millennium Tower in a year or two’s time. I expect based on this trend line that more of Millennium Tower will have found residents.

There’s a perception among some activists who I respect that luxury developments are mostly unoccupied, and the owners are investors, likely foreign, buying condos like stocks and letting them sit empty, and so cities allowing developers to build luxury housing just makes housing prices go up, and it effectively takes housing away from the people who would otherwise have lived there if the developers had built a more reasonably-priced building.

There’s a question which I don’t have the information to answer on how developers decide how many units to build (would they have built 442 units if they had positioned the building at market rate?). There’s also an argument I don’t have time to pursue about whether it’s better for investors to spend $3 million on a condo in downtown Boston or $1 million each on three houses in Jamaica Plain or Somerville, but I’ll leave those for another time.

This certainly shows that at least this one new development isn’t doing as much to increase the supply of Boston housing as it might appear on paper. It’s not housing a full 442 units’ worth of people, at least not yet.

It is, however, housing at least 185 units’ worth of people, which is nothing to dismiss either.

Featured image: Millennium Tower- Nov 2015. By Jp16103Own work, CC BY-SA 3.0, Link.

Common Sense on Apple’s Recycled Hardware Reuse Policy

I hit some nerves on Twitter when commenting on this Motherboard article on Apple’s recycling policy by Jason Koebler a couple weeks ago, so I want to talk about my issues with the piece in a little less constrained format than Twitter provides.

The usual disclaimer: as always, although I talk about my experience at Akamai here, I don’t work for them any more, and I speak only for myself.

First I want to summarize that article, because I read it three times and still came away with the wrong impression, until I went away, returned the next day, and discovered what my mistake had been. (Judging by some of the commentary on Twitter, I wasn’t alone either. I personally find the article at best careless in how it repeatedly enables that misunderstanding, but the point of this post is not to argue its merits as a written work.)

The article talks about two separate but related programs Apple runs, and doesn’t do a good job distinguishing them. In one, Apple both buys back Apple-branded hardware from its customers for refurbishment and resale, and Apple accepts Apple-branded and other manufacturers’ hardware back through its stores and its mail-in program for recycling.

Two, on top of its Apple-branded recycling programs, many states have laws requiring electronics manufacturers like Apple to accept e-waste for recycling in proportion to the sales of their electronics in the state. Apple, being a major electronics retailer in the US, accepts a large tonnage of electronics for recycling. These collection efforts are not Apple-branded. When your school or office does an e-waste collection drive, the hardware may be sent through this program to e-waste management companies under contract to Apple for recycling.

It also appears from the article like Apple uses the same recycling companies for at least some of its Apple-branded waste stream and the state-mandated waste stream. (My guess is that there are relatively few major recycling companies capable of handling Apple’s volume, but I don’t really know.) Once the hardware arrives, it sounds like it doesn’t matter where it came from—it’s under the same Apple contract.

The article is very concerned about the stipulations of that Apple contract. Here’s Mr. Koebler quoting a Michigan state report:

“Materials are manually and mechanically disassembled and shredded into commodity-sized fractions of metals, plastics, and glass,” John Yeider, Apple’s recycling program manager, wrote under a heading called “Takeback Program Report” in a 2013 report to Michigan Department of Environmental Quality. “All hard drives are shredded in confetti-sized pieces. The pieces are then sorted into commodities grade materials. After sorting, the materials are sold and used for production stock in new products. No reuse. No parts harvesting. No resale.

(Emphasis mine.)

Remember, this applies to all the hardware Apple collects, both Apple products and other manufacturers’, under Apple-branded recycling programs and through third-party programs.

This is bad, he explains, because it’s much better for the environment to reuse and repair electronics rather than recycle them.

Kyle Wiens, the CEO of iFixit, notes that recycling “should be a last option” because unrecyclable rare earth metals are completely lost and melted down commodities are less valuable and of generally of a lower quality than freshly mined ones. Repair and reuse are much better ways to extend the value of the original mined materials.

Good so far. It quickly becomes obvious that there’s another motive at work, however, and of course that motive is money.

To be clear, Apple’s practices are often against the wishes of the recycling companies themselves, who don’t like to shred products that are still valuable. In a weird twist of fate, I visited ECS Refining before I knew that it did recycling for Apple. While I was there, I watched workers crowbar and crack open recent-model MacBook Pro Retinas—worth hundreds of dollars even when they’re completely broken—to be scrapped into their base materials.

At the time, I asked ECS CEO Jim Taggart how he feels about “must shred” agreements when he sees products that could have data safely deleted before being turned into parts or repaired. He called such deals an “extreme position,” one his company doesn’t like signing but is a core requirement from some manufacturers.

Now none of this is unreasonable—people who care about the environment (and make their living doing it) want to minimize waste, and recycling companies are in a cutthroat, commoditized business and want to maximize their returns. It’s a rare instance where the economic thing is also the environmental thing, even. All of that’s expected, and fine as far as it goes.

The trouble starts, though, when Mr. Koebler omits Mr. Yeider of Apple’s stated rationale for its policy, which is the sentence immediately following the part of that Michigan report he quotes that talks about “No reuse. No parts harvesting. No resale.” However, a partial scan of the report is thoughtfully included below the paragraph in question, and it says this:

This methodology preserves the chain of custody and assures the protection of data contained in the machines.

Now what the heck does that mean?

What it means is broadly that Apple sees itself to have a duty towards the data on devices you turn over to them, or recycle through third-party programs which wind up in Apple’s hands. Specifically, they believe have a duty to keep that data secret. And they’re right, they do.

Certainly for hardware Apple receives as part of their Apple-branded buy-back and recycling programs, they have a duty to maintain a chain of custody of it from the moment it leaves the customer’s hands in the store or enters the mail system until it has reached some safe state. In-store, having done this myself with an old phone, usually the store employees will walk you through wiping your personal data off device before handing it over, but mistakes happen, and there are no guarantees for hardware which has been mailed in. For hardware which I the consumer have turned over to Apple for recycling, it’s a serious black eye for them if it turns up on Ebay, even if all the data has been wiped—because what if it hadn’t been? The Fappening pales in comparison.

I’m less clear what chain of custody means in the context of hardware Apple receives through the state-sponsored recycling programs, but I presume there is some point at which that hardware enters the possession of Apple-contracted recyclers, and from then on the same argument applies as for hardware obtained through Apple’s branded programs.

Now the article does nod to this periodically but dismisses it as a minor issue, and one the recycling companies are obviously capable of handling.

But in practice, the premature recycling of an iPhone or a MacBook is not ideal. MacBook hard drives can be removed and replaced. And the recyclers Apple uses all advertise industry-standard data destruction tools that can be used to safeguard consumer data without requiring the destruction of all of the rest of the computer or phone’s parts.

There are a few complications with the picture this paints. One is that (to pick an example at random) the newest Macbook’s (solid-state) hard drive is in fact soldered to the mainboard, making it much harder to replace. It’s still technically removable, in the sense that any chip on a circuit board is removable, but it requires a lot more work than just pulling a daughter card by hand, as could be done with older Macbooks (including the Air, which surprised me). The onboard solid-state storage in phones has of course always been soldered down.

Solid-state drives are also much harder to sanitize than older spinning-platter hard drives. Due to the way they work, highly sensitive data can remain in them after it has been deleted, or even after the drive has been formatted. (Unlike spinning platter drives where, despite what the conventional wisdom says, a single-pass format operation is fine.) There are no software tools which can fully sanitize a solid-state drive once it has been used, so anyone considering whether to allow a device to be reused needs to take the risk that a sufficiently advanced adversary could recover some of their data into consideration.

However, the recycling industry has not fully caught up with this. My own encounter with “industry-standard data destruction” for solid-state drives while I was at Akamai did not fill me with any kind of confidence. So little confidence, in fact, that I hired an amazing intern and we successfully prototyped a better method (tl;dr yes they will blend). The NSA’s unclassified data sanitization standard (which our process meets) requires shredding to a 2mm grain size or smaller. I think the “confetti-sized” pieces the Michigan report describes (which I interpret as ~5mm grain) are plausibly sufficient sanitization against non-nation-state adversaries, which, let’s be honest, is who most of us are up against, most of the time.

I could go on a lot longer about SSD destruction (ask my friends! I’m great at parties!), but the long and the short of it is—if I’ve promised someone that I’m going to make the data on their SSD go away, no really, forever, nothing short of physical destruction is going to let me tell them honestly that the job is done and their data is never coming back to haunt them.

Unexamined in all this is the question of who should make the decision whether to allow the device to be reused or require it to be destroyed. The article thinks it should be the recyclers. The status quo, which Apple’s contracts with their recycling vendors enforce, is that the consumer makes that judgment, and I think that that is exactly as it should be. As a consumer, both of Apple products and other electronics, I can choose to sell my device back to Apple or to a third-party service like Gazelle for reuse, if I accept the (very small, but not zero) risk of some of my data being recovered. Or, I can send my device to a recycling program, and, if it goes to a recycler contracted by Apple, I can be assured that my data is really gone. In both cases, I and I alone get to decide how I want my hardware handled.

Given the existence proof provided by services like Gazelle, I don’t see that there’s anything stopping the recycling companies, or refurbishers like the man quoted in the article, from establishing their own brands for electronics reuse, if there’s enough money in it and enough environmental argument for it. They don’t need access to the recycling stream, and Apple is right not to let them have it.

That said, I fully support people choosing to give their hardware up for reuse rather than recycling, as I have myself done in the past. Despite the concerns I mentioned earlier about SSD sanitization, practical issues there are—so far—extremely rare. I think for most of us most of the time, using the device’s operating system function to erase it is sufficient to ensure our data won’t wind up in the wrong hands if the device is subsequently reused. But it currently is, and should continue to be, my choice whether the device gets reused or not.

 

 

Postscript: I want to acknowledge a couple arguments which I think are interesting but which I decided not to pursue in full detail here.

One is the question of whether it’s in the best interests of Apple, Apple’s users, and the hardware ecosystem as a whole for Apple to let its recycling vendors dump product in volume, and old product at that, into the currently quite healthy secondary market for Apple devices—or the markets for other manufacturers’ devices, for that matter—but I do want to note in passing that it is not at all clear to me that that is the case. I would love to see someone with more of an economics background than I have take this on.

Another related question I’m setting aside is whether it’s really in Apple’s users’ best interest to be encouraged towards old hardware which no longer receives software support, although the sketch of that argument is that old Apple hardware which no longer receives security updates is legitimately dangerous to its users and to the ecosystem at large, and it is a public health good for Apple to remove it from circulation.