I just sent an unintentionally blistering response to a Facebook recruiter. Having invested the time in writing it, I remembered that I have a very disused blog, and perhaps people reading here would find it useful, either as fodder for your own such messages, or as a snapshot of my concerns regarding Facebook and fascism in America in 2018. If either of these apply to you, enjoy.
…It’s not a Modest Proposal, because that was originally meant in satire, however it’s been corrupted in these latter and debased days, and I’m quite serious here.
(I am less of an expert on this than other things I blog about, although more knowledgeable than some; no warranty expressed or implied, &c.)
Today, basically all software comes with a blanket waiver of liability. The owners and coders of it do not express or imply any warranty, etc, blah blah blah, if it kills you, you pay your own funeral bills, and also you’re dead. And this leads us to situations where we have insulin pumps running consumer-grade software which hit its end of life four years ago at the ripe old age (for a piece of consumer-grade software) of fifteen.
The NSA hoarding vulnerabilities angle on this is a red herring, and I wish people would drop it. As nice as it would be for the US government to invest more than they do today in defense of software, there’s always going to be an interest in offense against software, and if it’s not the NSA’s vulnerability stockpile getting breached, it’s the Bad Guys’®, however we define them today. Even with some kind of MLAT for software vulnerabilities, the Bad Guys® do not sign or abide by those treaties, and unlike building nuclear weapons, exploiting software at this scale is still the province of bored and clever CS undergrads. We must proceed from the assumption that big tranches of vulnerabilities in our software and exploits for those vulnerabilities exist and might get exposed all at once.
There’s been much speculation, basically all of it (including this) ill-informed about the various legal frameworks already in place, both for software and other more established engineering products, about what effect some nebulously expressed change in the liability laws or case-law around software would have on the industry and practice of software production.
And my modest contribution is this:
There are many different lines within the software ecosystem you can draw, and probably we will need to draw all of them, but safety-critical versus consumer (and then industrial control, and god knows what else) are some important fucking distinctions.
I don’t know what about the economics of medical devices today causes them to be such a shitshow that this liability regime isn’t in place already, although I assume it’s much like the shitshow of other electronic devices (eg. Android phones), where it’s a commodity market without a way of valuing security, and integrators cobble together whatever shit they can to check the feature boxes the marketing and sales departments want and keep their customers buying new shit fast enough to keep the company from going bankrupt, but not fast enough to give them margins such that they can afford to build not-shitty medical devices, because it’s apparently unreasonable to expect that these companies and the people working for them should value not killing other people who have no choice but to submit themselves to the tender ministrations of the healthcare industrial complex.
Possibly a liability system for safety-critical devices would cause them to rethink their shitty life choices, and, more importantly, realign their market so that they could act on what the goodness in me compels me to assume is the non-shittiness within their hearts.
This anyway is my best explanation for the health insurance industry of today, who have for most of my life been rapacious bastards who will put you on the streets for pre-existing conditions including depression, which is only the natural state of all beings confronted with the enormity of the problem of evil in the world, and who are now championing not going back to the bad old days, because there is a bit of humanity left in their Grinch hearts after all. (And also regulation like this is actually better for business, but shh, don’t tell the capitalists that, it confuses and frightens them.)
Reliable sources (the guy who runs the certification company) inform me that we can do this for airplane avionics software, and it’s only (what I presume is) the lack of a (regulated-to-be-)level playing field in the medical device industry which makes this hard today, so it seems plausible that some medical, legal, and technical folks inspired by aviation and other safety-critical industries could sit down and create some proposed legislation which Congress could adopt with minimal editorial oversight which would result in a better medical device industry, fewer hospitals crippled by ransomware attacks, lower insurance premiums, and fewer fucking dead people.
It’s not like people aren’t working on this: (link to I Am The Cavalry) (link to Cyber-ITL) (link to Engineering a Safer World). Somehow this work hasn’t made the requisite impact yet, and maybe WannaCry will open people up to it, and maybe it won’t, but a mob of people with torches and pitchforks at their legislators’ offices asking “what are you doing about medical device cybersecurity” won’t hurt.
Because any sober and fundamentally good-hearted person can see that it’s past fucking time we fixed this.
Inspired-slash-frustrated by a conversation on Twitter a few weeks ago, which I am too tired now to find, I went and dug into just how many people actually live, more-or-less full-time, in Boston’s Millennium Tower. I was reminded of this by a conversation today, and, since the question seems to keep coming up, I might as well post my back-of-the-envelope analysis here.
A very light sketch of the background: Millennium Tower is a 60-story, 442-unit super-swank luxury condo development in the heart of downtown Boston; the tallest to date in the city, and by far the most expensive. This makes it a very visible target for the ire of folks who are rightly concerned about rising housing prices in Boston, with housing prices up 50% over just the last five years and rents quickly following.
It’s been widely reported that many of the buyers at Millennium Tower are non-local, even international, and so while there may be 442 units in the building, the question is, are there 442 units’ worth of people actually living there? (Millennium Tower replaced a Filene’s department store, so it’s not like houses or smaller apartments were torn down and people were put out of housing by its construction, but you can rightly ask the same question of developments where they were.)
First an existence proof: you can in fact find properties in Millennium Tower available for rent. I found these by searching Boston Craigslist for “Millennium Tower”: you’ll pay somewhere between $2500/mo (two units available at that price) and $4600/mo for an 800 sq. ft., 1-bedroom apartment. That $2500/mo figure is comparable to what I’ve heard for 1-bedrooms in Harvard and Porter Squares, which surprised me. Considering where and what Millennium Tower gets you, from a tenant perspective that $2500 is actually quite a steal, assuming it’s legit, which I have no reason to doubt. So that’s not just an existence proof but a really positive one. Rents are priced to move.
Now the question: just how many units in Millennium Tower are either owned or rented by people who live there full time? That’s a hard question to answer, so I’m going to answer the related and somewhat easier question of how many units are either owned by people as their principal residence, or rented (and I’m going to assume the rental tenants live there full time). That’s probably not entirely true, but I think it’ll let us start to put some bounds on our uncertainty.
To do that, I turn to the Bates Real Estate report on Millennium Tower from last month, which I encourage you to check out if you’re interested, as it’s the best source of this data I’ve found. (It’s free, you just need to give an email address.)
Bates looks at the first 425 sales (of the 442 total). Of those 425 sales, 20% (85) were declared a “homestead” by their buyers, which means a number of things under Massachusetts law, but relevantly for our purposes means they are claiming it as their principal residence. So that gives us a floor on the number of units where people live full-time. This is low relative to other comparable properties in Boston (the report cites 79% of the units at The Seville declared a homestead).
Now for rentals: as of this report, “more than 100” residences had found tenants. (I count 106 lines, but the report doesn’t make it clear that they correspond 1-to-1 with rentals, so I’ll take the more conservative number for now.) That’s another 23% of the units which have people likely living in them full-time.
So together this gives us a rough guess, that, as of March, about 43% of the units had people living in them full-time, and that that’s much less than other comparable, but more-established, properties in Boston.
The report makes a big deal of how fast the units sold (90% from October 2014 to September 2015, 11 months) relative to other comparable properties, so while it does seem that there’s a disproportionately high number of non-resident owners, some of what may be going on here is that it takes time to find tenants, and perhaps also the Boston market for luxury housing isn’t what the buyers may have hoped.
Those two units listed for $2500/mo are asking below what the lowest rent the report found circa March ($3500), so at least some buyers are in fact responding to market pressures and lowering rents to try to find tenants. Mr. Bates, if you read this, I’d love to read an update on Millennium Tower in a year or two’s time. I expect based on this trend line that more of Millennium Tower will have found residents.
There’s a perception among some activists who I respect that luxury developments are mostly unoccupied, and the owners are investors, likely foreign, buying condos like stocks and letting them sit empty, and so cities allowing developers to build luxury housing just makes housing prices go up, and it effectively takes housing away from the people who would otherwise have lived there if the developers had built a more reasonably-priced building.
There’s a question which I don’t have the information to answer on how developers decide how many units to build (would they have built 442 units if they had positioned the building at market rate?). There’s also an argument I don’t have time to pursue about whether it’s better for investors to spend $3 million on a condo in downtown Boston or $1 million each on three houses in Jamaica Plain or Somerville, but I’ll leave those for another time.
This certainly shows that at least this one new development isn’t doing as much to increase the supply of Boston housing as it might appear on paper. It’s not housing a full 442 units’ worth of people, at least not yet.
It is, however, housing at least 185 units’ worth of people, which is nothing to dismiss either.
I hit some nerves on Twitter when commenting on this Motherboard article on Apple’s recycling policy by Jason Koebler a couple weeks ago, so I want to talk about my issues with the piece in a little less constrained format than Twitter provides.
The usual disclaimer: as always, although I talk about my experience at Akamai here, I don’t work for them any more, and I speak only for myself.
First I want to summarize that article, because I read it three times and still came away with the wrong impression, until I went away, returned the next day, and discovered what my mistake had been. (Judging by some of the commentary on Twitter, I wasn’t alone either. I personally find the article at best careless in how it repeatedly enables that misunderstanding, but the point of this post is not to argue its merits as a written work.)
The article talks about two separate but related programs Apple runs, and doesn’t do a good job distinguishing them. In one, Apple both buys back Apple-branded hardware from its customers for refurbishment and resale, and Apple accepts Apple-branded and other manufacturers’ hardware back through its stores and its mail-in program for recycling.
Two, on top of its Apple-branded recycling programs, many states have laws requiring electronics manufacturers like Apple to accept e-waste for recycling in proportion to the sales of their electronics in the state. Apple, being a major electronics retailer in the US, accepts a large tonnage of electronics for recycling. These collection efforts are not Apple-branded. When your school or office does an e-waste collection drive, the hardware may be sent through this program to e-waste management companies under contract to Apple for recycling.
It also appears from the article like Apple uses the same recycling companies for at least some of its Apple-branded waste stream and the state-mandated waste stream. (My guess is that there are relatively few major recycling companies capable of handling Apple’s volume, but I don’t really know.) Once the hardware arrives, it sounds like it doesn’t matter where it came from—it’s under the same Apple contract.
The article is very concerned about the stipulations of that Apple contract. Here’s Mr. Koebler quoting a Michigan state report:
“Materials are manually and mechanically disassembled and shredded into commodity-sized fractions of metals, plastics, and glass,” John Yeider, Apple’s recycling program manager, wrote under a heading called “Takeback Program Report” in a 2013 report to Michigan Department of Environmental Quality. “All hard drives are shredded in confetti-sized pieces. The pieces are then sorted into commodities grade materials. After sorting, the materials are sold and used for production stock in new products. No reuse. No parts harvesting. No resale.“
Remember, this applies to all the hardware Apple collects, both Apple products and other manufacturers’, under Apple-branded recycling programs and through third-party programs.
This is bad, he explains, because it’s much better for the environment to reuse and repair electronics rather than recycle them.
Kyle Wiens, the CEO of iFixit, notes that recycling “should be a last option” because unrecyclable rare earth metals are completely lost and melted down commodities are less valuable and of generally of a lower quality than freshly mined ones. Repair and reuse are much better ways to extend the value of the original mined materials.
Good so far. It quickly becomes obvious that there’s another motive at work, however, and of course that motive is money.
To be clear, Apple’s practices are often against the wishes of the recycling companies themselves, who don’t like to shred products that are still valuable. In a weird twist of fate, I visited ECS Refining before I knew that it did recycling for Apple. While I was there, I watched workers crowbar and crack open recent-model MacBook Pro Retinas—worth hundreds of dollars even when they’re completely broken—to be scrapped into their base materials.
At the time, I asked ECS CEO Jim Taggart how he feels about “must shred” agreements when he sees products that could have data safely deleted before being turned into parts or repaired. He called such deals an “extreme position,” one his company doesn’t like signing but is a core requirement from some manufacturers.
Now none of this is unreasonable—people who care about the environment (and make their living doing it) want to minimize waste, and recycling companies are in a cutthroat, commoditized business and want to maximize their returns. It’s a rare instance where the economic thing is also the environmental thing, even. All of that’s expected, and fine as far as it goes.
The trouble starts, though, when Mr. Koebler omits Mr. Yeider of Apple’s stated rationale for its policy, which is the sentence immediately following the part of that Michigan report he quotes that talks about “No reuse. No parts harvesting. No resale.” However, a partial scan of the report is thoughtfully included below the paragraph in question, and it says this:
This methodology preserves the chain of custody and assures the protection of data contained in the machines.
Now what the heck does that mean?
What it means is broadly that Apple sees itself to have a duty towards the data on devices you turn over to them, or recycle through third-party programs which wind up in Apple’s hands. Specifically, they believe have a duty to keep that data secret. And they’re right, they do.
Certainly for hardware Apple receives as part of their Apple-branded buy-back and recycling programs, they have a duty to maintain a chain of custody of it from the moment it leaves the customer’s hands in the store or enters the mail system until it has reached some safe state. In-store, having done this myself with an old phone, usually the store employees will walk you through wiping your personal data off device before handing it over, but mistakes happen, and there are no guarantees for hardware which has been mailed in. For hardware which I the consumer have turned over to Apple for recycling, it’s a serious black eye for them if it turns up on Ebay, even if all the data has been wiped—because what if it hadn’t been? The Fappening pales in comparison.
I’m less clear what chain of custody means in the context of hardware Apple receives through the state-sponsored recycling programs, but I presume there is some point at which that hardware enters the possession of Apple-contracted recyclers, and from then on the same argument applies as for hardware obtained through Apple’s branded programs.
Now the article does nod to this periodically but dismisses it as a minor issue, and one the recycling companies are obviously capable of handling.
But in practice, the premature recycling of an iPhone or a MacBook is not ideal. MacBook hard drives can be removed and replaced. And the recyclers Apple uses all advertise industry-standard data destruction tools that can be used to safeguard consumer data without requiring the destruction of all of the rest of the computer or phone’s parts.
There are a few complications with the picture this paints. One is that (to pick an example at random) the newest Macbook’s (solid-state) hard drive is in fact soldered to the mainboard, making it much harder to replace. It’s still technically removable, in the sense that any chip on a circuit board is removable, but it requires a lot more work than just pulling a daughter card by hand, as could be done with older Macbooks (including the Air, which surprised me). The onboard solid-state storage in phones has of course always been soldered down.
Solid-state drives are also much harder to sanitize than older spinning-platter hard drives. Due to the way they work, highly sensitive data can remain in them after it has been deleted, or even after the drive has been formatted. (Unlike spinning platter drives where, despite what the conventional wisdom says, a single-pass format operation is fine.) There are no software tools which can fully sanitize a solid-state drive once it has been used, so anyone considering whether to allow a device to be reused needs to take the risk that a sufficiently advanced adversary could recover some of their data into consideration.
However, the recycling industry has not fully caught up with this. My own encounter with “industry-standard data destruction” for solid-state drives while I was at Akamai did not fill me with any kind of confidence. So little confidence, in fact, that I hired an amazing intern and we successfully prototyped a better method (tl;dr yes they will blend). The NSA’s unclassified data sanitization standard (which our process meets) requires shredding to a 2mm grain size or smaller. I think the “confetti-sized” pieces the Michigan report describes (which I interpret as ~5mm grain) are plausibly sufficient sanitization against non-nation-state adversaries, which, let’s be honest, is who most of us are up against, most of the time.
I could go on a lot longer about SSD destruction (ask my friends! I’m great at parties!), but the long and the short of it is—if I’ve promised someone that I’m going to make the data on their SSD go away, no really, forever, nothing short of physical destruction is going to let me tell them honestly that the job is done and their data is never coming back to haunt them.
Unexamined in all this is the question of who should make the decision whether to allow the device to be reused or require it to be destroyed. The article thinks it should be the recyclers. The status quo, which Apple’s contracts with their recycling vendors enforce, is that the consumer makes that judgment, and I think that that is exactly as it should be. As a consumer, both of Apple products and other electronics, I can choose to sell my device back to Apple or to a third-party service like Gazelle for reuse, if I accept the (very small, but not zero) risk of some of my data being recovered. Or, I can send my device to a recycling program, and, if it goes to a recycler contracted by Apple, I can be assured that my data is really gone. In both cases, I and I alone get to decide how I want my hardware handled.
Given the existence proof provided by services like Gazelle, I don’t see that there’s anything stopping the recycling companies, or refurbishers like the man quoted in the article, from establishing their own brands for electronics reuse, if there’s enough money in it and enough environmental argument for it. They don’t need access to the recycling stream, and Apple is right not to let them have it.
That said, I fully support people choosing to give their hardware up for reuse rather than recycling, as I have myself done in the past. Despite the concerns I mentioned earlier about SSD sanitization, practical issues there are—so far—extremely rare. I think for most of us most of the time, using the device’s operating system function to erase it is sufficient to ensure our data won’t wind up in the wrong hands if the device is subsequently reused. But it currently is, and should continue to be, my choice whether the device gets reused or not.
Postscript: I want to acknowledge a couple arguments which I think are interesting but which I decided not to pursue in full detail here.
One is the question of whether it’s in the best interests of Apple, Apple’s users, and the hardware ecosystem as a whole for Apple to let its recycling vendors dump product in volume, and old product at that, into the currently quite healthy secondary market for Apple devices—or the markets for other manufacturers’ devices, for that matter—but I do want to note in passing that it is not at all clear to me that that is the case. I would love to see someone with more of an economics background than I have take this on.
Another related question I’m setting aside is whether it’s really in Apple’s users’ best interest to be encouraged towards old hardware which no longer receives software support, although the sketch of that argument is that old Apple hardware which no longer receives security updates is legitimately dangerous to its users and to the ecosystem at large, and it is a public health good for Apple to remove it from circulation.
A story about the philosophy behind my VPN advice post, and a much deeper elaboration of my followup, for security people, public health people, and interested laypeople. Neither post is required reading for this one.
And a disclaimer, as always: I no longer work for Akamai, and I speak here only in my personal capacity. I don’t represent Akamai here in any way.
Most of the time I was at Akamai, I worked for Michael Stone, who has influenced my thinking about security and its place within the broader problem of safety a great deal. (To even understand and acknowledge that security exists within a broader context, I owe to him.)
One of Michael’s great strengths as a problem-solver, which he has worked consciously to develop, is that he has a large collection of what he calls “frames” or “lenses”—ways of thinking about problems, and other model problems to which to draw analogies—drawn both from within technology and outside it. In collecting these, he recognizes that often the first step to answering a question is to ask (or frame) it in the right way. His frames are “right ways” which have proven useful to others in the past.
Michael has a family background in public health, particularly from his mother’s work, but it was only slowly through my time working with him that we came to recognize that those ideas might have value to us. It was during the Heartbleed crisis, in April 2014, when I remember us first talking seriously about applying frames from public health to computer security issues. As that crisis wore on, it became clear just how effective they were at helping us understand and manage the complexity of the problems we were faced with.
Heartbleed affected some companies a great deal, and some companies not very much at all. For Akamai, which had tens of thousands of vulnerable servers, and thousands of potentially compromised certificates, Heartbleed was a significant crisis. Separating servers and certificates into vulnerable populations, considering various potential courses of action as interventions, and putting together triage plans were some of the ways that the frames helped us communicate with each other, our management, and our customers.
Five months later, the Shellshock crisis showed that the frame was also productive to apply not just inside the company but outside as well, to an Internet containing populations of servers running vulnerable services. We asked ourselves, what kinds of interventions could Akamai provide to protect our customers while they patched? Could that have a material effect on the health of the whole Internet? Would the benefits outweigh the side-effects? Since then, I increasingly approach all security advice through the frame of public health.
To caveat what follows: I’m not a public health professional, and I have only an interested layperson’s understanding of these concepts. If you are a public health professional, and I have misused terms or misapplied concepts, I would love correction, either privately or in the comments.
The reversion of Federal broadband privacy rules and the recommendation of VPN services, which I wrote about last week, is a model opportunity to apply the frame of public health to a security advice question. We have a population—broadband users in the United States. We have a threat, in the sale of their aggregate browsing information to marketing organizations by their ISPs. And we have a proposed intervention—using a VPN.
By framing the problem this way, I immediately cut out a great deal of complexity. No longer do I need to consider all possible users of VPNs, or all possible uses to which they might put them. Users have different priorities, and different uses dictate different priorities. Sometimes these priorities are even at cross purposes! The number of combinations is large, but I only need to consider the one. As a communicator, defining my audience well helps me communicate better with everyone, not least because those to whom I am speaking can easily recognize that fact, and those to whom I am not speaking can know that they are safe to ignore me.
Then we consider the nature of the particular threat facing our population of broadband users in the United States—the sale of their browsing information to marketing organizations and perhaps other, less savory, actors, by their ISPs. This is a concerning threat, although it must be weighed against the sale of similar information by ad networks and social media platforms, and will vary from ISP to ISP depending on what each chooses to stipulate in their particular terms of service.
And last we consider the proposed intervention, a VPN service: its costs, its potential benefits, and its potential side-effects. VPNs can be free, although as I’ve mentioned that is a red flag. Acceptably-good VPNs for our purposes need not be expensive, though; they are often made, after all, out of commodity VPSes and open source software, which can be had cheaply. And as to potential benefits, a properly configured open source VPN software package running on a commodity VPS will protect a vulnerable US residential broadband user from having their browsing data rolled up and sold by their ISP for marketing purposes.
As to potential side effects, though, as I laid out in my previous posts, we know that many VPNs do not provide the protections they claim to provide, and many are actively malicious. In that context, recommending a VPN is like recommending an entirely unlicensed drug in an unregulated market where it is known that many substances sold as that drug are of wildly varying purity and often laced with other, harmful substances. Some doctors may still agree to monitor individual patients’ use of such drugs, when they believe that their patients understand the risks, the doctors can make efforts to test the drugs for impurities before use, they can monitor the effects, and the drug’s benefits to the patient still outweigh the risks.
However, few public health officials, faced with anything less than life-or-death stakes, would recommend such a drug as an intervention by public policy. Both a public health official and a primary care doctor share an obligation to “first, do no harm,” and yet that applies very differently to a population than to an individual. (As an example of a case where public health officials did choose to suggest for individuals to do exactly what I outline above, consider this article on the use of PrEP in Great Britain. But even there, in a literally life-or-death situation, they did not recommend or require something they weren’t fully confident of.) And for essentially all US broadband users, having our browsing data aggregated by our ISPs and sold for marketing purposes is not a life-or-death proposition.
So this is why I do not recommend US residential broadband users use a VPN, as a matter of what we might call public health policy in computer security. And I think that, as computer security professionals, we will give better advice, which is more actionable to our users, if we at least think of security advice questions ourselves in this frame, and use it to structure our advice, and perhaps use terms drawn from public health when we speak in public about security advice questions as well.
It’s not that there isn’t an explicit user statement or threat model here—I lay out a very clear one. But this framing allows me to consider a broad group of users facing a narrow threat and so issue extremely specific and actionable advice. I might even feel safe giving this advice without prefacing it with the threat model, if I feel my advice follows the “first, do no harm” principle well enough, and people outside the target population will not be hurt if they follow it.
I’ve attended a few security trainings where the leaders tried to enumerate every possible combination of user and use case and give advice for each, and that resulted in what I found to be very confusing security advice. It also overrepresented the needs of the one person with challenging or interesting, often technical, needs, over those of the twenty people whose needs were less obviously so.
I have also frequently seen security people respond to every security question with “well, it depends on your threat model,” which is surely a reaction to bad blanket security advice given in the past. While it is strictly true as a statement, and is something I highly encourage for 1-on-1 conversations (as much to break the advice-givers out of our technical mindsets as anything), in group settings we can find some reasonable aggregates and make some reasonable assumptions based on context and perhaps a bit of research.
Also, using this public health frame encourages me to consider not just whether the proposed solution could work, but in what ways it might introduce new problems. This makes it clear to me, not just as an advice-giver, but as a developer of security solutions and a manager of such teams, that there is more to my work than whether the solution is narrowly effective. I must consider how much it costs—in money, in time, in my users’ emotional and cognitive energies (sometimes referred to as ‘spoons‘). I need to consider how it might break, how it might be subverted, how my users’ needs and habits might change as a result of its use.
When we started applying this frame to software security, even I, as a layperson, understood roughly that public health officials needed to do this work I’ve just outlined when considering their interventions. Michael convinced me that, not only did we need to do this work in computer security as well—and in practice I do not find it onerous—but the fact that we weren’t doing this work is why our interventions so regularly failed, and did so in ways which regularly hurt and sometimes killed people. And the key insight he had, from which all of this understanding sprung, was to frame the problem as a public health problem—or even to ask, “Could we?”
I encourage you to try this frame, the next time you have cause to provide security advice to a group of people. If you do, I would love to hear how well it applies for you: what works, what doesn’t, where you get stuck. And finally, I hope that this idea of frames spurs us to ask the next questions, about what else this frame can teach us and what other frames exist in the world which might apply to our work. Who else has in the past solved similar problems to the ones we face now?