“A day is coming, when, in the eye of the law, literary property will be as sacred as whisky, or any other of the necessaries of life.” —Mark Twain”
Inventors, programmers, writers, and musicians often get the same advice for the new Internet economy and the world of downloading and digital copying. The advice is to embrace new business models which give us exposure and access, maybe even fame or cryptocurrency in place of boring old money. We can sell labor like consulting services and live performances, T-shirts and books too, but we better give up hopes of obsolete and shameful licensing fees and royalties.
Here’s the foremost legal theorist of the Free Software Movement insisting that the collapse of copyright law will be good for musicians too ( from Liberation Musicology published in The Nation magazine in 2001).
“This increase in efficiency means that composers, songwriters and performers have everything to gain from making use of the system of unowned or anarchistic distribution, provided that each listener at the end of the chain still knows how to pay the artist and feels under some obligation to do so, or will buy something else–a concert ticket, a T-shirt, a poster–as a result of having received the music for free. Hundreds of potential “business models” remain to be explored … “
Back in 2000, when I started a software company, I was advised to give away software and sell services. The argument was that my company would be known as the experts on what we had invented and customers would pay a premium for our work. Each user in the chain would know how to pay the inventor and feel under an obligation to do so and we’d be profitably exploring a new business model. Well, it seemed plausible enough at the time if you were not too smart (raises hand). But it turns out that neither corporations nor consumers — the customers — are interested in this model.
Experience taught us that the business logic of corporate customers was not favorable to small company inventors who sold services. Our small business with a new product came with a certain level of risk that customers could reduce by asking some huge corporation to support the software instead of relying on a tiny startup. On the other hand, low wage consultants and competing venture funded startups could always under-price us particularly since we did all the work and bore all the cost of inventing and making the software. And to complete the picture, users in the chain who had money at stake were very quick to come up with convenient reasons why they did not have an obligation to pay us. Our ideas were, according to them, wrong, obvious and well known, not new at all. Our personalities were lacking in charm, our sense of style was off, our demands unreasonable. Any competent engineer, we were informed, could reproduce what we did over a weekend. In short, we didn’t deserve payment of any kind. I have heard very similar justifications for copying music in similarly resentful tones. People downloaded our code, rearranged it, and announced they had something better that owed nothing to our crappy inventions anyway. We were left with a very small bit of the market – unusual people who were willing to take a chance. We met some cool people, but it was no way to make a living .
Professor Moglen was correct, however, in the sense that this system was very efficient for the big companies that used our work. Instead of royalties and licensing fees, they only needed to pay for labor in a market where everyone was undercutting everyone else. Big companies with brands, marketing power, distribution, large legal teams, and lots of different ways to charge can often find all sorts of ways to wring money from products (“monetize”). In fact, the ways in which big companies like Amazon, IBM, and Google exploit free software both as users and contributors are fascinating for people who find innovative business models fascinating. Record companies can offer low royalty payments to Spotify and make it back on equity, non-license fees, and sheer volume ( as David Byrne explained). They can even mass produce T-shirts. RedHat and Oracle can sell free software by leveraging marketing, brand name, and risk minimization. RedHat’s Bob Young used to have a standard speech explaining both that he, Young, was dumber than his audience of programmers and that Red Hat’s business was like ketchup where Heinz could charge a premium because of brand alone. The first part was an outrageous lie well crafted to appeal to an audience of egotistical and naive programmers, but the second part was true enough. Even IBM used to give away software back when it was essentially the only computer hardware vendor in the world. The software helped them sell hardware and kept competing software vendors from taking too much of the revenue stream. For that matter, selling bottled tap water can be a very profitable business for Pepsi, but only because it has the distribution channels and marketing reach. In the end, though, all this clever monetization can leave artists and inventors who are not marketing wizards with a small share of the T-shirt revenue- if that.
Eventually we lost the faith, stopped releasing our code and started enforcing our intellectual property rights. Not everyone was happy. A high level corporate manager in a big company told me “we know about your technology and we understand its advantages but your patents make it awkward for us to use.” I asked what was awkward about paying a license fee, and he told me that his company was not about to pay license fees to some small time losers from nowhere. But here’s the important fact: Copyrights and patents, although they are often misused, can make it more awkward for big companies to shift revenue from the artists or inventor’s pocket to the corporate pocket in the course of some “agile efficient monetization of the marketplace”. Efficiency is one of those virtues that depends on where you sit. It’s certainly less efficient for a giant corporation to devote time and effort to licensing technology from small companies or paying ungrateful artists for their music than to simply make use of it. It’s expensive for a large firm to vet a product from a small company for utility and possible legal exposure and, at least initially, low cost to just download something for free.
On the other hand, it’s a lot more efficient for an inventor, programmer, writer, or musician to get a royalty check than to sell T-shirts. More lucrative, anyway.
There are downsides to depending on other people to do the work for nothing, but they are downstream ones. A manager or programmer at a tech company would have received a chilly response from her bosses a couple of years ago if she had explained that the company was depending on a minor component for logging, produced by some volunteer team, and that the corporation should go through all the effort needed to onboard that tiny team as a vendor and purchase and vet that software even though it could be downloaded for free. Years later, it turned out the software in question opened a major security hole in corporate firewalls and nobody who decided to download it for free was fired.
Property versus intellectual property
But is any of this morally correct? For many people software patents are bad, in fact, intellectual property itself is bad. One complaint involves the supposedly dubious status of intellectual property as property. The argument goes something like this:
“the owner of a digital recording or program doesn’t lose anything when a copy is made, unlike the owner of a piano or a bicycle who no longer has physical possession if ownership is transferred. You still have a recording of your movie or musical performance or program even if someone is letting people download copies for nothing. See – it’s not actual property.”
But even with physical objects, ownership and transfer is not really that simple. Suppose my family and I move into your house without your permission. You still “have” the house – the physical object is still there. If there is sufficient space or if it’s a rental house, you may not even be inconvenienced. What you’ve lost is something far more abstract than a physical object. You lost a right to control use or to demand payment for use – exactly what the musician lost when someone pirated her song. And what about intangible property? Bank accounts, loans and other financial assets are intangible pseudo-objects that have less physical presence than a bunch of bits encoding a song or program. If your financial adviser embezzles your retirement savings to bet on the dog track or your bank invents bogus charges that they deduct from your account, no physical object passes out of your possession but you’d most likely consider yourself deprived of something. What you were deprived of is a right to some money, itself a strange non-material abstraction. Property rights are all about fictional entities, pseudo objects, rights to forbid others to do something and so on – there is nothing specially unmaterial or spectral about intellectual property compared to other kinds of property.
We also heard that some kinds of IP are more worthwhile than others. It’s not especially surprising, I suppose, that these theories often seem to align with the financial interests of the person making the argument (me included, of course). For example, in Germany software patents are widely derided, but patents on pharmaceuticals (controversial elsewhere) are not. Germany is a net payer on one of those kinds of patents and a net payee on the other kind. Guess which is which. Similarly, I’ve had interesting discussions about software intellectual property with people who work for computer hardware companies and sincerely believe software intellectual property to be dubious at best. For them, somehow, a design expressed in a hardware programming language which is then converted into a recipe for computer chips can be worthy of protection, but a design expressed in a different programming language that is then converted into a process for controlling the machinery that makes those same chips is different in kind. The most entertaining argument I saw, however, was from an enthusiastic supporter of free software who wrote an indignant email to me from his company account, lambasting my use of software patents. When I pointed out that his corporate employer owned a lot more software patents than we did, he explained that his employer was a commercial enterprise and my business was not. Musicians seem to often get these kinds of lectures too.
The internet changes everything
People also argue that the nature of the internet and of digital representations of music and programs and movies makes that old-fashioned copyright and patent stuff obsolete. Here’s Professor Moglen again:
“our world consists increasingly of nothing but large numbers (also known as bitstreams), and that – for reasons having nothing to do with emergent properties of the numbers themselves – the legal system is presently committed to treating similar numbers radically differently. No one can tell, simply by looking at a number that is 100 million digits long, whether that number is subject to patent, copyright, or trade secret protection, or indeed whether it is “owned” by anyone at all. “
No one can tell, simply by looking at it, whether a particular block of land or house or apartment is “owned” by one private individual or another, though. Does simply looking at an automobile or a pile of I-beams or a basket of mangos, produce information on whether that object is owned, or leased, or stolen? There is nothing fundamental about a number that tells you whether it encodes a program or musical composition protected by copyright, or a method protected by patent, but there’s nothing “emergent” about a vat of chemicals or a ball point pen that tells you whether it embodies a patented chemical process or mechanical design. Ownership is not a physical property of an object or pseudo-object, ownership comes out of social relationships of human beings. Moglen continues:
” Like everything else in the digital world, music as seen by a CD player is mere numeric information; a particular recording of Beethoven’s Ninth Symphony recorded by Arturo Toscanini and the NBC Symphony Orchestra and Chorale is (to drop a few insignificant digits) 76749873424, while Glenn Gould’s peculiarly perverse last recording of the Goldberg Variations is (similarly rather truncated) 767459083268. Oddly enough, these two numbers are “copyrighted.” This means, supposedly, that you can’t possess another copy of these numbers, once fixed in any physical form, unless you have licensed them. And you can’t turn 767459083268 into 2347895697 for your friends (thus correcting Gould’s ridiculous judgment about tempi) without making a “derivative work,” for which a license is necessary.”
It’s illuminating to transfer this argument back to an older technology.
“Like everything else in the printed world, novels are mere sequences of letters; a particular printing of Fledgling might contain “When your rage is choking you, it is best to say nothing.”, while “the banana company did not have, never had had, and never would have any workers in its service” is from One Hundred Years of Solitude. You can’t fix Octavia Butler’s anger or Garcia-Marquez’s antipathy to banana companies and make other copies of these sequences of letters for your friends unless you have licensed them.”
We had copyrights for written works long before there were those electronic computers and pesky binary numbers that are supposedly eroding the whole basis of copyright law. And, by the way, that line about “correcting” Gould hit home for me. If you think you can play the piano better than Glenn Gould, go ahead. If you don’t like his work, don’t buy it. If you want to be “creative” with it and distribute it, pay a license fee to his heirs. The concept of “derivative work” in intellectual property law is there because people were publishing “improved” versions of Mark Twain’s work too. Intellectual property is important primarily because of the money, but it’s also important because it allows the people who create the product to exercise some control of how it is used. Neil Young was able to prevent Spotify from using his music to promote disinformation because he owned the rights. For our real-time software, one question I used to ask free software advocates was to explain why it would be immoral for us to forbid use of the software to control land mines.
Free software, public domain software, creative commons licenses – all of those can be good things. Irving Wladawsky-Berger’s recognition of how free software could help IBM’s business was pretty much brilliant although one could easily come up with more cynical formulations of his explanation. I even agree with those who say music and inventions are never wholly original, but build on the accomplishments of others. Of course they are. None of that justifies compulsory pre-emptive “freeing” of creative works or inventions by consumers of that work who just feel like they know better or, more likely, that their corporate employer would find it easier. Commerce is all about incentives and the incentive of a corporate lawyer or manager to insist that the company should pay an individual inventor or small business because it is the right thing to do is not all that high, believe it or not. Same with the incentive of a consumer to buy a song or movie that can be downloaded for free. And when some companies find it more profitable to use your work without paying, that makes it even more difficult for people at other companies to argue in favor of cutting you a big check. If all your friends stream new music from a company that systematically underpays artists, you tend to look like a chump for paying more. The same peer pressure operates in corporate purchasing. A lot of people actually want to do the right thing, but very few will buck the trend to do so.
And what’s the right thing?
“To promote the Progress of Science and useful Arts, by securing for limited Times to Authors and Inventors the exclusive Right to their respective Writings and Discoveries.”
The legal system of intellectual property is far from perfect. Actually, it’s scarily imperfect. Several times only dumb luck saved us being ground up between the vast resources of big companies and the vast expense of the legal system. But I was able to help create interesting technology, and get paid for it, and stay independent, because the much maligned intellectual property laws do something to protect inventors, musicians, artists, film makers and others who don’t want to work for a giant multinational, but do want to make things.
A free software advocate once told me that Mozart didn’t need copyrights. Years later I read Mozart’s letters and was appalled at how much time he had to spend groveling to wealthy sponsors. And maybe those of us who are less talented than Mozart but also create works that other people want to use or listen to or read also should get paid by labor right, not because someone with money wants to be a sponsor. So when next you hear about terrible Disney and their apparently infinite copyrights on Mickey, or about some patent troll demanding to be paid, or some musician suing a streaming service, think also about those of us who just want to be paid when someone uses our writings, music, creations, and discoveries. Proudhon may have been right and property may be theft, but in a society based on thievery, it is quite convenient to have some property.
YOUR GRACE, MOST WORTHY PRINCE OF THE HOLY ROMAN EMPIRE!
I will not presume to trouble Your Grace with a full description of our unhappy circumstances, which my father has set forth most accurately in his very humble petition which was handed to you on March 14th, 1777. As, however, your most gracious decision was never conveyed to him, my father intended last June once more most respectfully to beg Your Grace to allow us to travel for a few months in order to enable us to make some money; and he would have done so, if you had not given orders that in view of the imminent visit of His Majesty the Emperor your orchestra should practise various works with a view to their performance. Later my father again applied for leave of absence, which Your Grace refused to grant, though you permitted me, who am in any case only a half-time servant, to travel alone. Our situation is pressing and my father has therefore decided to let me go alone. But to this course also Your Grace has been pleased to raise certain objections. Most Gracious Prince and Lord! Parents endeavour to place their children in a position to earn their own bread; and in this they follow alike their own interest and that of the State. The greater the talents which children have received from God, the more are they bound to use them for the improvement of their own and their parents’ circumstances, so that they may at the same time assist them and take thought for their own future progress. The Gospel teaches us to use our talents in this way. My conscience tells me that I owe it to God to be grateful to my father, who has spent his time unwearyingly upon my education, so that I may lighten his burden, look after myself and later on be able to support my sister. For I should be sorry to think that she should have spent so many hours at the harpsichord and not be able to make good use of her training.
Your Grace will therefore be so good as to allow me to ask you most humbly for my discharge, of which I should like to take advantage before the autumn, so that I may not be obliged to face the bad weather of the ensuing months of winter. Your Grace will not misunderstand this petition, seeing that when I asked you for to travel to Vienna three graciously declared that I had nothing to hope for in Salzburg and would do better to seek my fortune elsewhere. I thank Your Grace for all the favours I have received from you and, in the hope of being able to serve you later on with greater success, I am
The developers of Google’s Spanner database built an interesting system but chose to build their own clock synchronization technology instead of getting a more precise off the shelf solution and chose to make extensive use of the notoriously complex Paxos distributed consensus algorithm even though they were incorporating clock sync into their system. Whatever the rationale at Google, building a clock sync enabled high speed distributed transaction system is much less complex if you don’t make those choices.
Off the shelf technology that can synchronize clocks to under a microsecond in high end systems and can even produce tight clock sync in the decidedly not optimal situation of cloud virtual machines is readily available (www.fsmlabs.com). And with clock sync, it is possible to enormously simplify distributed consensus. In fact, it’s hard to understand why Paxos is so widely used. Paxos is a highly complex (ingeniously complex) and painful to implement attempt to solve a problem that doesn’t exist.
The humorously titled “Paxos made simple” paper starts off discussing the problem of distributed consensus in completely asynchronous networks where timeouts won’t work and clocks are not synchronized at all. In practice, computer networks are not totally asynchronous, they do permit precise clock synchronization (and that technology has gotten much better since Paxos was introduced in 1998), and Paxos doesn’t even work in a totally asynchronous network. Page 7 of “Paxos made simple” points out that the algorithm can fail to converge on a solution, it is not “live”. Lamport, the author, then suggests electing a single coordinator or primary agent – using timeouts! Once you admit that timeouts work, you can replace Paxos with far simpler algorithms, some well known in database systems since the 1970s. Precise clock sync permits much more radical simplifications.
(recently saw a post claiming that Damore’s firing was an indication of closed minds at Google, so it seemed important to repost this – vy)
The Google manifesto is a crackpot political argument combined with basic scientific ignorance, not a scientific argument. The people claiming otherwise are either lying or terrible at reading. Damore’s essay is packed full of paraphrases of Jordan Peterson’s body of work, right wing political slogans, and resentment against women. Damore claims that any program to encourage hiring and promotion of women and minorities constitutes discrimination against white men. According to him a company mentoring program to encourage women programmers would be both wrong and illegal. He says that there is no gender wage gap and women get lower wages because they don’t want to work as hard or take risks like men do. He claims that feminism is a ploy by “Marxist intellectuals” to replace their old class struggle pitch. He says that Google management are left-wing extremists, motivated by a “veiled left ideology”, whose diversity initiatives are “encroaching extremist and authoritarian policies”. He says that he is in favor of diversity, but against actually doing anything to encourage diversity because:
Discriminating just to increase the representation of women in tech is as misguided and biased as mandating increases for women’s representation in the homeless, work-related and violent deaths, prisons, and school dropouts.”
Damore’s deplorable basket of crackpot resentments is not science. Period. The defenses of Damore by people like University of Texas Computer Science Professor Scott Aaronson and UW’s Stuart Reges and Peter Singer are either grossly dishonest or complete failures of scholarship. Aaronson writes to say that all Damore was doing was ” mooting the hypothesis of an evolutionary reason for average differences in cognitive styles between men and women,” and that “the basic thesis being discussed […] that natural selection doesn’t stop at the neck, and that it’s perfectly plausible that it acted differently on women and men in ways that might help explain many of the population-level differences that we see today“. He laments the “clickbait narrative” of Damore’s critics (although he doesn’t cite or quote any of them because that would ruin his baseless temper tantrum). But Damore is not saying anything is “perfectly plausible”. He is insisting that Google should abandon any attempt to hire or advance women or minorities. Stuart Reges wrings his hands in anguish:
For the last ten months I have been discussing this issue at the Allen School of Computer Science & Engineering where I work. I have tried to understand why Damore’s opinions generated such anger and have struggled to decide what I want to do in response.
He tried, so sincerely, to understand why people would get angry reading stuff like:
Yes, in a national aggregate, women have lower salaries than men for a variety of reasons. For the same work though, women get paid just as much as men. Considering women spend more money than men and that salary represents how much the employee sacrifices (e.g. more hours, stress, and danger), we really need to rethink our stereotypes around power. – Damore
Really Stuart? Ten months of sincere effort and you can’t figure out what about Damore’s memo would make people mad?
As it became clear that the working class of the liberal democracies wasn’t going to overthrow their “capitalist oppressors,” the Marxist intellectuals transitioned from class warfare to gender and race politics. The core oppressor-oppressed dynamics remained, but now the oppressor is the “white, straight, cis-gendered patriarchy.” – Damore
Although Aaronson won’t cite any of the critics of Damore, he lists a number of “courageous” defenders. For example, here is Peter Singer with more of this nonsense:
James Damore, a software engineer at Google, wrote a memo in which he argued that there are differences between men and women that may explain, in part, why there are fewer women than men in his field of work. For this, Google fired him.
Reges has a similar claim:
In this case, suggesting that men and women are different either in interests or abilities is considered blasphemy. – Reges
Obviously what Reges, Singer, and Aaronson wrote is not true. Insisting that there is no wage gap or that any diversity program at all constitutes discrimination against white guys or that Google management is on some left-wing crusade, is not evolutionary biology and it’s not arguing that “there are differences between men and women” – a totally uncontroversial claim. It’s not acceptable for these academics to misrepresent Damore’s resentful tract as some musings about evolutionary biology.
But what about all the scientific citations in Damore’s memo? Giving those all benefit of the doubt, if Aaronson and Singer and all the other handwringing apologists had tried to characterize Damore’s memo more more honestly as, say: “a combination of a political rant and an argument about evolutionary biology”, all of their indignant claims to be defending free scientific inquiry would have been a lot harder. They would have had to let go of the strawman and come up with arguments less dependent on emotion and name-calling. Here’s Aaronson
Let that be the measure of just how terrifyingly efficient the social-media outrage machine has become at twisting its victims’ words to fit a clickbait narrative—a phenomenon with which I happen to be personally acquainted. Strikingly, it seems not to make the slightest difference if (as in this case) the original source text is easily available to everyone.
This is to defend someone who wrote that one problem with Google’s efforts to hire women is that:
“We can make software engineering more people-oriented with pair programming and more collaboration. Unfortunately, there may be limits to how people-oriented certain roles at Google can be and we shouldn’t deceive ourselves or students into thinking otherwise (some of our programs to get female students into coding might be doing this).”
Or here’s Singer:
So on an issue that matters, Damore put forward a view that has reasonable scientific support, and on which it is important to know what the facts are.
Really? There’s “reasonable scientific support for “For the same work though, women get paid just as much as men”? No there is not. How about “political correctness is a phenomenon of the Left and a tool of authoritarians.” Some nice double blind studies on that one? What about:
I’ve heard several calls for increased empathy on diversity issues. While I strongly support trying to understand how and why people think the way they do, relying on affective empathy—feeling another’s pain—causes us to focus on anecdotes, favor individuals similar to us, and harbor other irrational and dangerous biases. Being emotionally unengaged helps us better reason about the facts.
Reasonable scientific support for that? Reges wants pity
When I tried to discuss Damore at my school, I found it almost impossible. As a thought experiment, I asked how we could make someone like Damore feel welcome in our community. The pushback was intense. My question was labeled an “inflammatory example” and my comments were described as “hurtful” to women. – Reges.
No kidding. The head of undergraduate instruction asks how to make a misogynist feel welcome in the community and people push back. Shocking. The entire basis of Reges, Aaronson’s and Singer’s defenses of Damore depends on whitewashing away all the ideology in his essay, taking his variant of “I’m not a racist but ... ” at face value, and then pretending that his wacky “biology” is respectable. Only if you do all that can you denounce the irrational anti-scientific anti-free speech mob from your claimed position as an Olympian Scientist and Defender of Free Speech. How original: to defend a sexist rant by claiming its critics are hysterical.
“Biological males that were castrated at birth and raised as females often still identify and act like males – James Damore”
Wow, imagine not being struck by how weird that is. There is a biological argument in Damore’s essay, but it’s ridiculous. Nobody argues that there are no biological differences between men and women. Nobody. Damore comes up with: “Differences in distributions of traits between men and women may in part explain why we don’t have 50% representation of women in tech and leadership.” Well, who said that those differences definitely did not explain any differences in representation? Nobody, that’s who. Google engineering is 19% female. Google’s management is 22% female. The issue is not why Google doesn’t have 50% representation, but why it has 19% representation.
Damore cites a number of studies that show or claim to show that certain traits are weakly correlated with gender. The gender based differences in those studies are in single digits so even if we accepted them as authoritative (they are not) they would not begin to explain the gender differences at Google. More striking, not one of those studies concerns programming ability or performance. When people claim Damore’s cited studies represent the scientific consensus (which is not true) they leave out that those studies have absolutely nothing to say about either programmers or diversity programs. Let me just stress that, because Damore is repeatedly claimed to have been relying on science: Not a single one of the studies that Damore cites claims to show a relationship between gender and ability to code.
Not one. Traits like which toys rhesus monkeys pick by gender may be related to human performance writing Java code but there is no evidence of that and it’s pretty implausible. Why being “more thing oriented” would make men more likely to be managers of people at Google or more interested in distributed consensus algorithms is also not addressed. But even if we were as naive as Professors Aaronson, Reges, and Singer, and accepted this collection of studies as meaning what Damore claims they mean, it takes a stupendous level of mathematical ignorance to agree that such studies would explain Google’s gender imbalance.
There are 52 cards in a deck of cards, so a blind pick from a shuffled deck produces an ace with probability 4/52. Yet, I can guarantee you I can pick 4 aces in a row out of the deck every time if I look at the faces of the cards. Google and other companies don’t hire based on random sampling from the population. They recruit, interview and test and, because they are a highly profitable company, they can pay more and offer better benefits to attract employees they want. They look at the cards. If they want aces, they can get them. Most of the programmers in the world are average or worse. Does Google have to accept that half of its programmers will be below average? This is obvious, but in a rush to take Mr. Damore’s rant as a serious contribution, Damore’s defenders, who claim to be speaking the name of science, apparently can’t do basic statistical reasoning.
The really interesting question is why this document has any defenders at all and, particularly, why all the concerned hand wringing from academics.
“Civilization’s going to pieces,” broke out Tom violently. “I’ve gotten to be a terrible pessimist about things. Have you read ‘The Rise of the Coloured Empires’ by this man Goddard?”
“Why, no,” I answered, rather surprised by his tone.
“Well, it’s a fine book, and everybody ought to read it. The idea is if we don’t look out the white race will be–will be utterly submerged. It’s all scientific stuff; it’s been proved.”
“Tom’s getting very profound,” said Daisy with an expression of unthoughtful sadness. “He reads deep books with long words in them. What was that word we—-“
“Well, these books are all scientific,” insisted Tom, glancing at her impatiently. “This fellow has worked out the whole thing. It’s up to us who are the dominant race to watch out or these other races will have control of things.”
“We’ve got to beat them down,” whispered Daisy, winking ferociously toward the fervent sun. – F. Scott Fitzgerald, Great Gatsby.
(h/t to vp at Lobste.rs for reminding me of this )
The first time I saw this method was when I went to work for Parallel Computer Systems, , later called Auragen, in the famous tech startup center of Englewood Cliffs, New Jersey. I commuted there from the East Village*. The company built a parallel computer based on Motorola 68000s with a replicated message bus. The bus guaranteed message delivery to 3 destinations would either succeed to all three or fail to all three. This property is called “reliable broadcast”. All interprocess communication was by message transfer (a fashionable idea at the time). Each process had a backup. Whenever a primary process sent a message, the message was also delivered to the backup and to the destination backup. If the primary failed, the backup could be run. The backup would have a queue of messages received by the primary and a count of messages sent by the primary. When the recovering backup tried to transmit a message, if the count was greater than zero, the count would be decremented and the message discarded because it has already been transmitted by the old primary. When the recovering secondary did a receive operation, if there was a message on the input queue, it would get that message. In this way, the recovering backup would repeat the operations of the primary until it caught up. As an optimization, the primary could be periodically checkpointed and queues of duplicated messages could be discarded.
The operating system was an implementation of UNIX. In practice, it was discovered that making each UNIX system call into a message exchange, which was an idea advocated in the OS research community at the time, caused serious performance problems. The replicated state machine operation depended on this design in order to make the state machine operation deterministic. Suppose the primary requested, for example, the time and then made a decision based on the time. A recovering secondary would need exactly the same time to guarantee that it produced the same results as the primary. So every interaction between application and OS needed to be recorded in a message exchange. But a message exchange is nowhere near as fast as a system call (unless the OS developers are horrible).
The performance issue was mitigated by some clever engineering, but was a problem that was discovered in parallel by a number of development teams working on distributed OS designs and micro-kernels which were in vogue at the time. Execution of “ls -l” was particularly interesting.
Anyways, here’s the description from the patent.
To accomplish this object, the invention contemplates that instead of keeping the backup or secondary task exactly up to date, the backup is kept nearly up to date but is provided with all information necessary to bring itself up to the state of the primary task should there by a failure of the primary task. The inventive concept is based on the notion that if two tasks start out in identical states and are given identical input information, they will perform identically.
In particular, all inputs to a process running on a system according to the invention are provided via messages. Therefore, all messages sent to the primary task must be made available to the secondary or backup task so that upon failure of the primary task the secondary task catches up by recomputing based on the messages. In essence, then, this is accomplished by allowing every backup task to “listen in on” its primary’s message.
United States Patent 4,590,554 Glazer , et al.May 20, 1986
Inventors:
Glazer; Sam D. (New York, NY), Baumbach; James (Brooklyn, NY), Borg; Anita (New York, NY), Wittels; Emanuel (Englewood Cliffs, NJ)
*True story: I applied for the job after finding an advert in a discarded copy of the NY Times on the floor of a Brooklyn apartment while visiting friends. I sent via US mail a resume poorly typed on a manual typewriter- I’m tempted to say “composed by the light of a tallow candle” but that would be over the top- and forgot to enclose the second page.
This article on the interesting Wave Transactional File System inspired me to look up an earlier file system that also used copy on write semantics.
From:
Anita Borg, Wolfgang Blau, Wolfgang Graetsch, Ferdinand Herrmann, and Wolfgang Oberle. 1989. Fault tolerance under UNIX. ACM Trans. Comput. Syst. 7, 1 (January 1989), 1-24. DOI=http://dx.doi.org/10.1145/58564.58565
4.3 Availability of the File System
Since a recovering file server reconstructs its buffers by reading blocks from the file system, the file system in the state as of the last sync must be available. The existence of that version of the file system is also necessary during recovery as the file server redoes requests. For example, if a file has been deleted since sync and a read request is reissued, the disk driver, and thus the recovering file server, will behave differently than the primary. Unfortunately, the contents of the disk can change between syncs, at least during the Fsync that constitutes the first phase of the sync operation.
The solution is to use a copy-on-write strategy between syncs, rather than overwriting existing blocks. Logically this corresponds to keeping two versions of a file system.3 An early version of the file system organization described here is discussed in Arnow [ 11].
There are two root nodes on disk. At any given time one of them is valid for recovery. We refer to the other as the alternate root. Associated with each root is state information (the state tables described above), the most recent being that associated with the currently valid root. Changes to the file system are done relative to a copy of the valid root kept in memory in the primary file server’s address space, and in a nondestructive manner, as seen in Figure 2(a-d). Freed blocks, which contain the old data, are added to a semi-free list, and cannot be reallocated until after the next sync. Therefore, the unmodified file system still exists rooted in the valid on-disk root node.
If a crash occurs at any time between syncs, the recovering file server is able to determine which root to use because of information sent on the primary’s last sync. It reads in the correct state information and reconstructs its buffers accordingly. Disk blocks that were used by the primary since the last sync appear to it as free blocks.
The difficult case is when a crash occurs during a sync. To see that the solution works in this case, consider the sequence of actions that take place during a sync. First, all dirty blocks except the root are written to disk, and old blocks are added to the semi-free list. Second, the state information is collected and written to the alternate state area. Third, the in-memory root is written to the alternate on disk root block, Finally, the sync message is constructed and sent to the backup. It contains the information necessary to update message queues as well as specifying which on-disk state information and root block to use on recovery.
Once the sync message has been sent, the semi-free list is added to the free list and the primary continues. Just before the sync message is sent, there are two copies of every modified data and indirect block. At any time before the sync message is sent, the old consistent state is available. Any time after it is sent, the new state and file system will be used and message queues consistently updated. An additional benefit of this organization is that the file system as a whole is considerably more robust than a standard UNIXstyle file system. Even if the entire system is shut down in an uncontrolled way as the result of multiple faults or operator error, there will always be an entire consistent file system on disk.
Come back after a trip to see Marc Andreessen’s team of twitter posters complaining wryly about how government is so gosh durn big and citing experts like Milton Friedman.
A difficulty of modern economics: I have yet to talk to a non-Keynesian who believes Keynesians intend to cut fiscal spending post-recovery.
That is, Marc Andreesen, the former National Center for Supercomputing Applications programmer who helped write the Mosaic Web Browser for the WWW that CERN scientists developed on top of the Internet technology that DARPA and NSF researchers and bureaucrats created on top of prior government funded technology like integrated circuits, and then built a company that made him rich with the team that worked together at NCSA, engaging in the typical rich person bitching about the gummint. Which shows how little reality matters to people.
This is state of the art for systems software now – which is not all that impressive.
Glantz explained that Ikea has more than 3,500 Red Hat Enterprise Linux (RHEL) servers deployed in Sweden and around the world. With Shellshock, every single one of those servers needed to be patched and updated to limit the risk of exploitation. So how did Ikea patch all those servers? Glantz showed a simple one-line Linux command and then jokingly walked away from the podium stating “That’s it, thanks for coming,” as the audience erupted into boisterous applause. On a more serious note, Glantz said that it took approximately 2.5 hours to test, deploy and upgrade Ikea’s entire IT infrastructure to defend against Shellshock. Eweek.
GT Advanced declared bankruptcy and blamed Apple for its problems. Apple called GT Advanced’s story “defamatory”. I have no idea about the specifics in this case but I do know about big companies pushing insanely onerous and self-defeating terms on small ones. Here’s the original claim by GT Advanced:
At the start of negotiations, Apple offered to buy 2,600 sapphire growing furnaces from GT Advanced, which GT Advanced would operate on behalf of Apple, the “ultimate technology client to land,” according to Squiller.
“In hindsight, it is unclear whether Apple even intended to purchase any sapphire furnaces from GTAT,” he wrote.
But after months of hard negotiating, Apple offered a deal under which it would shift away economic risk by lending GT Advanced the money to build the furnaces and grow the sapphire, and then sell it exclusively to Apple for less than market value, Squiller wrote.
GT Advanced was effectively forced to accept the unfair deal in October 2013 because its intense negotiations with Apple had left it unable to pursue deals with other smartphone makers, he said.
Back when we were selling a real-time OS, I contacted an ex-boss who now had a high position in a telecommunications company to see if he could help us sell into it. His response was “don’ t touch this place, it is expert in destroying small vendors.” And, whatever the actual story with GT and Apple, the storyline is not at all unusual. The elements of a smaller company spellbound by prospects of a huge deal/giant customer, followed by time consuming negotiations, followed by onerous demands – been there, done that. We once were negotiating with a huge semiconductor company about a big deal that, over time, got worse and worse for us. We “finalized” with some terms that we thought might be survivable. And then the semiconductor company negotiators, one of whom by this point our negotiators were privately calling “Beelzebub”, announced they had to take the deal “to management” and came back with much more absurd demands. We were able to walk away but we saw other small companies make deals with Beelzebub and then fail. There is a strong impulse in some big companies, among some business units, to squeeze small company vendors way too far.
Once, after a sales visit to a big Wall Street Firm, two experienced sales people for a second supplier told me they were shocked by what I had said to the customer and advised me never to make the same error. What was my mistake? I had told the customer that we had produced a product that worked a lot better than what they were currently using, cost them a lot less, and was highly profitable to us. “Never do that”, counseled my colleagues, “they want to believe that, at best, you are breaking even on the deal, otherwise they think they left money on the table.”
The absurdity of these kinds of “negotiations” is that they are highly unprofitable for big companies. The potential savings are usually negligible for the bigger company. Negotiating purchase agreements is expensive for big companies. If they are even in the position of dealing with a small vendor, there must be some compelling business reason for getting whatever the small vendor is selling. This also means that a deal that would be unprofitable, perhaps damaging, to the vendor would put a critical part of the supply chain at risk for the purchaser. But instead of closing the deal and moving on, some purchasing groups in some companies want to grind the vendor down to “show value” to their management or maybe even just out of habit. Sometimes this requires the smaller company to walk away, sometimes to go over the heads of the purchasing group to business units waiting for the product. We’ve probably lost some deals that would have worked out well in the end by refusing to accept onerous terms, but we’ve also walked away from deals that would have been fatally unprofitable. The good thing is that walking away from an overbearing big customer is usually a good starting point for a sales effort aimed at its competitors.
A glitch in time synchronization caused German exchange operator Deutsche BoerseAG DB1.XE -0.16% ‘s Eurex Exchange arm to halt trading for slightly more than an hour early Monday, the latest in a thread of technical issues at global exchanges.
The Eurex derivatives trading market was halted at 0620 GMT, 20 minutes after trading started, “in order to protect the integrity of the market,” Deutsche Boerse said.
“An incorrect time synchronization within the system” triggered the market halt. The problem was solved, pre-trading started at 0720 GMT and, as of 0730 GMT all products were again tradable, the exchange said.
A person close to the matter said the glitch was caused by a problem with the system clock, not extreme data load, noting that current trading volumes are far below previous peaks.
From what we understand, the problem was due to a number of GPS time servers that dropped leap year adjustments, so jumping the time back 36 seconds. Then PTP2 software adjusted server time to jump backward 36 seconds. It’s not the first failure of this sort (or the last). The legacy technology for time synchronization lacks the cross check, failover, and alerting that are built into TimeKeeper. That technology was not designed with enterprise in mind and although it has been heavily modified over time, it is very difficult to engineer a solution to basic architectural mismatch. For example, TimeKeeper is architected to treat all time sources the same, but software like PTPd and NTPd is designed for a specific protocol. So TimeKeeper can use a feed from one protocol to crosscheck another, but that requires a clumsy grafting process in one of the protocol specific time synchronization programs. As another example, the PTP standard has a completely inappropriate fault-tolerance technique baked into the standard – a technique that is a holdover from the PTP standard origin in device control. The standard was designed for systems with really simple networks and time consumers. A single cable with some welding machines on the end is perhaps a typical intended use case. The idea was that the time sources would advertise how good they were and the consumers would simply pick the one that said it was the best. This is an absurd policy for an enterprise network with super-powerful server computers receiving time packets across complex networks dotted with routers and switches. TimeKeeper was designed to ignore this “best master” protocol for fault tolerance and to utilize the smarts of the consumers to detect problems in the feed and to select among alternatives.
That’s not how it looks from here but I think part of the muddiness in the software patent argument is the result of arguments that really attack the entire idea of patents but are advanced as being specific to software patents. The whole idea of a patent is that you do foster innovation and competition by “handing out monopolies”. Maybe that idea is wrong, but if you think it is wrong you are likely opposed to patents – period. Tim Lee and many of the other
Famous anti-innovation patenter.
people who dislike software patents confuse the issue by simultaneously claiming that (1) software patents are inherently worse/different than other patents and (2) that software patents choke off innovation because of properties that turn out to apply to all patents. Opposing patents in general, however, is a more radical proposition than opposing software patents – perhaps more a more radical proposition than people feel comfortable making.
My position is that many patents are wrongly granted for “inventions” that are neither novel nor un-obvious and that the system for adjudicating patents is way too slow, error prone, and expensive. But patents themselves serve a useful purpose. The obvious example is Excel which is effectively a monopoly without the benefit of any patents at all. The work of the innovators was, without patent protection, rapidly copied by companies with better marketing and greater resources. Innovation stopped. End of story.
And “business method” patents, in general, are not really software or computer patents at all. Usually they are efforts to patent a well known method of doing business adding some technology to the mix to buttress a claim of novelty. One could have similarly claimed a hundred years ago that making sales calls via telephone was an invention or that delivering adverts by TV instead of by radio was an invention.
2 a.m. ET