“A day is coming, when, in the eye of the law, literary property will be as sacred as whisky, or any other of the necessaries of life.” —Mark Twain”
Inventors, programmers, writers, and musicians often get the same advice for the new Internet economy and the world of downloading and digital copying. The advice is to embrace new business models which give us exposure and access, maybe even fame or cryptocurrency in place of boring old money. We can sell labor like consulting services and live performances, T-shirts and books too, but we better give up hopes of obsolete and shameful licensing fees and royalties.
Here’s the foremost legal theorist of the Free Software Movement insisting that the collapse of copyright law will be good for musicians too ( from Liberation Musicology published in The Nation magazine in 2001).
“This increase in efficiency means that composers, songwriters and performers have everything to gain from making use of the system of unowned or anarchistic distribution, provided that each listener at the end of the chain still knows how to pay the artist and feels under some obligation to do so, or will buy something else–a concert ticket, a T-shirt, a poster–as a result of having received the music for free. Hundreds of potential “business models” remain to be explored … “
Back in 2000, when I started a software company, I was advised to give away software and sell services. The argument was that my company would be known as the experts on what we had invented and customers would pay a premium for our work. Each user in the chain would know how to pay the inventor and feel under an obligation to do so and we’d be profitably exploring a new business model. Well, it seemed plausible enough at the time if you were not too smart (raises hand). But it turns out that neither corporations nor consumers — the customers — are interested in this model.
Experience taught us that the business logic of corporate customers was not favorable to small company inventors who sold services. Our small business with a new product came with a certain level of risk that customers could reduce by asking some huge corporation to support the software instead of relying on a tiny startup. On the other hand, low wage consultants and competing venture funded startups could always under-price us particularly since we did all the work and bore all the cost of inventing and making the software. And to complete the picture, users in the chain who had money at stake were very quick to come up with convenient reasons why they did not have an obligation to pay us. Our ideas were, according to them, wrong, obvious and well known, not new at all. Our personalities were lacking in charm, our sense of style was off, our demands unreasonable. Any competent engineer, we were informed, could reproduce what we did over a weekend. In short, we didn’t deserve payment of any kind. I have heard very similar justifications for copying music in similarly resentful tones. People downloaded our code, rearranged it, and announced they had something better that owed nothing to our crappy inventions anyway. We were left with a very small bit of the market – unusual people who were willing to take a chance. We met some cool people, but it was no way to make a living .
Professor Moglen was correct, however, in the sense that this system was very efficient for the big companies that used our work. Instead of royalties and licensing fees, they only needed to pay for labor in a market where everyone was undercutting everyone else. Big companies with brands, marketing power, distribution, large legal teams, and lots of different ways to charge can often find all sorts of ways to wring money from products (“monetize”). In fact, the ways in which big companies like Amazon, IBM, and Google exploit free software both as users and contributors are fascinating for people who find innovative business models fascinating. Record companies can offer low royalty payments to Spotify and make it back on equity, non-license fees, and sheer volume ( as David Byrne explained). They can even mass produce T-shirts. RedHat and Oracle can sell free software by leveraging marketing, brand name, and risk minimization. RedHat’s Bob Young used to have a standard speech explaining both that he, Young, was dumber than his audience of programmers and that Red Hat’s business was like ketchup where Heinz could charge a premium because of brand alone. The first part was an outrageous lie well crafted to appeal to an audience of egotistical and naive programmers, but the second part was true enough. Even IBM used to give away software back when it was essentially the only computer hardware vendor in the world. The software helped them sell hardware and kept competing software vendors from taking too much of the revenue stream. For that matter, selling bottled tap water can be a very profitable business for Pepsi, but only because it has the distribution channels and marketing reach. In the end, though, all this clever monetization can leave artists and inventors who are not marketing wizards with a small share of the T-shirt revenue- if that.
Eventually we lost the faith, stopped releasing our code and started enforcing our intellectual property rights. Not everyone was happy. A high level corporate manager in a big company told me “we know about your technology and we understand its advantages but your patents make it awkward for us to use.” I asked what was awkward about paying a license fee, and he told me that his company was not about to pay license fees to some small time losers from nowhere. But here’s the important fact: Copyrights and patents, although they are often misused, can make it more awkward for big companies to shift revenue from the artists or inventor’s pocket to the corporate pocket in the course of some “agile efficient monetization of the marketplace”. Efficiency is one of those virtues that depends on where you sit. It’s certainly less efficient for a giant corporation to devote time and effort to licensing technology from small companies or paying ungrateful artists for their music than to simply make use of it. It’s expensive for a large firm to vet a product from a small company for utility and possible legal exposure and, at least initially, low cost to just download something for free.
On the other hand, it’s a lot more efficient for an inventor, programmer, writer, or musician to get a royalty check than to sell T-shirts. More lucrative, anyway.
There are downsides to depending on other people to do the work for nothing, but they are downstream ones. A manager or programmer at a tech company would have received a chilly response from her bosses a couple of years ago if she had explained that the company was depending on a minor component for logging, produced by some volunteer team, and that the corporation should go through all the effort needed to onboard that tiny team as a vendor and purchase and vet that software even though it could be downloaded for free. Years later, it turned out the software in question opened a major security hole in corporate firewalls and nobody who decided to download it for free was fired.
Property versus intellectual property
But is any of this morally correct? For many people software patents are bad, in fact, intellectual property itself is bad. One complaint involves the supposedly dubious status of intellectual property as property. The argument goes something like this:
“the owner of a digital recording or program doesn’t lose anything when a copy is made, unlike the owner of a piano or a bicycle who no longer has physical possession if ownership is transferred. You still have a recording of your movie or musical performance or program even if someone is letting people download copies for nothing. See – it’s not actual property.”
But even with physical objects, ownership and transfer is not really that simple. Suppose my family and I move into your house without your permission. You still “have” the house – the physical object is still there. If there is sufficient space or if it’s a rental house, you may not even be inconvenienced. What you’ve lost is something far more abstract than a physical object. You lost a right to control use or to demand payment for use – exactly what the musician lost when someone pirated her song. And what about intangible property? Bank accounts, loans and other financial assets are intangible pseudo-objects that have less physical presence than a bunch of bits encoding a song or program. If your financial adviser embezzles your retirement savings to bet on the dog track or your bank invents bogus charges that they deduct from your account, no physical object passes out of your possession but you’d most likely consider yourself deprived of something. What you were deprived of is a right to some money, itself a strange non-material abstraction. Property rights are all about fictional entities, pseudo objects, rights to forbid others to do something and so on – there is nothing specially unmaterial or spectral about intellectual property compared to other kinds of property.
We also heard that some kinds of IP are more worthwhile than others. It’s not especially surprising, I suppose, that these theories often seem to align with the financial interests of the person making the argument (me included, of course). For example, in Germany software patents are widely derided, but patents on pharmaceuticals (controversial elsewhere) are not. Germany is a net payer on one of those kinds of patents and a net payee on the other kind. Guess which is which. Similarly, I’ve had interesting discussions about software intellectual property with people who work for computer hardware companies and sincerely believe software intellectual property to be dubious at best. For them, somehow, a design expressed in a hardware programming language which is then converted into a recipe for computer chips can be worthy of protection, but a design expressed in a different programming language that is then converted into a process for controlling the machinery that makes those same chips is different in kind. The most entertaining argument I saw, however, was from an enthusiastic supporter of free software who wrote an indignant email to me from his company account, lambasting my use of software patents. When I pointed out that his corporate employer owned a lot more software patents than we did, he explained that his employer was a commercial enterprise and my business was not. Musicians seem to often get these kinds of lectures too.
The internet changes everything
People also argue that the nature of the internet and of digital representations of music and programs and movies makes that old-fashioned copyright and patent stuff obsolete. Here’s Professor Moglen again:
“our world consists increasingly of nothing but large numbers (also known as bitstreams), and that – for reasons having nothing to do with emergent properties of the numbers themselves – the legal system is presently committed to treating similar numbers radically differently. No one can tell, simply by looking at a number that is 100 million digits long, whether that number is subject to patent, copyright, or trade secret protection, or indeed whether it is “owned” by anyone at all. “
No one can tell, simply by looking at it, whether a particular block of land or house or apartment is “owned” by one private individual or another, though. Does simply looking at an automobile or a pile of I-beams or a basket of mangos, produce information on whether that object is owned, or leased, or stolen? There is nothing fundamental about a number that tells you whether it encodes a program or musical composition protected by copyright, or a method protected by patent, but there’s nothing “emergent” about a vat of chemicals or a ball point pen that tells you whether it embodies a patented chemical process or mechanical design. Ownership is not a physical property of an object or pseudo-object, ownership comes out of social relationships of human beings. Moglen continues:
” Like everything else in the digital world, music as seen by a CD player is mere numeric information; a particular recording of Beethoven’s Ninth Symphony recorded by Arturo Toscanini and the NBC Symphony Orchestra and Chorale is (to drop a few insignificant digits) 76749873424, while Glenn Gould’s peculiarly perverse last recording of the Goldberg Variations is (similarly rather truncated) 767459083268. Oddly enough, these two numbers are “copyrighted.” This means, supposedly, that you can’t possess another copy of these numbers, once fixed in any physical form, unless you have licensed them. And you can’t turn 767459083268 into 2347895697 for your friends (thus correcting Gould’s ridiculous judgment about tempi) without making a “derivative work,” for which a license is necessary.”
It’s illuminating to transfer this argument back to an older technology.
“Like everything else in the printed world, novels are mere sequences of letters; a particular printing of Fledgling might contain “When your rage is choking you, it is best to say nothing.”, while “the banana company did not have, never had had, and never would have any workers in its service” is from One Hundred Years of Solitude. You can’t fix Octavia Butler’s anger or Garcia-Marquez’s antipathy to banana companies and make other copies of these sequences of letters for your friends unless you have licensed them.”
We had copyrights for written works long before there were those electronic computers and pesky binary numbers that are supposedly eroding the whole basis of copyright law. And, by the way, that line about “correcting” Gould hit home for me. If you think you can play the piano better than Glenn Gould, go ahead. If you don’t like his work, don’t buy it. If you want to be “creative” with it and distribute it, pay a license fee to his heirs. The concept of “derivative work” in intellectual property law is there because people were publishing “improved” versions of Mark Twain’s work too. Intellectual property is important primarily because of the money, but it’s also important because it allows the people who create the product to exercise some control of how it is used. Neil Young was able to prevent Spotify from using his music to promote disinformation because he owned the rights. For our real-time software, one question I used to ask free software advocates was to explain why it would be immoral for us to forbid use of the software to control land mines.
Free software, public domain software, creative commons licenses – all of those can be good things. Irving Wladawsky-Berger’s recognition of how free software could help IBM’s business was pretty much brilliant although one could easily come up with more cynical formulations of his explanation. I even agree with those who say music and inventions are never wholly original, but build on the accomplishments of others. Of course they are. None of that justifies compulsory pre-emptive “freeing” of creative works or inventions by consumers of that work who just feel like they know better or, more likely, that their corporate employer would find it easier. Commerce is all about incentives and the incentive of a corporate lawyer or manager to insist that the company should pay an individual inventor or small business because it is the right thing to do is not all that high, believe it or not. Same with the incentive of a consumer to buy a song or movie that can be downloaded for free. And when some companies find it more profitable to use your work without paying, that makes it even more difficult for people at other companies to argue in favor of cutting you a big check. If all your friends stream new music from a company that systematically underpays artists, you tend to look like a chump for paying more. The same peer pressure operates in corporate purchasing. A lot of people actually want to do the right thing, but very few will buck the trend to do so.
And what’s the right thing?
“To promote the Progress of Science and useful Arts, by securing for limited Times to Authors and Inventors the exclusive Right to their respective Writings and Discoveries.”
The legal system of intellectual property is far from perfect. Actually, it’s scarily imperfect. Several times only dumb luck saved us being ground up between the vast resources of big companies and the vast expense of the legal system. But I was able to help create interesting technology, and get paid for it, and stay independent, because the much maligned intellectual property laws do something to protect inventors, musicians, artists, film makers and others who don’t want to work for a giant multinational, but do want to make things.
A free software advocate once told me that Mozart didn’t need copyrights. Years later I read Mozart’s letters and was appalled at how much time he had to spend groveling to wealthy sponsors. And maybe those of us who are less talented than Mozart but also create works that other people want to use or listen to or read also should get paid by labor right, not because someone with money wants to be a sponsor. So when next you hear about terrible Disney and their apparently infinite copyrights on Mickey, or about some patent troll demanding to be paid, or some musician suing a streaming service, think also about those of us who just want to be paid when someone uses our writings, music, creations, and discoveries. Proudhon may have been right and property may be theft, but in a society based on thievery, it is quite convenient to have some property.
YOUR GRACE, MOST WORTHY PRINCE OF THE HOLY ROMAN EMPIRE!
I will not presume to trouble Your Grace with a full description of our unhappy circumstances, which my father has set forth most accurately in his very humble petition which was handed to you on March 14th, 1777. As, however, your most gracious decision was never conveyed to him, my father intended last June once more most respectfully to beg Your Grace to allow us to travel for a few months in order to enable us to make some money; and he would have done so, if you had not given orders that in view of the imminent visit of His Majesty the Emperor your orchestra should practise various works with a view to their performance. Later my father again applied for leave of absence, which Your Grace refused to grant, though you permitted me, who am in any case only a half-time servant, to travel alone. Our situation is pressing and my father has therefore decided to let me go alone. But to this course also Your Grace has been pleased to raise certain objections. Most Gracious Prince and Lord! Parents endeavour to place their children in a position to earn their own bread; and in this they follow alike their own interest and that of the State. The greater the talents which children have received from God, the more are they bound to use them for the improvement of their own and their parents’ circumstances, so that they may at the same time assist them and take thought for their own future progress. The Gospel teaches us to use our talents in this way. My conscience tells me that I owe it to God to be grateful to my father, who has spent his time unwearyingly upon my education, so that I may lighten his burden, look after myself and later on be able to support my sister. For I should be sorry to think that she should have spent so many hours at the harpsichord and not be able to make good use of her training.
Your Grace will therefore be so good as to allow me to ask you most humbly for my discharge, of which I should like to take advantage before the autumn, so that I may not be obliged to face the bad weather of the ensuing months of winter. Your Grace will not misunderstand this petition, seeing that when I asked you for to travel to Vienna three graciously declared that I had nothing to hope for in Salzburg and would do better to seek my fortune elsewhere. I thank Your Grace for all the favours I have received from you and, in the hope of being able to serve you later on with greater success, I am
The developers of Google’s Spanner database built an interesting system but chose to build their own clock synchronization technology instead of getting a more precise off the shelf solution and chose to make extensive use of the notoriously complex Paxos distributed consensus algorithm even though they were incorporating clock sync into their system. Whatever the rationale at Google, building a clock sync enabled high speed distributed transaction system is much less complex if you don’t make those choices.
Off the shelf technology that can synchronize clocks to under a microsecond in high end systems and can even produce tight clock sync in the decidedly not optimal situation of cloud virtual machines is readily available (www.fsmlabs.com). And with clock sync, it is possible to enormously simplify distributed consensus. In fact, it’s hard to understand why Paxos is so widely used. Paxos is a highly complex (ingeniously complex) and painful to implement attempt to solve a problem that doesn’t exist.
The humorously titled “Paxos made simple” paper starts off discussing the problem of distributed consensus in completely asynchronous networks where timeouts won’t work and clocks are not synchronized at all. In practice, computer networks are not totally asynchronous, they do permit precise clock synchronization (and that technology has gotten much better since Paxos was introduced in 1998), and Paxos doesn’t even work in a totally asynchronous network. Page 7 of “Paxos made simple” points out that the algorithm can fail to converge on a solution, it is not “live”. Lamport, the author, then suggests electing a single coordinator or primary agent – using timeouts! Once you admit that timeouts work, you can replace Paxos with far simpler algorithms, some well known in database systems since the 1970s. Precise clock sync permits much more radical simplifications.
(recently saw a post claiming that Damore’s firing was an indication of closed minds at Google, so it seemed important to repost this – vy)
The Google manifesto is a crackpot political argument combined with basic scientific ignorance, not a scientific argument. The people claiming otherwise are either lying or terrible at reading. Damore’s essay is packed full of paraphrases of Jordan Peterson’s body of work, right wing political slogans, and resentment against women. Damore claims that any program to encourage hiring and promotion of women and minorities constitutes discrimination against white men. According to him a company mentoring program to encourage women programmers would be both wrong and illegal. He says that there is no gender wage gap and women get lower wages because they don’t want to work as hard or take risks like men do. He claims that feminism is a ploy by “Marxist intellectuals” to replace their old class struggle pitch. He says that Google management are left-wing extremists, motivated by a “veiled left ideology”, whose diversity initiatives are “encroaching extremist and authoritarian policies”. He says that he is in favor of diversity, but against actually doing anything to encourage diversity because:
Discriminating just to increase the representation of women in tech is as misguided and biased as mandating increases for women’s representation in the homeless, work-related and violent deaths, prisons, and school dropouts.”
Damore’s deplorable basket of crackpot resentments is not science. Period. The defenses of Damore by people like University of Texas Computer Science Professor Scott Aaronson and UW’s Stuart Reges and Peter Singer are either grossly dishonest or complete failures of scholarship. Aaronson writes to say that all Damore was doing was ” mooting the hypothesis of an evolutionary reason for average differences in cognitive styles between men and women,” and that “the basic thesis being discussed […] that natural selection doesn’t stop at the neck, and that it’s perfectly plausible that it acted differently on women and men in ways that might help explain many of the population-level differences that we see today“. He laments the “clickbait narrative” of Damore’s critics (although he doesn’t cite or quote any of them because that would ruin his baseless temper tantrum). But Damore is not saying anything is “perfectly plausible”. He is insisting that Google should abandon any attempt to hire or advance women or minorities. Stuart Reges wrings his hands in anguish:
For the last ten months I have been discussing this issue at the Allen School of Computer Science & Engineering where I work. I have tried to understand why Damore’s opinions generated such anger and have struggled to decide what I want to do in response.
He tried, so sincerely, to understand why people would get angry reading stuff like:
Yes, in a national aggregate, women have lower salaries than men for a variety of reasons. For the same work though, women get paid just as much as men. Considering women spend more money than men and that salary represents how much the employee sacrifices (e.g. more hours, stress, and danger), we really need to rethink our stereotypes around power. – Damore
Really Stuart? Ten months of sincere effort and you can’t figure out what about Damore’s memo would make people mad?
As it became clear that the working class of the liberal democracies wasn’t going to overthrow their “capitalist oppressors,” the Marxist intellectuals transitioned from class warfare to gender and race politics. The core oppressor-oppressed dynamics remained, but now the oppressor is the “white, straight, cis-gendered patriarchy.” – Damore
Although Aaronson won’t cite any of the critics of Damore, he lists a number of “courageous” defenders. For example, here is Peter Singer with more of this nonsense:
James Damore, a software engineer at Google, wrote a memo in which he argued that there are differences between men and women that may explain, in part, why there are fewer women than men in his field of work. For this, Google fired him.
Reges has a similar claim:
In this case, suggesting that men and women are different either in interests or abilities is considered blasphemy. – Reges
Obviously what Reges, Singer, and Aaronson wrote is not true. Insisting that there is no wage gap or that any diversity program at all constitutes discrimination against white guys or that Google management is on some left-wing crusade, is not evolutionary biology and it’s not arguing that “there are differences between men and women” – a totally uncontroversial claim. It’s not acceptable for these academics to misrepresent Damore’s resentful tract as some musings about evolutionary biology.
But what about all the scientific citations in Damore’s memo? Giving those all benefit of the doubt, if Aaronson and Singer and all the other handwringing apologists had tried to characterize Damore’s memo more more honestly as, say: “a combination of a political rant and an argument about evolutionary biology”, all of their indignant claims to be defending free scientific inquiry would have been a lot harder. They would have had to let go of the strawman and come up with arguments less dependent on emotion and name-calling. Here’s Aaronson
Let that be the measure of just how terrifyingly efficient the social-media outrage machine has become at twisting its victims’ words to fit a clickbait narrative—a phenomenon with which I happen to be personally acquainted. Strikingly, it seems not to make the slightest difference if (as in this case) the original source text is easily available to everyone.
This is to defend someone who wrote that one problem with Google’s efforts to hire women is that:
“We can make software engineering more people-oriented with pair programming and more collaboration. Unfortunately, there may be limits to how people-oriented certain roles at Google can be and we shouldn’t deceive ourselves or students into thinking otherwise (some of our programs to get female students into coding might be doing this).”
Or here’s Singer:
So on an issue that matters, Damore put forward a view that has reasonable scientific support, and on which it is important to know what the facts are.
Really? There’s “reasonable scientific support for “For the same work though, women get paid just as much as men”? No there is not. How about “political correctness is a phenomenon of the Left and a tool of authoritarians.” Some nice double blind studies on that one? What about:
I’ve heard several calls for increased empathy on diversity issues. While I strongly support trying to understand how and why people think the way they do, relying on affective empathy—feeling another’s pain—causes us to focus on anecdotes, favor individuals similar to us, and harbor other irrational and dangerous biases. Being emotionally unengaged helps us better reason about the facts.
Reasonable scientific support for that? Reges wants pity
When I tried to discuss Damore at my school, I found it almost impossible. As a thought experiment, I asked how we could make someone like Damore feel welcome in our community. The pushback was intense. My question was labeled an “inflammatory example” and my comments were described as “hurtful” to women. – Reges.
No kidding. The head of undergraduate instruction asks how to make a misogynist feel welcome in the community and people push back. Shocking. The entire basis of Reges, Aaronson’s and Singer’s defenses of Damore depends on whitewashing away all the ideology in his essay, taking his variant of “I’m not a racist but ... ” at face value, and then pretending that his wacky “biology” is respectable. Only if you do all that can you denounce the irrational anti-scientific anti-free speech mob from your claimed position as an Olympian Scientist and Defender of Free Speech. How original: to defend a sexist rant by claiming its critics are hysterical.
“Biological males that were castrated at birth and raised as females often still identify and act like males – James Damore”
Wow, imagine not being struck by how weird that is. There is a biological argument in Damore’s essay, but it’s ridiculous. Nobody argues that there are no biological differences between men and women. Nobody. Damore comes up with: “Differences in distributions of traits between men and women may in part explain why we don’t have 50% representation of women in tech and leadership.” Well, who said that those differences definitely did not explain any differences in representation? Nobody, that’s who. Google engineering is 19% female. Google’s management is 22% female. The issue is not why Google doesn’t have 50% representation, but why it has 19% representation.
Damore cites a number of studies that show or claim to show that certain traits are weakly correlated with gender. The gender based differences in those studies are in single digits so even if we accepted them as authoritative (they are not) they would not begin to explain the gender differences at Google. More striking, not one of those studies concerns programming ability or performance. When people claim Damore’s cited studies represent the scientific consensus (which is not true) they leave out that those studies have absolutely nothing to say about either programmers or diversity programs. Let me just stress that, because Damore is repeatedly claimed to have been relying on science: Not a single one of the studies that Damore cites claims to show a relationship between gender and ability to code.
Not one. Traits like which toys rhesus monkeys pick by gender may be related to human performance writing Java code but there is no evidence of that and it’s pretty implausible. Why being “more thing oriented” would make men more likely to be managers of people at Google or more interested in distributed consensus algorithms is also not addressed. But even if we were as naive as Professors Aaronson, Reges, and Singer, and accepted this collection of studies as meaning what Damore claims they mean, it takes a stupendous level of mathematical ignorance to agree that such studies would explain Google’s gender imbalance.
There are 52 cards in a deck of cards, so a blind pick from a shuffled deck produces an ace with probability 4/52. Yet, I can guarantee you I can pick 4 aces in a row out of the deck every time if I look at the faces of the cards. Google and other companies don’t hire based on random sampling from the population. They recruit, interview and test and, because they are a highly profitable company, they can pay more and offer better benefits to attract employees they want. They look at the cards. If they want aces, they can get them. Most of the programmers in the world are average or worse. Does Google have to accept that half of its programmers will be below average? This is obvious, but in a rush to take Mr. Damore’s rant as a serious contribution, Damore’s defenders, who claim to be speaking the name of science, apparently can’t do basic statistical reasoning.
The really interesting question is why this document has any defenders at all and, particularly, why all the concerned hand wringing from academics.
“Civilization’s going to pieces,” broke out Tom violently. “I’ve gotten to be a terrible pessimist about things. Have you read ‘The Rise of the Coloured Empires’ by this man Goddard?”
“Why, no,” I answered, rather surprised by his tone.
“Well, it’s a fine book, and everybody ought to read it. The idea is if we don’t look out the white race will be–will be utterly submerged. It’s all scientific stuff; it’s been proved.”
“Tom’s getting very profound,” said Daisy with an expression of unthoughtful sadness. “He reads deep books with long words in them. What was that word we—-“
“Well, these books are all scientific,” insisted Tom, glancing at her impatiently. “This fellow has worked out the whole thing. It’s up to us who are the dominant race to watch out or these other races will have control of things.”
“We’ve got to beat them down,” whispered Daisy, winking ferociously toward the fervent sun. – F. Scott Fitzgerald, Great Gatsby.
(h/t to vp at Lobste.rs for reminding me of this )
Everyone can recognize some kinds of spurious reasoning – like the following: (A) Most programmers are not great.
(B) Google tries to hire only excellent programmers
(C) A implies B is hopeless and counterproductive and demoralizing
Clearly, step (C) is nonsense – employers are not required to randomly hire and the distribution may make the effort to hire good programmers harder, but it tells us nothing about how valuable such an effort is and whether it can succeed. So (C) is an example of what we call in formal logic, “unsound deduction”. Yet a much worse argument got a lot of support from people who are supposed to be in the business of logical thinking. This argument starts with a false premise and then uses the same unsound deduction.
(A) Biology ensures more men than women want to be programmers.
(B) Google wants some balance
(C) A implies B is hopeless and counterproductive
Many people, supposedly educated people, have claimed this to be a simple statement of scientific fact that is denied only by “left wing ideology” or something. Ben Franklin was right when he explained that the advantage of being a reasoning creature is you can always find a reason to justify what you want. In this case, some people want to justify feelings of resentment – despite high paying jobs and good working conditions. The defense of the Google Manifesto as “science” though is particularly noxious both because of the unsound deduction at its core and because it’s clearly a political argument – clearly political, but with muddy and poorly thought out political ideas (which is what resentment gives you). Here is a sample:
when it comes to diversity and inclusion, Google’s left bias has created a politically correct monoculture that maintains its hold by shaming dissenters into silence. This silence removes any checks against encroaching extremist and authoritarian policies. For the rest of this document, I’ll concentrate on the extreme stance that all differences in outcome are due to differential treatment and the authoritarian element that’s required to actually discriminate to create equal representation.
Someone should have put in more time working and less listening to hate radio.
Principal type-schemes for functional programs∗ Luis Damas† and Robin Milner First published in POPL ’82: Proceedings of the 9th ACM SIGPLAN-SIGACT symposium on Principles of programming languages, ACM, pp. 207–212
Consequently, to overcome this restriction, the implementations of Rust’s standard libraries make widespread use of unsafe operations, such as “raw pointer” manipulations for which aliasing is not tracked. The developers of these libraries claim that their uses of unsafe code have been properly “encapsulated”, meaning that if programmers make use of the APIs exported by these libraries but otherwise avoid the use of unsafe operations themselves, then their programs should never exhibit any unsafe/undefined behaviors. In effect, these libraries extend the expressive power of Rust’s type system by loosening its ownership discipline on aliased mutable state in a modular, controlled fashion: Even though a shared reference of type &T may not be used to directly mutate the contents of the reference, it may nonetheless be used to indirectly mutate them by passing it to one of the observably “safe” (but internally unsafe) methods exported by the object’s API – from
Proposition: there are no major applications in any safe language that do not depend on either C libraries or “unsafe” magic keywords or both.
The first time I saw this method was when I went to work for Parallel Computer Systems, , later called Auragen, in the famous tech startup center of Englewood Cliffs, New Jersey. I commuted there from the East Village*. The company built a parallel computer based on Motorola 68000s with a replicated message bus. The bus guaranteed message delivery to 3 destinations would either succeed to all three or fail to all three. This property is called “reliable broadcast”. All interprocess communication was by message transfer (a fashionable idea at the time). Each process had a backup. Whenever a primary process sent a message, the message was also delivered to the backup and to the destination backup. If the primary failed, the backup could be run. The backup would have a queue of messages received by the primary and a count of messages sent by the primary. When the recovering backup tried to transmit a message, if the count was greater than zero, the count would be decremented and the message discarded because it has already been transmitted by the old primary. When the recovering secondary did a receive operation, if there was a message on the input queue, it would get that message. In this way, the recovering backup would repeat the operations of the primary until it caught up. As an optimization, the primary could be periodically checkpointed and queues of duplicated messages could be discarded.
The operating system was an implementation of UNIX. In practice, it was discovered that making each UNIX system call into a message exchange, which was an idea advocated in the OS research community at the time, caused serious performance problems. The replicated state machine operation depended on this design in order to make the state machine operation deterministic. Suppose the primary requested, for example, the time and then made a decision based on the time. A recovering secondary would need exactly the same time to guarantee that it produced the same results as the primary. So every interaction between application and OS needed to be recorded in a message exchange. But a message exchange is nowhere near as fast as a system call (unless the OS developers are horrible).
The performance issue was mitigated by some clever engineering, but was a problem that was discovered in parallel by a number of development teams working on distributed OS designs and micro-kernels which were in vogue at the time. Execution of “ls -l” was particularly interesting.
Anyways, here’s the description from the patent.
To accomplish this object, the invention contemplates that instead of keeping the backup or secondary task exactly up to date, the backup is kept nearly up to date but is provided with all information necessary to bring itself up to the state of the primary task should there by a failure of the primary task. The inventive concept is based on the notion that if two tasks start out in identical states and are given identical input information, they will perform identically.
In particular, all inputs to a process running on a system according to the invention are provided via messages. Therefore, all messages sent to the primary task must be made available to the secondary or backup task so that upon failure of the primary task the secondary task catches up by recomputing based on the messages. In essence, then, this is accomplished by allowing every backup task to “listen in on” its primary’s message.
United States Patent 4,590,554 Glazer , et al.May 20, 1986
Inventors:
Glazer; Sam D. (New York, NY), Baumbach; James (Brooklyn, NY), Borg; Anita (New York, NY), Wittels; Emanuel (Englewood Cliffs, NJ)
*True story: I applied for the job after finding an advert in a discarded copy of the NY Times on the floor of a Brooklyn apartment while visiting friends. I sent via US mail a resume poorly typed on a manual typewriter- I’m tempted to say “composed by the light of a tallow candle” but that would be over the top- and forgot to enclose the second page.
This graph from Peter Diamandis about how Kodak entered the sinkhole is kind of amazing. Diamandis explains Kodak’s failure to swerve in what is I think the orthodox Silicon Valley analysis:
[in 1996] Kodak had a $28 billion market cap and 140,000 employees.
In 1976, 20 years earlier, Kodak had invented the digital camera. They owned the IP and had the first mover advantage. This is a company that should have owned it all.
Instead, in 2012, Kodak filed for bankruptcy, put out of business by the very technology they had invented.
What happened? Kodak was married to the “paper and chemicals” (film development) business… their most profitable division, while the R&D on digital cameras was a cost center. They saw the digital world coming on, but were convinced that digital cameras wouldn’t have traction outside of the professional market. They certainly had the expertise to design and build consumer digital cameras — Kodak actually built the Apple QuickTake (see photo), generally considered the world’s first consumer digital camera. Amazingly, Kodak decided they didn’t even want to put their name on the camera.
There is more of the same (2012 and before) in the MIT Technology review. This is a totally convincing story (it had me convinced), but it leaves out three things:
the boring old chemicals division of Eastman Kodak which was spun off in 1993 (three years earlier) is still around, profitable ( $10B/year revenue) and dwarfs what’s left of the original company,
and Fuji films, Kodak’s also ran in chemical film space managed to leap over the sinkhole and prosper, but not by relying on digital cameras.
Kodak did briefly become a market leader in digital cameras but ran into a more fundamental problem.
Back in 1996, the business press and analysts thought Kodak was doing the right thing by divesting its chemical business.
NEW YORK — Eastman Kodak Co., struggling against poor profit and high debt, Tuesday took a big step in its corporate restructuring, announcing that it will divest Eastman Chemical Co. and in one fell swoop wipe out $2 billion of debt.
Such a spinoff would not have occurred just a few years ago, analysts said, and the move signals that Chief Executive Kay R. Whitmore is responding to new, tougher markets and stockholder pressure to improve financial results quickly.
“They are now recognizing that they are not a growth company, that they must go through this downsizing,” analyst Eugene Glazer of Dean Witter Reynolds said in an interview on CNBC.
Kodak’s shares, up sharply Monday in anticipation of the announcement, ended down $1.375 to $52.375 on the New York Stock Exchange.
[…] “We determined that there was little strategic reason related to our core imaging and health business for Kodak to continue to own Eastman,” Whitmore said at a news conference.
Kodak, best known for photography products but also a major pharmaceutical and chemicals group, has endured slow growth for years. Its photography business has been hit by changing demographics, foreign rivals and new technologies such as camcorders. Whitmore said Kodak sales, especially in photography and imaging, were weak.
[…] Costs will be reduced elsewhere in the company, Whitmore said. Other executives also have said Kodak will cut spending on research and development of new products.
In retrospect, dumping the cash generating parts of the business and cutting R&D was not the best plan even if Wall St. analysts loved the idea. But it’s easy to be a genius after the fact as Willy Shih points out:
Responding to recommendations from management experts, from the mid-1990s to 2003 the company set up a separate division (which I ran) charged with tackling the digital opportunity. Not constrained by any legacy assets or practices, the new division was able to build a leading market share position in digital cameras — a position that was essentially decimated soon thereafter when smartphones with built-in cameras overtook the market.
Yes, those camera phones – which not too many people saw coming in the 1990s. Not only that, but Kodak’s path in digital imaging was not obvious.
The transition from analog to digital imaging brought several challenges. First, digital imaging was based on a general-purpose semiconductor technology platform that had nothing to do with film manufacturing — it had its own scale and learning curves. The broad applicability of the technology platform meant that it could be scaled up in numerous high-volume markets (such as microprocessors, logic circuits, and communications chips) apart from digital imaging. Suppliers selling components offered the technology to anyone who would pay, and there were few entry barriers. What’s more, digital technology is modular. A good engineer could buy all the building blocks and put together a camera. These building blocks abstracted almost all the technology required, so you no longer needed a lot of experience and specialized skills.
Semiconductor technology was well outside of Kodak’s core know-how and organizational capabilities. Even though the company invested lots of money in the basic research and manufacturing of solid-state semiconductor image sensors and developed some notable inventions (including the color filter array that is used on virtually every color image sensor), it had little hope of being a competitive volume supplier of image sensor components, and it was difficult for Kodak to offer something distinctive.
And Shih, perhaps unintentionally, reinforces Diamandis’s point that the top company managers failed to face up to the problem.
For many managers of legacy businesses, the survival instinct kicked in. Some who had worked at Kodak for decades felt they were entitled to be reassigned to the new businesses, or wished to control sales channels for digital products. But that just fueled internal strife. Kodak ended up merging the consumer digital, professional, and legacy consumer film divisions in 2003. Kodak then tried to make inroads in the inkjet printing business, spending heavily to compete with fortified incumbents such as HP, Canon, and Epson. But the effort failed, and Kodak exited the printer business after it filed for Chapter 11 bankruptcy reorganization in 2012.
Management chaos and “spending heavily to compete with fortified incumbents”.
Yep.
With the benefit of hindsight, it’s interesting to ask how Kodak might have been able to achieve a different outcome. One argument is that the company could have tried to compete on capabilities rather than on the markets it was in. This would have meant directing its skills in complex organic chemistry and high-speed coating toward other products involving complex materials — a path followed successfully by Fuji. However, this would have meant walking away from a great consumer franchise. That’s not the logic that managers learn at business schools, and it would have been a hard pill for Kodak leaders to swallow.
it would have been a hard pill for Kodak leaders to swallow.
But wasn’t that their job? So to conclude this exercise in cut and paste, what about Fuji? The Economist had an interesting take:
the digital imaging sector accounts for only about one-fifth of Fujifilm’s revenue, down from more than half a decade ago.
How Fujifilm succeeded serves as a warning to American firms about the danger of trying to take the easy way out: competing through one’s marketing rather than taking the harder route of developing new products and new businesses. […]
Like Kodak, Fujifilm realised in the 1980s that photography would be going digital. Like Kodak, it continued to milk profits from film sales, invested in digital technologies, and tried to diversify into new areas. Like Kodak, the folks in the wildly profitable film division were in control and late to admit that the film business was a lost cause. As late as 2000 Fujifilm counted on a gentle 15 or 20-year decline of film—not the sudden free-fall that took place. Within a decade, film went from 60% of Fujifilm’s profits to basically nothing.
If the market forecast, strategy and internal politics were the same, why the divergent outcomes? The big difference was execution.
Fujifilm realised it needed to develop in-house expertise in the new businesses. In contrast, Kodak seemed to believe that its core strength lay in brand and marketing, and that it could simply partner or buy its way into new industries, such as drugs or chemicals. The problem with this approach was that without in-house expertise, Kodak lacked some key skills: the ability to vet acquisition candidates well, to integrate the companies it had purchased and to negotiate profitable partnerships. “Kodak was so confident about their marketing capability and their brand, that they tried to take the easy way out,” says Mr Komori.
Fujifilm realised it needed to develop in-house expertise in the new businesses.
This article on the interesting Wave Transactional File System inspired me to look up an earlier file system that also used copy on write semantics.
From:
Anita Borg, Wolfgang Blau, Wolfgang Graetsch, Ferdinand Herrmann, and Wolfgang Oberle. 1989. Fault tolerance under UNIX. ACM Trans. Comput. Syst. 7, 1 (January 1989), 1-24. DOI=http://dx.doi.org/10.1145/58564.58565
4.3 Availability of the File System
Since a recovering file server reconstructs its buffers by reading blocks from the file system, the file system in the state as of the last sync must be available. The existence of that version of the file system is also necessary during recovery as the file server redoes requests. For example, if a file has been deleted since sync and a read request is reissued, the disk driver, and thus the recovering file server, will behave differently than the primary. Unfortunately, the contents of the disk can change between syncs, at least during the Fsync that constitutes the first phase of the sync operation.
The solution is to use a copy-on-write strategy between syncs, rather than overwriting existing blocks. Logically this corresponds to keeping two versions of a file system.3 An early version of the file system organization described here is discussed in Arnow [ 11].
There are two root nodes on disk. At any given time one of them is valid for recovery. We refer to the other as the alternate root. Associated with each root is state information (the state tables described above), the most recent being that associated with the currently valid root. Changes to the file system are done relative to a copy of the valid root kept in memory in the primary file server’s address space, and in a nondestructive manner, as seen in Figure 2(a-d). Freed blocks, which contain the old data, are added to a semi-free list, and cannot be reallocated until after the next sync. Therefore, the unmodified file system still exists rooted in the valid on-disk root node.
If a crash occurs at any time between syncs, the recovering file server is able to determine which root to use because of information sent on the primary’s last sync. It reads in the correct state information and reconstructs its buffers accordingly. Disk blocks that were used by the primary since the last sync appear to it as free blocks.
The difficult case is when a crash occurs during a sync. To see that the solution works in this case, consider the sequence of actions that take place during a sync. First, all dirty blocks except the root are written to disk, and old blocks are added to the semi-free list. Second, the state information is collected and written to the alternate state area. Third, the in-memory root is written to the alternate on disk root block, Finally, the sync message is constructed and sent to the backup. It contains the information necessary to update message queues as well as specifying which on-disk state information and root block to use on recovery.
Once the sync message has been sent, the semi-free list is added to the free list and the primary continues. Just before the sync message is sent, there are two copies of every modified data and indirect block. At any time before the sync message is sent, the old consistent state is available. Any time after it is sent, the new state and file system will be used and message queues consistently updated. An additional benefit of this organization is that the file system as a whole is considerably more robust than a standard UNIXstyle file system. Even if the entire system is shut down in an uncontrolled way as the result of multiple faults or operator error, there will always be an entire consistent file system on disk.
I have a post up on FSMLabs web site about the use of GPS and other satellite time for MiFID II timestamp compliance. It’s fascinating how much effort has recently gone into trying to convince people that MiFID II will require direct time from a national lab or certified via a national lab despite the clear wording in MiFID II proposed regulations. To me, the deal is sealed in the Cost Benefit Analysis in which the ESMA regulators write
“The final draft RTS also reduces costs of the initial draft RTS proposed in the CP by allowing UTC disseminated via satellite systems (i.e. GPS receiver or the use of other satellite systems when available)”
That is not a promise one can easily walk away from. ESMA justifies the regulations with a cost/benefit analysis in which the costs for time stamping are limited by license to use GPS time. Of course, legal reasoning and logic are not always the same, but I’m trying to figure out how ESMA regulators could claim that they didn’t mean it, or why they would have such a motivation.
