keeping simple

Blog

  • Spinning the press about technology

    We’ve been an engineering driven company from day one – something that leads us to be frustrated with “analysts” and reporters who seem unwilling to challenge marketing. For example, here’s how a recent Investor’s Business Daily article starts:

    “With a fast-growing presence in everything from servers to cell phones, the Linux operating system appears ready for prime time. But is it ready for real time?

    “MontaVista Software thinks so…”

    Hello! Anyone there able to use Google? Why is it that reporters who write such stuff won’t do a little research so that they can tell their readers Linux has been “ready for real-time” in the most demanding applications for over a decade?

  • Auragen computers remembered

    In the early 1980s, I worked for a start-up called Auragen Computers based in Fort Lee, New Jersey. We were making a 68K based fault tolerant UNIX based on a smart idea by Sam Glazer. Most of the software engineers lived in New York City and commuted out. Start-ups with New Yorkers have a different “culture” than start-ups with Californians. One of the senior technical staff was once asked to meet with investors to talk about why development was taking so long. After some fruitless attempts at discussion he explained, kindly, “I don’t think I can make this clear in 5 minutes. It took me 10 years to understand this stuff and I’m a lot smarter than you.” Auragen did some good work, but failed to get the product out to market in time. A good lesson to learn.
    Auragen gave me my first job in the computer industry and a great education in operating systems. I still remember sitting in a room with Jim Baumbach, Anita Borg, David Arnow, and Sam Glazer soon after I started there, and confronting the reality that everyone in the room was a lot smarter, better educated, and more witty than me (my only consolation was that, in their early 30s, they seemed pretty damn old).

    See this paper: A message system supporting fault tolerance

    and this post: The replicated state machine model of fault tolerance from the 1980s

     

  • AMD powered 16 core systems are real

    We’re finishing some tests on the IWill 8-chip 16 core server and seeing outstanding timing. RTCore can share Linux with cores or reserve them for real-time. On reserved cores the jitter time we have found to be fundamental never reaches two microseconds, no matter what the system load. On non-reserved, the IWill systems look like they are staying under 10 microseconds – we will see if that holds up over a few days. The fact that it has done so well is a tribute to AMDs fast interconnect architecture. Of course the utility of multi-core in high end control or simulation is that you can have dedicated real-time and powerful computing in the same box.
    In RTCore (both RTLinux and RTCoreBSD) you can create threads on any processor and then create a thread that is “reserved”. While that thread is live, the BSD or Linux client is never scheduled on the thread’s processor core. So it is easy to reserve and free cores as demanded.

    At under 1KW, the Iwill is not even that bad on power consumption (although we have means of bringing that down and newer processors should be cooler). All in all, it is obvious that a new era has arrived for the higher end of control systems, network devices, and other systems that blend computation with control. As we test out the real-time networking capabilities later this week, I will come back to talk about some other, possibly surprising, areas of application.

  • The long haul in the embedded software business.

    Chris Lanfear from VDC systems asks

    FSMLabs has the product line, but lacks the market presence and awareness that other companies have invested in with venture-backed capital. We believe the company has largely bootstrapped itself over the years and while this conservative approach has a number of benefits, it is difficult to achieve the market presence of a MontaVista that way. That said, the problems that MontaVista appears to have are only compounded by investors with high-return expectations. Maybe the folks in Socorro have a better plan?cite

    And that’s reasonably on target. We have grown slowly, invested in technology and engineering, remain relatively clueless about “market presence”, and have a pretty straightforward business operation (make products with some compelling technical advantage, sell them at a profit, sell support at break even or better, move towards more integrated products, leverage the open source
    base so we don’t re-create commodity stuff, … ). Chris underestimates the number of mission critical applications using RTLinux, but see “clueless about marketing” above. Our sales guys have to sell the stuff on price and quality – a daunting task in this market.

    Chris mentions our lack of VC investment, and one of the reasons we have not taken VC investment is that we have resisted pressures to make up stories about fabulous wealth in blinks of an eye and “innovative” business models that create profits from “market presence”. We sell highly complex, technical software that requires a long slow engineering and test process. Our “business model” of looking for compelling technical advantage pushes us towards hard problems (you can’t obtain a compelling technical advantage on work that can be done easily).

  • PowerPoint and RocketScience II

    PowerPoint and RocketScience II

    In a previous note I objected to venture capitalist Guy Kawasaki’s requests (demands) for shallow and glib sales pitches in PowerPoint, but Professor Edward Tufte is also asking for sales pitches, just ones with more sober disguise. When you read “My other models for NASA are Feynman’s lectures on physics, and the A3 page (or 11 by 17 in) folded in half” the excellent quality of the writing helps slip by the emotive appeal to the form of Fenyman’s lectures and crisp clean sheets of paper as if something were implied about the quality of the content. Any University library computer science section can supply an apparently infinite number of illustrations of how perfectly good paper, verb predicate sentences, and beautifully typeset symbols can be employed to transmit vacuous nonsense.
    What Tufte identifies as marketing values – “fast pace, misdirection, advocacy not analysis, slogan thinking, branding [and] exaggerated claims” – can easily be found in academic conferences. Computer science is not unique. Physicists joke about the rate of increase of the size of Physics Letters, calculating the date when it will be growing faster than the speed of light, and noting that no laws of physics will be violated since “no information is carried”.
    Kawasaki is asking the desperate “entrepreneurs” to pitch their products and companies in 10 quick pictures. Could a presentation like that convey an idea of what HP or Visicalc or Apple or Adobe or Microsoft was trying to do? And all that simple stuff has already been done. Shallow pitches cannot give enough information to distinguish between a brilliant new idea for a storage system and some tired rehash of what everyone else did twenty years ago. But Tufte is also judging books by the illustrations. Feynman was a showman. I don’t believe that he would have refused PowerPoint, smoke machines, laser light shows, a samba band, or live dancers (especially not live dancers) in his lectures. The clarity, intellectual rigor, and honesty of the work (what Feynman described as bending over backwards to look at any possible weakness in the argument) has little to do with the medium or the venue.

    Kawasaki and bullshitTufte are both very sharp and provide interesting observations on presentation syntax. But the semantics is the real deal. Presentations are collaborations between audience and presenter. If the audience is passive, lacks expertise, and doesn’t want to do the work required for critical analysis, the presenters will be drawn to the audience level. If the presenter is making a less than honest pitch, a 40 page tightly written document carries with it no warranty. The failure of managers to understand the problems with the Columbia space shuttle would not have been overcome by banning PowerPoint. Professor Frankfurt points out that bullshit “is one of the most salient features of our culture”. Seeing through bullshit is tough work and it can’t be delegated to formatting.

    (put the graphic back 0ct 2 2014)

     

  • Digital Rights Management and Logic

    The Sony DRM fiasco is due to a common failure of requirements management logic. If an engineered system relies on certain properties, whenever you add a new requirement, you need to check consistency. You have a boat that has EnoughCargoSpace and ReasonableEnergyCost and you decide to add the requirement CantLeak. So you encase the entire boat in a meter of steel. Amazingly the boat no longer meets its other requirements. The weightless nature of software encourages people to forget that there are inescapable tradeoffs. If you impose a rule “B: The DRM system is all powerful” then you have contradicted “A: No Software Can Turn Off Security.” DRM engineers need to come to terms with “Some of the properties of a computer are more critical than protecting IP.” In my drm paper (published in LinuxDevices in 2002) I discussed potential security and safety issues when DRM interacts with embedded software. It’s bad enough to open a door to virus software on home PCs, but imagine the effects in medical devices and elevators!

    See also Moshe Yudkowsky on the ColdPlay story

    Update: See also – no kidding.

    Update: So how much of the Song BMG fiasco was caused by bad engineering practices and how much was caused by bad management practices? These two types of bad are often synergistic.

    Update: Boing Boing covers continuing fallout.

    Update: See the Linux Devices story on DRM

  • Power Point, Rocket Science and dangers of compelling stories

    In American English, you can say that something is not too difficult by saying “it’s not rocket science.” We don’t have a good idiom for saying the opposite – that something is hard to understand, not bullet-pointable. Edward Tufte dislikes PowerPoint because it can be used in a way that obscures critical data under a pile of graphics and misleading bullet points.

    My other models for NASA are Feynman’s lectures on physics, and the A3 page (or 11 by 17 in) folded in half. You can see where we’re at. If they would just write sentences, with subjects and predicates, rather than those damn bullet points.
    At some of these organizations, a technical report is called a “pitch” and is presented in 10-15 minutes, or presented simply as a PP deck to look over, or shown as a one-slide executive summary, or circulated by email-attached PP slides for the cognoscenti. Some of that reporting is done in a crisis; the Boeing PP slides were prepared in 2 or 3 days when the Columbia was in trouble but still flying. ( cite)

    Guy Kawasaki wants shorter, sharper, presentations:

    a PowerPoint presentation should have ten slides, last no more than twenty minutes, and contain no font smaller than thirty points.

    from Guy Kawasaki

    I’m dubious that improving the technology or execution of presentations is a solution to anything. Certainly improving the technology of selling political candidates has not improved their quality. Kawasaki is asking for a short sweet pitch. He’s asking to be told a compelling story about how he can make lots of money fast. And the NASA bureaucrats wanted compelling stories about how they could complete the mission. Making the NASA presentations shorter and sharper would not have helped. Maybe the pitches tell Kawasaki what he needs to know – part of what he wants to know is whether the presenter is a good salesperson. Sales and sales pitches are part of how we communicate. But some things are rocket science or just as hard and for those things pitches are dangerous. For rocket science or the equivalent you need tough, demanding, focused presentations that painstakingly dig out the tradeoffs and uncertainties. These presentations don’t have to be dull, but they are too qualified and too complex to be good pitches. Compelling stories may not be true stories. Pitches that engage us emotionally may also lead us down the wrong path. Something may feel right but be wrong. I often ask for more detail, more caveats, more analysis both on engineering issues and business issues. In both places, short sweet and compelling can mean  too good to be true.

    I’m biased of course, because many of our competitors* have much better pitches than ours. Fortunately for us, although not necessarily for everyone involved, physics trumps salesmanship in our field – eventually and sometimes spectacularly. (There’s a followup to this note here)

    *[ this was back in the RTLinux days – VY]  (added this and removed broken link 10/3/2014)

  • Formal methods and academic computer science

    Holloway [22] points out that the typical argument in favor of formal methods (that software is bad, unique, and discontinuous; that testing is inadequate; and that formal methods are essential to avoid design flaws) is logically flawed, and unnecessarily complex (in logical terms). He proposes a simpler argument, which is both simple and logically valid: software engineers want to be real engineers; such engineers apply mathematics; and since formal methods is the mathematics of software engineering, software engineers should use formal methods.

    from http://www.jpbowen.com/pub/fmics05.pdf

    Can you imagine Maxwell or some advocate of his making a similar argument about why electrical engineers should use electromagnetic theory? The circular argument here depends on the declaration that “formal methods is the mathematics of software engineering”.  No it’s not. The mathematics of some engineering discipline helps engineers solve actual problems in that discipline.

  • Why programs fail and get hacked

    “People’s distrust of the high-pressure engines was confirmed when the boiler of a stationary engine exploded at Greenwich on 8 September 1803. It was the usual tale; the boy who had been trained to work the engine went off to catch eels and a labourer stopped the engine without releasing the safety valve.” Richard Hills, Power from Steam, Cambridge 1989.

    I posted this to Risks digest in 1995. Software is still in the pre-safety equipment steam boiler stage.

  • Priority Inheritance: Hack or Error

    The subject of priority inheritance has come up again on the Linux kernel mailing list and Torvalds correctly notes

    “Friends don’t let friends use priority inheritance”. Just don’t do it. If you really need it, your system is broken anyway.

    “Priority inheritance” is similar to “soft real-time” in that both are efforts to avoid building software to specification. I put together a detailed critique in “Against Priority Inheritance”. There are two basic problems with priority inheritance (1) the algorithms are inefficient, complex, and inherently buggy and (2) any claimed benefits can be obtained in a more durable and efficient manner via good design.

    The RTLinux core does not support priority inheritance for a simple reason: priority inheritance is incompatible with reliable real-time system design. Priority inheritance is neither efficient nor reliable. Implementations are either incomplete (and unreliable) or surprisingly complex and intrusive.

    Doug Locke’s response to “Against Priority Inheritance” seems to me to have to evade the issue in the standard way: read his response and see for yourself. It’s not enough to say “yes, of course you need a proper implementation” since one of the points of the paper is that proper implementations are fragile, expensive, and unreliable.

    Update: If you want technical details on the can-o-worms PI opens – read the great description by Bryan Crantril at Sun.