It’s not like I didn’t see doomed robotics projects in the US National Labs.

According to a bulletin issued by the U.K. Air Accidents Investigation Branch (AAIB), an electrical fault led to a series of display and control problems. These included the intermittent failure of both pilots’ electronic displays and the uncommanded application of left rudder trim.  [..] The crew also reported hearing “chattering” from the circuit breaker panels. During the incident the crew flew the A321 manually, referring to the standby instruments which continued tofunction normally. The problem was rectified after the crew switched off the No. 1 generator in response to an ‘ELEC GEN 1 FAULT’ message on the Airbus’s ECAM warning display, says the AAIB. The cause has been traced to an electrical problem and the AAIB says that Airbus indicates that a reset of the Flight Augmentation Computer, caused by an electrical power interruption, can result in incremental offsets in rudder trim.

As a visitor to Japanese auto companies 10 years ago, I was blown away by the brilliance of the industrial design and the enormous investment in that, and shocked at the software development effort – one in which the software team was clearly considered to be low status. Not that my, more limited, experience with US companies was much more impressive. In fact, it was interesting that as Japanese auto companies were reaching out to at least look at advanced control software, US companies relegated that work to the outer periphery. Of course, this was some time ago, but obviously a $100M investment in top level control software development and test at Toyota a decade ago would have been a good investment – even if they had not given any of it to me.

Once I was visiting an office products manufacturer and discussing their new project. The hardware team had chosen to use a processor for which there was no solid compiler suite, no existing strong web or network or other components etc. because it was cheap. I explained that the choice of processor was fine with us, but that our prices for providing basic RTOS support would be high due to the necessity of compensating for this environmental limitation. The team broke into a raucous argument between the software developers and the hardware engineers – with the software people basically yelling “we told you idiots!”.

One of the things that struck me at a Japanese auto company was how the manufacturing discipline that allowed them to make such high quality hardware was not even attempted with software. Software is still not taken seriously as a manufacturing area – and the consequences are not happy ones. See the article by founders of Coverity for a great tour of the strange views tolerated within software development

Do bugs matter? Companies buy bug-finding tools because they see bugs as bad. However, not everyone agrees that bugs matter. The following event has occurred during numerous trials. The tool finds a clear, ugly error (memory corruption or use-after-free) in important code, and the interaction with the customer goes like thus:

“So?”

“Isn’t that bad? What happens if you hit it?”

“Oh, it’ll crash. We’ll get a call.” [Shrug.]

Kendo enforces a deterministic interleaving of lock acquisitions and specially declared non-protected reads through a novel dynamically load-balanced deterministic scheduling algorithm. The algorithm tracks the progress of each thread using performance counters to construct a deterministic logical time that is used to compute an interleaving of shared data accesses that is both deterministic and provides good load balancing. Kendo can run on today’s commodity hardware while incurring only a modest performance cost ( http://www.gigascale.org/pubs/1883/asplos073-olszewski.pdf)

There is similarly motivated work on Java going on at UIC: http://dpj.cs.uiuc.edu/DPJ/Home.html

Both works refer to Lee’s paperwhich I discussed earlier. Nondeterministic threading is a  historical accident in computer engineering. Operating systems introduced time-sharing methods so single thread programs could be run in parallel during I/O delays and then so that multiple users could reasonably fairly share a processor and then so the OS and service programs could provide multiple services to a user at the price of slowing down user applications.  Exposing this system to users has been a mixed blessing. Certainly in the controls world, non-determinism is a dangerous fault.

Consider a simple design:

1. Task A: collect data from an Analog/Digital converter, sampling at rate 1/t seconds and filling a small buffer (1/t)*n seconds.
2. Task B:  Aggregate A data and produce 3 streams of processed data that are functions of the input data and some settings.
3. Task C:  produce control data from the processed data at an output rate of 1/t2 seconds
4. Task D: provide a secure internet port which will process requests to interrogate data, to setup data push at some rate maximum 1/t3 pushes per second and update the settings with worst case delay from arrival of input packet to reset of 1/t4 seconds.
5. Task E: operate a touch screen display that has some of the same functionality as task D

That seems like it should be straightforward, but it’s not: it’s a one year project that has a 70% chance of failing. Making sure shared data is efficiently and correctly shared, validating that the scheduling and timing is ok, and optimizing for hardware limits and power are all hard.  And that seems wrong because there is nothing in such a project that has not been done thousands of times before.  As the electric power industry stumbles into modern software based control,  the slow development times and poor reliability of products developed under this model is a serious problem.

* NOTE: RTLinux is a trademark of WindRiver Systems.

And like Intel, I would argue that mobile software companies are instrumental in making silicon solutions pervasive, because they tick two major check boxes: reference design and support.

The hidden asset of mobile software companies
Mobile embedded software companies (e.g. Myriad, Access, Aplix, NXP software, Azingo) have a unique understanding of products as a hardware/software system. They understand how silicon can be leveraged, encapsulated and productised to fast-track device delivery. Beyond specific IP, these companies have the capability to build the very first product that will feature a new chipset. This is the reference design check box.

A reference design is not a half-baked breadboard; it is a scale 1:1 device, ready to ship.

The absence of such capability has drained Texas Instrument, once the mobile silicon leader. Alternatively, true reference design has been instrumental in the success of Qualcomm and Mediatek.(cite)

Also see previous note

If anything, Wind River’s inability to breakout, despite a once Microsoft-like position of dominance, is a by-product of their failure to meaningfully go “up the stack” and away from their historical focus on the silicon layer as a primary differentiation point.

In other words, if Wind River had enabled the next generation of Cisco and Apple killers by providing more differentiated OEM-in-a-Box offerings, ala what Google is now trying to do with Android, they would not be staring at a $900M market cap and relatively flat revenues, margins and stock price.

Well, that’s an argument. Embedded is a tough beat, but Wind did not exactly set the world on fire developing new concepts and software. Intel’s history in software is not glorious either – so it’s going to be an interesting challenge for the new entity.

I’m not a fan of the “algebraic” tagged model that Lee suggests as an alternative – too much of the flavor of “formal methods” via the denotational semantics base.  In fact, Liu’s thesis, referenced here, struggles mightly with the unsatisfactory nature of the mathematical framework to get somewhere. Do we really have to create lemmas about posets to describe 2 simple processes? It seems to me that the confusion of the underlying mathematical basis has to be resolved before we can figure this out. Or maybe not.

Consider a really dumb system that keeps a “gain” variable, outputs a single analog voltage, and inputs both an input signal and “messages” that tell it what to do. The input events can be thought of as samples of the analog signals on the “pins”.

Inputs are maps: Pins -> Signals  where each input represents one unit of time and the set Pins contains “Vin” and some unknown number of pins (which may not be actual physical pins) comprising the message port.

Given a sequence of inputs, w, we suppose we have a function LastGain(w) that extracts the value of gain we were told to set in the most recent message and SinceLastCommand(w) that counts samples (time units) since the last command message.  Let’s be more specific on the “Vin”

LastVin(wa) = a(Vin).

StableVin(wa) = (1+ StableVin(w))*[a(Vin) ==  LastVin(w)]  where [exp] is 1 if exp is true and 0 otherwise

StableVin(emptysequence)=0

Then we can require that Dev(w)= g*v if

StableVin(w)>= t1 and v=LastVin(w)

and g=LastCommand(w) and SinceLastCommand(w) >= t2

From my point of view, a large part of the evolution in this market has been a painful and expensive process of learning that chip and device companies and operators cannot cost effectively dabble in the operating system business when the devices get to the current level of sophistication.

(painting above by Goya shows management of very large multinational who made a lot of bad decisions. This painting is in Madrid, but has little to do with the OSIM conference exactly.)