As a visitor to Japanese auto companies 10 years ago, I was blown away by the brilliance of the industrial design and the enormous investment in that, and shocked at the software development effort – one in which the software team was clearly considered to be low status. Not that my, more limited, experience with US companies was much more impressive. In fact, it was interesting that as Japanese auto companies were reaching out to at least look at advanced control software, US companies relegated that work to the outer periphery. Of course, this was some time ago, but obviously a $100M investment in top level control software development and test at Toyota a decade ago would have been a good investment – even if they had not given any of it to me.

Once I was visiting an office products manufacturer and discussing their new project. The hardware team had chosen to use a processor for which there was no solid compiler suite, no existing strong web or network or other components etc. because it was cheap. I explained that the choice of processor was fine with us, but that our prices for providing basic RTOS support would be high due to the necessity of compensating for this environmental limitation. The team broke into a raucous argument between the software developers and the hardware engineers – with the software people basically yelling “we told you idiots!”.

One of the things that struck me at a Japanese auto company was how the manufacturing discipline that allowed them to make such high quality hardware was not even attempted with software. Software is still not taken seriously as a manufacturing area – and the consequences are not happy ones. See the article by founders of Coverity for a great tour of the strange views tolerated within software development

Do bugs matter? Companies buy bug-finding tools because they see bugs as bad. However, not everyone agrees that bugs matter. The following event has occurred during numerous trials. The tool finds a clear, ugly error (memory corruption or use-after-free) in important code, and the interaction with the customer goes like thus:

“So?”

“Isn’t that bad? What happens if you hit it?”

“Oh, it’ll crash. We’ll get a call.” [Shrug.]

Re: Schedule idle
MOLNAR Ingo (mingo@chiara.csoma.elte.hu)
Wed, 11 Nov 1998 04:09:32 +0100 (CET)
[...]
> _please_ We can do better than this. Only semaphores (not spinlocks) need
> to have the priority inheritance. [...]
nope there are _not_ only semaphores, but many other types of locks.
> [...] This can be done with lists off the
> semaphore and tasks… [...]
it’s _not_ easy to extend Linux semaphores to handle priority inheritance. currently semaphore operations can be done via hw-atomic test-and-set instructions. If we do anything more complex, we cannot use simple instructions anymore. Linux semaphores are 2 instructions for an up() and 2 for a down(), and thats one of our crown jewels

the whole point is not quite valid, RT and filesystem IO doesnt mix well anyway … the solution: use system calls that are guaranteed to not block, either by design, or by system policy (ie. separate filesystem on a RAMDISK) … or use a device that doesnt introduce large latencies. (RAMdisk or solid state disk)

And then ….

Kendo enforces a deterministic interleaving of lock acquisitions and specially declared non-protected reads through a novel dynamically load-balanced deterministic scheduling algorithm. The algorithm tracks the progress of each thread using performance counters to construct a deterministic logical time that is used to compute an interleaving of shared data accesses that is both deterministic and provides good load balancing. Kendo can run on today’s commodity hardware while incurring only a modest performance cost ( http://www.gigascale.org/pubs/1883/asplos073-olszewski.pdf)

There is similarly motivated work on Java going on at UIC: http://dpj.cs.uiuc.edu/DPJ/Home.html

Both works refer to Lee’s paperwhich I discussed earlier. Nondeterministic threading is a  historical accident in computer engineering. Operating systems introduced time-sharing methods so single thread programs could be run in parallel during I/O delays and then so that multiple users could reasonably fairly share a processor and then so the OS and service programs could provide multiple services to a user at the price of slowing down user applications.  Exposing this system to users has been a mixed blessing. Certainly in the controls world, non-determinism is a dangerous fault.

Mr. Myhrvold: All of this fear is from people who have guilty knowledge of their own actions. There are lots of major tech companies that grew from zero to gigantically successful in a very short period of time without investing in their own inventions. They got there by using other people’s inventions.

[cite]

No kidding.

Even though it has no weight and is invisible, a program is device of sorts.  If we think of a program as a mathematical object, the methods of formal logic – of meta-mathematics – are the natural methods to use. If we think of a program as a device, like a piano or a bathtub or an automobile, then what mathematicians call “foundational questions” are far away. People are very resistant to the idea of a program as a “thing” for the obvious reasons, but mathematical objects don’t have the weird properties and defects of manufactured objects (or physical objects for that matter). A ball bearing is not a sphere and an implementation of a stack is not an ordered sequence. Recursion in programs is different from recursion in mathematical functions: f(0):=1, f(x+1):= (x+1)*f(x) is  a complete definition of a mathematical object but only an approximation of some of the properties of the program written the same way. Any programmer will know that f(10000) almost certainly fails – and that’s an important part of the specification of the program.

The confusion between programs and mathematical objects is a pervasive obstacle in fields as apparently unrelated as program verification, software pricing, and patent law. In program verification we bog down in foundational methods of formal logic because that is the obvious tool to study mathematical objects, but it’s certainly not the obvious tool for doing mathematics or describing manufactured objects.   In software pricing, the resistance of manufacturers to let mere software figure in to costs in terms of how complex it is and how much it adds to value rather than by weight repeatedly leads to product development schedules that invest too little time and/or money in the most important components. In patent law, confusion between unpatentable mathematical methods and technology for programs is used to deny software innovators the same rights that are uncontroversially granted to innovators who design cardboard boxes