It’s not like I didn’t see doomed robotics projects in the US National Labs.

“This will blow away any RTOS,” said Cauchy. “We speed up execution by doing things like vector interrupts, I/O memory mapping, and turning off timers.”

By Gad!

The ALMA team has released ACS 8.0 on Red Hat 4.4, downgrading the Linux version from the foreseen 5.2 version. This choice, with the consequent back-porting of the code to the older OS version, had to be taken because of major problems encountered by the Control and Correlator teams in porting RTAI real time code to Red Hat 5.2. As soon as these problems will be solved, ACS 8.0 will be released also for Red Hat 5.2. Report

These folks deliberately chose to base their work on a  version of  software we developed that had somehow been transformed to have none of our copyright notices, ignored our complaints about their lack of interest in scientific integrity, and refused to even discuss a commercial solution with the inventors, and have by now invested untold amounts of public money trying to keep the mess working.

“What appears to be a defense contractor in Bethesda, Md., had a file-sharing program on one of their systems that also contained highly sensitive blueprints for Marine One,” Boback said.

Tiversa also found sensitive financial information about the cost of the helicopter on that same computer, WPXI-TV reported.

I’m not a fan of the “algebraic” tagged model that Lee suggests as an alternative – too much of the flavor of “formal methods” via the denotational semantics base.  In fact, Liu’s thesis, referenced here, struggles mightly with the unsatisfactory nature of the mathematical framework to get somewhere. Do we really have to create lemmas about posets to describe 2 simple processes? It seems to me that the confusion of the underlying mathematical basis has to be resolved before we can figure this out. Or maybe not.

Consider a really dumb system that keeps a “gain” variable, outputs a single analog voltage, and inputs both an input signal and “messages” that tell it what to do. The input events can be thought of as samples of the analog signals on the “pins”.

Inputs are maps: Pins -> Signals  where each input represents one unit of time and the set Pins contains “Vin” and some unknown number of pins (which may not be actual physical pins) comprising the message port.

Given a sequence of inputs, w, we suppose we have a function LastGain(w) that extracts the value of gain we were told to set in the most recent message and SinceLastCommand(w) that counts samples (time units) since the last command message.  Let’s be more specific on the “Vin”

LastVin(wa) = a(Vin).

StableVin(wa) = (1+ StableVin(w))*[a(Vin) ==  LastVin(w)]  where [exp] is 1 if exp is true and 0 otherwise

StableVin(emptysequence)=0

Then we can require that Dev(w)= g*v if

StableVin(w)>= t1 and v=LastVin(w)

and g=LastCommand(w) and SinceLastCommand(w) >= t2

Years ago I proposed the following code snippet as a minimal standard for a useful verification method. Still not quite there.

/* you are not expected to understand this */

if(save()){

          load_memory_management(); /* map in new current */

          return;

}else {

         current = new;

        resume();

        panic("RESUME RETURNED! KERNEL BROKENn");

}

Not so complicated, but if you can’t do this, you cannot handle operating systems at all- or any threading.

In response to the comment. This is a paraphrase of Dennis Ritchie’s  Unix switch code and comment.  “Save” saves the registers and the return address for “current” and always returns 0. “Resume” restores the saved registers and return address for current (by doing so, changes stacks) and returns 1 – to that saved return address which is the test of the “if”. So the “if” statement is always evaluated as false when saving and is always evaluated as true when resuming and “resume” never returns to the else.  I always took Ritchie’s comment to be a warning that the code and the previous comments were not easy to understand. The state change here is basic thread switching but it is slippery.

John Regehr writes:

On the other hand, there is plenty of useful work to be done on supporting time sensitive applications (I’ll just avoid saying “soft real-time”) even when no guarantees are made. Let’s take the example of an OS that disables interrupts when any process is executing inside the kernel. This is not hypothetical, I was recently talking to people designing such an OS because they are formally verifying it; they did not believe they were capable of verifying an OS that accepts interrupts more liberally. Anyway, needless to say the average-case response time to interrupts of this OS will suck. Now if we have some breakthrough in verification that permits interrupts to be accepted most places in this kernel, average case responsiveness will improve though worst-case may
not. Is this useless? Of course not, if it makes it possible to watch videos or listen to music or similar, on this OS. For applications of this kind a mathematical guarantee is simply overkill and what we really care about is more like “works almost all of the time in practice, given actual workloads.”

This is perfectly reasonable, in fact, there are many interesting properties of operating systems that are not real-time properties at all if we define real-time properly so that it does not end up meaning “any performance properties.” What’s unreasonable is this idea that crippling a piece of software to make it possible to “prove” that it “works” gains much of anything. That story about the drunk looking for his keys under the light-post is supposed to be a joke, not a design methodology. It’s fine to argue that by simplifying a system one can validate it while not losing too much performance or functionality for the purpose – but all too often the second part of the argument is omitted. What is the point of a system that is provably “correct” at a performance level that assures it cannot be used for its ostensible purpose? Worse, this methodology is often used in order to make it possible to apply “mathematical” methods which have not been ever shown to be more reliable than testing or code reviews or even good luck charms (the claim that “formally validated code is better than other code is, at best, unproven).

The phrase “mathematical guarantee ” is worth some deeper analysis.

• Mathematical. The types of “hard specifications” that I am advocating do not need to be overly “mathematical” and certainly don’t need to be “formal” to be useful. On the contrary. The “formal methods” advocates have made a fetish of formalization to the detriment of useful engineering specification and that causes people to shy away from attempting to be more precise. What we want from a specification, at minimal, is that it should be falsifiable. Perhaps an incremental approach to adding fast paths to operating system services to reduce interrupt latencies can make the system described above “more responsive” for “standard workload”. If so, we should be able to characterize “more responsive” in some falsifiable way. If the specification is just “there is no wall-clock observable delay on the echoing of typed characters while stress program S runs in the background and a tester types a file in the ed program” or that “programs A,B, and C start within 5 seconds of double clicking the icon” then it is at least a falsifiable specification and is a concrete goal for the software developer. In any case, we can see that “average” is not what is needed, since “average” implies that if the system runs for 1 hour, a 5 minute total stop can be amortized away by 55 minutes of high speed operation. Perhaps what we want is as simple as “the video player on an otherwise unloaded system will not stop more than 10 seconds cumulatively over a one hour period where stops are delays in frame display that are over 1/60th of a second.” But this “soft” real-time requirement carries a real-time property with it – as argued previously.
• Guarantee. There is a difference between a design guarantee and an evaluation guarantee. A TCP/IP network stack is designed to produce an in-order, reliable, stream of data between sender and receiver. But it is possible that in some situations, under some loads and with some definition of “reliable”, a simple datagram stack can produce the same property. The TCP/IP stack is designed to guarantee a property that the simpler stack may be shown to have in some circumstances. An evaluation guaranty is an attempt to show that a system has some property. One of the most useful ideas in Common Criteria is the evaluation assurance level (EAL) which is supposed to indicate the reliability of such a guaranty. One might find “we ran it a couple of times and it looked good” to be less or more reassuring than “we produced a 300 page document that strenuously exercised the symbol production features of LaTex” but these are both “evaluation guarantees”. Note that design guarantees should be backed up by evaluation guarantees that the design is correct and implemented correctly.

“Real-time software” normally refers to software that offers design guarantees on latency and scheduling times, data rates, or other time sensitive operations. Validating that the design succeeds requires some combination of testing, mathematical proof, and design and code review. But we may also be able to validate real-time properties of code that has no design real-time properties at all. One could take an instance of a LAMP stack (Linux + Apache + MySQL + PHP) and subject it to extensive testing on a particular hardware platform and observe that in practice it satisfies a real-time specification. Suppose our measurements show that all pages of a certain limited complexity will be “served” within 1 second over a 1GB network that is reserved for the LAMP system and where requests are at a rate bounded by R and there are no more than 30 seconds cumulative failure on this property over 1 hour. That’s a falsifiable, concrete, real-time specification – mathematical, even [this is in response to a note from B., a very knowledgeable system designer who may or may not be quotable].

“Soft real-time” is a perfect example of the “soft design” noted in an earlier post. There are perfectly good ways of characterizing quality of service (QOS) assurances precisely. Doug Jenson proposes one possible definition:

The general case of a deadline (which is a soft deadline) has utility measured in terms of lateness (completion time minus deadline), tardiness (positive lateness), or earliness (negative lateness). Larger positive values of lateness or tardiness represent lower utility, and consequently larger positive values of earliness represent greater utility.

That is, he says a real-time system will have a function

0 =< Utility(Error) =< 1

where, Error > 0 for a late computation and Error < 0 for an early computation. A hard real-time system will have Utility(Error)=0 whenever |Error| is greater than some acceptable limit. That’s a good start, and there are other obvious ways to quantify. In practice, utility functions may require history: dropping the fourth frame of video during a 1 second interval is different than dropping the first frame during that period. We might specify 75 frames/second which means about 13 milliseconds a frame for flicker free video. Then we could say that the average is x frames per second and there are no outliers more than n standard deviations from the mean. Or we could require that over any interval of time t seconds there will be at most n delays of more than E milliseconds. There will be a big difference between requirements for editing (which may permit no frame delays more than 100 microseconds) and consumer viewing which will be a moving target but may allow dropping frames that are more than 2 milliseconds late, but not permit more than 1 frame to be dropped every 2 seconds or something like that. Note that one of the distinguishing characteristics of any type of real-time system is that timing is not subject to amortization. Being 10 seconds early and then 10 seconds late, does not mean that the system is perfectly on time.

In practice, we rarely see quantitative specifications of real-time behavior in “soft real-time” systems probably because such specifications would reveal the engineering flaw in most soft-real-time systems. In order to make any QOS assurances you need to be able to make hard assurances and as soon as specifications are written down in any detail at all, this inconvenient problem becomes all too clear. If our specification is that over a one second period, no more than 2 frames will be more than 100 microseconds late, then if the first two frames come in 110 microseconds after deadline, the all of the remaining frames in the second must be under 100 microseconds late. The distinguishing characteristic of “soft” real-time, in practice, is that a soft-real-time system can tolerate some timing errors before falling back on a more rigid timing requirement. Here’s a summary.

A hard real-time system has firm worst case timing properties that must always be met to avoid failure

A soft real-time system is a hard real-time system with some recoverable error modes

But if we accept the above definition, then designing “soft real-time” systems will be seen to be more demanding than designing hard real-time systems. Specifications like “seems peppy” will no longer be acceptable and the types of sloppy mechanisms that can be shown to reduce the tail of distributions in some circumstances will be less appealing.

If you look at Xen code, for example, you can see that it copies huge chunks of Linux. The argument seems to be that if Linux does not have security and fails to efficiently allocate resources, Linux+ModifiedLinux will do a better job, somehow. There’s no reason to believe this will happen and many reasons to believe that it will, in fact, make security more elusive. That’s not to either argue against virtualization as a useful technique: our RTMS uses a type of virtualization and we have a lot of customers who love VMWare, but the hype-ervisor is disconnected from reality. The delusion that making arbitrary divisions in software can reduce complexity is a persistent one, but it has no basis.

What is the fundamental performance limiting factor that has dominated that last 30 years of computer architecture? The obvious answer is the disparity between processor and memory/storage speed. We connect processors to cache, to more cache, to even more cache, to some bus, to memory, to disk/network, suffering costs at each step as we try to keep the damn processor busy. My guess is that memory locality is totally different now than it was back when virtual memory systems were first designed – much less pronounced. My short review of the literature shows little research on, for example, memory locality of databases or operating systems since the 1980s. (Is that wrong?) But we do know enough to understand why the entire concept of distributed shared memory is wrong although there needs to be a catchy name for this category of error. The error is to see that some feature of computer hardware makes some type of software engineering difficult and then provide a software mechanism that makes that type of engineering impossible. The complexity of the memory hierarchy and the complexity of making locality work pose enormous challenges for software designers. The purpose of distributed shared memory is to make it impossible for software designers to control the cache performance of their programs by hiding caching behavior under a non-deterministic opaque layer.

(see comment by John Regehr)