Categories
Tag: real-time
Real-time operating systems are either a solved problem or a backwater of engineering design. Threads, semaphores, mutexes, some basic I/O, priority scheduling all of this has been more or less standardized in the POSIX 1003.13 smaller profiles (51,52) for many years. The basic programming model has not changed in years. Even FSM’s original RTOS and QNX, the two most unusual RTOS’s, are pretty similar from a programming point of view except for the split between real-time and non-real-time in our old product. My suspicion is that the programming model provided by these RTOS designs can be replaced by something better mostly because I think synchronization is a painful and error prone exercise that only gets worse as systems become more complicated. On the other hand, although many of our customers really loved it, and I think it was a huge advantage, the split mode approach in the old RTLinux* was a difficult learning experience for a lot of people.
Consider a simple design:
- Task A: collect data from an Analog/Digital converter, sampling at rate 1/t seconds and filling a small buffer (1/t)*n seconds.
- Task B: Aggregate A data and produce 3 streams of processed data that are functions of the input data and some settings.
- Task C: produce control data from the processed data at an output rate of 1/t2 seconds
- Task D: provide a secure internet port which will process requests to interrogate data, to setup data push at some rate maximum 1/t3 pushes per second and update the settings with worst case delay from arrival of input packet to reset of 1/t4 seconds.
- Task E: operate a touch screen display that has some of the same functionality as task D
That seems like it should be straightforward, but it’s not: it’s a one year project that has a 70% chance of failing. Making sure shared data is efficiently and correctly shared, validating that the scheduling and timing is ok, and optimizing for hardware limits and power are all hard. And that seems wrong because there is nothing in such a project that has not been done thousands of times before. As the electric power industry stumbles into modern software based control, the slow development times and poor reliability of products developed under this model is a serious problem.
* NOTE: RTLinux is a trademark of WindRiver Systems.
[note: edited to remove garbage characters]
If anything, Wind River’s inability to breakout, despite a once Microsoft-like position of dominance, is a by-product of their failure to meaningfully go “up the stack” and away from their historical focus on the silicon layer as a primary differentiation point.
In other words, if Wind River had enabled the next generation of Cisco and Apple killers by providing more differentiated OEM-in-a-Box offerings, ala what Google is now trying to do with Android, they would not be staring at a $900M market cap and relatively flat revenues, margins and stock price.
Well, that’s an argument. Embedded is a tough beat, but Wind did not exactly set the world on fire developing new concepts and software. Intel’s history in software is not glorious either – so it’s going to be an interesting challenge for the new entity.
This paper by Prof. Edward Lee explains something of why “threads” are such a painful abstraction. As Prof. Lee notes, threads intrinsically create unspecified program operation (which he calls non-determinism) and resource conflicts which we then attempt to “prune” via synchronization and complex tools. In an earlier note, I argued that we should design real-time multi-threaded applications to minimize the need for synchronization, but Prof. Lee goes further to point out that the thread model itself encourages a disorganized program structure. Along those lines, one of the basic difficulties in real-time application design is non-deterministic resource allocation. How can we ever be sure that, for example, a multi-threaded app where threads can allocate memory has sufficient memory for critical threads to proceed?
I’m not a fan of the “algebraic” tagged model that Lee suggests as an alternative – too much of the flavor of “formal methods” via the denotational semantics base. In fact, Liu’s thesis, referenced here, struggles mightly with the unsatisfactory nature of the mathematical framework to get somewhere. Do we really have to create lemmas about posets to describe 2 simple processes? It seems to me that the confusion of the underlying mathematical basis has to be resolved before we can figure this out. Or maybe not.
[partially completed dumb example deleted 2/20/2016]