Some old stuff I found in the office
Category: rtlinux
Adding real-time support to general purpose operating systems
What is claimed is:
- A process for running a general purpose computer operating system using a real time operating system, including the steps of:
providing a real time operating system for running real time tasks and components and non-real time tasks;
providing a general purpose operating system as one of the non-real time tasks;
preempting the general purpose operating system as needed for the real time tasks; and
preventing the general purpose operating system from blocking preemption of the non-real time tasks.
An early RTLinux paper with Michael Barabanov.
real_time_1996
Only 20 years later, the idea is now not too scary.
Real-Time Linux
Michael Barabanov Victor Yodaiken March 3, 1996
1 Introduction
If you wanted to control a camera or a robot or a scientific instrument from a PC, it would be natural to think of using Linux so that you could take advantage of the development environment, X-windows, and all the networking support. But, Linux cannot reliably run these kinds of hard real-time applications. A simple experiment will illustrate the problem. Take a speaker and hook it up to one of the pins from the parallel port. Then run a program that toggles the pin. If your program is the only one running, the speaker will produce a nice somewhat steady tone. Not completely steady, but not bad. When Linux updates the file system every couple of seconds, you might notice a small change in tone. If you move the mouse over a couple of windows the tone becomes irregular. If you start netscape in one of the windows, you will hear intervals of silence as your program waits for higher priority processes to run.
The problem is that Linux, like most general purpose operating systems, is designed to optimize average performance and to try to give every process a fair share of compute time. This is great for general purpose computing, but for real-time programming precise timing and predictable performance is more important than average performance. For example, if a camera fills a buffer every millisecond, then a momentary delay in the process reading that buffer may cause data loss. If a stepper motor in a lithography machine must be turned on and off in precise intervals in order to minimize vibration and to move a wafer into position at the correct time a momentary delay may cause an unrecoverable failure. And consider what might happen if the task that causes an emergency shutdown of a chemistry experiment must wait to run until Netscape redraws the window. It turns out that redesigning Linux to provide guaranteed performance would take an enormous amount of work. And taking on such a job would defeat our original purpose. Instead of having an off the shelf general purpose OS, we would have a custom made special purpose OS that would not be riding the wave of the main Linux development effort. So what we did was slip a small, simple, real-time operating system underneath Linux. Linux becomes a task that runs
only when there is no real-time task to run and we pre-empt Linux whenever a real-time task needs the processor. The changes needed in Linux itself are pretty minimal. Linux is mostly unaware of the real-time operating system as it goes about its business of running processes, catching interrupts, and controlling devices. But real-time tasks can run to a quite high level of precision. In our test P120 system, we can schedule tasks to run within a precision of about 20 microseconds.
And more ….
Categories
Real-time Linux
My opinion has always been that the Linux-RT project was based on an unfixable engineering error.
A few words on the status and the future of RT: ----------------------------------------------- The situation since last years RTLWS (https://lwn.net/Articles/572740/) has not improved at all, it's worse than before. While shortly after RTLWS quite some people promised to whip up proper funding, nothing has materialized and my personal situation is worse than before. I'm really tired of all the politics involved, the blantant lies and the marketing bullshit which I have to bear. I learned a few month ago that a certain kernel vendor invented most of RT anyway and is the expert in this field, so the customers dont have to worry about my statements. Just for the record: The initial preempt-RT technology was brought to you mostly by Ingo Molnar, Steven Rostedt, Paul Mckenney, Peter Zijlstra and myself with lots of input from Doug Niehaus, who researched full in kernel preemption already in the 1990s. The technology rewrite around 3.0-rt was done by me with help from Peter and Steven, and that's what preempt-RT today is based on. Sure, people can believe whatever marketing bullshit they want, but that doesn't make the truth go away. And the truth is, that those who claim expertise are just a lying bunch of leeches. What really set me off was the recent blunt question, when I'm going to quit. What does this mean? Is someone out there just waiting that I step down as preempt-RT maintainer, so some corporate entity can step up as the saviour of the Linux RT world? So instead of merily leeching someone seeks active control over the project. Nice try. http://lwn.net/Articles/604632/
For some reason, all copies of an early variant of RTLinux called “myrtlinux” by its Italian “author” have disappeared from the web, but thanks to some archives we can find some fragments from old days. For example, within a year or two of this email, the story had changed rather dramatically. To me, there were important lessons about the memories of academics who could get grants by not remembering and the principles of “free software” advocates who could get paid by being indignant in the right direction. (sorry if this is too cryptic, but I want to get some of this material back on line for a variety of reasons).
From: mante () aero ! polimi ! it (Paolo Mantegazza) Date: 1998-03-16 15:52:14 [Download message RAW] Hello, fresh news on using the FPU within RTLinux. I've been able to convince a collegue, performing active vibration control of plates to decrease the transmitted noise, to use RTLinux with the FPU support of my variant. (I was smart in having the other take the risk, wasn't I?). Please note that is the same technique I posted some time ago for an rt_prio_sched of the standard release. They are sampling the response of three piezosensors, carry out some signal conditioning and simple compensation and evaluate a direct proportional feedback to actuate three piezoactuators. They use a 200 Mhz plain Pentium and conditioning and feedback are performed by using the FPU without any problem, with X running and with the system lively, at 18 Khz. The signal are checked on a scope. For those interested the piezo sensors and actuators are three ceramic disks bonded to the panel. The DA acquisition are carried out with a DAS1600 card along with two DA conversions, the third DA output with some other card whose brand I forgot. This is a preliminary activity in view of the implementation of a neural controller. The only way they found to crash the system was to unplug the computer by stepping onto a flying socket. After these simple tests their doubts on using RTLinux are fading away and I am happy that they can trust me somewhat more now. I have not checked the system personally yet, but this is just a confirmation of my testings. Having understood that "fixed FPU support" meant "sound and working" I'm eagerly waiting for the new 2.2.xx related RTLinux release with a native improved FPU support. Ciao, Paolo.
Categories
More petty
The ALMA team has released ACS 8.0 on Red Hat 4.4, downgrading the Linux version from the foreseen 5.2 version. This choice, with the consequent back-porting of the code to the older OS version, had to be taken because of major problems encountered by the Control and Correlator teams in porting RTAI real time code to Red Hat 5.2. As soon as these problems will be solved, ACS 8.0 will be released also for Red Hat 5.2. Report
These folks deliberately chose to base their work on a version of software we developed that had somehow been transformed to have none of our copyright notices, ignored our complaints about their lack of interest in scientific integrity, and refused to even discuss a commercial solution with the inventors, and have by now invested untold amounts of public money trying to keep the mess working.
[ revised version of an older post]
“Soft real-time” is a perfect example of the “soft design” noted in an earlier post. There are perfectly good ways of characterizing quality of service (QOS) assurances precisely. Doug Jenson proposes one possible definition:
The general case of a deadline (which is a soft deadline) has utility measured in terms of lateness (completion time minus deadline), tardiness (positive lateness), or earliness (negative lateness). Larger positive values of lateness or tardiness represent lower utility, and consequently larger positive values of earliness represent greater utility.
That is, he says a real-time system will have a function
0 =< Utility(Error) =< 1
where, Error > 0 for a late computation and Error < 0 for an early computation. A hard real-time system will have Utility(Error)=0 whenever |Error| is greater than some acceptable limit. That’s a good start, and there are other obvious ways to quantify. In practice, utility functions may require history: dropping the fourth frame of video during a 1 second interval is different than dropping the first frame during that period. We might specify 75 frames/second which means about 13 milliseconds a frame for flicker free video. Then we could say that the average is x frames per second and there are no outliers more than n standard deviations from the mean. Or we could require that over any interval of time t seconds there will be at most n delays of more than E milliseconds. There will be a big difference between requirements for editing (which may permit no frame delays more than 100 microseconds) and consumer viewing which will be a moving target but may allow dropping frames that are more than 2 milliseconds late, but not permit more than 1 frame to be dropped every 2 seconds or something like that. Note that one of the distinguishing characteristics of any type of real-time system is that timing is not subject to amortization. Being 10 seconds early and then 10 seconds late, does not mean that the system is perfectly on time.
In practice, we rarely see quantitative specifications of real-time behavior in “soft real-time” systems probably because such specifications would reveal the engineering flaw in most soft-real-time systems. In order to make any QOS assurances you need to be able to make hard assurances and as soon as specifications are written down in any detail at all, this inconvenient problem becomes all too clear. If our specification is that over a one second period, no more than 2 frames will be more than 100 microseconds late, then if the first two frames come in 110 microseconds after deadline, the all of the remaining frames in the second must be under 100 microseconds late. The distinguishing characteristic of “soft” real-time, in practice, is that a soft-real-time system can tolerate some timing errors before falling back on a more rigid timing requirement. Here’s a summary.
A hard real-time system has firm worst case timing properties that must always be met to avoid failure
A soft real-time system is a hard real-time system with some recoverable error modes
But if we accept the above definition, then designing “soft real-time” systems will be seen to be more demanding than designing hard real-time systems. Specifications like “seems peppy” will no longer be acceptable and the types of sloppy mechanisms that can be shown to reduce the tail of distributions in some circumstances will be less appealing.
Spent a couple of very interesting days at the OSIM conference in Madrid as part of my consulting for WindRiver which has a very powerful market position in cellular handsets now – partly due to their acquisition of RTLinux for embedded last January. Interesting to see these companies negotiate the vast complexity of the technology being deployed in handsets and the equally complex picture of software intellectual property and the famous “value line” – which really has way too many dimensions to be reduced to a line and should be renamed the “value manifold”. Both Trolltech and Funambol touted dual-license models where the some feature of the open source product pushes possible customers to buy non-open source licenses. You could cynically characterize these models as “we get free market presence and the people who violate the license would have stolen the software anyway.” Pragmatic. People from Gnash were advocating a pure open source model on the old fashioned “sell services” plan – claiming that the embedded flash environment was so fractured and that services were needed for many projects. In both these cases: my understanding may differ from their understanding, so please don’t ascribe my opinions to anyone else.
From my point of view, a large part of the evolution in this market has been a painful and expensive process of learning that chip and device companies and operators cannot cost effectively dabble in the operating system business when the devices get to the current level of sophistication.
(painting above by Goya shows management of very large multinational who made a lot of bad decisions. This painting is in Madrid, but has little to do with the OSIM conference exactly.)
Categories
Security notes
David Elliot Bell has an interesting essay on the US government and software security – little “inside baseball”, but informative. He cites my critique of MILS. One of Bell’s points, which is briefly mentioned in my note, is that composition is harder than many imagine. Systems A and B may meet some security level individually, but not when connected to each other.
Categories
Plagiarism watch
Thanks to google code watch I see that an open source real-time Linux project has reached new heights of originality – changing the magic number that they copied from RTLinux 10 years ago to a new number. The constant definition
#define RT_TASK_MAGIC 0x754d2774
which was in very early versions of RTLinux and appeared every previous version of the claimed “completely original” RTAI releases has been replaced by
#define RT_TASK_MAGIC 0x9ad25f6f
And people say there is no originality in open source software! For the non-programmers, a “magic number” is a elementary way of identifying a data structure to help in debugging – chosen to be unusual but not bound by any functional requirement at all. It’s just an arbitrary marker. The appearance of 2 identical magic numbers, with identical symbolic names, used in the identical way, in similar data structures, in code that essentially does the same thing can only be due to copying. Of course, this is a well known example, we’ve been pointing it out to people for years.