Category: rtlinux

Categories
operating systems rtlinux

Pitiful formal methods in computer science

FSMLabs is making its way through the DO178 requirements and we’ve given a lot of thought to Common Criteria over the last couple years. One of the advantages of DO178 is that it does not accept the totally untested and frankly unlikely premises of “formal methods”.

The reason mathematics has advanced so much was not because of the Euclidean axioms-lemma-theorem straitjacket, but in spite of it. Luckily, when we actually discover mathematics, we do it the Babylonian way, empirically and algorithmically. It is only when it is time to present it, that we put on the stifling Greek formal attire.

so says Doron Zeilberger

And the mystery quote. Who said this?

“That mathematics reduces in principle to formal proofs is a shaky idea” peculiar to this [20th] century. In practice, mathematicians prove theorems in a social context. […] It is a socially conditioned body of knowledge and techniques.”

Categories
rtlinux software business systems

Above all, the SUSPENSE: Wind River’s DSO with Robert Mitchum and hippies! (updated)

I’ve been meaning to write about DSO – WindRiver’s attempt to define synthetic “standards” (see the linuxdevices article for technical detail and a more serious approach). “Is DSO hype or substance?” asked Wind River CEO Ken Klein during the sponsored address, and sought to persuade the engineers in the audience that it is indeed substance and not just “marketing spin or renaming embedded systems with a three-letter acronym.” [cite: electronic news]

Robert Mitchum

Klein is seeing what many of us see – the tools and design approaches that worked for little embedded systems don’t work anymore. But because you are now able to run Linux or BSD or Windows on an embedded system, you don’t need an invented “standard” set of components – you can use the actual standardized components that have been developed for enterprise and desktop computing. There is obviously room for new components, but Java and Apache and Libc and the usual databases and Linux/BSD network stacks and so on are all useful in embedded systems. Our approach from day one in RTLinux has been to avoid reinventing those components and just add real-time capabilities to working solutions.

i am dso

The WindRiver booth at the Embedded Systems Conference in San Jose featured glowering faces waving fists that had “DSO” tatooed on them. Reminded me of “Night of the Hunter: “The film’s slogan on a major poster proclaimed: “The wedding night, the anticipation, the kiss, the knife. BUT ABOVE ALL…THE SUSPENSE!” The image showed actor Mitchum hugging a distressed Shelley Winters, with the L-O-V-E tattooed hand embracing her back, and the H-A-T-E tattooed hand grasping a knife. “(cite) So this must some of that brand management that I keep hearing about, but it’s a little too sophisticated for an engineering type to follow. If you look at the DSO web-site things get even more involved as they introduce the hippie angle: “This is the Woodstock of DSO. And just as it was with that event, 10 years from now, everybody will claim to have been here.” (here). Goodness gracious. Where is Wavy Gravy when you need him? I dunno. I was at ESC and didn’t pick up much of a sense that things were really happening in the DSO pavilion (they had some comfortable chairs, I’ll grant that).

See also ValleyWag’s rude take (on the conference and DSO, not the movie).

Categories
rtlinux software business systems

The long haul in the embedded software business.

Chris Lanfear from VDC systems asks

FSMLabs has the product line, but lacks the market presence and awareness that other companies have invested in with venture-backed capital. We believe the company has largely bootstrapped itself over the years and while this conservative approach has a number of benefits, it is difficult to achieve the market presence of a MontaVista that way. That said, the problems that MontaVista appears to have are only compounded by investors with high-return expectations. Maybe the folks in Socorro have a better plan?cite

And that’s reasonably on target. We have grown slowly, invested in technology and engineering, remain relatively clueless about “market presence”, and have a pretty straightforward business operation (make products with some compelling technical advantage, sell them at a profit, sell support at break even or better, move towards more integrated products, leverage the open source
base so we don’t re-create commodity stuff, … ). Chris underestimates the number of mission critical applications using RTLinux, but see “clueless about marketing” above. Our sales guys have to sell the stuff on price and quality – a daunting task in this market.

Chris mentions our lack of VC investment, and one of the reasons we have not taken VC investment is that we have resisted pressures to make up stories about fabulous wealth in blinks of an eye and “innovative” business models that create profits from “market presence”. We sell highly complex, technical software that requires a long slow engineering and test process. Our “business model” of looking for compelling technical advantage pushes us towards hard problems (you can’t obtain a compelling technical advantage on work that can be done easily).

Categories
operating systems rtlinux

Priority Inheritance: Hack or Error

The subject of priority inheritance has come up again on the Linux kernel mailing list and Torvalds correctly notes

“Friends don’t let friends use priority inheritance”. Just don’t do it. If you really need it, your system is broken anyway.

“Priority inheritance” is similar to “soft real-time” in that both are efforts to avoid building software to specification. I put together a detailed critique in “Against Priority Inheritance”. There are two basic problems with priority inheritance (1) the algorithms are inefficient, complex, and inherently buggy and (2) any claimed benefits can be obtained in a more durable and efficient manner via good design.

The RTLinux core does not support priority inheritance for a simple reason: priority inheritance is incompatible with reliable real-time system design. Priority inheritance is neither efficient nor reliable. Implementations are either incomplete (and unreliable) or surprisingly complex and intrusive.

Doug Locke’s response to “Against Priority Inheritance” seems to me to have to evade the issue in the standard way: read his response and see for yourself. It’s not enough to say “yes, of course you need a proper implementation” since one of the points of the paper is that proper implementations are fragile, expensive, and unreliable.

Update: If you want technical details on the can-o-worms PI opens – read the great description by Bryan Crantril at Sun.

Categories
rtlinux systems

Comparing VXworks to RTLinux

A couple of years ago, one of our salesmen asked us to comment on a comparison between VxWorks and RTLinux performance that had a prospective customer worried. When we tracked down the article, we were dumbfounded that it was being taken seriously. The article was a student paper from a mid-level class in Networks with some obvious limitations that I’m sure are known to the author. (I am very thankful my undergraduate student papers were written before the Internet so they don’t show up years later.) The paper compares the old RTLinuxFree 3.1 to VxWorks. Here are some curiosities.

  • The author writes:
    “We configured both RTOSes to use round-robin scheduling policy to determine context switch time.” That’s impressive given that RTLinuxFree 3 did not ever support round-robin scheduling.
  • Measurements show RTLinux with better priority inheritance performance! RTLinux has never implemented priority inheritance for reasons described in my paper Priority Inheritance: Hack or Error.
  • There is a comparison of interrupt latency on an MPC8260 that is dubious: we measured RTLinux 3.x with about 1/10 of the latency he shows.
  • He compares the performance of a message receive call, despite the absence of such a call from RTLinux 3.1.
  • and so on …

The paper is available on a server in Korea and it shows up near the top of searches. There is a reason we don’t do that comparison (VxWorks license), but there is no good reason that project teams seriously considering options can’t do the comparison for themselves.

Tags: , .

Categories
rtlinux

Handbook of Networked and Embedded Control Systems

You can find an article on writing real-time control loops in RTLinux inside the giant Hristu-Varsakelis, Levine Handbook of Networked and Embedded Control. The article is by Edgar Hilton, Matt Sherer and myself and covers simple loops, loops with data and control going to the non-real-time “client” and loops using the Interface Kit (XML/RPC).

Categories
rtlinux

Proving Operating Systems Correct:#8

One of the longstanding problems with operating systems is that there is no way to validate their correctness in the same way that engineers can calculate the ability of a beam to carry a weight or a wire to carry a current. Well, the problem is worse than that, because we don’t even have a precise way of stating exactly what we want the OS to do. We have APIs, a good thing, but suppose that you want to show that an implementation of pthread_create is correct. The first question is correct according to what specification and then you start trying to figure out a precise way to state the requirements implied by the man page. Not easy. At least for me. This paper is a start.

Categories
rtlinux software business

The romance of trade and fuzzy slippers

Commerce used to be more fashionable. The French historian Fernand Braudel writes that in the 1630s “ten ships a year from the Indian ocean, from Calicut, Surat, or Msulipatam, and the odd Portugese ship out of Goa were still arriving in Mocha with cargoes of pepper, cinnamon, nutmeg, and cloves“. He quotes a French trader of the 17th century about Cairo: “the number of merchants of the Indies is beyond reckoning, in coffee, incense, gum, aloes of all sorts, senna and tamarind, saffron-bulbs, myrrh, ostrich plumes, fabrics of all kinds of thread and cotton, stuffs and porcelains.” I’d like to believe that 300 years from now, someone will read “real-time operating systems from New Mexico sent for cell phones in Nice, jet engine testers in Hartford, machine tools in Shanghai, network switches in Sunnyvale, and locomotives in Germany” and marvel at the odd commerce of a forgotten age. It does seem unlikely. the software business is just not as romantic as shipping coffee and tamarind via Mocha. But I doubt that those ancient merchants in Mocha could beat us on sheer absurdity of our Internet age world mix of cultures. G. and I went to visit the Japanese owned machine tool company in Shanghai, climbed the stairs to a foyer where, as per Japanese custom, shoes were exchanged for slippers provided by our hosts, and then we tried focus on a serious business discussion, suavely drinking green tea while dressed in suits and ties wearing fuzzy Budweiser-logo slippers on our feet.
I guess it could have been worse.fuzzy slippers

Categories
rtlinux

RTLinux easy but RT hard.

I wrote a chapter for an academic book recently and the publisher sent me a “copyright assignment” form ( don’t even start) that asked me, among other things, to provide a warranty that it was perfectly safe to use the methods I described. Well, to be plain – it’s not. Real-time software gives you a plethora of opportunities for causing damage (there was that programmer in Jersey, not using RTLinux, who discovered his mistake when the placer robot started tossing 50Kg blocks of steel at him). So please let’s not have a “real-time programming for dummies” anytime soon. dummies But it’s a lot easier to write correct real-time software, even for smart and well educated people, if the operating system is not making things unnecessarily complex. In fact, that’s a good description of our mission at FSMLabs: rooting out and destroying unnecessary complexity in real-time systems. The PSDD system by Barabanov is pretty impressive at that, but here’s how to write a simple data acquisition loop in vanilla RTLinux courtesy of the “main” system mostly due to Cort Dougan. This is a quantum jump over the original “Linux module” stuff we had in the old days and it opens up many new improvements. Note that standard out even works! Proof that “simple” and “raw” are different.

The basic loop looks like”while(running){ input data; forward data; wait until next period;}. A RTLinuxPro application to implement this consists of a Linux/BSD executable dot-rtl program and some Linux or BSD post processing. The dot-rtl program usually has the form ” thread; main(){}” where the main routine initializes data and starts the thread. A simple version would output to standard out. Then you could use a command line “myprogram.rtl > logfile” or “myprogram.rtl | postprocess”to log or postprocess the data. Here is an example running at a period of 200 microseconds with data consisting of a single 32bit value and a count that can be used as a kind of timestamp. Just add some error checking and go.

#include
#include
#include

pthread_t thread;
int running = 1;
extern void get_input(int *);

void *loop(void *t)
{
unsigned int data;
unsigned long count = 0;
struct timespec next;
rtl_clock_gettime(CLOCK_REALTIME, &next);

while ((volatile int) running) {
get_input(&data); //you write this one.
printf("%ul %un", count, data);
count++;
timespec_add_ns(&next, 500 * 1000);
rtl_clock_nanosleep(CLOCK_REALTIME, TIMER_ABSTIME, &next, NULL);
}
return NULL;
}

int main(void)
{
rtl_pthread_create(&thread, NULL, loop, (void *) 0);
rtl_main_wait(); //wait for stop from user
running = 0; // not really necessary, but it seems cleaner to me.
rtl_pthread_cancel(thread);
rtl_pthread_join(thread, NULL);
return 0;

}

Categories
rtlinux

Embedded productivity and engineering skills

I was at some trade show wandering around with a cynic who was pointing out that every booth advertised productivity improvements. How many improvements we’ve made to productivity, and this has gone on for years” he said,“By now, we must be able to make incredible things with no effort.” In the embedded world, productivity is, of course, not incredible – or at least not incredibly good – but this can change. Using Linux or BSD as platforms in embedded development is a huge step although the Legacy RTOS (LRTOS) vendors and the 16bitters are fighting all the way. Our customers who come from the LRTOS world are often amazed to discover that many complex problems (like: display a graphical window on a remote machine) are trivial on a modern POSIX operating system. And the productivity gains of hardware platform independence are also big: You mean I can start software development now on a PC or reference board and just run the same software on our hardware when (or if) our hardware is ready? You mean we don’t have to start six months late?

A standard data acquisition program on RTLinuxPro is so easy it’s hard to believe.
See the post on control loops