Category: business

Programmers will readily tell you that “Windows sucks” or “Linux sucks” or “POSIX sucks” why this or that software is badly designed, bloated, slow, buggy, un-needed, ugly, and generally disgusting. But, Windows is actually an immensely useful and sophisticated program with astounding capabilities – Linux is too. POSIX is a comprehensive solution to many difficult problems. These programs and APIs are widely used because they solve real problems. Large, complex, “high capability” software is never as simple and clean as it seems like it should be to programmers, but I think that’s because the Le Corbusier fallacy is so hard to resist. It is possible, with some effort and skill, to write a prototype Operating System or Data-base or web browser that is fast, lean, clean, and beautifully structured. But the demands of the applications will conflict with the clean etc. design and force all sorts of changes. The Le Corbusier fallacy is the belief that the aesthetics of the architect trump the sloppy requirements of the mere users. In actual architecture, this belief results in unlivable high rise slums , but in programming we get either failed projects or “prototypes” that don’t actually serve any purpose. Programmers are too ready to dismiss working code that actually does something useful because they are entranced with imagined clean, pure, designs. I’m certainly not arguing against designing software or in favor of sloppy design, far from it. Software design must be motivated by external considerations – the utility of the software, not by some internal idea of what is “clean”.

The telecommunications stack we have been building and starting to deliver to customers is an unbelievably complex software structure that is capable of incredible feats. Imagine a Windows application, using .Net and SQLServer that maintains customer data, a SuSe Linux based Oracle SS7 application, a fault tolerance and remote maintenance scheme running on Carrier Grade Linux, and large number of pedal-to-the-metal RTLinux POSIX threads maintaining VOIP connections – all on a single multicore multiprocessor connected by shared memory with TCP/IP running over it. This system is not something that can be easily captured in some whiteboard drawing of boxes. It’s a city, not a housing project.

Paul McKenney once wrote:

Despite such complications, priority inheritance works reasonably well for exclusive locks, and is a major component of Ingo Molnar’s CONFIG_PREEMPT_RT patch. There are strongly held opinions both for and against priority inheritance, for example. http://www.linuxdevices.com/articles/AT7168794919.html

in which Victor Yodaiken considers priority inheritance to be harmful, and, as near as I can tell, soft realtime to be irrelevant. […] The big advantage of priority inheritance is that it is simple for its users. Use of priority inheritance does degrade scheduling latency compared to a carefully hand-crafted solution, and priority inheritance’s implementation is difficult for reader-writer locks, to say nothing of memory allocation or communications primitives.Nevertheless, priority inheritance does seem to have a significant role to play in mainstream “metal hard” realtime. It is not perfect, but, then again, what is?

And I think this is a good summary of the differences between the “soft”( muddle through) and “hard” (design properly) approaches to real-time. In FSMLabs design approach, “simple for users” is good, but not good enough to overcome “doesn’t work reliably”. Priority inheritance itself is not a big deal: we can add it, and have added it, as a more complicated mode for mutexes when needed to make legacy code port. But in our world, correctness is more important than convenience. Perhaps only highly qualified engineers who understand timing should write real-time code directly: others can use pre-made components or automatic tools (like Matlab simulink) or wait until there are more advanced test tools. The idea of using an engineering technique known to be incorrect simply because it is convenient, is I think an indication of the weak state of software engineering. This same caution applies elsewhere: the effects of kernel programmers trying to write web pages or source code management methods on the fly are also not pretty. It would be convenient if I could develop data base systems without understanding data base theory – but I can’t. McKenney’s “despite such complications” follows a note of some of the fatal problems I had covered in the cited article and elsewhere – but a solution to these problems is nowhere found.

The weakness of the soft real-time approach results in the weakness of the PREEMPT_RT systems in Linux (although it is clear that the developers are starting, finally, to copy a working solution – RTLinux and to slowly abandon their design.) Anyways, a draft analysis of what is wrong with PREEMPT_RT can be found here. Although I doubt McKenney is consciously modifying his argument on the basis of the business model of his employer, it is interesting to consider who profits from a technology that requires customers to both purchase large amounts of engineering services (to tune and configure) and to over-specify hardware.

It is now possible to put together sophisticated and powerful embedded and control systems that are mostly composed of pre-existing working software. One of the demonstrations we showed at AMD’s recent embedded workshop was a small two processor (4 core) box running RTLinux and Windows XP. Embedded development teams for RTCore based products can mirror the decoupling of the RTCore architecture – one team can do the low level control application and other teams can work on all the non-time-dependent “enterprise” software. A printer controller might be developed by connecting a real-time component under RTCore that talks to sensors and actuators, a database and remote update/maintenance subsystem under Linux and an operator interface running under Windows XP. These three components can be developed independently by different teams as long as architects have correctly designed the interfaces. The RTCore component on current Opterons can guarantee either sub 10 microsecond or 1 microsecond worst case interrupt latency – depending on whether you share the processor core with Linux or dedicate a processor core to real-time. The Linux application could be made web accessible using Apache, for example, with a backend database either MySQL or Oracle or even DB2. The Windows application can go right to the video and our experience is that the double buffering of the Linux file system and Windows itself improves performance when running a virtualized windows kernel. A Sun Java application on the Linux side may connect to C# under Windows on one side and C++ or C executing in the RTCore environment on the other.

Let’s look at two example applications – a printer and an automobile “infotainment” system. One way to build a printer is as a system [FPGA: RTCore: Management: Operator] where RT software in RTCore manages a controlling FPGA on one side (that manages the sub microsecond requirements of the hardware) and provides paths for the Linux based Management and Windows based Operator subsystems to submit commands and get data. For a lighter infotainment system, perhaps using a single processor, we might need [Devices: RTCore: Display] where the RTcore component acts as a device accelerator, replacing dedicated hardware (such as a smart MPEG ) with software, a Windows based Display manages multimedia, and Linux is just used essentially as a BSP.

Security can be easily improved by making the three components cross check each other. For example, a remote update connecting to the Java based server under Linux may ask the Windows based and Rtcore based components to cross check certificates. An attacker would have to deal with three different operating system environments working together. Reliability can also be improved – e.g. by having Rtcore force a reboot of Linux or Windows if progress is not being made.

The underlying theme here which shows how far this new type of embedded software has moved from the traditional “hand crafted” embedded development process. Programmers have large functional components available to use in development instead of starting from the traditional impoverished low level embedded environment. The implications for both productivity and creativity are significant.

FSMLabs is finally able to discuss the single core cell phone handset solution we developed with Infineon Technologies last year. This system is cool technology – literally – since it allows parts count reduction on the handset and that reduces power consumption and that reduces heat production. More to the point, in a volume driven business, removing some hardware components is big news. I hope that some of the “no software royalties” hard liners will now pause and consider why paying royalties (or a production license) in order to not have to ship chips and batteries for those chips may be, you know, profitable.

The technology is very interesting. Essentially, we squeeze a lot of performance out of an ARM9 – making it jump back and forth between user interface, radio control, and wireless stacks. Turns out, the ARM9 is up to the job – it was being held back by poorly designed software.

There have been many product announcements of the imminent release of Linux based single core cell phone solutions. I think that in this industry people sometimes forget that there are hard technical problems. Just having a visionary CEO say “make it so” and let loose the loudly singing marketing flock doesn’t actually get the bits to move right. Our software worked because of the smart engineering effort of Cort Dougan, Michael Barabanov, and Zwane Mwaikambo and the engineers they supervised and the work of the engineers at Infineon and also the excellent evaluation work done by Infineon engineering management that allowed them to avoid the mistake that everyone else made in searching for paths to this product. (of course there is a lot more to the phone than FSMLabs work, but RTLinux is agile enough to move the processor to the right place at the right time.)