MiFID II, GPS and UTC time

I have a post up on FSMLabs web site about the use of GPS and other satellite time for MiFID II timestamp compliance.  It’s fascinating how much effort has recently gone into trying to convince people that MiFID II will require direct time from a national lab or certified via a national lab despite the clear wording in MiFID II proposed regulations. To me, the deal is sealed in the Cost Benefit Analysis in which the ESMA regulators write

“The final draft RTS also reduces costs of the initial draft RTS proposed in the CP by allowing UTC disseminated via satellite systems (i.e. GPS receiver or the use of other satellite systems when available)”

That is not a promise one can easily walk away from. ESMA justifies the regulations with a cost/benefit analysis in which the costs for time stamping are limited by license to use GPS time.  Of course, legal reasoning and logic are not always the same, but I’m trying to figure out how ESMA regulators could claim that they didn’t mean it, or why they would have such a motivation.


 “South Sea Bubble” by Edward Matthew Ward via href=”https://commons.wikimedia.org/wiki/File:South_Sea_Bubble.jpg#/media/File:South_Sea_Bubble.jpg” Wikimedia

Kirkhart’s conjecture

Is that there is some insight into the nature of primes that comes from considering the classes of numbers that share the same set of prime factors. For example

2x3x5 = 30, 4x3x5 = 60, 2x9x5 = 90, 2x3x25 = 150, 4x3x25=300 …

all of these have the same set of prime divisors {2,3,5}

in fact, each of these classes of numbers is generated by a unique square-free-number.


Relaxed memory order

Two interesting posts on the order in which reads and writes happen and are made visible on shared memory multi-processors.  X86 has a strong memory model in which if X=Y=0 and then processor core 1 writes 1 to  X and then Y, if processor core 2 reads X and sees a 1, then its read of Y will find 1 not 0.  This “publication safety” issue is different on other multiprocessors and must be hell to implement on x86.


Multicores and publication safety 


Algebra for computer science and function nesting

The fundamental mathematical basis of computer science is that code and data are the same things. A sequence of bits can represent real numbers, integers, vectors, groups, matrices, video , audio, or programs, algorithms or even proofs. When we try to describe these systems mathematically, then it should not be surprising when we encounter a value that is also a map and a map with an image that consists of more maps.  For example, in a UNIX/MULTICS type file system if the file system is represented by a map F:Paths → Data, the data for a directory could also be considered to be or to represent a map. If F(home/snowden/passwords) is block of text, then F(home/snowden) is or encodes a map from strings to some information about where to find the file. In the original UNIX file systems F(home/snowden) is a map from strings to inode numbers. So (F(home/snowden))(passwords) is the inode number of the file that contains the passwords. UNIX style tree structured file systems are generally embedded in simpler file systems that map inode numbers to file contents S:Inode → Data. So the resolution of file names to data involves something like this

F(home/snowden/passwords) =  S(S(S((S(root)(home)))(snowden))(passwords)))

In more detail:

i1 = S(root)(home); i2 = S(i1)(snowden);  i3 = S(i2)(password);

data = S(i3). 

This kind of construct is familiar to programmers, but seems weird in math even though it’s well defined and not all that deep.


Patent 5,995,745

Adding real-time support to general purpose operating systems

A general purpose computer operating system is run using a real time operating system. A real time operating system is provided for running real time tasks. A general purpose operating system is provided as one of the real time tasks. The general purpose operating system is preempted as needed for the real time tasks and is prevented from blocking preemption of the non-real time tasks.



What is claimed is:

  1. A process for running a general purpose computer operating system using a real time operating system, including the steps of:

providing a real time operating system for running real time tasks and components and non-real time tasks;

providing a general purpose operating system as one of the non-real time tasks;

preempting the general purpose operating system as needed for the real time tasks; and

preventing the general purpose operating system from blocking preemption of the non-real time tasks.

Continue reading Patent 5,995,745

RTLinux early paper from 20 years ago

An early RTLinux paper with Michael Barabanov.

Only 20 years later, the idea is now not too scary.


Real-Time Linux
Michael Barabanov Victor Yodaiken March 3, 1996

1 Introduction

If you wanted to control a camera or a robot or a scientific instrument from a PC, it would be natural to think of using Linux so that you could take advantage of the development environment, X-windows, and all the networking support. But, Linux cannot reliably run these kinds of hard real-time applications. A simple experiment will illustrate the problem. Take a speaker and hook it up to one of the pins from the parallel port. Then run a program that toggles the pin. If your program is the only one running, the speaker will produce a nice somewhat steady tone. Not completely steady, but not bad. When Linux updates the file system every couple of seconds, you might notice a small change in tone. If you move the mouse over a couple of windows the tone becomes irregular. If you start netscape in one of the windows, you will hear intervals of silence as your program waits for higher priority processes to run.

The problem is that Linux, like most general purpose operating systems, is designed to optimize average performance and to try to give every process a fair share of compute time. This is great for general purpose computing, but for real-time programming precise timing and predictable performance is more important than average performance. For example, if a camera fills a buffer every millisecond, then a momentary delay in the process reading that buffer may cause data loss. If a stepper motor in a lithography machine must be turned on and off in precise intervals in order to minimize vibration and to move a wafer into position at the correct time a momentary delay may cause an unrecoverable failure. And consider what might happen if the task that causes an emergency shutdown of a chemistry experiment must wait to run until Netscape redraws the window. It turns out that redesigning Linux to provide guaranteed performance would take an enormous amount of work. And taking on such a job would defeat our original purpose. Instead of having an off the shelf general purpose OS, we would have a custom made special purpose OS that would not be riding the wave of the main Linux development effort. So what we did was slip a small, simple, real-time operating system underneath Linux. Linux becomes a task that runs
only when there is no real-time task to run and we pre-empt Linux whenever a real-time task needs the processor. The changes needed in Linux itself are pretty minimal. Linux is mostly unaware of the real-time operating system as it goes about its business of running processes, catching interrupts, and controlling devices. But real-time tasks can run to a quite high level of precision. In our test P120 system, we can schedule tasks to run within a precision of about 20 microseconds.

And more ….


I have seen some ultrafinitists go so far as to challenge the existence of 2100 as a natural number, in the sense of there being a series of “points” of that length. There is the obvious “draw the line” objection, asking where in 21, 22, 23, … , 2100 do we stop having “Platonistic reality”? Here this … is totally innocent, in that it can be easily be replaced by 100 items (names) separated by commas. I raised just this objection with the (extreme) ultrafinitist Yessenin-Volpin during a lecture of his. He asked me to be more specific. I then proceeded to start with 21 and asked him whether this is “real” or something to that effect. He virtually immediately said yes. Then I asked about 22, and he again said yes, but with a perceptible delay. Then 23, and yes, but with more delay. This continued for a couple of more times, till it was obvious how he was handling this objection. Sure, he was prepared to always answer yes, but he was going to take 2100 times as long to answer yes to 2100 then he would to answering 21. There is no way that I could get very far with this. Harvey M. Friedman “Philosophical Problems in Logic” (found in Wikipedia



Windows support in Timekeeper and MiFID II Compliance

TimeKeeper® PTP/NTP time synchronization software now works on  Microsoft Windows® Operating Systems including Windows 7, Windows 8 and Windows Server 2012. This has been a lot longer in coming than we originally expected because most of our early customers turned out to be Linux only. But bigger customers meant an increasing push for TK support for Windows and tightening regulation of timestamp accuracy meant customers now have more urgent reasons to upgrade time on all their systems. You can see how well it works if you can pick out that straight lime-green line in the graph above – that is TK running on one Windows box with alternative time sync software running on the others.

Production tests(not lab tests) show that synchronization in the sub 10 microsecond or better range is easily achievable. TimeKeeper Windows also provides high accuracy for cloud based and other virtual machine instances.

MiFID2 and security – keeping track of the money


A shorter version of this post is on the  FSMLabs web site.  MiFID2 is a new set of regulations for the financial services industry in Europe that includes a much more rigorous approach to timestamps.  Timestamps are in many ways the foundation for data integrity in modern processing systems – which are distributed, high speed, and generally gigantic. But when regulations or business or other constraints require timestamps to really work, the issues of fault tolerance and security come up. It doesn’t matter how precise your time distribution is if a mistake or a hacker can easily turn it off or control it.

TimeKeeper incorporates a defense-in-depth design to protect it from deliberate security attacks and errors due to equipment failure or misconfiguration. This engineering approach was born out of a conviction that precise time synchronization would become a business and regulatory imperative.KeystoneCops

  1. Recent disclosures of still more security problems in the NTPd implementation of NTP show how vulnerable time synchronization can be without proper attention to security. PTPd and related implementations of the PTP standard have similar vulnerabilities.
  2. Security and general failure tolerance should be on the minds of firms that are considering how to comply with the MiFID2 rules because time synchronization provides both a broad attack surface and a single point of failure unless properly implemented.

The first step towards time non-naive time synchronization is a skeptical attitude on the parts of IT managers and developers. Ask the right questions at acquisition and design time to prevent unpleasant surprises later.

One of the most dangerous aspects of the just disclosed NTPd exploit is that NTPd will accept a message from any random source telling it to stop synchronizing with its actual time sources. Remember, NTPd is an implementation of NTP, other implementations may not suffer from the same flaw. That d is easy to overlook, but it’s key. TimeKeeper’s NTP and PTP implementations will, for example, ignore commands that do not come from the associated time source and will apply analytical skepticism to commands that do appear to come from the source. TimeKeeper dismisses many of these types of attacks immediately and will start throwing off alerts to provoke automated and human counter-measures. The strongest protection TimeKeeper offers, however, comes from its multi-source capabilitiesthat allow it to compare multiple time sources in real-time and reject a primary source that has strayed.

Correct time travels a long, complex path from a source such as a GPS receiver or a feed like the one British Telecom is now providing. Among the questions system designers need to ask are the following two.

  1. Is the chain between source and client safeguarded comprehensively and instrumented end-to-end?
  2. Is there a way of cross-checking sources against other sources and rejecting bad sources?

Without positive answers to both of these questions, the time distribution technology is inherently fragile and robust MiFID2 timestamp compliance will be unavailable.

The painting is: “Quentin Massys 001” by Quentin Matsys (1456/1466–1530) – The Yorck Project: 10.000 Meisterwerke der Malerei. DVD-ROM, 2002. ISBN 3936122202. Distributed by DIRECTMEDIA Publishing GmbH.. Licensed under Public Domain via Commons – 

MiFID2 Timestamp regulations


There are a number of places in the new guidelines that increase the rigor required for timestamping data. One key part covers SI’s (systematic internalizers) who operate kind-of like private exchanges. TimeKeeper’s ability to produce traceable audit and to use multiple sources is designed for precisely this kind of application.

Moreover, the inclusion of the timestamp in the pre-trade information published by the SI is a key information for the client to better analyse ex-post the quality of prices quoted by SIs, and in particular to assess with accuracy the responsiveness of the SI and the validity periods of quotes. Without a timestamp assigned by the SI itself, market participants would need to rely on the information potentially provided by data vendors, the timestamps of which would be less accurate, especially when quotes are published through a website as pointed out by some respondents to the question on access to the quotes of SIs

Image is by Hans Holbein the Younger (1497/