Accurate and reliable knowledge of time is fundamental to cyber-physical systems for sensing, control, performance, and energy efficient integration of computing and communications. This simple statement underlies the RoseLine project. Emerging CPS [Cyber Physical Systems – vy] applications depend on precise knowledge of time to infer location and control communication. There is a diversity of semantics used to describe time, and quality of time varies as we move up and down the system stack. System designs tend to overcompensate for these uncertainties and the result is systems that may be over designed, in-efficient, and fragile. The intellectual merit derives from the new and fundamental concept of time and the holistic measure of quality of time (QoT) that captures metrics including resolution, accuracy, and stability.
The project will build a system stack that enables new ways for clock hardware, OS, network services, and applications to learn, maintain and exchange information about time, influence component behavior, and robustly adapt to dynamic QoT requirements, as well as to benign and adversarial changes in operating conditions. Application areas that will benefit from Quality of Time will include: smart grad, networked and coordinated control of aerospace systems, underwater sensing, and industrial automation. The broader impact of the proposal is due to the foundational nature of the work which builds a robust and tunable quality of time that can be applied across a broad spectrum of applications that pervade modern life. Roseline.
Category: timekeeper
News from last year.Aug. 26, 2013 11:52 a.m. ET
A glitch in time synchronization caused German exchange operator Deutsche BoerseAG DB1.XE -0.16% ‘s Eurex Exchange arm to halt trading for slightly more than an hour early Monday, the latest in a thread of technical issues at global exchanges.
The Eurex derivatives trading market was halted at 0620 GMT, 20 minutes after trading started, “in order to protect the integrity of the market,” Deutsche Boerse said.
“An incorrect time synchronization within the system” triggered the market halt. The problem was solved, pre-trading started at 0720 GMT and, as of 0730 GMT all products were again tradable, the exchange said.
A person close to the matter said the glitch was caused by a problem with the system clock, not extreme data load, noting that current trading volumes are far below previous peaks.
From what we understand, the problem was due to a number of GPS time servers that dropped leap year adjustments, so jumping the time back 36 seconds. Then PTP2 software adjusted server time to jump backward 36 seconds. It’s not the first failure of this sort (or the last). The legacy technology for time synchronization lacks the cross check, failover, and alerting that are built into TimeKeeper. That technology was not designed with enterprise in mind and although it has been heavily modified over time, it is very difficult to engineer a solution to basic architectural mismatch. For example, TimeKeeper is architected to treat all time sources the same, but software like PTPd and NTPd is designed for a specific protocol. So TimeKeeper can use a feed from one protocol to crosscheck another, but that requires a clumsy grafting process in one of the protocol specific time synchronization programs. As another example, the PTP standard has a completely inappropriate fault-tolerance technique baked into the standard – a technique that is a holdover from the PTP standard origin in device control. The standard was designed for systems with really simple networks and time consumers. A single cable with some welding machines on the end is perhaps a typical intended use case. The idea was that the time sources would advertise how good they were and the consumers would simply pick the one that said it was the best. This is an absurd policy for an enterprise network with super-powerful server computers receiving time packets across complex networks dotted with routers and switches. TimeKeeper was designed to ignore this “best master” protocol for fault tolerance and to utilize the smarts of the consumers to detect problems in the feed and to select among alternatives.
Distributed transactions have historically been implemented by the database community in the manner pioneered by the architects of System R* [22] in the 1980s. The primary mechanism by which System R*-style distributed transactions impede throughput and extend latency is the requirement of an agreement protocol between all participating machines at commit time to ensure atomicity and durability. To ensure isolation, all of a transaction’s locks must be held for the full duration of this agreement protocol, which is typically two-phase commit.
The problem with holding locks during the agreement protocol is that two-phase commit requires multiple network round-trips between all participating machines, and therefore the time required to run the protocol can often be considerably greater than the time required to execute all local transaction logic. If a few popularly accessed records are frequently involved in distributed transactions, the resulting extra time that locks are held on these records can have an extremely deleterious effect on overall transactional throughput. [ Calvin: Fast Distributed Transactions for Partitioned Database Systems. Alexander Thomson et al]
This is one reason why we have all the new databases without transaction support – because high transaction rates in a distributed environment (e.g. web or IOT applications in the Cloud) cannot be scaled in face of lock overhead.
A common configuration for FSMLabs TimeKeeper customers is to cross-couple time sources in New Jersey and New York City or London and Slough or Chicago and Aurora or Singapore and Sidney- any two trading locations that are connected with high quality network. Sometimes the network connection does not even have to be that great. TimeKeeper will cross-check time sources, complain when things look wrong, and failover when needed. Multiple time sources also produces a timestamp provenance. Trading applications will have a record showing that the timestamps they produce were in agreement with two or more independent sources. A number of firms scale this basic design to multiple sites: increasing the depth of fault-tolerance and the strength of the provenance. Cross-coupling time feeds also provides early warning on a number of network problems. Several customers saw TimeKeeper warnings about secondary sync and found on investigation that their network providers were changing equipment or rerouting without notice.
The technology disconnect
What it did was reinforce a point about the sociology of management: From cars to space shuttles, from offshore oil wells to nuclear reactors, the people who make the decisions are often out of step with the mechanical details.” – Mathew Wald, New York Times. 2014/06/09.
We sell pretty complex software that synchronizes clocks of computers, down to nanoseconds in many networks. Our customers are often firms where where IT staff, let alone higher management, are consumed by pressing issues, none of which have to do with the nuts-and-bolts of how ideal clock algorithms and time protocols interact with network equipment, operating systems, and application servers. Our easiest sales are to companies where some person is able to get perspective on the technical issue and relate it to business priorities. Otherwise, things get lost between IT staff running hard to keep up with day-to-day and big picture matters, and higher management who are “out of step with the mechanical details”. This is why the most successful big technology companies have built up corps of “technology fellows”, in practice if not formally. These are experts who can take a sufficiently deep dive into technology to appreciate the problem, who also understand business priorities, and who have management confidence. If someone with those skills had been involved in GMs key ignition lock discussions the company could have made far better decisions. As more and more firms become dependent on critical technology infrastructure, they will need to acquire people with such skills, in-house or not.
also on LinkedIn
The story we were told at a bank that I cannot name is that all their time synchronization was the domain of an engineer tucked away for 30 years in the home office, a fellow known as Professor Time. The system he had built was remarkable in its complexity and fragility. Nobody seemed to understand how it worked. Accuracy was highly variable. There was no management, no documentation. We never got to meet the Professor, but I always thought of him as something like this.
In the past, the enterprise time synchronization market was composed of vendors that sold boxes to IT teams that were responsible for architecting a solution from GPS clocks and free software that was not at all designed for the job. Time synchronization is a specialized field that is a lot more complex than it may appear. And quality has become more of an issue as trading speeds and volume have increased so much. Some firms have responded to the change by scaling up their synchronization staffing. Some have relied on luck. Some hope that heavily marketed new technology in PTP aware routers will solve their problems.
Over the last couple of years, we have been building out technology to provide financial trading firms and other organizations that need precise timing with an alternative to the boxes+ custom-in-house approaches. We have built client and server software that is fault-tolerant, cross-checking, with sophisticated alarms, easy configuration, and graphical web management/data-analysis tools. And we’ve put that software in powerful server computers that can serve time directly at 10Gbps (and better) and that have all the standard enterprise features (like lots of storage for archival records and dual power supplies). All the parts are designed to work with each other and to connect in a flexible, resilient time distribution network. Our new partnership with Spectracom brings their extensive hardware expertise, distribution and support infrastructure into the mix.
Measuring NTPd in the Cloud II
The problem with ntpd is the sudden performance degradations which can occur when conditions deviate from ‘ideal’. We have detailed these robustness issues of ntpd in prior work, including [17, 18]. Simply put, when path delay variability exceeds some threshold, which is a complex function of parameters, stability of the feedback control is lost, resulting in errors which can be large over small to very long periods. Recovery from such periods is also subject to long convergence times […]
Now to the results. As expected, and from the very
first migration, ntpd exhibits extremely large errors (from
-1 to 27 s!) for periods exceeding 15 minutes (see zoom
in middle plot) and needs at least another hour to converge
to a reasonable error level.from Virtualize Everything but Time, Broomhead ,Cremean,Ridoux,Veitch
TimeKeeper is a “from ground up” implementation of clock synchronization over both PTP and NTP and does not share NTPd limits.
Measured on Amazon EC2.
Standard deviation from reference time over 20 hours
TimeKeeper: 23 microseconds
NTPd: 6000 microseconds (6 milliseconds)
The nature of data security is constantly changing for the financial industry. With the ever-growing need for tighter IT infrastructure security and the increasing prevalence of BYOD-friendly workplaces, financial institutions are turning to managed services to help handle these changes. In fact, a TABB Group forecast predicts that by 2016, 50% of financial institutions will be opting for managed services to help account for their IT infrastructure management. Similarly, a survey by TSIA cites managed services revenue today as the fastest-growing in the service line (which includes professional services, support services, and field services). [Onramp]
Cassandra Cluster Synchronization
Cassandra is a highly-distributable NoSQL database with tunable consistency. What makes it highly distributable makes it also, in part, vulnerable: the whole deployment must run on synchronized clocks.
It’s quite surprising that, given how crucial this is, it is not covered sufficiently in literature. And, if it is, it simply refers to installation of a NTP daemon on each node which – if followed blindly – leads to really bad consequences. You will find blog posts by users who got burned by clock drifting. [Holub]
Cassandra labels updates with times and then uses those times to figure out the freshest update. Suppose machine A and B are both working with a database of records. B updates record R, then A updates record R and 3 out of 5 machines holding copies of R get this second update. Then B asks for record R and the read operation gets 5 responses – it can throw away the stale copies of R and forward the fresh copy because the timestamp on the update from A is more recent than those on the older copies. But this only works if the times are right – which means you need solid clock synchronization. Time synchronization clients like NTPd and PTDp (and variants) will silently fail, so the system can lose synchronization without notice. This will silently corrupt data. One of the most sophisticated parts of TimeKeeper is the algorithm to detect incorrect times, to failover where possible, and to alarm where not possible.
2 a.m. ET