Tag: ntp

Categories
timekeeper

The Enterprise Profile for PTP and TimeKeeper

One of the most interesting things we saw in the proposed IEEE 1588 enterprise profile was a bold suggestion on fault tolerance that looked familiar. Here’s FSMLabs press release from September 2011

TimeKeeper 5.0 offers the ability to monitor multiple time distribution channels, even those operating on different time distribution standards or of different quality due to distance or network issues. As an example, a TimeKeeper client may monitor two different Precision Time Protocol (PTP) “master clocks” and three different Network Time Protocol (NTP) servers. In addition, if the time quality of TimeKeeper’s primary sources becomes questionable, TimeKeeper can now switch from tracking one time source to another, according to a fail-over list provided at configuration time.

This press release described products that were already in the field in production. I remember that although customers liked this capability, talks at timing conferences often provoked complaints from engineers who insisted that the PTP “Best Master Clock” protocol already solved the problem. Anyways, it was gratifying to see that by February 2015 a similar, scaled down, capability was being proposed for the PTP Enterprise Profile. 

Clocks SHOULD include support for multiple domains.  The purpose is to support multiple simultaneous masters for redundancy. Leaf devices (non-forwarding devices) can use timing information from multiple masters by combining information from multiple instantiations of a PTP stack, each operating in a different domain. Redundant sources of timing can be ensembled, and/or compared to check for faulty master clocks. The use of multiple simultaneous masters will help mitigate faulty masters reporting as healthy, network delay asymmetry, and security problems.  Security problems include man-in-the-middle attacks such as delay attacks, packet interception / manipulation attacks. Assuming the path to each master is different, failures malicious or otherwise would have to happen at more than one path simultaneously. Whenever feasible, the underlying network transport technology SHOULD be configured so that timing messages in different domains traverse different network paths.

Note that there are three things missing from this proposal that were in TimeKeeper 5.0 back in 2011: the ability to use NTP sources as well as PTP, the ability to use multiple PTP sources in the same domain, and working software. Stating “SHOULD” in a standard is a long way from “works in the field” but recognition of the problem is a good step.

 

Categories
timekeeper

Measuring NTPd in the Cloud II

 The problem with ntpd is the sudden performance degradations which can occur when conditions deviate from ‘ideal’. We have detailed these robustness issues of ntpd in prior work, including [17, 18]. Simply put, when path delay variability exceeds some threshold, which is a complex function of parameters, stability of the feedback control is lost, resulting in errors which can be large over small to very long periods. Recovery from such periods is also subject to long convergence times […]

Now to the results. As expected, and from the very
first migration, ntpd exhibits extremely large errors (from
-1 to 27 s!) for periods exceeding 15 minutes (see zoom
in middle plot) and needs at least another hour to converge
to a reasonable error level.

from  Virtualize Everything but Time, Broomhead ,Cremean,Ridoux,Veitch

TimeKeeper is a “from ground up” implementation of  clock synchronization over both PTP and NTP and does not share NTPd limits.

 

Categories
timekeeper

NTPd DOS attacks

Miscreants who earlier this week took down servers for League of Legends, EA.com, and other online game services used a never-before-seen technique that vastly amplified the amount of junk traffic directed at denial-of-service targets.

Rather than directly flooding the targeted services with torrents of data, an attack group calling itself DERP Trolling sent much smaller sized data requests to time-synchronization servers running the Network Time Protocol (NTP). By manipulating the requests to make them appear as if they originated from one of the gaming sites, the attackers were able to vastly amplify the firepower at their disposal. A spoofed request containing eight bytes will typically result in a 468-byte response to a victim, a more than 58-fold increase.              [Ars Technica]

Servers running TimeKeeper would diagnose such attacks early and ignore them.  In fact, whether TimeKeeper is serving NTP or Precision Time Protocol (PTP) it will shrug off many of the exploits that can bring down networks running free software time synchronization. The ubiquitous NTPd free server and client software is designed to expect a well controlled network with a limited range of possible errors and, certainly, without malicious requests.  TimeKeeper is designed for an enterprise environment where infrastructure failures are not tolerable.    The protocols simply define how clock updates are distributed on the network and the format of the packets, they do not solve problems of network robustness or security. Security, failover, audit trail, manageability and other critical enterprise service properties are the responsibility of the software that implements these protocols.

Managers of enterprise networks that depend on time distribution and synchronization need to determine whether these are critical services or not. If not, then lack of instrumentation, legacy security holes, and in-house patching may be acceptable. On the other hand, if timing is a critical service, then enterprise qualified software components are not optional. This issue should be of particular concern to firms that offer their own customers time as a service because failures on the customer side might be perceived by customers as a failure in the service itself and those failures may even blowback into the provider network.

 

 

Categories
timekeeper

Challenges upgrading time synchronization in a global financial company

The real-world challenges of trying to upgrade time-synchronization without TimeKeeper are described in an excellent 2012 technical paper about a project at the company IMC Global Finance [IMC].  The article documents a two year, highly resourced project run by an expert engineering team – a nice data point for perhaps best case time synchronization using the open source alternatives to TimeKeeper. The tradeoff evaluation, of course, will be different in every enterprise, but the blunt analysis of limits to the existing open source technology contained in the IMC article is refreshing:

In all, this paper contributes concrete examples where PTP’s byzantine robustness, scalability and efficiency characteristics range between absent to poor – and attempts to raise awareness on the steps needed to build PTP solutions with the characteristics that global users want.

PTP, as it is currently defined, is not (yet) a viable solution beyond smallish LANs

The cautionary statements in the IMC paper will come as a surprise to many in the industry because PTP has been aggressively marketed as the solution to time synchronization. In practice, PTP is just a protocol specification –it is the implementation that matters, and many of the limitations that confounded the IMC developers are not limitations of the protocol, but limitations of the implementation in the open source clock synchronization software they utilized.

Clock Synchronization Background

Let’s step back and look at the underlying issue of clock synchronization. Application programs running on separate computers rely on “clocks” that are found in each computer and adjusted by software. If those clocks are not synchronized it is impossible to consolidate records because we won’t know when events happened or even the order of events. Furthermore, without synchronized clocks, application programs that work together cannot determine how long collaborative transactions took or how long data takes to get from machine to machine.  Clock synchronization is therefore essential to both data integrity and system management.  Synchronizing clocks down to levels of microseconds or below has become a requirement in some parts of financial trading (down to milliseconds or below in cloud based computing) and, increasingly, in other areas as well. So the basic problem is how to synchronize these clocks and keep them synchronized – particularly because the hardware clocks on computers tend to accumulate errors rapidly. That challenge is compounded in a cloud compute environment.

To synchronize, each application server must continuously correct its clock relying on updates sent over the network from some reference time source. One common type of reference time source is a device that gets the time from a Global Positioning System (GPS) or other satellites and then shares that time with “clients” over the network.  The older, ubiquitous, standard for sharing those time updates is called the Network Time Protocol (NTP) but there is a newer, heavily marketed, alternative called the IEEE 1588 Precision Time Protocol (PTP). The open source NTPd will consume and produce NTP packets while there are several open source variants of PTPd that do the same for PTP packets. TimeKeeper can work with both protocols – and with multiple PTP “profiles” – and   also inter-operates with the open source implementations.

PTP limits and fault-detection and recovery

TimeKeeper is protocol agnostic – we provide sub-microsecond clock synchronization, fault-tolerance, scalability, and process documentation with either PTP (in several versions) or NTP. Each protocol has different advantages in different situations, and some problems are best solved above the network protocol layer – transparently to applications. And PTP is a standard that needs a lot of help in the enterprise. Perhaps the most important example is that PTP “best master clock” protocol makes fault-detection difficult or impossible and can easily put users in a state of violating existing regulations. The IMC developers discovered this problem mid-project and found it to be costly and difficult to meet regulatory and audit requirements.

The implications of this aspect of the PTP protocol only became clear after PTP was deployed worldwide, using multiple GMs to service thousands of clients. On several occasions, a GM bug caused the (single) time source to send time without leap seconds information, for two (!) hours. As the active GM continued to send “Announce” packets as normal, with the same BMC values (priority, clock class, etc.), the inactive GMs had no reason to take over. All clients, however, saw a 34 second offset without any indication that this time might be invalid or suspicious in any way. Consequently, they either “corrected” this situation by stepping (jumping) the clock backwards, or by slewing (slowing down) the clock at maximum speed to the “new” value. Both cases are unacceptable in our regulatory environment (namely FINRA rule 7430 [3])

TimeKeeper fixes this problem above the protocol layer. Sophisticated filtering methods dispose of bad updates, cross checks of time sources detect failing or compromised sources and there are configurable triggers of when to fail over to a backup source. TimeKeeper can also notify automated monitoring software and network operators when it sees problems.  The number of independent sources monitored by TimeKeeper is essentially arbitrary: some customers have five sources to provide a very high level of reliability. Because we solve the unreliable source problem above the protocol layer, a TimeKeeper client can use PTP and NTP sources to cross-check each other. That is TimeKeeper has the property that the IMC developers eventually proposed as a future direction for PTP:

[and so, in the future]  Therefore, the PTP client needs a heuristic to judge the ’trustworthiness’ of its current GM, as compared to the group, and to be able to switch when needed

tk38x

TimeKeeper, as illustrated in the graphic above can track many sources and cross-check them. But there is currently no notion of independent clocks within the PTP standard so the IMC developers were constrained to less effective, higher overhead solutions:

These multiple issues were first controlled by deprecating clock stepping, and later by deprecating long (i.e., more than 2 seconds) slews. At the same time, the specific issues were addressed in cooperation with the vendor [the hardware vendors –vy]. In addition, the identified problems resulted in multiple improvements to the configuration of the production time infrastructure and the network itself, especially on the multicast-forwarding configuration.

 While all these measures have objectively improved the situation, they are still insufficient to cover all known and unknown corner cases, as the slave clocks may still drift for long periods of time (at best), or be slowly slewed to any desired offset by a faulty, hacked or GPS-spoofed server (at worst). In all cases, this susceptibility introduces a significant operational overhead to keep the time synchronization system operating and monitored 24 hours/7 days a week.

Managers who choose to use the open source PTPd software in place of TimeKeeper are unknowingly betting that in the future PTP will develop some technology for addressing the problem of unreliable sources and that this future will materialize before their firms encounter a significant embarrassment.

A second look at PTP

The PTP standard has some advantages, but it is essential for technical management to understand the limitations of PTP which was originally intended to synchronize clocks on simple local area networks (LANs) that could not be more different than what we have in the enterprise. The protocol designers wanted to be able to synchronize clocks of control devices – sensors and actuators – connected to a single Ethernet network cable. They specified that the reference time sources should “multicast” time updates, essentially sending out time update packets that would be seen by all the devices on the cable. They located most of the intelligence in the time source – the grandly named GrandMaster – because they assumed the clients were simple control devices.  As the IMC developers put it:

Acknowledging that PTP was initially designed as a LAN protocol[…], the issues can be broadly divided into a) issues on the PTPv2 standard itself, b) issues that have to be addressed when PTP is expanded to work over WANs, and c) issues that caused the biggest operational impact on the (tested) implementations.

Now compare this target network with, say, IMCs network, which is similar to many networks in financial trading and other enterprises.

IMC has built and continuously maintains a state-of-the-art technological infrastructure that keeps us competitive. In particular, IMC features a global network that directly connects to over 40 exchanges worldwide, in all major financial locations and spanning all time zones. This network has dozens of datacenters (DCs), either co-located or in proximity to the financial exchanges, all with state-of-the-art switching backbones and inter-connected with a variety of both leased and partially shared high-speed interconnection lines. Together, all these DCs host thousands of servers, most of them performing critical real-time activities and all of them requiring strict traceability of their clocks to UTC for both current (e.g.,[3]) and anticipated regulatory/compliance reasons, for risk mitigation and for internal performance testing.

IMCs network was, before the project outlined in this paper, already set up to synchronize clocks using the older NTP protocol – which fits in the enterprise more easily.  Because the widely used open source NTP implementation software (which is also used inside of most network clock appliances) cannot deliver the time accuracy required in modern transaction systems, it is sometimes claimed that the protocol is limited in accuracy. This is false. TimeKeeper can easily deliver sub-microsecond accuracy over NTP. In fact, the two protocols are quite similar – the biggest difference is that there is hardware support for PTP in some networking equipment. However, there is also hardware support for NTP in some widely available network equipment and excellent results are routinely produced by TimeKeeper in this instance too.

When firms like IMC use TimeKeeper to upgrade clock synchronization, they can stay with NTP or gradually migrate or use a mixture of NPT and PTP. The IMC paper describes a number of issues that could motivate network managers might to be cautious about a migration to PTP. For example consider this note about the PTP multi-cast specification.

Requiring a single multicast address and group caused severe operational problems for IMC, as it requires all hosts to be both senders and receivers of a single shared WAN multicast group (with the clock separation methods described in section III.B). This requires an “all-to-all” semantic that is, in practice, far more complex to build and maintain than the regular “one-to-many” multicast semantics, primarily because of asymmetric routing issue

Ironically, one of the methods the IMC developers used to try to control the effects of PTP multicast is to embrace one of the many non-standard “hybrid modes”  that have proliferated in PTP implementation.  With this mode, PTP acts a lot like NTP, partially abandoning the multicast base of PTP to use a unicast handshake that is identical to the base NTP update transaction.

Conclusion

Ultimately, infrastructure and network managers need to look at the business logic of their time synchronization requirements and then examine the cost/benefit tradeoffs of different possible solutions. For some, the absence of software licensing costs for PTPd variants will outweigh the higher engineering investment, weaker technology, and commitment to discuss what might be considered proprietary issues on the open source development mailing lists. The IMC paper provides some documentation on the results that can be expected.

For more information contact sales@fsmlabs.com

NOTES

[IMC] Challenges deploying PTPv2 in a Global Financial Company. Pedro V Estrela, Jan L. Bonebakker, 2012 International IEEE Symposium on Precision Clock Synchronization for Measurement, Control and Communication

Categories
timekeeper

Time Synchronization in the Cloud

Synchronizing clocks in the cloud, especially for virtual machines, is way beyond the capabilities of ordinary synchronization methods. Tests show that virtual machines in the cloud relying on NTPd can fall off the reference time by tens of minutes over a single day. Even bare metal cloud platforms are significantly worse than dedicated machines.  This creates an interesting dynamic because distributed applications are becoming more and more dependent on tight time synchronization and there are a large number of existing applications that need at least millisecond level synchronization.  In the cloud environment, the design limitations of the alternative time protocol, PTP, mean it cannot be relied upon to give provide better performance either. PTP was initially designed to servos and data acquisition devices on a single shared ethernet – weaknesses like dependence on multicast (broadcasting)  and top  down failover methods are a problem in the general enterprise, but really become impediments in the cloud.  That’s why we took a semantic approach, fixing time distribution above the level of the protocol. The low level, bit level, packet level, cannot deliver the sophisticated time analysis, fault recovery, and management needed for complex environments like the cloud.

Categories
timekeeper

Mitre and GPS spoofing

Some slides on GPS timing vulnerabilities.

Categories
timekeeper

The business case for being time protocol agnostic.

Time Synchronization and Distribution is a business critical issue but it is easy to become bogged down in arcane technology/marketing controversies.  One of those controversies is over the choice between low level network protocols used to deliver time to application server computers.  The most widely used time synchronization protocol in the enterprise is called the Network Time Protocol (NTP) and there is a newer protocol called IEEE 1588 Precision Time Protocol. For most financial trading firms however, the critical questions are not down in the weeds of networks protocols, but in the level of business process: how accurate and reliable time synchronization will be for trading applications and how much alternative solutions cost to implement and to maintain.  TimeKeeper time synchronization technology – hardware appliances, software clients, and software servers – is all protocol agnostic and can operate both protocols concurrently so that our customers can make decisions based on business logic instead of on technology details.

Customers and network equipment providers often express surprise (or, sadly,  skepticism) when we say that we can provide the same deep sub-microsecond accuracy for NTP that we provide for PTP but it should not be all that surprising.  From a network packet technology point of view the two protocols are similar and some of the same hardware support that helps improve PTP performance is also available for NTP.  Each protocol has its advantages and disadvantages that manifest differently depending on user requirements and network architecture. In practice, we find that the most demanding high precision trading systems benefit from being able to use both protocols to back up and complement each other.

The perception that NTP is an inferior protocol is due to wide use of equipment that implements NTP badly. Network clocks that run NTP on 30 year old free software designed for keeping desktop PCs synchronized to within seconds, executing on an underpowered Linux embedded computers with network adapters chosen for lowest cost will usually have terrible NTP performance.  The poor performance is a result of the quality of the implementation and is not intrinsic to the protocol.

TimeKeeper uses sophisticated algorithmic methods, smoothing and filtering technology, and a highly optimized code base to implement both protocols and our own methods for fault tolerance and error detection. We do not rely on any legacy code for either the server side (which sends out time) or the client side (which consumes time from a time server). TimeKeeper software will use hardware timestamping on both protocols if possible. TimeKeeper GrandMaster appliances use exceptionally precise hardware timestamping for both protocols at 10 Gigabits per second or 1 Gigabit per second. Even using TimeKeeper on just one end of a NTP stream will greatly improve time accuracy (and reliability). Using TimeKeeper on both ends, for example, using a TimeKeeper GPS GrandMaster as server and TimeKeeper Client software on the client computer will dramatically improve performance. Sub-microsecond precision is quite practical (and no tuning or configuration wizardry is needed). And TimeKeeper MultiSource Client software can be setup to consume multiple time sources, some PTP and some NTP.

Because TimeKeeper is protocol agnostic, network managers can radically upgrade time synchronization without major labor and capital expenditures. PTP prefers to be used in a multicast environment – it was designed for simple networks consisting of a single shared Ethernet. NTP was designed for long haul networks. An existing time distribution network based on NTP can be upgraded transparently by adding TimeKeeper client and server software and GrandMasters as needed or even incrementally. PTP can be turned on and off for parts of the network as business logic dictates. Clients can be configured to get a PTP feed from a local PTP GrandMaster source and to also consume an NTP feed from an existing source. Existing NTP Network clocks can even be connected to TimeKeeper servers and converted into high quality PTP/NTP servers.  TimeKeeper adapts to the architecture of the existing network and to the demands of business process, it does not disrupt existing networks.  TimeKeeper is designed to assist management who are focused on issues such as what level of accuracy, record keeping, and fault tolerance their trading systems need, rather than on low level technology details.