Loose cables at CERN and Time synchronization is hard

According to sources familiar with the experiment, the 60 nanoseconds discrepancy appears to come from a bad connection between a fiber optic cable that connects to the GPS receiver used to correct the timing of the neutrinos’ flight and an electronic card in a computer. After tightening the connection and then measuring the time it takes data to travel the length of the fiber, researchers found that the data arrive 60 nanoseconds earlier than assumed. Since this time is subtracted from the overall time of flight, it appears to explain the early arrival of the neutrinos. New data, however, will be needed to confirm this hypothesis [ Science Insider]

Time synchronization is a real-time problem, meaning that every link is a single point of failure unless you engineer a lot of cross checking.  In our FSMLabs TimeKeeper software, we work in an environment that is a lot more complex and heterogeneous  than the CERN environment – fortunately we don’t need to get to those levels of precision. But we are operating at the under 1 microsecond level in many systems and it is enormously difficult to keep a running system balanced as network links misbehave, fans turn on and off,  and processor load changes. One thing we are increasingly doing is to monitor multiple time sources and track their relationships – to improve system resilience.