Tag: big data

Categories
timekeeper

Cassandra Cluster Synchronization

Cassandra is a highly-distributable NoSQL database with tunable consistency. What makes it highly distributable makes it also, in part, vulnerable: the whole deployment must run on synchronized clocks.

It’s quite surprising that, given how crucial this is, it is not covered sufficiently in literature. And, if it is, it simply refers to installation of a NTP daemon on each node which – if followed blindly – leads to really bad consequences. You will find blog posts by users who got burned by clock drifting. [Holub]

Cassandra labels updates with times and then uses those times to figure out the freshest update. Suppose machine A and B are both working with a database of records. B updates record R, then A  updates record R and 3 out of 5 machines holding copies of R get this second update. Then B asks for record R and the read operation gets 5 responses – it can throw away the stale copies of R and forward the fresh copy because the timestamp on the update from A is more recent than those on the older copies.   But this only works if the times are right – which means you need solid clock synchronization. Time synchronization clients like NTPd and PTDp (and variants) will silently fail, so the system can lose synchronization without  notice.  This will silently corrupt data. One of the most sophisticated parts of TimeKeeper is the algorithm to detect incorrect times, to failover where possible, and to alarm where not possible.

See: TimeKeeper, Clouds, Big Data

Categories
timekeeper

Data tiedowns with reliable time stamping

Management teams are  growing more reliant on the ability to immediately  access and quickly sort through massive amounts of data to find the information they need – Data Governance for Financial Institution

A “data tiedown” is a reliable and cross-checked timestamp that secures a data item or group of data items to one or more fixed times – such as time of collection, generation, or storage. Data analysis cannot compensate for bad data and distributed processing makes it easy to generate bad data whether from new inputs or by corrupting existing data. As transaction rates increase and data size grows, traditional locking techniques are both more expensive to implement and easier to get wrong. Timestamps can be an important part of data integrity assurance by imposing order on data, even if it is generated or stored by multiple processing elements that are only loosely coupled. But timestamping is in itself a fragile process.  One common error mode for networks relying on IEEE 1588 Precision Time Protocol (PTP) for clock synchronization involves a loss or incorrect calculation of the number of leap seconds since the epoch.  If data integrity depends on timestamps a sudden jump of 35 or 16 seconds can have a devastating impact. So the integrity of timestamps is something that also  needs to be safeguarded. One way to do that is to do what TimeKeeper does – track multiple reference time sources so that the clock can be continuously cross checked against other clocks.  The timestamp is then provided with a record of how well it matches secondary, tertiary, or deeper sources. When these all match, the timestamp has high reliability. Errors can also be traced – there is a forensic methodology available to validate time and locate problems. Data is then tied to a timestamp and the timestamp is tied to a log that verifies its accuracy – the combination is an example of a data tiedown.

 

Categories
timekeeper

Time maps for visualizing time distribution networks

TimeKeeper 6.3.0 map - demo
Without this information, a time distribution network is a black box. TimeKeeper looks at all the sources of time, in any protocol, that it can see around it, and tracks each source back to its source. These maps, especially in larger and older firms, find all sorts of interesting things. They find “redundant” time sources that are not. They find odd cycles that can cause dumb NTP implementations to oscillate between sources. They can find unexpectedly long legs (edge length corresponds to one way delay). In one corporate network they discovered a GPS source that was, to the surprise of users, failing over to a modem connected to a trans-Atlantic feed.  These pictures are a way of creating actionable  information from reams of difficult to understand data.

TimeKeeper 6.3.0 map

Categories
systems

Obama’s big data team

Those applications used a huge chunk of Amazon’s Eastern data center. “We had thousands of nodes,” said VanDenPlas. “We pushed 180TB of traffic with billions and billions of requests. We had 60% of all of Amazon’s medium [instances] in US East.”

Read more: http://sdt.bz/37299#ixzz2I0RN4r00

Categories
systems

Big Data and politics

For the general public, there was no way to know that the idea for the Parker contest had come from a data-mining discovery about some supporters: affection for contests, small dinners and celebrity. But from the beginning, campaign manager Jim Messina had promised a totally different, metric-driven kind of campaign in which politics was the goal but political instincts might not be the means. “We are going to measure every single thing in this campaign,” he said after taking the job. He hired an analytics department five times as large as that of the 2008 operation, with an official “chief scientist” for the Chicago headquarters named Rayid Ghani, who in a previous life crunched huge data sets to, among other things, maximize the efficiency of supermarket sales promotions.

Exactly what that team of dozens of data crunchers was doing, however, was a closely held secret. “They are our nuclear codes,” campaign spokesman Ben LaBolt would say when asked about the efforts. Around the office, data-mining experiments were given mysterious code names such as Narwhal and Dreamcatcher. The team even worked at a remove from the rest of the campaign staff, setting up shop in a windowless room at the north end of the vast headquarters office. The “scientists” created regular briefings on their work for the President and top aides in the White House’s Roosevelt Room, but public details were in short supply as the campaign guarded what it believed to be its biggest institutional advantage over Mitt Romney’s campaign: its data.

Read more: http://swampland.time.com/2012/11/07/inside-the-secret-world-of-quants-and-data-crunchers-who-helped-obama-win/#ixzz2BYScPM1z

 

Categories
systems

Julia programming language and hadoop

While we’re being demanding, we want something that provides the distributed power of Hadoop — without the kilobytes of boilerplate Java and XML; without being forced to sift through gigabytes of log files on hundreds of machines to find our bugs. We want the power without the layers of impenetrable complexity. We want to write simple scalar loops that compile down to tight machine code using just the registers on a single CPU. We want to write A*B and launch a thousand computations on a thousand machines, calculating a vast matrix product together. Julia

Aside from whatever merit the language has, the note about “kilobytes of boilerplate Java and XML” seems both wildly optimistic and on point.