I think there’s a third class of low-latency analytics to be performed against data as it’s generated. The DBToaster project (http://www.cs.cornell.edu/bigreddata/dbtoaster/) gives a good example. I’m curious to see the size of each of these domains (essentially < 1 sec, < 1 min, < 1 hr).

Later,
Jeff

i have posted a correction on my post about the vector clock stuff and explained why it happened. we were deep in discussions about Cassandra – and it doesn’t use vector timestamps.

thank goodness u agree about Consistency. So does Avinash. What i have tried to point out is that Dynamo paper’s section on quorums and consistency is confusing like hell. It leads readers to believe that they can get consistency – when they can’t (with 100% odds). If u look at Jonathan’s arguments – he’s continuing to insist that there are proper read/write quorums in Dynamo/Cassandra – whereas there aren’t. The term ’sloppy quorum’ is used for a reason and the system is only ‘eventually consistent’ for the same reason.

i haven’t said that relaxed consistency is not attractive for some applications. i am also not saying that dynamo is only deployed within a single data center. what i am saying though is that consistency needs to be relaxed only when partitioning is possible – and that this can be built as a separate layer above a data store with CA. the other thing that i keep stressing is that having bounds on inconsistency is a matter of practical importance. while recovering from an event like a disaster, one is faced with a choice of bringing online significantly old data and availability in the face of disaster. In such admin initiated actions – it’s very important to have some idea of how much data could be potentially lost. The reason simply being that if data is significantly out of date – one might rather choose to be unavailable for the couple of days that it takes for the disaster to repair.

i continue to disagree about partitions within a single data center. u had mentioned whether i had on the ground experience in a web company. it might help to know then that my comments about core switches and partitioning is not some figment of imagination – but derived from actual events from our site – one of the largest in the world. any kind of network partition in our data centers is usually catastrophic. we are simply unable to lose network access to one of our core services (from say our web tier) and continue functioning normally (from that data center). this would be fairly typical of any web site. so we must build arrangements that prevent network partitions in a data center. rack failures (which are usually switch failures) are another case (that are almost like partitions) – but this problem is easily solved by replicating across racks (a la hdfs). important central servers have to be attached to multiple switches.

i think this is a critical point (without which the argument for starting with CA only falls apart). FWIW – i have had almost total success internally with this argument (people immediately agree from experience that partitions are simply intolerable within a data center).

(btw – on a related point – S3 is eventually consistent as well – and it’s a total pain to deal with that aspect of it (first hand experience working out Hive integration with Amazon guys). i almost felt sorry for Amazon engineers as they kept explaining how screwed up the semantics were. some day amazon will have to fix it (competition is coming)).