i have posted a correction on my post about the vector clock stuff and explained why it happened. we were deep in discussions about Cassandra - and it doesn’t use vector timestamps.

thank goodness u agree about Consistency. So does Avinash. What i have tried to point out is that Dynamo paper’s section on quorums and consistency is confusing like hell. It leads readers to believe that they can get consistency - when they can’t (with 100% odds). If u look at Jonathan’s arguments - he’s continuing to insist that there are proper read/write quorums in Dynamo/Cassandra - whereas there aren’t. The term ’sloppy quorum’ is used for a reason and the system is only ‘eventually consistent’ for the same reason.

i haven’t said that relaxed consistency is not attractive for some applications. i am also not saying that dynamo is only deployed within a single data center. what i am saying though is that consistency needs to be relaxed only when partitioning is possible - and that this can be built as a separate layer above a data store with CA. the other thing that i keep stressing is that having bounds on inconsistency is a matter of practical importance. while recovering from an event like a disaster, one is faced with a choice of bringing online significantly old data and availability in the face of disaster. In such admin initiated actions - it’s very important to have some idea of how much data could be potentially lost. The reason simply being that if data is significantly out of date - one might rather choose to be unavailable for the couple of days that it takes for the disaster to repair.

i continue to disagree about partitions within a single data center. u had mentioned whether i had on the ground experience in a web company. it might help to know then that my comments about core switches and partitioning is not some figment of imagination - but derived from actual events from our site - one of the largest in the world. any kind of network partition in our data centers is usually catastrophic. we are simply unable to lose network access to one of our core services (from say our web tier) and continue functioning normally (from that data center). this would be fairly typical of any web site. so we must build arrangements that prevent network partitions in a data center. rack failures (which are usually switch failures) are another case (that are almost like partitions) - but this problem is easily solved by replicating across racks (a la hdfs). important central servers have to be attached to multiple switches.

i think this is a critical point (without which the argument for starting with CA only falls apart). FWIW - i have had almost total success internally with this argument (people immediately agree from experience that partitions are simply intolerable within a data center).

(btw - on a related point - S3 is eventually consistent as well - and it’s a total pain to deal with that aspect of it (first hand experience working out Hive integration with Amazon guys). i almost felt sorry for Amazon engineers as they kept explaining how screwed up the semantics were. some day amazon will have to fix it (competition is coming)).

Mobs and “movements” don’t like nuance.

It is surprising to me that you’ve read the relevant papers as your analysis of Dynamo ignored vector clocks and the various conflict detection and resolution mechanisms while asserting conflict resolution was a glaring problem. A significant chunk of the paper deals with nothing but conflict detection and resolution, even though it is rare in practice.

I don’t believe anyone is arguing that you should use Dynamo or Cassandra is guaranteed consistency is required. What people _are_ arguing is 1) there are interesting applications that don’t require such guarantees, 2) those applications often have high writability/low latency requirements, 3) partitions happen (even in a single data center). As I said above, it would be foolish to use a systems with relaxed consistency guarantees when consistency is paramount for your application.

On Amazon’s use of Dynamo, you are making a leap unsupported, even contradicted, by evidence: Werner did not say ‘Dynamo failed, so we stopped using it’, he said ‘We did the best we could at the time, it worked for years, but based on 5 years of experience we have now built something better’. I would be extremely surprised if they abandoned many of the principles embodied in Dynamo.

Finally, and again, your understanding of the underlying infrastructure is rather odd. Partitions can and do happen within a single datacenter. Your argument that 1) things like Dynamo are only ever deployed in a single datacenter and 2) partitions don’t happen in a single datacenter, hence 3) staying writable while partitioned is irrelevant is patently false on every point.

So, I agree: let the facts speak for themselves. You cherry pick and grossly misstate information to support your position, then cast aspersions at those who disagree with you. Dynamo and Cassandra were or are in production making enormous amounts of money for several companies. Any claims you make that they don’t work for the jobs for which they were built are bogus and say more about your attachment to being right than interest in an engineering discussion.

b

i don’t have a paper trail in PODC and SOSP - but here’s my background. i bought up the hadoop cluster at Facebook. I can comfortably claim to be one of the critical factors in making our Hadoop cluster scale from 80 to thousands of nodes today. as the lead of this team and effort for almost a couple of years - i was indeed carrying the pager (or more accurately the cell phone) and i have spent numerous nights attending to our cluster, solving all sorts of problems and keeping our users happy. i have dealt with this shit - and i know it’s not pleasant. while i was doing all this - i was also responsible for conceiving Hive and one of it’s primary developers.

secondly, what i am saying is sound academically. for example - read the VLDB 08 paper on PNUTS from Yahoo from respected academicians. It’s close to an implementation that i would design myself. while it doesn’t attack Dynamo directly - it does cover the problems with eventual consistency for developing applications. i have a reasonably strong education in computer science - and i still haven’t forgotten (entirely) reading the quorum consensus papers from Herlihy back in school.

i have used vector clocks myself when i worked at Netapp. We had a very simple 2 node HA cluster - but state synchronization was always a issue. I independently thought of and implemented vector clocks to reconcile cluster state with eventual consistency semantics. That level of consistency was just fine for the state we were dealing with. this was many many years back. We also implemented a commercial Disaster recovery system - it had to balance the tradeoffs between CAP (I didn’t know this term then). A very smart colleague of mine solved the problem very elegantly using hierarchical quorums. Flat quorum groups just didn’t work.

Consider that these HA and DR systems are used in many business critical applications (Netapp sold quite a load of them) and one of the reasons the DR solution worked well was because we were able to provide admins a good balance of C, A and P.

As regards my conclusions - Avinash has acknowledged flat out in internal mailing lists that Cassandra should not be used if data is desired to be consistent. I think you should reconsider the sources you trust - he has after all written the bulk of Cassandra code and was one of the Dynamo authors as well. (the fact that the leading committer of the open source Cassandra trunk doesn’t understand these issues points out how bad the situation is)

Consider also that we are hearing that Dynamo is slowly being deprecated inside Amazon. It would not be fair for me to comment further on this (let’s hear from Amazon guys).

Consider also that i am constantly pointing out BigTable/GFS as a better abstraction with partition tolerance being built as a layer on top of this. So i am indeed again referring to prior work with tremendous credentials.

Note that while Dynamo is withering inside Amazon (a cloud computing stalwart) - BigTable powers a strong commercial grade development platform (AppEngine).

i would let the facts speak for themselves.

The situation you describe is a partition: A & B are writable, while C is only readable [and you read from C; and you can avoid this by adjusting R/W/N]. Ignoring for the moment that such a scenario is extremely unlikely, recall that being always writable is an explicit design goal of Dynamo, _even at a cost to consistency_. This makes it appropriate for certain applications and inappropriate for others. Arguing that Dynamo and Cassandra are fatally flawed because they don’t offer the same consistency guarantees as Oracle RAQ is similar to arguing that Oracle RAQ is fatally flawed because it isn’t always writable under partition. Use the right tool for the right job, don’t insist a tool be universally applicable.

b