Why your Cassandra needs local NVMe and Rok

Do you run a NoSQL database, like Cassandra or MongoDB, on the cloud or on-prem and it is terribly slow?

Probably, you are already paying a lot of money for the infrastructure, and slow is definitely something you don’t need.

Cassandra is a very popular NoSQL database, and it is widely deployed on AWS (Amazon Web Services) and GCP (Google Cloud Platform). The insoluble dilemma of deployment, which is constantly discussed the past few years, is whether you should be running Cassandra over local (ephemeral disks on AWS, local SSDs on GCP, local DAS on prem) or shared storage (EBS on AWS, PDs on GCP, SDS/SAN on prem).

The former option offers great performance, but no flexibility, while the latter is very flexible, but cripples performance and adds cost.

DataStax documentation recommends running Cassandra on ephemeral disks instead of using shared storage. However, a lot of people (see companies like Spotify, CrowdStrike, Librato) have been moving to shared storage lately, to gain from its flexibility. For example to migrate nodes easily, and take backups.

In terms of CPU and RAM, you have a wide range of choices. The cloud providers offer various types of instances, so that won’t be a problem. But what about IOPS? The flexible, shared storage that cloud providers offer is great for keeping your data safe and always available. However, it comes with an important downside, it is poor in IOPS.

What if you could have your application running on extremely fast, local NVMe SSDs, while keeping the flexibility of shared storage at the same time?

Well this is what we are building, and we call it Rok.

Rok is decentralized storage for the cloud native world. We believe modern, highly mobile apps need to discover persistent data instantly and access it fast, anywhere they run. That’s why we designed the first software product to combine the performance of local storage with the flexibility of shared storage, while enabling seamless collaboration on data, across your global infrastructure. Rok allows you to run your stateful containers over fast, local NVMe storage on-prem or on the cloud, and still be able to snapshot the containers and distribute them efficiently: across machines of the same cluster, or across distinct locations and administrative domains over a decentralized network. We think performance and flexibility shouldn’t be mutually exclusive anymore. One should have both. Everywhere.

So, by running Cassandra with Rok you get all the advantages of running over local NVMe storage:

  • extremely high IOPS
  • I/O latency in the order of μs
  • massive scale-out, with excellent scalability as the cluster grows
  • significant cost savings compared to running over shared storage

while keeping all the advantages of shared storage:

  • local backups
  • offsite backups
  • node migrations

Moreover, you get something that wasn’t possible before:

  • collaboration at global scale

This means that you can take a snapshot of your NoSQL database along with its data, and share it with another user of a completely distinct administrative domain, at a distinct location. This is a perfect match for test & dev use cases, analytics, or forensics.

The answer is: local storage with the flexibility of shared storage using Rok

Now that I have your attention, I will let the numbers speak for themselves.

I will present an analysis of cost and performance for a 100TB cluster on AWS and GCP.

Cassandra on AWS

We are going to compare the cost and performance of two Cassandra clusters on AWS. Each one has 100TB of raw capacity.

Shared StorageNVMe + RokComparison
Storage TypeEBS (io1)Local NVMe
Storage Capacity (raw)100 TB100 TB
Instance Typec4.4xlargei3.4xlarge
Number of instances2727
Total vCPUs432432same
Total GB of RAM8103,2944x better
Nominal aggregate write IOPS432 K9,720 K22x better
Nominal aggregate read IOPS432 K22,275 K51x better
Cost per month$50,838$18,01664% cheaper

Comparison of running Cassandra cluster on AWS over shared storage (EBS) and local NVMe shows that the latter approach results in 22 times more nominal aggregate write IOPS, 51 times more nominal aggregate read IOPS, and 64% cost reduction

We can see that using Rok and local NVMe-backed instances on AWS, you get more than 51x the nominal aggregate read IOPS, and more than 22x the nominal aggregate write IOPS, with over 60% cost reduction, keeping all the flexibility you need.

Cassandra on GCP

We are going to compare cost and performance of two Cassandra clusters on GCP. Each one has 100TB of raw capacity.

Shared StorageNVMe + RokComparison
Storage TypePDsLocal NVMe
Storage Capacity (raw)100 TB100 TB
Instance Typen1-standard-8n1-standard-8
Number of instances3434
Total vCPUs272272same
Total GB of RAM1,0201,020same
Nominal aggregate write IOPS510 K12,240 K24x better
Nominal aggregate read IOPS510 K23,120 K45x better
Cost per month$23,942$16,17832% cheaper

Comparison of running Cassandra cluster on GCP over shared storage (PDs) and local NVMe shows that the latter approach results in 24 times more nominal aggregate write IOPS, 45 times more nominal aggregate read IOPS, and 32% cost reduction.

We can see that using Rok and local NVMe-backed instances on GCP, you get more than 45x the nominal aggregate read IOPS, and 24x the nominal aggregate write IOPS, with more than 30% cost reduction, keeping all the flexibility you need.Comparison of running Cassandra cluster on GCP over shared storage (PDs) and local NVMe shows that the latter approach results in 24 times more nominal aggregate write IOPS, 45 times more nominal aggregate read IOPS, and 32% cost reduction.

Conclusion

Now, that the Rok data management platform adds the flexibility you need on this setup, and with the NVMe prices going down, running over local NVMe storage is a very compelling option. We strongly recommend NVMe-backed instances combined with Rok; the performance boost you will experience along with the associated cost savings will surprise you.

If you have any questions, or want to learn more about the proposed solution, don’t hesitate to drop us a line at contact@arrikto.com or get started with MiniKF.

Free Technical Workshop

Turbocharge your team’s Kubeflow and MLOps skills with a free workshop.