Why We Chose Azure Table Storage over Amazon DynamoDB

In creating a new cloud service, we knew using the right software platform was going to play a huge role in how successfully we could implement Cognito Forms. We needed a cost effective, scalable solution that would allow us to be deeply competitive with comparable products. Our first step—choosing between a SQL and NoSQL data platform.

SQL or NoSQL?

To make this determination, we first defined two core requirements we needed with our database:

A scalable product that can effectively run with high volume

Cognito Forms is designed for millions of users and almost limitless data. Also, our Cognito Forms app is wildly customizable. With this in mind, we realized having a structured database would present a challenge. SQL is easy to query, but noticeably slower and harder to manage with massive amounts of users with a large volume of data. Although harder to query, NoSQL can be partitioned, or grouped, based on load. This is helpful when trying to find data we use most often.

Scalable winner: NoSQL

A cost effective storage solution to be competitive

Although the feature set of NoSQL databases is considerably smaller, NoSQL databases are roughly two orders of magnitude cheaper than enterprise class SQL databases. Research verified that using a NoSQL non-relational database would save us approximately $5,000 a month per terabyte of data. NoSQL’s price was fixed, while the cost would increase for every terabyte used with a SQL database. Although choosing a NoSQL database would require significant upfront development, we decided it was worth the time and effort. This was essentially our opportunity cost for achieving inexpensive data storage, allowing us to be competitive in the market from the start.

Price winner: NoSQL

Amazon Dynamo DB or Microsoft Azure Table Storage?

Having determined that we favored a NoSQL solution for Cognito Forms, we began our search for the ideal cloud NoSQL provider. There was little analysis on the web comparing NoSQL offerings for our level of expected volume. While Amazon is the industry leader for cloud services, we were able to consider competitors due to only needing a small subset of their offerings. We narrowed down our research to two services, Amazon DynamoDB and Microsoft Azure Table Storage.

Amazon DynamoDB

PRO	CON
Geo-redundant Servers	Engineer Friendly instead of Developer friendly
Mature cloud provider	Result set size limit in query
20 data centers in United States	IAAS: Infrastructure as a service
SSD hard drives	Predictive pricing model

Upon listing out some of the major pros and cons for DynamoDB, as they related to our development team and product needs, we quickly realized it wasn’t a good fit. Although Amazon is broadly established, several of their DynamoDB policies worked against Cognito Forms. For example, with the result set being limited in a query, our developers would be required to manually resubmit the query until obtaining the sought after results. DynamoDB can only scan a megabyte’s worth of data at a time, taking a simple query to time consuming lengths. A major downfall of DynamoDB is the predictive pricing model. Instead of paying for what you actually use in a day, you must pay for what you think you may use. This is not an ideal pricing scenario for applications with variable usage.

Microsoft Azure Table Storage

PRO	CON
Geo-redundant Servers	Only 4 data centers in United States
Platform as a Service	Many features are still in beta (auto-scaling backups)
Payment based on actual usage	Not as many third-party offerings (SMS texting, etc.)
More efficient to query
Integrates directly with our development environment

Azure Table Storage, on the other hand, provides some useful services for Cognito Forms. Instead of paying for the predicted amount of data throughput you will use in a day, you pay for what you actually use. Doesn’t that make sense? No mind-reading required. Another benefit is that Azure is a Microsoft tool and we develop in .NET and other Microsoft technologies. Azure integrates seamlessly with our existing development environment.

Azure is more of a Platform as a Service than Amazon DynamoDB. It deploys updates to our servers for us, allowing us to focus our time on other areas of Cognito Forms. When looking at the cons, we quickly realized they weren’t as significant as the pros. Azure, having many features still in beta, including their backup offering, could be daunting; however, geo-redundant servers are already a form of backups that exists within Azure Table Storage. Copies of our data would be stored on servers across the United States, providing a disaster recovery system in itself.

Winner?

To determine the winner, we built our application so that it could perform on both platforms. If we were going to make this huge decision, we were going to make sure we covered all the bases. We set up hosting on both platforms and ran load testing to decipher if one would out-perform the other. Our study concluded that Amazon DynamoDB required constant monitoring to adjust throughput to handle the pressure of the high volume, while Azure seamlessly accepted the excess data without flaw. They both had their pros and cons, but in the end, Azure was the clear winner.