It was the beginning of the 2010s. Big data and all of its applications were all you could hear about. And I, who at that time, was pivoting towards getting a job in the industry couldn’t have chosen a better time to do so. At least, that was the promise.
Determined to crack open what big data was all about, I enrolled in the big data coursera track. Through a series of videos and online tutorials, engineers very excitedly went into great detail about big data formats, computational architecture, clusters, nodes and many other words that were foreign to my novice ears. The peak of the course came when we were tasked to write our own MapReduce job in Cloudera. For me it felt like a lot of hard work just to get a simple count.
Then luck struck. Twice. First, I did get a data analyst role. And second, the job did not entail writing MapReduce jobs! However, the big data movement was very much still alive touting its next promise: warehouses. The promise was enticing: forget about managing clusters. Simply concentrate on what you can do best: writing elegant SQL.
That promise has held up for the most part. I have spent most of my career working with different warehouses; starting with Redshift, followed by a brief stint with BigQuery and currently working with Snowflake. Despite some minor differences, the common thread is that I can simply concentrate on writing SQL while the underlying architectural complexity is hidden away from me.
Warehouses work well. However, this simplicity comes with a price tag. Sometimes, a big one. A colleague of mine learned it the hard way. A single query to get a monthly count was racking up hundreds of dollars. To limit this potentially unchecked spending, companies have to set up a whole system of guardrails, query optimization and resource monitoring.
So while warehouses made it technically easy to store and access your data, the real barrier was costs. Big data comes with a big budget. And big budgets are nowhere to be found in startups and mid-size companies.
Is there another way? In his post big data is dead, Jordan Tigani, co-founder of MotherDuck and former founding engineer of Google BigQuery, certainly believes so. He makes two compelling arguments:
You probably don’t have big data
Even in the unlikely case you have big data, most of your queries are accessing only a small fraction of it
That begs the question: when is data big? A simple answer is whenever you cannot process or analyze it on your own laptop. Turns out that according to a Redshift fleet study analyzed by Tigani, only around 5% of datasets in Redshift actually have big data. The remaining 95% can probably be analyzed on your own machine.
Meanwhile, our laptops have quietly become more powerful and their computational capacity has grown at a faster speed than our data. Data for which originally the MapReduce framework was invented can now often be analyzed on your laptop in seconds.
Even when our data is big, it is generally the case we are not accessing all of it at once. Think about how data is used in your company. With the exception of some deep dive analysis, the most accessed data is usually the most recent. This is the data that powers your dashboards and normally is already pre-aggregated in some kind of daily, monthly or yearly metrics.
This was also found in the Redshift analysis cited before. Only 0.03% of queries executed by customers were against more than 10TB of data. A whopping 67% were querying less than 1GB worth of data.
If we don’t really have big data, has the industry been optimizing for the wrong use case all along? In other words, are we building our whole infrastructure around a Ferrari when probably a reliable Toyota would do the job?
This is exactly the bet DuckDB and MotherDuck are making. Their promise relies on squeezing the most out of your single machine. Once you remove all the unnecessary architectural complexity, queries are not only faster but also substantially cheaper. If you do not have big data — and most likely you don’t — it is time to reconsider. Are you sure you need the costly Ferrari, or has the time come to switch to a Toyota?