Warm, fluffy stacks, covered in sweet and sticky maple syrup. Oh, sorry. We’re not talking about pancakes today. Instead, we’re diving into what is a modern data stack.
We unpack what a stack is, how data is involved, and what makes the whole thing modern. So, grab yourself a pile of flapjacks, Johnny cakes, or pannkakor (the Swedish word for pancakes), and let’s dig in.
What is a stack, and what’s a data stack?
A stack is a modern, sometimes techie, term for a group of software products that (when connected) can facilitate different business processes. For example, a marketing stack could include your website analytics software, a CRM platform, social media management software, and more. They are the tools a marketer uses to automate or conduct their daily business tasks more efficiently.
A data stack, meanwhile, is a series of technology products that are specifically focused on data collection, storage, manipulation, and movement. For instance, to compile the data from your different digital advertising platforms, you’ll need a software product or two (or sometimes more) to make that happen.
Is a data warehouse part of the modern data stack?
The tools in a data stack can be everything from spreadsheet software, to data visualization tools, to a data warehouse, and even advanced business intelligence software. The data stack consists of all the tools you need to understand your business data and make data-driven decisions. Think of data integration, data transformation and data analytics software.
What is a modern data stack?
To understand what a modern data stack is, let’s first look to the past. In the “old days,” a data stack was created by a business on a local (on site) server. They were incredibly technical and complex to build. As such, they were built and managed by data scientists and engineers who were often a part of the IT department.
The goal of these traditional stacks was to bring all of the data to a single end destination — sometimes called an “end-to-end” approach. Typically, all data would be collected into a sort of data warehouse where it could be accessed, manipulated, and analyzed by a core team of engineers.
In contrast, modern data stacks take advantage of cloud (or remote) server technology. They can handle much greater amounts of data and at lower cost. Plus, they can be easier to implement and utilize more widely understood (though still specialized) technologies like SQL.
An era of emerging democratization
As these more modern data stacks were being built and implemented, a wider user base emerged. While these modern data stacks are still largely owned and operated by engineers, data analysts could begin to work with them - as long as they were comfortable working with SQL, BigQuery, and more.
This began a period of increasing democratization in utilizing data stacks and the tools within.
Even marketing teams are now looking to take advantage of modern data stacks. After all, modern marketing teams work with various diverse digital advertising platforms, email platforms, content management systems, and more. Of course. they would seek a broader system to manage these areas of their day-to-day business. More on this in a bit, though.
|Traditional data stack||Modern data stack|
|Theme||End-to-end centralization||Unbundling of technologies|
|Target user||Database administrators, data engineers, IT||Central analytics and Business Intelligence staff|
|Products||ETL, Server based Data Warehouse and BI||Automated ETL, Cloud-native Data Warehouses and BI|
|Leaders||Business Objects, Tableau Qlik, Informatica, Teradata, Oracle||Looker, BigQuery, Redshift, Snowflake, dbt, Fivetran, Matillion|
Cloud computing was critical for the evolution of modern data stacks
The underlying ideas of business intelligence and the strategies and technologies that enable data-driven decision-making formed in the 1980s. At the time, new technologies began to increase in computing power and speed, allowing analysts to expand their capabilities to review many different areas of a business's operations.
Since then, the concept of business intelligence has remained generally the same. However, cloud computing really changed the game for engineers. For starters, BI teams weren’t bound by local storage limitations, meaning they could start handling a seemingly limitless flow of information. Plus, cloud computing was much faster than working with a local server.
At this point, the cost of storing these vast flows of data dropped immensely. Servers could run in parallel across thousands of machines (think Amazon Web Services or AWS, for instance). These cloud-based data warehouses were a game changer; they were even formally named: massively parallel processing (MPP) databases.
For our purposes, though, we’ll call them data warehouses.
New data tools create a new paradigm
As these new, cloud-based systems proliferated, new players and products entered the data stack arena. Google’s ever-expanding suite of analytics tools blazed a new trail, and specialized products like data pipes began to pop up across the industry.
And thus, the era of the modern data stack was born. Companies would employ a diverse array of cost-efficient and high-speed technologies to manage and analyze vast amounts of business data (so long as you knew some specific coding languages, of course).
The traditional data stack vs. the modern data stack
All of that backstory is nice, but you may wonder what specific drawbacks of the traditional model does a modern data stack address. Let’s take a look:
Core challenges of a traditional data stack
• Difficult and time consuming to create
• Difficult to “untangle” issues in the data flow
• Slow response to new information
• Required highly specialized experts to extract any valuable insights
Benefits of a modern data stack
• Much easier implementation
• Long-term commitments are replaced by plug-and-play flexibility
• Moves from a niche IT project to a business-wide approach
• Allows more teams to give greater consideration to the possibilities (and responsibilities) of data
Our partner Snowflake wrote a detailed report about the modern data stack, including information on what companies are in it (in the field of marketing data).
Text continues below image.
What trends do we see emerging?
Now that we know the history of data stacks and how modern stacks benefit the broader business, we sat down with one of our product managers. And we asked to get their view on where the space is headed next. Together, they identified three key areas:
1. No-code platforms, which mean less dependency on data scientists
Remember those marketing teams that we mentioned earlier? They would love to get their hands on some great data management tools. However, advanced coding and data engineering skills are outside their wheelhouse. That’s why we sees the emergence of code-free solutions as a potential game changer. More and more dashboards, reporting, and automated analytics tools are emerging for non-technical users.
3. Unbundling the stack
As we covered, data stacks used to be the realm solely of data engineers within the IT department. However, this tightly knit use case is beginning to become (in a sense) unbound. Teams like human resources, finance, product, and marketing see the value that good data analysis can provide.
3. Democratization of data
Typically, executives can get what they ask for pretty quickly — and understandably so. If they need a report covering the business’s core financial indicators, they can normally expect to receive it from the BI team ASAP. Other teams in the organization can be underserved, though. Marketing teams could be clamoring to understand their performance better, but the many requests to the BI team create a natural bottleneck.
Expect more and more tools to emerge that address this issue, allowing these non-technical teams to service their own data needs easily.
Those tools, like Funnel, can dramatically empower these teams to implement data analysis more deeply into their operations and decision making. Even the famous tech-focused venture capital fund Andreessen Horowitz, has dubbed these newly empowered data-driven employees ‘operational analysts.’ Expect to see this trend continue.
Another emerging trend that our product managers highlighted was the shift away from viewing your data stack as a stack altogether. Instead, they explained, data users will begin to see their many tools as a data mesh.
We guess that means pancakes are set to reclaim their “stack throne,” then.