Warm, fluffy stacks, covered in sweet and sticky maple syrup. Oh, sorry. We’re not talking about pancakes today. Instead, we’re diving into the world of the modern data stack.
We unpack what a stack is, how data is involved, and what makes the whole thing modern. So, grab yourself a pile of flapjacks, Johnny cakes, or pannkakor (the Swedish word for pancakes), and let’s dig in.
What is a stack, and what’s a data stack?
A stack is a modern term for a group of software products that (when connected) can facilitate different business processes. For example, a marketing stack could include your CMS, website analytics software, CRM platform, social media management software, and more. They are the tools a marketer uses to automate or conduct their daily business tasks more efficiently.
A data stack, meanwhile, is a series of technology products that are specifically focused on raw data collection, storage, manipulation, and movement. For instance, to compile the data from your different digital advertising platforms, you’ll need a software product or two (or sometimes more) to make that happen.
So, now you understand what is meant with a stack in a business context, and what a data stack is. But what is this specific modern data stack? Let's explore it in depth.
What is the modern data stack?
To understand what the modern data stack is, we will first explain the 'legacy stack' of the past.
In the “old days,” a data stack was created by a business on a local, on site server. They were incredibly technical and complex to build. As such, they were built and managed by data scientists and engineers who were often a part of the IT department.
The goal of these traditional stacks was to bring all of the data to a single end destination. This is why this kind of data stack is sometimes called an “end-to-end” approach. Typically, all data would be collected into a sort of data warehouse where it could be accessed, manipulated, and analyzed by a core team of engineers.
In contrast, the modern data stack takes advantage of cloud data warehouses and modern, remote server technology. These new servers can handle much greater amounts of data and at lower cost. Plus, they can be easier to implement and utilize more widely understood (though still specialized) technologies like SQL. The modern data stack is also easier to maintain than the old, traditional data stack.
An era of emerging democratization
As these more modern data stacks were being built and implemented, a wider user base emerged. While these modern data stacks are still largely owned and operated by engineers, data analysts could begin to work with them - as long as they were comfortable working with SQL, BigQuery, and more.
This began a period of increasing democratization in using data stacks and the tools within.
Even marketing teams are now looking to take advantage of modern data stacks to better analyze data. After all, modern marketing teams work with various diverse digital advertising platforms, email platforms, content management systems, and more. Of course. they would seek a broader system to manage these areas of their day-to-day business. More on this in a bit, though.
|Traditional data stack
|Modern data stack
|Unbundling of technologies
|Database administrators, data engineers, IT
|Central analytics and Business Intelligence staff
|ETL, Server based Data Warehouse and BI
|Automated ETL, Cloud-native Data Warehouses and BI
|Business Objects, Tableau Qlik, Informatica, Teradata, Oracle
|Looker, BigQuery, Redshift, Snowflake, dbt, Fivetran, Matillion
Cloud computing was critical for the evolution of modern data stacks
The underlying ideas of business intelligence and the strategies and technologies that enable data-driven decision-making formed in the 1980s. At the time, new technologies began to increase in computing power and speed, allowing analysts to expand their capabilities to review many different areas of a business's operations.
Since then, the concept of business intelligence has remained generally the same. However, cloud computing really changed the game for engineers. For starters, BI teams weren’t bound by local storage limitations, meaning they could start handling a seemingly limitless flow of information. Plus, cloud computing was much faster than working with a local server.
At this point, the cost of storing these vast flows of data dropped immensely. Servers could run in parallel across thousands of machines (think Amazon Web Services or AWS, for instance). These cloud-based data warehouses were a game changer; they were even formally named: massively parallel processing (MPP) databases.
For our purposes, though, we’ll call them data warehouses.
New data tools create a new paradigm
As these new, cloud-based systems proliferated, new players and products entered the data stack arena. Google’s ever-expanding suite of analytics tools blazed a new trail, and specialized products like data pipes began to pop up across the industry.
And thus, the era of the modern data stack was born. Companies would employ a diverse array of cost-efficient and high-speed technologies to manage and analyze vast amounts of business data (so long as you knew some specific coding languages, of course).
The traditional data stack vs. the modern data stack
All of that backstory is nice, but you may wonder what specific drawbacks of the traditional model does a modern data stack address. Let’s take a look:
Core challenges of a traditional / legacy data stack
• Difficult and time consuming to create
• Difficult to “untangle” issues in the data flow
• Slow response to new information
• Required highly specialized experts to extract any valuable insights
Benefits of a modern data stack
• Much easier implementation
• Long-term commitments are replaced by plug-and-play flexibility
• Moves from a niche IT project to a business-wide approach
• Allows more teams to give greater consideration to the possibilities (and responsibilities) of data
Our partner Snowflake wrote a detailed report about the modern data stack, including information on what companies are in it (in the field of marketing data).
Text continues below image.
What trends do we see emerging?
Now that we know the history of data stacks and how modern stacks benefit the broader business, we sat down with one of our product managers. And we asked to get their view on where the space is headed next. Together, they identified three key areas:
1. No-code platforms, which mean less dependency on data scientists
Remember those marketing teams that we mentioned earlier? They would love to get their hands on some great data management tools. However, advanced coding and data engineering skills are outside their wheelhouse. That’s why we sees the emergence of code-free solutions as a potential game changer. More and more dashboards, reporting, and automated analytics tools are emerging for non-technical users.
2. Unbundling the stack
As we covered, data stacks used to be the realm solely of data engineers within the IT department. However, this tightly knit use case is beginning to become (in a sense) unbound. Teams like human resources, finance, product, and marketing see the value that good data analysis can provide.
3. Democratization of data
Typically, executives can get what they ask for pretty quickly — and understandably so. If they need a report covering the business’s core financial indicators, they can normally expect to receive it from the BI team ASAP. Other teams in the organization can be underserved, though. Marketing teams could be clamoring to understand their performance better, but the many requests to the BI team create a natural bottleneck.
Expect more and more tools to emerge that address this issue, allowing these non-technical teams to service their own data needs easily.
Those tools, like Funnel, can dramatically empower these teams to implement data analysis more deeply into their operations and decision making. Even the famous tech-focused venture capital fund Andreessen Horowitz, has dubbed these newly empowered data-driven employees ‘operational analysts.’ Expect to see this trend continue.
Also read our related post: What is a data stack?
Another emerging trend that our product managers highlighted was the shift away from viewing your data stack as a stack altogether. Instead, they explained, data users will begin to see their many tools as a data mesh.
We guess that means pancakes are set to reclaim their “stack throne,” then.
Is a data warehouse part of the modern data stack?
An important part of any data stack is a central place for data storage. While many companies use a data warehouse for this, there are also other possibilities, such as a data lake, data lakehouse, or a marketing data hub.
How do no-code platforms fit into the modern data stack?
No-code platforms are emerging as a significant component of the modern data stack, especially in the context of data democratization. These platforms allow individuals without advanced coding or data engineering skills to interact with, analyze, and visualize data. By providing a user-friendly interface and pre-built functionalities, no-code platforms enable teams across various departments—be it HR, finance, or marketing—to leverage data for decision-making.
What is a data mesh, and how does it differ from a data stack?
A data mesh is an architectural paradigm that promotes a decentralized approach to data management and data governance. Unlike a traditional data stack, which is centralized and managed by a specific team, a data mesh encourages various departments to act as 'data product owners.' These teams are responsible for the quality, governance, and usability of the data they produce. This shift enables a more agile and scalable data infrastructure, allowing for quicker adaptation to changing business needs and technologies.
What is data transformation?
Data transformation is the process of converting raw data into a format that is suitable for analysis. It's a key step in the data pipeline and can involve cleaning, aggregating, and reformatting data. In the context of a modern data stack, transformation tools often offer automated and real-time capabilities, making it easier for organizations to derive actionable insights from their data.