The allure of building custom, in-house marketing data pipelines can be compelling for many organizations. From a zoomed out perspective, it promises greater control, ability to customize to specific needs, centralization of data and maybe some cost savings.
However, while these benefits are theoretically attainable, they come at the price of navigating a myriad of challenges, especially when dealing with the intricate nature of marketing data. There are several aspects of marketing data which makes it more challenging to work with compared to other types of data you find in an organization. We’ll also look at how Funnel can help data engineers overcome these challenges.
Let's delve into the main challenges a data engineer will encounter in the setup and maintenance of in-house marketing data pipelines:
1. Setting up API connections
The foundation of this project will be building the API connections. It’s important to get exact specifications of requirements from the end users. In the case of marketing data, it’s typically the marketing team or senior leadership at the organization. The scope might range from just a few to tens of different platforms, and it could span many accounts per platform -- depending on your business size and marketing setup.
After the platforms and accounts have been mapped out, the level of granularity needed has to be considered. It is imperative that this happens early in the process, because the complexity involved in both constructing and maintaining marketing data pipelines increases significantly with the depth of granularity needed. Very basic reports containing campaign, costs, clicks, and impressions data are usually straightforward, but if there are additional requirements (like conversion data, geographical data, demographical data, and so on), it will often require piecing together results from multiple API-calls.
Facebook Ads is an example of an API that can be challenging to work with, even for experienced data engineers. Building the initial integration is a feasible task for most data engineers, but maintaining a steady pipeline is the challenge. Quota and rate limits need to be considered. And when data fetches fail, there needs to be a mechanism for retries. We will delve deeper into these and other challenges of maintaining data pipelines as we progress through the rest of the list.
2. Navigating API documentation
The front-end UI and back-end of marketing platforms tend to look very different, because they are generally developed and maintained by different teams on the platform side. This means that it can be challenging to understand how to translate a request from the marketing team (which is based on the UI), into what type of API call(s) is needed to retrieve data from the backend
3. Scheduling and handling retroactive data updates
Marketing data is dynamic. This means that values are expected to change as time progresses, and this needs to be considered for each platform that is integrated. On top of scheduling regular data refreshes, these also need to include a mechanism to handle instances where data is updated retroactively (i.e., re-downloading and updating existing data periodically). This ensures data consistency and accuracy over time.
4. Error and quota management
Errors are inevitable. A robust system should be in place to handle errors, perform retries, implement backoffs, and always respect rate limits and quotas set by platforms. This prevents overloading systems and safeguards against potential data loss. If these are not calibrated correctly, it can cause significant disruptions to data pipelines.
5. Budgeting time and resources for maintenance
APIs evolve. Whether it's due to platform upgrades or changing data needs, regular maintenance and updates are mandatory to ensure data flows are uninterrupted.
Larger API’s tend to update to a new version on roughly a quarterly basis. With each new API update, a number of fields are added, deprecated, renamed, or recategorized. Depending on how exposed you are to these changes, it will incur hours or weeks of work for your data engineers to have everything functioning as expected. It can also lead to new field requests from end users, more on that under point 12 below.
6. Non-communicated API changes
Platforms often communicate API changes at the same time as they are rolled out. Proactive monitoring and rapid response mechanisms are essential to avoid protracted data interruptions.
7. Security: Access tokens and credential management
Security is paramount. Ensuring that access tokens, credentials, and other sensitive data are securely stored and managed is non-negotiable. Furthermore, API access is usually coupled with working credentials for the platform in question, which means that when credentials lose access (as in the case when a person leaves the company), credentials need to be rotated to new working ones so as to not lose data access.
8. Data mapping
Marketing data is largely heterogeneous, but there are commonalities across different platforms that often provide value when they are mapped together. Cost, clicks, and impressions are basic examples of such metrics that exist in most types of marketing platforms where mapping them together produces a holistic overview of marketing efforts.
Depending on the number of platforms and ad accounts you have, harmonizing a metric like 'clicks' across platforms is usually straightforward. However, more advanced challenges can involve splitting out campaign name structures to allow granular reporting across platforms. Apart from the most basic cases, data mapping typically requires a keen understanding of the datasets you work with. This ensure you don't map together data that is not homogenous -- and thereby provide inaccurate results that lead to potentially harmful decisions.
9. Currency conversion
For global businesses, dealing with multiple currencies is standard, yet cumbersome. Implementing reliable and up-to-date currency conversion processes ensures that financial metrics are always accurate and comparable. This is another challenge which becomes exponentially more challenging the more currencies that are involved
10. Managing data endpoints
Once data is fetched, cleaned, and structured, it often needs to be sent to various tools for analysis, visualization, or storage. Efficiently managing these transfers while ensuring data integrity is essential.
11. Building new API connections
As marketing efforts expand, new platforms and tools come into play. The ability to rapidly integrate new API connections ensures the pipeline remains relevant and comprehensive. As we saw in the first point above, the challenge level depends on the platform.
12. Building support for new connectors, reports, and fields
With evolving business needs, the type of insights required will change. Similarly, new marketing platforms pop-up regularly, and existing platforms evolve and expand their offering. These new opportunities generate data, which, in turn, is made available via the respective API’s. This requires either building support for new connectors or adding fields and reports to existing ones. For most marketing teams, these are frequent occurrences. In order to keep the marketing team working quickly, the data integration needs to follow suit, but it’s easy for requests to pile up as a ticket backlog.
13. Stakeholder management
Stakeholder management involves identifying, understanding, and effectively engaging with your marketing team and executives who rely on the output of what is being built. Depending on the size and geographical distribution of your team, the time it takes to align might range from a few hours every month to regular check-ins every week.
Cost benefit analysis: Is building in-house marketing data pipelines worth it?
As we've explored, the intricacies of marketing data present both unique challenges and ongoing commitments. Yes, with a talented data engineering team and sufficient time, an in-house solution can be crafted. ETL tools like Airbyte, Supermetrics and Fivetran make the initial setup feasible. But it's important to understand that getting data into the data warehouse is a relatively small feat. The real value lies in what you are able to do, and spend time on, after the data is there.
Considering the recurring tasks and potential for unexpected issues, it's crucial to ask yourself, "Is the ongoing investment of your data engineering time the best way to achieve efficient, error-free, and comprehensive marketing data management?"
Remember, data engineers are a premium asset. This asset, alongside potential data scientists and analysts, should be used for activities that generate business value. Balancing their workload and expertise against the recurring demands of maintaining marketing data pipelines can be a challenge in itself.
Why Funnel is the game-changer for data engineers
Enter Funnel: the platform designed with the specific aim of streamlining the complexities of marketing data management. The marketing data hub acts as your copilot, ensuring that all of the hangups associated with self-built data pipes are avoided.
Its 200+ connections are managed by our team, ensuring they remain up-to-date with the latest platform requirements. Plus, Funnel’s integrations and no-code approach make it easy for engineers, marketers, and business teams to work harmoniously.
In essence, Funnel is more than just a tool – it's a comprehensive solution designed to ensure that data engineers and marketing teams can harmoniously co-create, manage, and optimize data pipelines with minimal friction.
Related reading: An introduction to marketing data warehouses
Disclaimer: The featured image for this article was created using generative AI.