CPG Data Silos With AI and Analytics

Editor's note: In the evolving CPG landscape with trends like BNPL, D2C, omnichannel retailing, and agile operations, personalization takes centre stage. At the heart of this digital transformation is Data.

This blog highlights the significance of breaking data silos to ensure accurate data flow to the right party at the right time. It will assist in the pursuit of customer-centricity and unlock new opportunities through detailed insights. Pertinent focus areas to enable right execution include data governance, cloud solutions, and AI.

“Often, we find that a consumer company has the data it needs to unlock business improvement, but the data resides in different business groups within the company” - McKinsey article

Corporates today, are using various systems to make things work. Each department, like sales or supply chain, has its own set of tools to get their jobs done. But sometimes, all these different systems can cause a problem known as "data islands." This means that the data we need gets stuck in separate places and it's hard to see the big picture.

Large organisations use multitude of systems and if these systems don’t communicate, different departments will never agree on what data is correct.

Let's consider an example to enhance our understanding: In the process of Sales and Operations Planning (S&OP), the central team comprises leaders from sales and operations. It's essential for them to have a clear understanding of data and terms before embarking on future planning. While these departments typically utilize different systems for their daily tasks within most organizations, effective planning demands a smooth integration of sales and operations data and systems, requiring minimal alterations.

On average large organizations use 367 software apps and systems. As organizations grow, ensuring the connectivity among systems becomes increasingly challenging, necessitating extensive interconnections. With increasing M&As and higher consolidation, the challenge becomes even harder.

Now, let's dig deeper and explore the different types of systems that companies use.

1. Trade Promotion Optimization (TPO) System: Blue Planner - This system aids in optimizing trade promotions to boost sales and profits through effective planning and analysis.

2. Category Management System: JDA Category Management - JDA offers a comprehensive platform to manage and analyze product categories for enhanced decision-making and improved sales strategies.

3. Point of Sale (POS) System: Square POS - Square's system provides seamless transaction processing and business management capabilities at the point of sale.

4. E-commerce Platform: Shopify - Known for its user-friendly interface and robust features, Shopify empowers businesses to create and manage online stores efficiently.

5. Enterprise Resource Planning (ERP) System: SAP ERP - SAP's ERP system integrates various business functions like finance, inventory, and HR, providing a centralized platform for streamlined operations.

As companies grow, they need to make sure that information can move freely between these systems. They need to prevent data from getting trapped in separate places.

These data silos are typically created due to legacy systems, decentralized data management, and departmental boundaries. Let's explore more into the core reasons that give birth to silos.

1. Lack of Data Lineage and Common Data Dictionaries:

Consider a corporate merger and acquisition process where executives from both companies come together. Clear understanding of financial data and terminology is imperative before they can strategize. Despite each company using distinct operational tools, a successful integration of financial data and systems is vital for effective decision-making.

2. Reliance on Individual Knowledge:

System knowledge is concentrated within specific individuals without proper documentation. When these knowledgeable individuals leave, the system becomes less open to changes. Teams resort to creating their own data solutions, often relying on spreadsheets.

3. Master Data Inconsistencies:

Discrepancies within master data can emerge across diverse teams, verticals, or departments within an organization. These variations lead teams to create their own data versions due to the differing master data.

4. Lack of End-to-End Visibility:

Gaps in visibility across the entire value chain of operations.

Data silos arise due to missing data at different stages of the process.

Example: CPG companies having primary and secondary sales data but lacking tertiary sales data.

According to Forrester, 30% of time of an Analyst is spent trying to find the right data and information – sapping 2.4 hours daily.

Navigating Data Silos: Internal and External Challenges for CPG Companies

For the CPG sector the presence of data silos extends beyond the organization's internal systems. Brands often encounter external data silos due to limited access to vital information held by external entities. This situation commonly arises when CPG companies lack direct control over data stored by retail outlets that sell their products. Consequently, CPG enterprises frequently resort to procuring second-party data from these retailers, specifically for data and analytical insights extending beyond the point of sale.

Implications for Business Efficiency and Decision-Making

• Data inaccuracy leading to low trust on data
• Manually transferring data is time-consuming, expensive and error prone
• Limits productivity
• Provides a lacklustre customer experience
• Limits collaboration across the business

which results in incorrect analysis and leads to poor decision-making. Businesses lose opportunities due to a lack of access to accurate data timely. According to Gartner's research, small to mid-sized businesses may face losses of up to $15 million per year, because of making ineffective business decisions due to outdated/ inaccurate data.

Nestle is centralizing its customer data and consent management. The company is trying to reduce the number of marketing applications it uses, which will help to reduce the number of data silos. It is now taking a more strategic approach to customer data, viewing it as a corporate asset rather than just a collection of data points.

One of the companies used Jira for customer servicing, and a sales CRM Previously, these systems were not communicating with each other, so the sales and customer service departments operated independently. Salespeople were unaware of customer service tickets raised by existing customers and were trying to upsell new products. From the POV of the customer, they were getting calls for a more expensive product even though they had challenges in the existing one. This caused friction between the company and the customers. However, by integrating their systems using APIs, they successfully enabled seamless communication and bidirectional updates between departments. This meant that salespeople could see that a ticket has been raised.

Connected planning and analysis is necessary for enterprises and to break down the incongruent systems it is important to focus on an Integrated Data layer. It can come in various formats which are explanied in the next section.

Data Lake:

Data lakes consolidate diverse data sources into a centralized repository, regardless of format or structure. They enable data storage at scale, accommodating both structured and unstructured data. By providing a unified storage platform, data lakes promote collaboration and ease of access for various teams.

Data Mesh:

Employing a data mesh approach to tackle data silos involves decentralizing data ownership to individual teams. These "data product teams" curate and manage specific data domains, promoting collaboration and accountability. By standardizing data interfaces and formats, integration becomes smoother, enhancing data accessibility and coherence. This approach breaks down silos, fosters collaboration, and accelerates insights across the organization.

Enterprise Data Warehouse (EDW):

EDWs centralize data from multiple sources, harmonizing it to create a single source of truth. They offer structured and well-defined data models that enhance consistency and accuracy. EDWs support complex queries and reporting, reducing the need for data extraction from different systems.

Data Fabric:

Data fabric solutions offer a unified data integration platform that connects various data sources and systems. They enable seamless data movement, transformation, and access across the organization. Data fabric enhances data visibility, governance, and control, reducing data fragmentation. Data-integration platform.

Centralizing your customer data and consent management is a valuable step toward enhancing your data operations. Talend Data Fabric, offered by Qlik, offers a cloud-based solution for data centralization. It streamlines data integration, ETL processes, data governance, security, and regulatory compliance. Notably, certain companies have achieved a 10% annual reduction in maintenance costs.

Source- qlik

Data fabric can effectively break silos of legacy systems by providing a unified data source. Silos lead to data fragmentation; Data fabric acts as a virtual layer and connects disparate data sources. Organizations can eliminate the need for data duplication, achieve real-time data synchronization, and enable data sharing across systems.

Additionally, data fabric provides data governance capabilities, ensuring data quality, security, and compliance across integrated systems. 

Enterprise data governance is crucial for organizations aiming to establish a single source of truth. To achieve this, they can leverage automated no-code ETL tools that streamline data integration from various sources. By utilizing these tools, businesses can extract data, apply necessary transformations, and load it into centralized data repositories. 

With the right interface, even non-technical users can participate in the data management process, ensuring data accuracy and consistency. Furthermore, the integration of open-source tools and data visualization platforms enhances data accessibility and enables effective decision-making. Together, these elements empower organizations to embrace data-driven strategies while maintaining efficient data governance and management practices.

Why is Expert intervention needed while dismantling silos?

Data Privacy and Security: With the increasing focus on data privacy regulations (e.g., GDPR, CCPA), organizations must implement robust security measures and ensure compliance when sharing and analysing data across departments.

Data Integration and Interoperability: Legacy systems and disparate data formats can pose challenges when integrating data from different sources. Organizations must invest in technologies that facilitate seamless data integration and interoperability.

Unstructured data: most systems also store unstructured data which can be hard to integrate into cross dimensions, this may lead to loss of information. This is a leakage that can be reduced by utilizing AI/ML as mentioned above. 

• More efficient use of and access to data between systems for simple platform scalability.
• Never-before-seen control over the identification and movement of data is essential to creating a unified/single view of the consumer ahead of the end of the cookie.
• Data cleansing, standardization, validation, and householding (including identity resolution) a customer data infrastructure (CDI) solution.
• Segmentation, analysis, and activation (including dynamic personalization and lookalike modelling), a data activation tool. 
• The ability to easily stitch unified data into rich profiles for a 360-degree customer view, optimize targeting across channels, support the full customer journey in real-time, and analyze results with a key focus on cross-selling to existing purchasers. 

1. Data Integration and Aggregation: AI can automate the process, regardless of data formats or structures. ML algorithms can analyze data schema, identify relationships, and reconcile inconsistencies, enabling a unified view of data from different silos.

2. Data Cleansing and Enrichment: AI algorithms can automatically cleanse and enrich data by identifying and resolving inconsistencies, duplicates, and missing values. This process ensures data accuracy and completeness, making the integrated data more reliable and useful for analysis.

3. Intelligent Data Matching and Entity Resolution: AI can assist in matching entities across various sources, such as customers, products, and suppliers. Machine learning models can learn patterns to identify data related to the same entity, overcoming data silos, and ensuring data integrity.

4. Automated Data Governance and Compliance: AI/ML models can learn data governance policies, detect anomalies, and enforce rules, ensuring secure data sharing across departments.

5. Intelligent Recommendation Systems: AI-powered recommendation systems can suggest relevant data sources, reports, or insights to users based on their roles, preferences, and historical usage patterns. Beck's automation is a notable example where AI lead the whole business value chain from production, recipe, to marketing and designing. It the “AI made BEER.”

We have been serving clients in the CPG/FMCG space for over a decade, providing them with invaluable data insights by helping them set up data warehouses to integration to assiting them with visualisations.

Leverage our expertise in setting up robust data warehouses, seamless integrations, and impactful visualizations. Our unparalleled proficiency insetting up Data Lakes and next-gen AI ensures faster and more efficient synchronicity, empowering your business with actionable insights. Elevate your data strategy today – let's embark on a journey to unlock growth and innovation. Contact us now to explore the limitless possibilities of data-driven success!