Data Architecture, the corporate infrastructure connecting business and data strategies, will face competing priorities in 2024. On the one hand, nearly half of organizations will gravitate toward modernizing their data architectures to increase operational real-time analytics and enable AI and ML (machine learning) capabilities. Simultaneously, with concerns about AI impacts, about 80% will prioritize security and Data Governance in 2024.
While data architects have had to find a compromise between security and analytics, 2024 promises to be a more delicate balancing of the two, depending on the state of a firm’s Data Quality. How good a company’s Data Quality is will set the pace for adapting 2024 Data Architecture trends.
Data Quality issues have risen by 15 hours between 2022 and 2023. In 2023 and beyond, 25% or more of revenue will be subjected to Data Quality issues.
With lean budgets due to economic uncertainty, goals to capitalize on AI projects, and a focus on Data Quality, organizations will proceed thoughtfully in setting up for and implementing the following Data Architecture trends:
- Applying accountable Data Governance
- Modernizing technical infrastructure to handle real-time data
- Implementing distributed architectures cautiously
- Increasing interest in machine customers or custobots
- Including third-party data with an organization’s data ecosystem
Applying Accountable Data Governance
The connection between good Data Governance and Data Architecture is no secret. About 54% of executives have made Data Governance a top priority for 2024 to 2025, according to Donna Burbank, a recognized industry expert with over 25 years of experience. Instead, managers face the challenge of driving accountable Data Governance that serves the organization’s data infrastructure.
William McKnight, an advisor to many of the world’s best-known organizations, emphasizes that Data Governance, to be helpful, must establish responsibility and tangible delivery. This accountable Data Governance must also align with an organization’s Data Quality needs for advancing its Data Architecture.
The evolution of Data Governance tools has advanced Data Quality processes within Data Architecture components, and this will continue. Gartner observes that embracing active metadata, managed through Data Governance, will become vital to modernizing Data Architecture in the 2020s.
Nevertheless, emphasizing Data Governance tools alone will fail to give Data Architecture a significant advantage. Dr. Peter Aiken, an acknowledged Data Management (DM) authority, defines Data Architecture as a structure of data-based assets supporting the implementation of an organizational strategy.
To address this gap, accountable Data Governance is a critical bridge for organizations using their Data Architectures more effectively, requiring good Data Quality to support strategy implementations. So, as organizations step toward accountable Data Governance in 2024, it will make sense to implement a solid framework tied to organizational strategies to advance Data Architecture.
Modernizing to Handle Real-Time Data
A rising streaming marketplace of IoT devices like sensors, smart home devices, social media feeds, etc., promises to lure companies into getting more efficient insights from real-time analytics through their data architectures. Companies that do so stand to enter an arena growing at a compound annual rate of 21.5%.
Moreover, unstructured data, including social media, represents less than 33% of managed data today. This figure indicates a need to modernize Data Architecture, such as more efficiently utilizing cloud computing for quick and relevant real-time insights.
Consequently, organizations must manage and scale data integrations and upgrade their pipelines for sufficient data processing and delivery. According to a Komprise 2023 survey, 47% of IT and businesses identified the top unstructured data challenge, moving data without disrupting users. Therefore, the state of Data Quality will set the pace and resource availability for implementing Data Architecture components to handle real-time data.
With increased AI and machine learning tools central to processing data in real time, pressure to use these technologies will grow as organizations give over 40% of their core IT spending to AI by 2025. This drive to modernize Data Architecture plus get a handle on Data Quality will lead to infrastructure turbulence and mixed results in handling real-time data in 2024.
Implementing Distributed Architectures Cautiously
Distributed data architectures, which use multiple platforms, not only handle real-time data, but also reduce the time to access data, offer redundancy, and increase flexibility. These benefits will spur companies to drive their implementations in organizations over 2024.
As companies consider which distributed architectures to apply, they will proceed cautiously. Since Data Quality issues comprise a significant proportion of technology budgets and affect business buy-in to IT projects, many companies will research more established and understood distributed architecture options.
Consequently, the cloud data warehouse will remain at the top of the list, with a 52% year-over-year growth in cloud data warehouse volume. Gartner sees 35% of a data center infrastructure as managed from a cloud-based control plane in 2027.
In addition to cloud data warehouses, 43% of firms will consider the data lakehouse. This data lakehouse implementation will provide an entry point into distributed architectures.
Companies requiring fewer resources to handle and having fewer Data Quality issues will investigate combinations of data fabric, algorithms that unify disparate data across systems, and data mesh, a microservices approach to enterprise Data Management.
For any distributed architecture, successful 2024 implementation will depend on having a solid Data Architecture framework that works well with other Data Management program components to ensure Data Quality. Without trust in data, any distributed architecture will function as a crown jewel to be “admired and analyzed, but not actionable.” This result bodes poorly for getting future Data Architecture funding for modernization.
A Rising Interest in Machine Customers or CustoBots
Gartner has identified nonhuman economic actors, machine customers or “custobots,” that can negotiate and purchase goods and services. Gartner predicts that corporations will generate over 21% of their revenue in 2030 through these algorithms.
While algorithms already interact with data systems’ APIs, the custobots will significantly change Data Architecture outcomes, behaviors, and activities. Microsoft Bing states:
“Machine customers challenge traditional Data Architecture paradigms … As custobots become more prevalent, data architects will be at the forefront of shaping data ecosystems catering to human and nonhuman participants.”
Although this scenario of AI cashing money from a bank account may seem very remote, companies will likely start trying generative AI to help with purchases. Google has already added a generative AI service that compares clothing prices for 2023 holiday shoppers.
Anticipate that custobots, like Google’s, will generate exponentially more data and algorithms through generative AI. Microsoft Bing notes, “Scalability becomes crucial to accommodate the growing number of custobots and their interactions.” Therefore, companies will want to watch this emerging trend in 2024 and plan for data infrastructures that deliver reliable Data Quality given the increased number of customers.
Including Third-Party Data with an Organization’s Data Ecosystem
In 2024, the marketplace will start seeing data ecosystems expand from an organization’s critical assets to including one or more third-party data sources. McKnight has identified aspects of this trend through the data cloud, a Data Architecture allowing organizations to share and exchange data with subsidiaries, partners, third parties, or users on the internet.
Academia has already seen the start of this trend through the Research Data Alliance (RDA), with 13,900 individual members developing and adopting an infrastructure promoting data-sharing.
In 2023, businesses have been researching the use of third-party data to handle rapidly rising data volumes and for new opportunities. This trend is expected to be lucrative, reaching $3.5 billion by 2028.
This evolution impacts organizations in at least three ways:
- Companies must have data architectures that already have and deliver good Data Quality to integrate third-party data and enrich their own.
- Companies with poor Data Quality will find it more difficult to advance their data infrastructure to exploit a more extensive, shared data ecosystem with multiple partners.
- Firms that share a data cloud must pay close attention to Data Quality and coordinate Data Architecture components with each other as they share their data resources.
Given these requirements, organizations should anticipate that 2024 will bring mixed results, with some organizations achieving amazing accomplishments and insights from this data sharing while others may struggle with rushed Data Architecture processes and implementations.
Conclusion
Organizations face complex Data Architecture priorities to modernize and grow revenue from opportunities with real-time data, custobots, and third-party data. Corporations want to achieve this goal while simultaneously managing risks and handling increasingly potential threats, like AI impacts.
The extent of Data Quality will set the pace of Data Architecture implementations and their successes. This trend will become evident as organizations modernize their technical infrastructure and adopt distributed architectures.
Those who continue to fail to address Data Quality issues will accommodate their Data Architecture at a slower pace. To remedy this situation, companies will more likely adopt accountable Data Governance. Even those with intricate and elegant Data Architectures will improve their accountable Data Governance to better budget and implement data infrastructure updates in 2024.
Image used under license from Shutterstock