The legacy architecture of some organizations’ data systems may need a serious upgrade to stay competitive. For example, the old architecture of a business may provide a clumsy fit when accessing cloud services and take longer to transfer data and perform tasks within the cloud. This results in higher cloud costs. Additionally, upgrading to a modern data architecture offers numerous benefits, ranging from faster overall system performance to improved customer experiences. 

Upgrading the data architecture often includes upgrading or replacing software (possibly through the use of the cloud) and sometimes the hardware.

The decision to upgrade to a modern data architecture should be based on the needs of the business. An architectural design should be upgraded and redesigned when it restricts the addition of new software and no longer supports the business’s needs or strategic objectives. 

Some of the components used in architectural designs include:

• Data pipelines: Data processing elements that are connected in series, with the data output of one element acting as the input for the next one
• Cloud storage: A cloud computing provider uses the internet to store data for customers
• APIs: Mechanisms that allow two software components to communicate using definitions and protocols
• Machine learning: Algorithms are used to support learning, in turn improving decisions and making more accurate predictions
• Data streaming: An automated software platform that supports sales and purchases and can be used to develop business intelligence
• Kubernetes: A platform for managing multiple containers

Upgrading to a modern data architecture requires determining what your current system can do, and what you want it to do. A modern organization design can combine scattered information from a variety of sources to provide useful business insights. Redesigning the architecture will improve data storage management and take care of data migration and reformatting of the data. It can also support the Internet of Things (IoT) and increase efficiency through the use of the cloud.

Reasons for Upgrading to a Modern Data Architecture

There may be a variety of reasons for maintaining legacy system architecture. It could be a sense of comfort with the outdated programming languages and old computer hardware. There may be financial reasons or timing concerns. 

However, the bottom line is that businesses cannot expect to remain competitive by preserving their legacy system architecture forever. The survival of organizations depends on acknowledging the need for change and recognizing the evolving goals of the business. This is especially true when competing with agile businesses that were “born in the cloud” and already have significant flexibility and a technological edge. 

Reasons for upgrading to a modern data architecture include:

The customer experience: Providing a good customer experience is necessary for developing a repeat customer base. Customer loyalty can be tricky on the internet. Websites offering unique services or products are more likely to draw repeat customers, providing they’ve had a good first-impression experience. 

For more general products and services, a potential customer searches the internet for the product they’re looking for, and screens them based on price and appearance. If they recognize a website they’ve done business with before, and they’ve had a positive experience, there’s a high probability they’ll return. If they had a bad experience, they’ll deliberately ignore that website.  

If disappointed customers have become the norm when doing business with your website, diminishing sales can be expected. A modern data architecture system is crucial to providing a good customer experience.

Regular downtime: An architectural framework suffering from significant amounts of downtime can throw schedules off and cause financial losses (uncompleted projects, a loss of sales, staff who can’t work).

Missed opportunities: While mild chaos may be the human norm, extreme chaos results in missed opportunities. A data system that causes problems on a regular basis can be a source of extreme chaos.

Modern Data Architecture Models

If you’ve realized now is the time to upgrade your legacy system architecture, then you’re ready to start. Some modern architectural models are listed below. However, you can always create your own much simpler model, which may better meet your needs.

Data mesh: This architectural design requires the cooperation of other business departments, branches and/or partners who share the same data. Each department, branch, or partner is responsible for maintaining the data in a specific uniform format – the same uniform format the other branches or partners use to store their data. This system allows organizations to store and share large amounts of data from a variety of designated locations.

The data mesh design allows access to large amounts of useful data for research, without the need (or time) to reformat unstructured data or data that is formatted differently. (Either the data is reformatted automatically – possibly an ETL pipeline – or is manually reformatted before the data goes to permanent storage.) The data mesh philosophy can be extremely useful to businesses that need scalable data storage and are expanding quickly. (This is a low-tech, and not necessarily expensive, solution.) 

The basic problems with data mesh are the need for cooperation from the other departments, branches, or partners, and changing the behavior of staff to support the new system.  

Data fabric: A form of data architecture that relies heavily on technology to reformat data from a variety of sources to build a storage system containing uniformly formatted data. The data fabric system combines certain Data Management technologies – such as data catalogs, data pipelining, Data Governance, data integration, and data orchestration. The primary goal of data fabric is the delivery of enriched and integrated data. 

Data fabric uses metadata to integrate, unify, and govern different data environments. It typically also uses machine learning to improve both data accessibility and security by automating, standardizing, and connecting the organization’s Data Management practices.

Data fabric is very technology-dependent, which raises the concern of vendor lock-in. Another issue is that, while the majority of data fabric tools support cross-platform interoperability, they do not necessarily work well with other, more common tools.

Data hubs: A data exchange center used to share and transfer curated data between different parties. It is a centralized data storage system, which manages the data and allows staff to see how data moves through the system. Data is acquired from a number of sources – both analytic and operational – through replication or publishing and subscription interfaces. A well-designed data hub may use artificial intelligence and machine learning, and should include features such as:

• Data storage
• Indexing
• Harmonization
• Processing
• Metadata
• Governance 
• Search queries

Data hubs are systems that receive data from a variety of sources and can be accessed by several users, primarily for business intelligence (BI) research. A modern data hub is a form of interconnected architecture with several sources and target databases. The primary goal of a data hub is not storage, but to process the data for BI purposes. (Storage in data hubs is often considered temporary, as data hubs are generally used for presentation purposes.) 

A common complaint about data hubs is that they take several years – perhaps over a decade – to pay for themselves.

Data lakehouses: Data lakehouses are considered a solution to the problems researchers have experienced when working with data lakes and data warehouses. 

The ever-growing amounts of unstructured data being gathered, stored, and used by organizations has become an irritating problem for researchers when working with a data warehouse or a data lake. The kinds of data being gathered currently include large amounts of data from the internet of things, as well as images, video, and audio, and other types of unstructured data. Data lakes store massive amounts of raw, unformatted data, but it can be complicated to locate specific files. Data warehouses, on the other hand, take up valuable time for cleaning, reformatting, and indexing huge amounts of raw data before storage, which may slow down research.

Data lakehouses are designed with their storage and computing processes separated. They store the data inexpensively, and index it for easy retrieval.

Although data lakehouses are a relatively new concept, they show great promise as a means of organizing data for research. They are still somewhat experimental but seem to be getting good reviews, overall.

Staying Competitive      

The philosophy of “If it’s not broken, don’t fix it,” does not apply to e-commerce, where competition is extreme. The “evolving business” philosophy is much more useful.

Mark Rogers of the Logicalis Group emphasizes that digital transformation is essential for success:  

“Change is now the norm. Just as we set a course based on our understanding of the technology landscape, that landscape changes. CIOs must accept that change is constant and work out how to get on the front foot – to shape change rather than being governed by it.”

Image used under license from Shutterstock.com