How to Kick Bad Data Retention and Management Habits

Businesses today are no longer just data-centric; many could more accurately be described as data-obsessed. That’s hardly surprising, particularly given the need to collect and retain information at the scale needed to meet compliance requirements and to deliver competitive advantage.

In these environments, data volumes are growing at exponential rates, and over-retention is quickly becoming a serious drain on resources as well as introducing additional risk. Indeed, keeping data for longer than required (for many businesses, years longer) is quite literally storing up a host of potential compliance and security issues, not to mention the significant costs of maintaining and scaling the underlying storage infrastructure needed to support it.

But how do these issues manifest themselves? Take the problems associated with unstructured data, which now account for anything up to 90% of all the data stored by the typical enterprise. That in itself wouldn’t be such a serious problem, were it not for the fact that many organizations have limited or no visibility into what they hold, who owns it, or whether it has any enduring value. This scenario is becoming increasingly problematic as storage volumes extend into the petabyte and, increasingly, the exabyte scale.

Business and technology infrastructure priorities are also adding to the overall data burden. For example, many organizations now operate more distributed environments, with data collection and management requirements driven by edge applications and the growing use of AI systems.

Looking more closely at the storage architecture trends at play, organizations typically operate a mixed estate of hardware and software from different vendors, across on-premises and cloud environments, all of which contribute to increased complexity and reduced visibility across the data landscape.

Compliance Headaches

As these data estates grow, there is a clear correlation between increasing complexity and the difficulty of maintaining compliance. If organizations aren’t even sure which datasets they hold or where that information is stored, how can they hope to demonstrate the required levels of data governance and control? In these circumstances, data sprawl can quite easily become the root cause of compliance breaches.

Moreover, weak governance processes and/or lack of data visibility can cause major problems around compliance auditing, particularly at scale. The bigger the data estate, the more onerous these processes become. In the finance or healthcare sectors, for example, organizations are typically required to demonstrate compliance with defined data-retention periods. However, when they cannot provide clear evidence of what data they hold or where it resides, they cannot then establish effective ownership or enforce lifecycle policies. In these circumstances, regulatory scrutiny can move beyond routine auditing to a more detailed examination of governance controls, with all the implications that entails.

Over-retention can also create additional exposure in the event of a data breach. This can happen if an incident suggests historical data should have been archived or deleted years earlier. Investigators are likely to question why the data was retained beyond its legitimate purpose, making an already challenging situation significantly more complex.

Healthy Habits

Given these important and increasingly urgent challenges, what needs to change in the way many organizations approach their data management strategy to ensure they remain compliant and efficient?

Firstly, data retention policies need to move forward from accumulation by default to policy-driven control. In practice, this means establishing a clear understanding of the data across the estate, applying consistent policies aligned with business value and regulatory requirements, and crucially, enforcing them at scale.

Rather than relying on manual processes or isolated tools, organizations need the ability to analyse unstructured data across environments and, based on defined policies, execute decisions in a controlled, auditable way. This shifts retention from a passive outcome of growth to an active process of managing data throughout its lifecycle.

From an infrastructure perspective, organizations ideally need enterprise-wide visibility into their entire data estate, including the ability to conduct metadata analysis to identify dormant, redundant, and orphaned files at scale. The lifecycle status of each dataset must be measurable and enforceable across hybrid and multivendor environments. This then enables them to implement appropriate policy-based data mobility and an archiving process to control and then minimize over-retention and data sprawl.

And then there’s the question of storage infrastructure, where currently, premium storage tiers are frequently used for inactive or low-value data. This can significantly increase both capital and operational expenditure, as capacity upgrades can be deferred and high-performance resources reserved for genuinely active workloads.

Get the management and technology strategy right, however, and organizations can develop much healthier data habits, where investment and performance are aligned with compliance requirements, creating a clear win-win over the long term.