The increasing prevalence of Big Data and Software Oriented Architecture has certainly made the Cloud appear to be one of the most viable mediums for accessing and managing data. Nonetheless, there are several ramifications of Cloud deployments that considerably exacerbate Metadata Management and significantly worsen aspects of overall service including:
Some of these issues pertain to public Clouds and service provider constraints; others relate to basic Data Governance concerns that, once external sources and Big Data are involved, spiral beyond control. Solutions range from more prudent management of relationships with service providers to an emerging toolset provided by both vendors and Cloud purveyors.
Earlier this summer Forrester issued research that evinced a marked dissatisfaction among Cloud customers pertaining to a dearth of Metadata issued on the part of service providers. A Network World article reveals that:
“…often information about each cloud workload isn’t clearly available to customers from the cloud vendors…all of this rich metadata is, obviously, available to cloud vendors who leverage it to deliver new products and services to customers. The inference being that cloud vendors are profiting from the very data their customers create.”
It is critical to note that the term Metadata in this instance refers to data about one’s Cloud service, and not necessarily the core definitions and consistencies of data elements that is a hallmark of prudent governance. Still, the former definitely impacts the latter as critical Metadata about performance and configurations, for example, can help to optimize data-driven processes and refine the way organizations leverage their data in the Cloud. One of the key points of the Forrester survey is that frequently with large, public Cloud providers, organizations are operating without this Metadata – while their purveyors are.
Subsequently, such Metadata complications – in the form of a general customer-based ignorance of Metadata pertaining to Cloud usage – can perhaps meet the nexus point of all of the aforementioned concerns in terms of regulatory compliance issues. Non-compliance results in costly financial repercussions and can present considerable problems with operations, while reflecting configuration and security troubles. A fair amount of the Metadata associated with these issues pertaining to Cloud service providers is generated in real time or near real time, which is critical for giving organizations an opportunity to react accordingly and remain compliant. The issuing of alerts related to such Metadata can greatly ameliorate these regulatory concerns; organizations should therefore insist on such notifications and ensure that they are part of their service provider contract. Implementing regulatory controls is a critical facet of Cloud service. Without Metadata about issues that affect regulatory compliance, organizations are perhaps incurring more risk than they are benefit.
Performance and Cost
Metadata difficulties associated with the Cloud typically present dual repercussions that involve both aspects of performance and price. In most instances, performance efficiency has an inverse relationship with cost in which the higher one is, the lower the other is. In several instances in the Forrester survey, respondents indicated that they were not fully apprised of costs prior to service. However, the exclusion of pivotal Metadata pertaining to varying facets of service and cost deprives the enterprise of awareness of how to best reduce costs and improve performance with a particular provider’s model.
Additionally, some of the more pervasive difficulties with Metadata in the Cloud extend beyond the efforts of service providers and reside squarely with the enterprise. Organizations without rigorous Metadata standards based on business definitions, requirements, and rules are at risk for suffering from Data Quality complications such as duplications, untrustworthy data, completeness, and timeliness. Moreover, these complications exist for public, private and hybrid clouds – independent of service providers. However, meticulously defined and maintained Metadata can help assuage these Data Quality issues and the overabundance of data which Big Data can engender by providing a basis for governance and Data Management systems that give data meaning. There are numerous Semantic standards based governance tools that can derive this meaning regardless of data structure, once an enterprise makes a dedicated effort to align its Metadata and base them on actual definition and requirements relevant to its business. It is equally important to base Metadata on specific uses for applications, and ideally try to make the data elements that inform those applications as enterprise-wide as possible to prevent silo approaches.
Vendor and Application Lock-in
Another facet of Cloud deployments that can present difficulties for Metadata Management pertains to the use of Metadata in specific Cloud applications. The propensity for vendor lock-in is perhaps one of the more prominent dangers that Cloud based applications present for Metadata, because many vendors and service providers have proprietary Metadata for their applications. Proprietary Metadata can not only result in situations in which the vendor or provider can dispute the ownership of an organization’s data after utilizing the provider’s service (since it now contains the provider’s proprietary Metadata). Additionally, it can result in situations in which an organization needs to continue to utilize that particular application to derive meaning from its data, because the data has been tailored to the Metadata standards of an external application. These situations are particularly relevant in instances in which enterprises are utilizing applications that may not be available outside of specific vendor or provider.
The majority of the complications surrounding Metadata that Cloud deployments produce simply require service providers to share an organization’s service Metadata with end users. Therefore, it is vital for companies to stipulate the sort of Metadata they require for operations, performance, and cost when negotiating service level agreements with providers. In many instances, however, there are still Metadata generated that an organization is unaware of and which providers will later on leverage into additional products and services. As such, prudent organizations should consider additional solutions from third-party vendors in the ever evolving market for Cloud services. According to InfoWorld: “There’s a large growth market for third-party tools that provide performance management, governance, accounting, and security services for public clouds.”
In addition to taking active efforts to manage Cloud service providers, the enterprise must also implement management tools specifically for data accessed through the Cloud. Many of these tools have a real-time value that can ameliorate regulatory and operational issues quickly in the form of issuing alerts and provisioning resources on demand to account for variations in workloads. Some products simply cache Metadata and enable users to analyze them and refresh as necessary. Oftentimes, these solutions interface directly with APIs to leverage Metadata directly from Cloud applications in any number of ways, from creating action to informing relevant personnel of recent developments. Such solutions create a situation in which organizations are better able to monitor the activities of their Cloud deployments and their Metadata in way that can increase the transparency of service providers.
Traditionally, issues of security have surrounded questions about utilizing the Cloud. Presently those issues have been augmented by those pertaining to Metadata and effective management of service providers. While incorporating third party Cloud management tools can help to reduce those issues, they also add to an organization’s cost and the overall intricacy of Cloud deployments. Such tools and Cloud services will work better in the long run if providers share more Metadata. With increased transparency on the part of Cloud providers and supplemental means for Metadata management tailored to the Cloud, organizations can overcome these complications.