Several key drivers within the data and software establishments are pushing the Data Management industry towards Agile Business Intelligence (BI). Increasing amounts of data and its importance in business practices requires expedient data integration flexible enough to keep pace with rapid changes in competitor strategies, customer habits, regulations, as well as national and global economics. The emergence of SCRUM methodologies has enabled software to develop at the pace of an organization’s needs. Iterations, self-service BI tools, and Data Virtualization have allowed BI to extract value out of Big Data technologies and other sources.
However, Agile BI is less a concrete term (denoting the facilitation of rapid, actionable insights in an increasingly mutable business world) than an ideal which largely varies by organization. It is a program (not a project) that evolves alongside an organization in accordance to its unique concerns which address areas of improvement and raise the top line. Whereas conventional BI programs can require several months or years to implement and produce tangible results, Agile BI can succeed within weeks by following The Twelve Principles of Agile Software, including:
- “Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage.”
- “Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.”
- “Business people and developers must work together daily throughout the project.”
- “Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.”
- “At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.”
Business, Operations, and IT Interactivity
There are certain technologies and methodologies intrinsically associated with Agile BI, but conventional BI programs can become Agile by following the aforementioned principles and utilizing various tenets of SCRUM. Although the goal of many of the former technologies is to produce methods whereby IT’s involvement is minimized, refinement of those technologies to suit the individual needs of an organization requires IT working as developers. In such a case, IT plays an instrumental role in implementing and refining features and aspects of BI to make it more applicable for users.
An implicit component of Agile BI is its utility for business and operational processes as well as users. The point is to facilitate decision-making information that supports action. Tailoring of various BI tools to provide the sort of data that is most frequently required of an organization – and which may be prone to sudden changes – primarily involves iterations in which users issue feedback to IT, who implement changes to improve analytics, reporting, and visualization processes.
The most pronounced difference between traditional and Agile BI is the length of time that individual iterations are implemented, referred to in the SCRUM Guide as sprints. According to Robert Solonynka of Data Experts in his Enterprise Data World 2013 session entitled “Making Agile Work with BI Projects: Tips & Best Practices”, conventional BI programs map out goals several months ahead and are considerably slowed by a lengthy documentation process; feedback and improvements based on the same goals in an Agile BI environment take weeks. The full SCRUM ideology and methodology need not be employed for organizations to implement an iterative process with short periods of feedback, which is used by IT to create advances or changes in the flexibility of a BI program.
A particularly critical component of the iterative process is testing (usually performed by IT) to ensure that adjustments from user feedback are properly accounted for in future prototypes of BI tools. Manual testing – which can be lengthy and impractical for deadlines of just a few weeks – should be substituted for automated testing. Several vendors including Zuzena, Oracle, and Microsoft issue products that can provide both automated and continuous testing. Testing is a key part of the overall dialogue between business and operations users and IT, which is vital to the ongoing Agile process.
The configuring and implementation of requirements is another important distinction between conventional and Agile BI. Whereas traditional program requirements are defined well in advance in meetings involving stakeholders and recorded as long-term objectives, Agile BI requirements fluctuate according to user needs and are based on short-term goals. Such requirements are generally evinced in the nature of the prototype created via user feedback, which naturally informs the next round of feedback based on actual usage of the tools. This way, the focus is on the BI product itself instead of on the process to create it.
The degree of refinement and modification to BI tools based on user input varies according to requirements. IT input can be as little as tweaking particular features and disabling those that are not required, or as involved as making substantial revisions in code. The collaborative process of Agile BI requires users to indicate what features they need and to prioritize them. In many instances, end-users will not be aware of the entire scope of features in particular BI tools, which is another reason that interaction between business, operations, and IT is vital to the success of Agile programs.
Aiding Agility: Contemporary Technologies
Contemporary BI tools certainly can assist the movement towards Agile BI by providing access to quantities and sources of data that are difficult to query with traditional proprietary data warehouses and relational databases. Data discovery tools are one of the more salient of such technologies which can determine and highlight difficult-to-discern relationships between multiple data sources. Data discovery tools can readily incorporate information from individual silos and help an organization form a complete picture on which to base decisions. Trillium’s TS Discovery and IBM’s Information Agenda are examples of some of the many BI components incorporating data discovery tools, reducing the involvement of IT and facilitating agility.
Contemporary solutions frequently incorporate data discovery tools with additional self-service BI technologies to increase agility because, as Forrester’s Boris Evelson points out, such products “enable all types of users (casual users, power users, and executives) to self-serve for new queries, reports, analytics, and dashboards.” Self-service tools include graphical user interfaces and dashboards that provide distinguishable visualization techniques, predictive analytics, and ad-hoc reporting – without the use of code. End users have greater control over what forms of data are generated and simply need to consult with IT departments to prioritize and access various features.
Lastly, Data Virtualization is instrumental in furthering Agile BI because it allows for users to access a multitude of different data services on demand. Data Virtualization eliminates the need for costly data storage and allows for users to access the most recent data to assist in real-time decision making. Virtualization works with any number of BI applications and dashboards, and allows for organizations to select the most pertinent data at the appropriate time, providing a degree of adaptability on which the Agile BI movement is based.
Ahead of the Curve
All of the aforementioned methods of creating and maintaining an Agile BI program inevitably involve meaningful interactions between IT, business, and operations personnel. Data Virtualization, data discovery tools, and self-service BI greatly reduce the need for iterations but not their importance – which becomes clear once processes, needs, and applications of data fluctuates due to internal or external developments. True agility can only be achieved through iterations (no matter how frequently or slight), which greatly minimize the gap between facilitating changes and implementation when profound alterations are required. Organizations foster agility by constantly identifying areas of improvement.
Ultimately, an Agile BI environment serves to empower business and operations personnel while boosting data’s value and influence in critical decision making. Organizations will benefit in the long run by staying abreast of marketplace developments and forging a reinforced mutability essential to extracting value from data. An important by-product of the Agile environment is the simplification of BI in general, a movement which is gaining ground through advances in mobile technologies and only adds to BI’s efficacy.