Data Management in Agile Development

By   /  September 3, 2013  /  No Comments

global data twoby Dwayne Phillips

Agile Development is currently a popular method for developing software. The popularity is because the Agile Methods are well suited to changing requirements: a common occurrence in today’s mobile apps. Questions remain regarding how to manage the data in an Agile development because documentation and documenting are given low priority.

Fundamentals of Agile Development

Agile Development was formally established in 2001 when a group of developers issued the Agile Manifesto. The methods, there are at least a dozen development methods that claim a link to Agile, emphasize a few concepts. Among these are:

  • Delivery of working software in short periods of time (four to six weeks)
  • Delivery of working software in iterations (a few more functions in each iteration)
  • Self-managing teams (no formal project manager)
  • A user representative works full-time with the team
  • Daily builds of working software

Note the use of the phrase “working software.” One of the primary values of the Agile Manifesto is “(we value) working software over comprehensive documentation.” It is the “comprehensive documentation” that provides the bulk of data on traditional or planned software development projects. The comprehensive documentation is practically absent on Agile Developments. There is, however, valuable data on Agile Developments and managing those data is a challenge.

Tutorials on Scrum, a popular Agile Method, can be found here and here.

Managing the Data: A Global View

First consider managing the large data on an Agile Development project. The large data come in three basic packages:

  1. The Project Backlog
  2. The Iteration Backlog
  3. The Software (from each iteration)

A “Backlog” is a list of functions and features that the users want in the software. The Project Backlog is what the users want in the final product of the Agile Development project. This final product is available after a set of iterations that satisfies the users. The Project Backlog can be expressed in a file (.docx, .html, etc.). Managing this type of file is nothing new and can be achieved using well known methods.

The Iteration Backlog is a list of functions and features that the users want in the software at the end of the current Iteration, i.e., that software produced in four to six weeks. The user representative and the team agree on what can be created during an Iteration. They take a workable part of the Project Backlog and implement an Iteration Backlog during an Iteration. Just like the Project Backlog, the Iteration Backlog can be expressed in a file and managed using well known methods.

The Software from the iteration comprises the code, tests, and tools required to transform the code into working software (compilers, etc.). The development team creates these data. Although the methods employed by the team differ greatly from those used on a traditional development project, the data are almost the same. Hence, these data are managed using well known tools and techniques.

Managing the Data: A Daily View

Now come the challenges to managing data on Agile Developments. The daily activities on an Agile Development project differ greatly from a traditional project. This is due to the emphasis on deliveries of working software in relatively short periods of time.

An Agile Development team begins its day in a meeting. The meeting, however, lasts only fifteen minutes and its hallmark is that everyone stands. Each team member states three items:

  1. Yesterday’s accomplishments
  2. Today’s tasks
  3. Any roadblocks

There are no PowerPoint presentations. Hence, there are no PowerPoint files to manage. There are no handouts. Hence, there are no pages to scan and manage. No one spends time writing what the team says. The team exchanges significant information, but nothing is recorded formally.

The team spends the day implementing the functions and features documented in the Iteration Backlog. That backlog, however, is only documented at a high level. The users record the details of the functions and features in User Stories, and the users often write the stories on cards that are tacked to the walls. Once a User Story is implemented, the team removes the card from the wall. The data from the Users Stories exists temporarily and only in scribbled form.

A key element in the working software aspect of Agile Development is the definition of what “works.” In Agile literature, this is known as the definition of “Done.” A User Story is done when the user representative declares it is done. The definition of done may have been scribbled on a card on the wall, but that card is discarded when done was declared.

Once a day, the team builds the software. This Daily Build is a usable product. It doesn’t do everything the user needs, i.e., it neither matches the Project Backlog nor the Iteration Backlog. It does, however, provide value to the user. A little thought reveals that the Daily Build often contains partially implemented User Stories. How much of a function is partially implemented is not recorded. A team member will explain that briefly at the next day’s standup meeting.

Implementing Data Management in Agile Projects

The foremost challenge in managing data on an Agile Development project is working with the attitudes of the team. Recall the Agile Manifesto value of “Working software over comprehensive documentation.” Violating this value is not an option. Any attempt to do so will likely cause the team to exit the project. A request of, “Please put that in an email and send it to me,” will likely be met with, “I’m working on software for the users.” The members of the Agile Development team will talk with the user representative to better understand what is desired; they will talk with one another about how to write the software and test it, but they won’t write documents for anyone to manage.

This point cannot be over emphasized. Team members will not record data in any manner resembling that of a traditional development. The data manager on an Agile Development project must find a way to capture data without aid from the team.

Begin with the daily standup meeting. The team members speak briefly and quickly so they can return to building software. An unobtrusive method of capturing the data is via someone writing what is said in shorthand. That sounds contradictory: use an old method to manage data from a new development methodology. Nevertheless, it works if someone is available who can write shorthand. A high-tech method of capturing the data is to record the standup meeting with either audio or video recorders. Those new tools work, but they may inhibit team members from speaking candidly, i.e., they will hide data.

Next come the User Stories. These scribbled cards on the wall can be captured with digital photographs. That is simple and does not intrude on the work of the team. It is fairly straightforward to type the scribbles into files (.docx, .html, etc.) that can be managed using traditional methods. Be sure to photograph the User Stories before they are removed from the walls. A User Story may appear, be implemented, and be discarded during a day, so the wall should be photographed more than once a day.

Along with the User Stories are the user definitions of “Done.” Divide “done” into at least two groups. The first group results from a numerical or text manipulation. A tester can create input data, consider numbers for this example, process the input data with software, and examine the output data. Store this input-to-output transformation in manageable files.

The second group of “done” results from a display on a screen. The definition of “Done” in this group is often no more than a user looking at a screen with a tester and saying, “Yes, that’s it. That’s what I want.” Capture the screen display in a manageable file.

Finally, there are the Daily Builds. The code, tests, and tools used to create them are created using the same types of tools and infrastructure used on traditional development projects. Data managers can use the same tools used on traditional projects. The frequency of management, however, must adapt. Backup everything everyday. Use a tool that allows reverting back to the state of any day of the project.


Agile Development is one of the more popular methods in use today. Ignore its use and developers may exit in search of organizations that do use it. In the same manner, mass exodus can come from ignoring the founding principles of the method. Honor the principles and the efforts of the development team. Adjust Data Management methods to accommodate what may seem like haphazard practices. Data Management and Agile Development can work together.

You might also like...

Data People Must Build the Bridge to Your Cyber Security People

Read More →