Data Is Risky Business: Thinking Beyond Systems for Data Governance

A few weeks ago, I wrote a blog post on the Castlebridge website where I explored what Stafford Beer can teach data governance practitioners. I considered Beer’s famous acronym POSIWID (the purpose of a system is what it does) and whether it could function as an “oil check” diagnostic for data governance frameworks. If your governance framework is dripping with policies and standards and basking in regular governance forum meetings and councils, but drowning in ungoverned data, unmitigated risks, or unforeseen data disasters, is the purpose governance or merely the appearance of governance? Are you playing all the right notes but not necessarily in the right order? This is a useful test, but it is, on its own, incomplete. Filling the gaps is the purpose of this article.

The Diagnostic Perspective

When diagnosing or assessing things, it’s worth reflecting on how the diagnostic tool frames the problem. POSIWID is a view from outside. You watch what a system does, over time, and you infer its purpose from the pattern of outputs. That is exactly the right way to audit a governance framework after the fact. But it tells you nothing about how the outputs got made.

It doesn’t explain why genuinely competent, well-intentioned people, sitting inside a genuinely well-designed control structure, quietly fail to do the thing the structure was designed to deliver. That level of diagnostic requires us to shift our perspective from the structures of the system to the actors in the system. This is why my earlier blog post introduced concepts from Mark Bevir’s Decentered Governance Theory and Karl Weick’s work on sensemaking in organizations. These perspectives are important as they inform a critical corollary to Beer’s POSIWID: the practice of the system is what people do (POSIWPD).

This corollary is not a restatement of POSIWID. POSIWID is essentially a retrospective diagnostic as it judges the system by its accumulated output (what it does). POSIWPD is constitutive and continuous. It says that the system does not exist independently of the situated actions of people that continuously remake the system, from moment to moment, decision to decision, in situations where the first thought is not of the organization chart or the formal letter of the law. A data governance framework is not a thing that produces outputs the way a machine produces widgets. It is a thing that is made, continuously, by people interpreting what they believe they are supposed to do, and it begins to falter and fail the moment enough of those people stop believing the interpretation is worth the effort.

Governance Is Enacted, Not Executed

Bevir’s argument for decentered governance theory, in its simplest form, is that governance is never just the formal structure on the org chart. It is produced by situated agents acting on beliefs shaped by inherited traditions and modified by the dilemmas they encounter along the way. I’ve discussed, in a different context.

Writing about the EU’s Digital Omnibus and the temptation to frame data protection obligations as tiresome overhead, I quoted Irish data privacy lawyer Simon McGarr’s observation that if you legislate for people to think about whether they ought to do a tiresome thing, they will conclude, at a rate approaching one hundred percent, that they ought not to do it. That is Bevir’s tradition and dilemma doing exactly what the theory says they do: The tradition is “compliance is a burden,” and the moment a genuine dilemma arrives (the awkward request, the edge-case disclosure, the client who wants an answer the policy doesn’t quite cover), the tradition, not the policy document, decides what happens next. The rule exists. The practice is what the person, standing in the room, believes the rule is for.

Put governance-as-structure and governance-as-practice side by side and the gap between them is where a key root cause of data governance failure lies. The structure nearly always exists (formalized or not). The RACI chart is documented. The escalation procedure is written down. What is often missing is a shared tradition, among the people who actually touch the data, the model, or the decision, that flagging a problem reads as competence rather than as troublemaking. The systems-thinking-inspired control loop can be perfectly wired and still fail, because the wiring carries a signal only if the person at the end of it believes sending the signal is what a good team player does when faced with a data-related dilemma.

Beyond Sensemaking: The Flight Deck and Heedful Interrelating

Weick’s work on sensemaking is also worth reflecting on when considering how the actions of people define the practice of governance. Weick observes that organizations construct a plausible, rather than an accurate, story about whether things are going well, and that the story is retrospective, built after the fact from whatever cues got noticed and retained. This explains why a crisis is usually the only thing that forces an organization to see what POSIWID would have told it all along about the operation of its data governance.

In Weick’s work, sensemaking is enabled through a process Weick and Karlene Roberts term “heedful interrelating.” This concept speaks far more directly to the question of what makes a decentered network of situated agents actually hold together under pressure when faced with dilemmas and challenges. Heedful interrelating was drawn from Weick and Roberts’ work studying high-reliability organizations such as aircraft carriers and nuclear power plants. What they identified was that the safe operation of these complex systems did not live in any one individual’s expertise, nor was it explicitly documented in a procedures manual. It emerged from the pattern of interrelating actions that people took, with each person contributing an action informed by their knowledge of the wider operation and their understanding of how their actions fit with everyone else’s. Weick and Roberts tell us that if this is done well across enough of the crew it resembles a collective mind distributed across the organization that is more capable of absorbing disruption or dealing with a dilemma than any individual’s competence or any single standard operating procedure on their own.

My doctoral research links the concepts of the decentered governance theory and sensemaking together as, taken together, they provide key pillars for an understanding of how the practice of the governance system emerges through the actions of individuals. Bevir’s decentered governance theory tells us that governance is enacted by people situated in a network of decision-makers in the organization who interpret what they need to do based on their personal and professional traditions and webs of belief about data and data governance. Weick’s sensemaking explains why the reality of governance can drift away from the written narrative. Heedful interrelating provides an explanation for what it looks like when a decentered network of decision-makers can communicate and collaborate effectively through aligned traditions, knowledge, and beliefs and because each of them attends continuously to their stewardship of data and how their actions contribute to delivering the goal of the governance system.

Whether it is a data quality analyst who flags an anomaly, a model owner who asks whether that anomaly matters for their use case, or a risk function that treats the question as worth answering quickly rather than filing for the next quarterly review, these are not three people independently executing three lines of a RACI chart. They are, or are not, heedfully interrelating. The RACI chart cannot capture that.

The People Problem, Confirmed

None of this needs to stay at the level of theory. Two pieces of empirical research published in 2025 land, independently and by different methods, on almost exactly the argument made above, and it is worth looking at them.

Walsh et al (2025) set out to examine what 20 case studies of data governance success or failure actually tell us about what makes a data governance program succeed or fail. Their conclusions were pretty blunt:

• Data governance theory does not resonate with practitioners.
• Management of people and their data behaviors has the most influence on program outcomes.
• Data Governance is “principally a people problem.”
• Research has tended to focus more on the data and technology, but has ignored the importance of roles, skills, communication, and collaboration.

Walsh et al highlight the importance of “cross-team collaboration, coupled with an open and cooperative mind-set.” That echoes Weick and Roberts’ heedful interrelating. Interdisciplinary teams are identified as “the secret sauce.” This echoes Bevir’s traditions and webs of belief. A team that collaborates effectively cross functional lines is practicing heedful interrelating to develop shared traditions and beliefs about data and its governance.

Anthony Mazzarella’s 2025 doctoral dissertation, Towards Resilient Data Governance: An Industry Study on Practitioner Perspectives on the Current State of Data Governance (University of Arkansas at Little Rock), approaches the same question from a different angle and through a survey conducted in partnership with the EDM Council. The convergence with Walsh’s conclusion is remarkable given how differently the two studies were built. Mazzarella’s headline finding, stated plainly in his discussion chapter, is that “governance functions as a human system of coordination,” and that “governance success relies more on coordination routines than on artifacts.” His data shows that the challenges practitioners actually report are not technical: Limited knowledge, lack of awareness, cultural resistance, and unclear goals dominate the list, and he notes that these, along with weak cross-department collaboration, “highlight coordination gaps rather than technical issues.”

Mazzarella’s conclusion maps almost exactly onto the design-vs.-practice distinction described earlier. Formal frameworks and technical tools “provide structure, but they only have impact when supported by collaboration, accountability, and learning.” Maturity, in his framing, should be judged “not just by the presence of documentation or frameworks but by consistent behavior — the regular exercise of decision rights, cross-functional collaboration, and structured feedback loops that facilitate learning and adaptation.” That is Beer’s fast loop, Bevir’s tradition, and Weick and Roberts’ heedful interrelating in action.

The Irony of Automating Governance

I ended the earlier piece with a warning about automating processes before understanding the informal governance layer holding them together, and about the risk of automating an accountability sink at scale rather than removing one. In that context, Lisanne Bainbridge’s 1983 paper “Ironies of Automation” made an observation about industrial control systems that has aged uncannily well for data governance and AI.

The more reliable and capable an automated system becomes, the more critical the human operator’s role becomes at the moments the automation fails or reaches the edge of its functional capability. Yet the very success of the automation is what strips the operator of the routine practice they need to maintain the skill and situational awareness that such moments demand. You do not stay sharp at catching the rare, novel failure by reading about it. You stay sharp by being close enough to the process, often enough, that your model of how it normally behaves is current, and your ability to notice when something is not normal is exercised rather than theoretical. To borrow from Deming: You can’t spot special causes of failure if you are not familiar with the common causes of failure or the performance capacity of the process.

Heedful interrelating is not a switch you flip on when a crisis arrives. It is a muscle, built and maintained through the ordinary, unglamorous business of routinely contributing, representing, and subordinating your actions to everyone else’s, on the unremarkable data quality issues and the routine model reviews, long before the genuinely novel disruption turns up. Automate the routine layer away by removing the human reviewer from the ordinary case and you don’t just lose the analytical contribution they were making on that ordinary case. You stop the practice that was keeping the network’s collective mind, in Weick and Roberts’ sense, actually exercised.

When the genuinely novel case does arrive, the very thing you were relying on to catch it and respond correctly has quietly atrophied from disuse. I have previously made a related argument about what happens when organizations strip out the human expert layer from analytical work to make room for generative AI: You don’t get faster expert judgment; you get an expertise pipeline that quietly stops producing experts. Bainbridge’s irony is the same argument applied one level up, to the collective practice of governance itself rather than to any individual’s expertise. The pipeline that produces resilient governance practice is not the training budget. It is the accumulated, unglamorous repetition of heedful interrelating on cases that were never going to make the incident report.

The purpose of the governance system is what it does. And what it does should be to develop competent and capable people who can govern data (and data-related technologies like AI) and who can understand why doing so is important.

Where This Leaves Practitioners

A control loop tells you whether governance is working. A network of people who genuinely believe escalating an issue is the right thing to do, and who are paying attention to how their actions fit with everyone else’s, is what makes the loop worth having in the first place. And it is the part of the system that needs to hold together when reality hands you the disruption nobody designed for.

A few things follow from taking that seriously, rather than leaving it as an aside. Treat traditions as something you design, through onboarding, through how leaders visibly model escalation, through which decisions get remembered as precedent, rather than as an accident of who has been there longest. Treat the handling of a dilemma as a governance activity worth making visible, rather than an exception path that quietly gets absorbed and forgotten. Build interpretive diversity into teams on purpose, through rotation and genuine dissent, rather than assuming the comfortable, plausible reading of an ambiguous signal will always be the accurate one. And before automating a process, ask not only who is accountable for it today, but whether the practice that currently keeps that accountability real is one your organization can afford to stop exercising.

That means building culture and changing mindsets to make sure that everyone is playing all the right notes in the right order.

References

Mazzarella, A. J. (2025). Towards resilient data governance: An industry study on practitioner perspectives on the current state of data governance [Doctoral dissertation, University of Arkansas at Little Rock].

Walsh, M. J., McAvoy, J., & Sammon, D. (2025). The data governance journey in practice: Insights from case study research. Information Systems Management, 42(3), 471–488. doi.org/10.1080/10580530.2025.2477459

Weick, K. E., & Roberts, K. H. (1993). Collective mind in organizations: Heedful interrelating on flight decks. Administrative Science Quarterly, 38(3), 357–381.

Bainbridge, L. (1983). Ironies of automation. Automatica, 19(6), 775–779.