The API Economy: A Big Ball of CRUD

By on

Click this link to learn more about the author Dave Duggal.

Quote: “The use of APIs has exploded with the growth of distributed computing, driven by the popularity of the Web, Cloud and now, the Internet of Things (IoT)”

Back in 1999 an academic paper, “The Big Ball of Mud” exposed fundamental limitations of ‘modern’ software development practices. Such papers reflect the tension between the theoretical roots of Computer-Science and the ever-widening commercial applications.

While “Software is eating the world,” software headaches are compounding as old practices can’t support today’s new requirements; they appear ill-suited for our increasingly dynamic, diverse and distributed age.

Value-chains are becoming increasingly complex orchestrations across multiple silos, partners, technologies and standards. As a result, a software application is no longer a thing, but rather a composite built from internal, third party and public APIs. Application behavior is dependent on the sum of interactions between participating endpoints. ‘The Big Ball of Mud’ is growing exponentially in a tangle of connections that now spans beyond organizational boundaries.

As organizations hurdle towards becoming digital businesses, they expect more of their software; pushing boundaries and exposing weaknesses. The limitation of conventional software development is becoming more widely perceived as they are holding back the next wave of productivity and connectedness. The challenge is to connect endpoints for meaningful behavior without resorting to static, non-responsive and brittle developments methods (i.e. CRUD), which simply won’t scale for today’s demands.

The roadblock to seamless end-to-end automation of Development and Operations (DevOps) is manual integration. While APIs create marketplaces of capabilities and information sources for composing modular solutions, working with APIs requires time-consuming and expensive manual integration of every endpoint. As a result the number of APIs grows and, as the rate of change increases, the integration effort explodes.

Across the country and the world, there are modern-day ‘labor camps’, albeit often well-appointed, anonymous mirrored-glass buildings packed predominantly by young people in jeans, t-shirts, hipster hats and stickers on their laptops, pounding away day and night to perform the tedious and redundant integration work. There are tens of thousands of these laborers. Fueled by fancy caffeinated drinks, they are the ghosts-in-the-machine, connecting your apps and data sources, like switchboard operators in the days of old, so we can pretend the world is seamlessly integrated. There has to be a better way.

The API Economy

The use of APIs has exploded with the growth of distributed computing, driven by the popularity of the Web, Cloud and now, the Internet of Things (IoT). It logically follows that the more we want to interact with a distributed software-defined world, the more interfaces we need.

So what are APIs? According to Wikipedia, An API expresses a software component in terms of its operation, inputs, outputs, and underlying types. An API defines functionalities that are independent of their respective implementations, which allows definitions and implementations to vary without compromising the interface. A good API makes it easier to develop a program by providing all the building blocks. A programmer then puts the blocks together.

While the entry provides a concise encapsulation of the nature of an API and the objective of facilitating composability of software development “building blocks”, most of us recognize that the real-world is not quite as plug-and-play as the definition suggests.

APIs: The Wild-wild West

There is no singular API standard. The designs of interfaces vary based on how they are specified, communicated, and what they expose and how they expose it. While REST APIs have eclipsed more formal SOAP/WSDL based interfaces (which supplanted CORBA), REST APIs are far from formalism. Developers have progressively traded up for lighter-weight, less formal protocols for convenience, but the net result is that every API is a snowflake. Each implementation of an interface is unique-APIs are as diverse as the functions they expose.

Wonderful, but how do developers cope if everything that is connected is a “one-off” manual integration? Moreover, how do the resulting composite applications constructed from these building blocks deal with change?

APIs: A Victim of Their Own Success

As organizations become ever more distributed, dynamic and diverse their business processes are becoming increasingly complex composites, but IT simply can’t keep up with the demand for manual API integration (and re-integration), it simply does not scale. This problem is at the heart of the tension between the business and IT.

Services and APIs are only loosely-coupled in conceptual architecture, but not in the real-world where people still have to manually discover and integrate them at the Application and Process layers. This is the reason why Service Orientation has yet to have delivered “Business Agility”.

From that high-level Enterprise Architecture perspective (as opposed to the bottom up IT view) it’s not useful or scalable to be stuck manually considering individual APIs. It would be better to have an abstraction, a model for looking at all of these endpoints as objects that expose functionality, but hide their complexity and automate management of the end-to-end API contract lifecycle in order to advance the Service Orientation architectures.

Dynamic APIs

By definition, the trending meme of a ‘Composable Digital Business’ suggests a flexible and dynamically integrated enterprise. It’s time to re-examine software application architecture and consider new software development practices. There is a nascent movement to apply Computer Science ‘first principles’ to automate IT processes in order to unleash business agility and development productivity. These are the core ideas behind EnterpriseWeb. We are contributing some of our groundwork to drive a standard, “Dynamic APIs,” to support model-driven API interoperability and evolution.

Leave a Reply