You are here:  Home  >  Data Education  >  BI / Data Science News, Articles, & Education  >  BI / Data Science Blogs  >  Current Article

Software Semantic Evolution: Part 2

By   /  December 4, 2015  /  6 Comments

Learn more about Yefim (Jeff) Zhuk. Click here to read Part 1Part 3, and Part 4 of this series.

Microservices, RAML, DataSense by MuleSoft, and the Next Step

RESTful API Modeling Language (RAML)

Have you worked for an enterprise that developed thousands of services? At some point you might have noticed that it is easier to create another one than find an existing service that covers the needed function. This very sad discovery is a good indication that service discovery needs improvements.

RESTful API Modeling Language (RAML) is designed to provide these improvements. RAML offers developers a formal way of describing RESTful APIs.

What is RAML?

RAML is built on the top of the standards such as YAML and JSON. RAML is a non-proprietary, vendor-neutral open spec. RAML gives developers freedom of providing their own semantics to define specific properties of services in a specific business domain. At the same time RAML includes basic characteristics necessary to invoke a service, such as basicUri (usually serves as the endpoint of REST service invocation), describes the post and get queries, and queryParameters that must be provided with the RESTful call.


JYZ Example

This example provides human readable descriptions as well as formal method definitions.

To get a well-formatted response use the PDF type with a simple request: http://ITofTheFuture.com/BASE/catalog/pdf

Note, that it is up to a developer to choose specific semantics for data property names, such as courseType, courseTitle, and courseInstructor. These naming conventions that look obvious and even trivial for one group of developers might miss expectations of another company, which has a different business dialect. And we will talk about semantic integration a bit later.

Mule Soft Introduced DataSense

Mule Soft made another step in this direction by creating Data Sense metadata for application designers.

MuleSoft is actively moving to Microservices. For developers this move to Microservices means API-led development. This is exactly what MuleSoft offers.

Similar to the discussion on service layers we had before, MuleSoft also separates service layers into three categories: Experience, Process, and System Layers.

The lowest service layer called System Layer represents underlying utilities and data services, including APIs to applications that provide data. The Process Layer is responsible for business processes that form workflows and eventually integrated into the Experience Layer consumed by end users.

MuleSoft Layers

DataSense is a better tool for developers who usually described their design ideas in Power Point and diagrams. Data Sense allows creating metadata to facilitate application design. Anypoint Studio can understand these metadata and can provide necessary translation data type and structure described there into application body.

At this point Anypoint Studio does some work on behave of developers. The tool intelligently discovers information about internal and external resources. Usually this was manually done by people. Imagine that there is a mobile application connected to Facebook. Facebook has its own API, data types and structure, which can be captured by DataSense. Anypoint Studio can provide this information back to you helping you to make better and quicker decisions about interfacing Facebook.

What Can be Done with DataSense and Anypoint Studio?

In the terms of Anypoint Studio, two major functions to discover and describe metadata are Perceptive Flow Design and DataSense Explorer.

Perceptive Flow Design

Mule can use an existing connection to the resource to retrieve metadata about message properties and payload.

This information will feed into DataWeave, a message transformation component. Then, the mapping data from one format to another happens almost automatically. At least part of work is done by a computer!

DataWeave Template Engine

You can type the word payload in the Anypoint Studio GUI to get a list of all the properties and methods associated to the payload. This is close to magic!

DataSense Explorer    

 DataSense Explorer

DataSense Explorer is part of Anypoint Studio. Explorer can visualize the message data structure at different points of the flow, when a developer is still designing the flow. A developer can select any element in the flow and the DataSense Explorer will display the structure of input and output data.

With the DataSense Explorer a developer can see the message contents at any given point in the flow. This is possible because the Explorer has access to the DataSense metadata of compatible connectors and knows about Session Variables, Inbound and Outbound and Payload properties.


DataSense and Studio Connection

DataSense allows developers to describe and discover information via the connector and connection to the application. Then DataSense passes this information about application entities and their structures to Anypoint Studio. Anypoint Studio presents the data at design time. Studio can even make suggestions about the expected values in fields returned by the connector. These suggestions are based on connector’s metadata and DataWeave’s intelligence.

What does it mean to implement DataSense?

The implementation consists of the following:  configuring metadata retrieval by creating the connector to supply this information, and configuring metadata awareness with annotations of operations (methods), providing to Anypoint Studio necessary information about the DataSense implementation.

Anypoint Studio is a rich extension of Eclipse with mostly well-known and almost intuitive windows. You can open the Import wizard from the File menu. With a pop-up wizard you can select an existing Anypoint Studio Project from External Location or open a new one. Once you select the Server Runtime as Mule Server 3.6.0 CE or EE, the studio will display the Mule Flows.

For precise settings you can use the mule-app.properties file with access credentials and more data describing the connector.

Connectors with Static and Dynamic Data Models

A connector might have a Static “strongly typed” data model or a Dynamic Data Model where data types are resolved at run-time

In the case of Static model, metadata retrieval as well as metadata awareness is immediately available by the strongly typed parameters.

In the case of Dynamic Data Model, some metadata will be resolved at run-time with two annotated methods, getMetadataKeys() followed by getMetadata().

The image below from MuleSoft examples shows one of the GUI windows offered by Studio for configuration.


The bottom line: DataSense and Studio are working together to discover and describe application interfaces. This work saves developer’s time and improves precision of the design by bringing an important layer of metadata information at design time.

The Trend

Do you see the trend? Development and even its modeling part become increasingly formal. Abstract ideas find their precise expressions and become tangible. Computer programs can understand and test these ideas. In a similar way the abstract concept of interfaces became part of Java language to allow compiler checking for correct implementations.

We describe software program behavior and support this ideology with Behavior-Driven Development tools, such as Apache Cucumber.

And we try to formalize design with simple language, so business people can understand and participate in the design. We did not want to introduce another weird xml languages (like RDF family). This intention is clear. Streamline the development flow and give the business more opportunities to directly participate. This is coming, but coming extremely slow. Here is just one of the reasons.

New tweaks of technology often require to restore old development arts in completely new environments. For example, application frameworks, such as Hibernate and others, hide from us complexity of data and degraded the need for data modeling skills.

Big Data renew this demand. When millions and billions of records are at stake, perfect data modeling is required. Perfect modeling with perfect understanding of NoSQL DB features and correct expectations of application queries.

There are more reasons telling us that it would be a stretch to expect business masses joining us in development trenches next week or next year.

So, what is the next step of software evolution?

About the author

Chief Architect at ITS, Inc., Yefim (Jeff) Zhuk, worked as Director of Enterprise Architecture at Sallie Mae, consulted Boeing and other corporate and government agencies in SOA and knowledge engineering, shared his expertise at Java One, Semantic Tech, and Boeing Conferences. In his books, patents and publications Yefim described a new field of Integrated Software and Knowledge Engineering. The methods and prototypes described him as Business Architecture Sandbox for Enterprise (BASE) and Conversational Semantic Decision Support place new technology seeds in the current business ground, helping transitioning to rule-based applications and Semantic Cloud Architecture.

  • Doug Egge

    Thanks for the article, Jeff.
    I would like to hear more of your thoughts on micro-services. I think that single responsibility micro-services have tremendous potential, but require a difficult paradigm shift to make them truly independent. In my experience, the concepts of not sharing state and eventual consistency are tough to sell, but that even nudging application teams in that direction brings a lot of value.

    • Jeff_Zhuk


      Thank you for a thoughtful comment.

      Here is what I think of Microservices:

      Microservices fight for independence against
      application flavors.

      Independence is expensive. Getting rid of application specifics,
      the essence of a service become smaller.

      Trying to play well in any environment, the shell of the service,
      the frame of the service package, is getting thicker.

      There is an associated cost and potential profit in years to come.
      While associated cost is well visible, it is harder to estimate Return on
      Investment (ROI). Unfortunately (or fortunately) accumulation of changes in
      technology and business direction throw away whole systems or brings a new
      development paradigm to redo them. This happens every three-five years.

      Massive Cobol systems are still running on mainframe not because
      of that technology superiority. They are just “too big to fail”.
      Symbolizing more liability than profitability, these heavy-weight old systems
      are hard to replace in one shot. This requires long term mentality and
      provisioning, which is also hard to find today in the corporate world.

      The bottom line: Microservices is the right step in many
      cases, although not in all. One of the benefits, which is a very important one,
      they make our constructions less monolithic, lighter and easier to change.

      Smaller pieces and bigger variety of them require better
      handling tools and more automation. This is still coming.

  • Farzana Kosgi

    Informative and interesting article and I agree with perfect data modeling requirement

    • Jeff_Zhuk


      The old art of data modeling was out-shadowed by Object-Relational frameworks, which did for us a lot of automation. (Maybe too much:-)
      With Big Data this art is coming back although in a modern refreshed dress. When millions and billions of records are in processing, their structure, although not necessary relational, is extremely important.

  • Don

    Hi, Jeff. Regarding streamlining the development flow and giving the business more opportunities to directly participate, do you think today’s business people are in a good position to understand the potential of Big Data and can assist in understanding the best way to model data?

  • Tony Santillanes

    What other companies, besides Mulesoft, are working on this evolution?

You might also like...

Data Governance vs. Data Architecture

Read More →