According to Eric Newcomer and Greg Lomow, authors of Understanding SOA with Web Services, service oriented architecture is “an architectural style that guides all aspects of creating and using business processes, packaged as services, throughout their lifecycle, as well as defining and provisioning the IT infrastructure that allows different applications to exchange data and participate in business processes regardless of the operating systems or programming languages underlying those applications.”
Although this definition seems to have a reasonable amount of industry acceptance, a variety of definitions exist, and you may see more than one reflected in the articles presented in this issue of the Data Strategy Journal. Regardless of definition, it seems everyone agrees that SOA is vast. SOA is more than IT or web services or a set of technologies; it is an organizational matrix that is applicable across the board - and it can be confusing.
In the hopes of providing you and your organization with insight into effective SOA implementation, we asked prominent consultants in the world of data services and service oriented architecture to answer a few key questions (via e-mail). Together these participants constitute this month’s Experts' Roundtable. Below you will find our experts and their answers.
MEMBERS OF THIS MONTH’S EDJ EXPERTS' ROUNDTABLE
Jim Bean - TOGAF 8 Certified Architect and author of four books on the topics of data modeling, database design, XML, Web Services, reuse, and Global Standards
Andrew Flower - Founder and lead consultant for Right Triangle Consulting
Ken Karacsony - Author, lecturer, Information Strategist and IT consultant
Joe LaFeir - Vice President of product development & CTO for RLPTechnologies
Doug Stacey - Data Architect in the Information Architecture group at Allstate Insurance Co
D.G.: What's the most common first step for organizations getting started with SOA? Is there a particular type of project that seems to break the ice better than others? (I.e. where is the low hanging fruit?)
The most important first step is creating the underlying infrastructure and governance to support SOA in your organization. Most people won’t want to hear this because it is difficult to sell, but the business will want to see results and what I am suggesting does not get them there immediately. SOA is not about any one single project, but rather an IT application framework and approach for delivering solutions to the business. Establishing the technical infrastructure and governance is a key first step to support an SOA strategy.
Coupling the infrastructure and governance with one or two small initiatives such as wrapping a high demand stable legacy application as a service or integrating a commercial business application as a service into your environment are ways to get the gears to turn. I would stay away from major re-engineering efforts as your entry into SOA. If a large initiative is required to get funding, create a small proof of concept early in the project to both prove the technology in your environment and to give your staff an opportunity to use the technology in a safe setting.
The reality of IT is that every IT related activity must demonstrate immediate value or people will question and resist it. The challenges involved in implementing SOA are significant and they will not be accepted unless there is some benefit realized fairly early on in the process. Show the rewards to the enterprise quickly by starting with business objects that have a broad enterprise appeal. Those business concepts and processes that are used by many different business groups are prime candidates for early SOA implementation. Frequently accessed business information means immediate return on an SOA implementation effort.
SOA is an enormous effort – beware being over-zealous in implementing SOA. History has demonstrated that the “big-bang” approach in IT rarely works. Small, incremental changes have a greater opportunity for success because they are more manageable. Fortunately, the incremental approach works well within the context of SOA because the architecture allows the company to implement one service at a time.
From a Data Services perspective there is no harm in starting piecemeal, on a project by project basis. Services to expose enterprise data can be built as needed. The important point when doing so is to take into consideration the ‘enterprise view’ when designing the service interface; trying to anticipate the needs of your broader consumer base rather than designing what I call “point to point” services.
D.G.: To what extent does SOA require a different mindset to succeed? If you could change the mindset of our readership to better prepare them for their first SOA project, how would you do it?
A large part of the integration problem is the result of our approach to rapid application development and short sighted solution to business problems. We have learned quite well how to deliver quickly, but often at the cost of redundancy, isolated applications, and an inability to share and reuse our data assets. When all facets of a well-defined SOA are properly applied, we have a method of resolving the integration problem and building an enterprise architecture that is effective, fluid, and articulate.
However, SOA is not just another development method or a set of technologies. It is an architecture that combines critical principles, standards, technical capabilities, and governance. If we retain our current development mindset such that we can continue to build and deploy more application and data solutions faster and without principles, standards, etc., we will perpetuate the integration problem and, over time, we’ll see an escalation in technology costs rather than the desired solution.
SOA is not about any one project it is an approach to delivering IT solutions. The approach delivers increasing value over time as the leverage of integrated services is realized and time to deliver solutions to the business is reduced. Implementing SOA will require a good degree of patience and discipline.
For many companies, SOA is a radical departure from traditional architectures that are based on tightly coupled application interfaces. Consequently, there may be a steep learning curve to understanding SOA. Training and education are absolutely essential to flatten the curve.
A top-down training approach is recommended. First, educate senior management on the fundamental tenets of SOA and the benefits of deploying it. This is critical. If the CIO, for example, is unable to grasp the basic methodologies and goals of the architecture, then he will not be able to support it.
Once you have trained upper management, proceed to lower-level managers. They must not only be educated in the overall goals and philosophy of SOA, but also trained in its practical details and how it will be implemented.
Finally, train your staff on the specifics of building and deploying SOA. This granular level of training needs to address the specific technologies to support the company's move to SOA. This will require the greatest amount of training.
Keep in mind that the initial training may not be a resounding success. The concepts of SOA are foreign to many IT professionals, who are probably more familiar with other architectural models.
Comprehending a new paradigm is often difficult. Futurist Joel Barker refers to this syndrome as the "paradigm effect." He explains that most people have certain boundaries that govern their perceptions of the world. When a new theory tests those boundaries, people may reject it because it doesn't fit in nicely with what they believe.
Conquering the paradigm effect requires commitment from management and a thorough training and education campaign.
Unless you take an enterprise view, rather than a project view, you’ll end up with a confusing mish mash of services with very low reuse and much analysis required down the line to figure out what existing services to reuse. The change in mindset may be one of “service governance,” or data governance, to insure that the most value is obtained from your SOA efforts. Otherwise, the effort could fail from an inability to demonstrate benefits and be tossed on the pile of past technology failures that held big promise but didn’t deliver.
The payoff is in reuse. If development continues to result in more services being built every time a new request comes along, the promise of SOA will not be realized. Service governance must be used to contain the proliferation of potentially redundant services. This doesn’t happen all by itself. It requires a concerted effort to inventory your services as they are built and to review that inventory before any new service is built to insure you aren’t overlapping the functionality of an existing service.
SOA offers the opportunity to do things we have always wanted to do: develop common data access, integration, quality services that are shareable and reusable. The same data governance principles apply, but now we have to consider function governance. Yes, the data professional has to think about the verbs (function) as well as the nouns. But we’ve been fighting for and pushing data governance and an enterprise perspective in our organizations for some time now, so we are well equipped to broaden our focus to other aspects that facilitate a more efficient enterprise.
D.G.: A big percentage of our readership is focused on data management. What are the key “data” issues that need to be accomplished for SOA success?
There are a number of data challenges that must be recognized and resolved within our SOA. First is the recognition that in the moderate to large enterprise we often have numerous silo'ed (vertical) data implementations. Each data implementation may have different naming, taxonomy, typing, metadata, and platform implementations. SOA and data services can help to resolve this integration challenge. This is the loosely coupled, interoperable, and standards compliant movement, exchange, sharing, and reuse of data. As data practitioners, we need to extend our practice to address these data issues within our SOA strategy.
Second, all SOA collaborations (that is, collaboration as regards the interaction of consumers and services) rely upon a well-defined, loosely coupled, interoperable, and standards compliant interface. That interface definition “is” metadata. WSDL, XML Schemas, SOAP, XML, service interface design, and canonical message models need to become core to the data practitioner lexicon and offerings.
The key data issue to deal with in terms of SOA is the concept of data in motion. With SOA the idea is to build composite applications by combining services to satisfy a business need, the loose coupling of services which allows for this flexibility will also require practitioners to pay particular attention to the modeling of the data that flows into and out of the services. Through the use of XML, process automation, and service orchestration technology, data management professionals have the technology tools to creatively distribute data to services and aggregate data from services.
SOA is not a replacement for fundamental data management techniques. Indeed, the benefits of SOA are predicated on intelligent data management. There are some keys to developing a thoughtful data management strategy. These include:
• Minimize data redundancy
• Establish information quality guidelines
• Assign Information stewardship
• Develop enterprise data models
• Create Meta data management
Just as redundancy in services makes absolutely no sense, neither does redundancy of data. Consolidate redundant databases where possible, and develop a common understanding of the data. This will greatly facilitate your move to SOA by simplifying the development, testing, and deployment of services for accessing the company’s single version of truth.
Data quality must also be addressed. First, it is important to understand the extent of data quality issues in the organization. A data quality effort should be undertaken to determine what data quality problems exist within the enterprise. Understanding these is critical in designing your SOA services, since it may point you towards areas of the business that can be reengineered.
Companies are finally beginning to understand the importance of information stewardship in managing data effectively. In support of SOA, a team of data stewards should be assembled around the major subject areas of the organization (e.g. Finance, Sales, Inventory, Customer, etc.). This will be a highly knowledgeable team of subject matter experts who will assist your information architecture team in achieving a common standard around data structure, meaning, and quality.
It is important to define the data in the organization to support the development of services in the framework of SOA. This is accomplished by developing enterprise-level data models that represent the real world objects in the company. This is a major effort and does not need to be done all at once. You may decide to perform your modeling activities in conjunction with the services that are being designed.
For example, if you are designing a service called “get_customer” you will need to model the customer subject area. There is a caveat, however. Your modeling effort must include representatives from the entire organization who rely on customer information. Remember, services are being designed for the enterprise which means that the data must also be designed – modeled – for the enterprise.
Meta data basically means “data about data.” Services will depend heavily on the definition and understanding of the enterprise data. This information should be centrally managed in a meta data repository. Common business terms and definitions must have consensus in the organization. Achieving this consensus is a task for your information architecture team and your subject area data stewards.
Data standardization and quality is very important. SOA gives us greater flexibility to use data, but that age old problem creeps in if that data is a bear to integrate.
D.G.: Our other audience represents the business. What do they need to do to help SOA succeed?
The best advice I can give here is a little patience during the early stages. Rushing the first application or two out will result in compromising the underlying infrastructure and will show up in the flexibility, or lack of, down the road. Early projects will be burdened with both the learning curve and likely take on a large portion of the infrastructure tax. Think of it like investing in the stock market.
SOA is not simply an IT initiative. That is to say that SOA requires strong collaboration between both the business and IT. In SOA, there is an inseparable connection between the business processes and the services that support the processes.
The business must be prepared and ready to improve and standardize the business processes in conjunction with implementing services. All too often, there is redundancy between similar business processes with subtle variations. This makes designing for SOA reuse difficult because it is impractical to develop an architecture based on subtle variation. That is why it is imperative to explore and redesign business processes and remove subtle variations for similar processes.
In addition to partnering with IT, the business also must understand that SOA is not a “quick-fix” and will take time to develop and implement. The CIO must communicate to the other “Cs” of the company that SOA is an investment in future savings. There is an initial cost associated with changing architectures to the services model, but the investment will pay huge dividends in supporting the organization later by establishing a stable and flexible architecture that will enable the company to adapt quickly to changing market conditions.
The business needs to be tolerant of the governance function sometimes dictating a solution that isn’t the absolute quickest way from point A to point B. In the long run they will benefit many times over from decreased maintenance costs and quicker application modification/development to respond to changing business needs. That will not be achieved however with a “cowboy” development approach with no participation from the data management professionals who have the ability to apply some governance over the development of services.
Value your data equally as your functionality. Historically, too much emphasis has been placed on applications that do specific things without considering the enterprises need to share data. When building and buying new systems, always consider how well the solution adheres to your enterprise data architecture. An overemphasis on the function (whether it is encapsulated in a service or not) ignores, in my opinion, the more valuable asset to the business- the data.
D.G.: What key skills and disciplines do data management professionals bring to SOA? How can they best leverage this experience into SOA professional success?
The service interface is critical to SOA success. The data practitioner has strong expertise in areas of data architecture, modeling, and scriptable metadata languages (as in DDL, etc.). These same skills need to be extended to include the similar and extended needs of SOA. The data practitioner must become an expert with WSDL, XML Schemas, SOAP, XML, interface design, and canonical message modeling.
The data practitioner is already skilled in the application of specific data models and database implementations. The next step is to augment the current skill set with data mapping and rationalization skills to help resolve data and data source integration challenges. A strong understanding of supporting transformation and translation technologies is a big plus.
Good data management fundamentals are still critical to the success of SOA based projects. Modeling data in motion can be a challenge; however, with solid data management discipline to drive consistency and structure to data, data management professionals can certainly be successful.
Skills such as logical and physical data modeling are absolutely invaluable to implementing SOA. It is important that the IT department work with the business to clearly identify and define all of the information requirements of the business. This is done as part of the conceptual and logical modeling sessions. Once the conceptual and logical modeling activities are complete, it is important to define the physical data models in order to support the development of services.
Data management professionals are often more used to looking at the big picture, of planning for the future when implementing for the present. The insistence of precise semantic meanings for data is second nature for a data management professional yet so critical to implementing data services in SOA.
D.G.: Any suggestions on how to scope an SOA project from a data perspective? What are the questions that need to be asked?
Because SOA is not about any single project, we need to look at the data in terms of a functional area of the business or the enterprise as a whole. While each project has its unique use or function it will be responsible for delivering, the true leverage of SOA will likely not be achieved if the data perspective of individual projects is too narrow. The data perspective should be broad with consideration for how the business produces and consumes data. Good fundamental data management practices still apply.
From a data perspective, scoping for SOA is no different than scoping for any other IT related project. Activities such as conceptual, logical, and physical modeling are integral to SOA success, so they must be planned for. Additionally, data architecture is very much required -- defining the “As-Is” and “To-Be” architectures that support the SOA initiative.
D.G.: What's the first tool investment a customer should make and why? (Looking for tool "category" rather than a specific vendor recommendation.)
The first tools decision and investment will likely be the use of an Enterprise Service Bus (ESB). If your technology strategy is to use an ESB to manage the integration of services it should be one of your early decisions. When considering an ESB also consider vendor capabilities for process automation. Understanding the requirements of the business is essential to selecting the appropriate technology.
I don’t think there is just one category to start with. I think you have to look at four things equally: Data access and integration services, function/application services, process orchestration services, and a technology platform to deploy these services.
Data access and integration services are probably nearer to the heart for us data professionals. This really is an introduction of deploying ETL technology, Messaging, Data Standardization, and EII solutions as services.
Function/application services are probably what most people think of first when they hear the acronym SOA. This is the deployment of application functionality as services.
Process orchestration is what makes services useful. Having a pool of services (function and data services) available is great but if I can’t orchestrate them and re-orchestrate them to execute a business process then I’m not getting the promised flexibility desired from SOA.
Finally, you need an application server and related software to manage and deploy these services where they can be found, as well as execute the services.
D.G.: When SOA projects fail, what are the most common reasons?
There are several easily identifiable causes of SOA failure. The first is not having a solid SOA definition, vision and strategy. If you do not understand what SOA is, how it can benefit your enterprise, and/or what you need to get there, failure will no doubt result.
Another cause for failure is letting your SOA strategy be led entirely by technicians. Technicians are strong partners and contributors to the SOA strategy. They have requisite skills and expertise. However, a successful SOA strategy will be led by business and technical “visionaries” and with strong contributions from a number of technical disciplines.
Another reason for failure is an over-riding emphasis on specific development practices or technology. If the SOA strategy is led by a heavily biased technical development team, failure is likely. SOA is not about any single or specific technology. Rather, it is about development principles, integration, reuse, standards, and governance. The real value of SOA is in resolving the integration and interoperability challenges – not in promoting a single technology or development platform.
When project becomes too focused on technology, new and emerging technology and technology concepts tend to attract the highly skilled technologist who can often dominate the direction of a project. Be sure to offset the strong influence of these technologists with strong project management and business representation.
Failing to plan for SOA is synonymous with “planning to fail,” because SOA is too big to work on without a plan or strategy. An organization needs to fight the urge to deliver quickly and spend the necessary time in the planning phase. This is one of the main goals of your SOA team. You may also need outside assistance from a consulting team to help plan. Time spent in the planning stage will pay huge dividends down the road. Do not try to build SOA too quickly – it is a complex solution to a complex set of problems. Rushing the design and implementation may only make things worse.
Most project failures are the product of not having a clear, well thought out strategy and a framework for making decisions which recognize that what you are doing impacts and is impacted by the rest of your enterprise. SOA forces an organization to think more about how their project is related to the whole enterprise. A SOA project can be successful in narrowly defined terms, but if it does not produce the promised flexibility and reusability, then it is a failure.
D.G.: Are there any SOA myths? Can you dispel these?
One of the most common SOA myths is that you can just buy SOA off the shelf. Unfortunately, this myth is often promoted by technology vendors. It is important to recognize that SOA is NOT just a technology solution. Rather, a service oriented architecture (SOA) is a combination of consumers and services that collaborate, are supported by a managed set of capabilities, are guided by principles, and governed by supporting standards.
Having an Enterprise Service Bus (ESB) means you’re doing SOA. The reality is the ESB is nothing more than middleware. I’m sure many people have built applications using an ESB that never realized any of the benefits of SOA. I’m also sure that many of you have examples of a poorly designed database, especially in the early days. Don’t get me wrong, the ESB is a key enabling technology, just like an RDBMS, but must be used to implement a well thought out architecture.
Perhaps the most prodigious myth regarding SOA is that it is synonymous with technology. A service-oriented architecture is not a technology; it is an architectural framework. An architectural framework is a design. Technology is a product or system that achieves a practical purpose. Technology certainly supports the implementation of a service-oriented architecture, but the technology does not constitute the architectural framework itself.
A service-oriented architecture is very much like a “high level” building architecture. It identifies the enterprise layout of applications and interfaces as well as the over-arching plan for technical architecture, information architecture, application architecture, business architecture, and integration architecture.
The tools, technologies, software, hardware, etc. follow the architecture and are similar to the bill of materials for the building plan. They support the architecture directly, but are not part of the architecture.
The important take away here is that the architecture ALWAYS precedes the physical design. You never want the technology to drive the architecture, which happens all too often in IT.
D.G.: For the skeptical business sponsor, what's your SOA elevator pitch? (Assume you're only going 10 floors.)
IT projects are complicated, fraught with risk, always seem to take longer than expected, and seem to cost more than expected. There is no silver bullet to solve these problems; however, we can implement approaches and systems to try to minimize them. SOA is a great example of an approach from IT to address these fundamental challenges with delivering IT solutions. The idea of SOA is to break the business problem down into smaller units of work (lets call them services), and to link them together to create solutions for the business. “Services” allows for reuse and consistent application of business rules. Enabling a new business process by stringing together a new combination of services is a lot easier and takes much less time than writing new applications. Seems pretty simple, and in concept it is. So why has it taken so long? The short answer is that the supporting technology and standards to make this approach feasible broadly, across technology platforms and vendors, has been maturing for some time and has come a long way in the last few years. Now we have the tools and accepted standards to make it a reality.
When selling SOA, remember not to “over sell” the concept. The business may be a little wary of yet another IT initiative. It is important to discuss both the benefits and challenges associated with a new architectural framework.
In selling the business on the benefits of SOA, it is important to address the frustrations of the business and demonstrate how SOA will address their frustrations. It may take some work to identify several key pain points facing the company.
If the business is grumbling that they are unable to adapt quickly to emerging business opportunities, demonstrate how SOA will make the business systems more flexible. If the complaint is that IT costs too much and fails to deliver on time and on budget, explain how SOA will help by easing systems integration issues and why developing reusable components will save money in the long run.
Whatever the problems facing the business, it is important to associate the value of SOA directly to the problems. If IT can clearly demonstrate how SOA will benefit the organization, then it will be an easy sell. However, if IT fails to quantify the benefits of SOA, then this is a strong argument against implementing SOA.
A service oriented architecture is going to decouple your business logic from the underlying technology. Once a portion of your business logic is implemented in such an architecture, you will be able to modify that logic to implement changing processes much more quickly than you’ve experienced in the past. No longer will a seemingly simple change from your perspective require technology changes that ripple through several systems and take months to complete.
You really should consider unlocking all that data (which is one of your company’s best assets by the way) by focusing on the enterprise being more efficient. Yes, you’ve already spent a ton of money on lots of different technology solutions, but they solved specific problems without enough concern for the enterprise value of the information they collect. Why don’t we spend a little time and, of course, money evaluating technology that facilitates sharing, reusability, and standards? Isn’t that what most businesses do to reduce cost and be more efficient?
ABOUT THE AUTHOR
Danielle Grilli lives in Los Angeles. She is a writer, editor, web developer, and instructor. You can reach her at firstname.lastname@example.org.