As explained in the previous section, ASG was born as a result of the convergence of SOA Governance and API Management. ASG combines the best of both worlds and disregards the worst. It can be used in any context where APIs and services are involved with application integrations (in the cloud, on-premise or both), mobile architectures, web architectures, and Internet of Things (IoT), to name a few.
For those more familiar with traditional SOA, ASG will seem a simplified version of SOA Governance that has been augmented with all of the nice features available in API Management (community support, metering and monetization, and contract management all via a web portal) and features that didn't deliver (for example, UDDI registries and its integration with asset repositories). For those coming from a mobile or API development background, the impact will probably be more notable given that design time is very lightweight in API Management but in the convergence it should get more rigorous.
Understanding what ASG is, its relation to SOA Governance and API Management as well as its objectives, is an important step forward; however, it is not a guarantee for success. A successful governance implementation must tick several boxes and answer several questions, such as:
What artifacts are required to deliver governance? It is vital for the success of a governance implementation to have a clear understanding of the artifacts to be delivered, what their purpose is, and what value they add to the overall governance implementation. The main artifacts are:
Governance strategy: It defines the goals and objectives for adopting ASG in the enterprise. Moreover, it defines the success criteria needed to ensure that the business benefits are realized by the adoption of SOA.
Reference architecture: It defines the core building blocks for SOA and API implementation.
ASG policies and standards: Guidelines such as patterns, anti-patterns, conventions, and best practices are considered or adopted when designing solutions.
Assets and taxonomies: Define all the assets available in the enterprise, their description, and type.
How can it be delivered? All of the necessary tools, processes, and procedures required when delivering governance and its objectives. The key artifacts that should be created or influenced are:
ASG Implementation Roadmap: it defines the activities required to deliver an ASG strategy and milestones for the implementation. This topic will be covered in more detail later in the chapter.
Design-time and runtime governance: These two fundamental concepts will be described in detail later in the chapter.
Who is responsible for delivering it? A description of all the participants required to deliver the artifacts previously listed, including their roles and responsibilities.
The answers to these questions, together with the concepts and tooling outlined in this book, provide the foundation for implementing a robust governance framework that underpins an end-to-end SOA and API ecosystem.
A governance framework defines the approach, artifacts, processes, tools, and people required to implement governance.
Having an effective and strong governance framework in place is extremely important as it delivers a common and consistent language for the enterprise to define and manage semantics, processes, standards, and accountability for the entire SOA and API lifecycle.
Indeed, without a well-defined governance framework, it is highly likely that the members of an organization, or its partners, will have their own understanding of what SOAs, APIs, and ASGs actually mean and how they are best implemented, leading to misalignment and duplication of effort across departments and between the collaborating organizations. In large and complex ASG implementations, this situation leads to poor implementations and almost inevitably, failure to deliver meaningful solutions.
A governance framework defines the roles, responsibilities, processes, and procedures (what-how-who) that are needed to enforce the governance of all aspects of the service and API lifecycle (what-how-who). It also defines the standard tools that should be used on each phase (this is particularly relevant in API Management for the community management aspects of it).
This book sets out to discuss the tooling and concepts provided by Oracle Corporation to achieve successful ASG. The following diagram depicts in more detail what a governance framework would look like when put into the context of the Oracle SOA Governance and API Management solutions. We will explore the concepts illustrated in this diagram throughout this chapter showing how each contributes to the overall framework.
The preceding diagram shows how the business objectives and strategy are the fundamental drivers of an ASG implementation. Without a clear focus on the business drivers and on how SOA and API solutions can help achieve these business goals, an ASG implementation will most likely fail to deliver and end up being perceived by the management as yet another expensive technology that adds little value. To avoid this, the governance framework should define metrics that can demonstrate how SOA solutions and APIs are being effectively utilized to deliver tangible benefits to the business.
The sections to come elaborate further on the building blocks that constitute an ASG framework.
Not having an ASG strategy of some sort means that SOA and API solutions are being built ad-hoc by projects somewhere in the enterprise, probably using their own tools and standards. While the project itself might benefit from these ad-hoc solutions, this is not good news for the enterprise as a whole, as running and operating non-standard solutions tends to become more and more complex in time, therefore increasing the total cost of ownership. This issue becomes even more evident as companies start moving their applications to the cloud. Many end up building tactical point-to-point interfaces between on-premise systems and cloud applications because there wasn't a strategy in place to advise on what approach to take or what technology to use. Similar problems occur when departments rush into building ad-hoc mobile apps. Because these apps were built tactically they tend to use their own middleware solutions to expose APIs.
Elaborating an ASG strategy is a good place to start to solve or prevent many of these and other issues. This is a very important start as it sets the foundation and direction on which other governance framework deliverables will be based.
Having said that, an ASG strategy should consider the following objectives:
Business and IT strategy requirements: Without understanding what the organization as a business is trying to achieve (the goals) and what IT strategy has been derived in support of these, it is impossible to deliver an ASG strategy that can effectively deliver value that is meaningful to the business and IT stakeholders. It is therefore imperative that the business goals and IT strategies are translated into a set of requirements that can subsequently be used to measure the success of an ASG framework implementation. A simple yet meaningful example is this table:
Use case discovery: Based on the derived ASG requirements, organize a series of workshops with business and IT stakeholders from different parts of the business to identify:
Maturity assessment: This consists of evaluating the current state of an SOA and API adoption within an organization using a maturity model as a reference to evaluate maturity. The ideal outcome of the assessment should be to understand:
How your organization's SOA and API maturity level compares with other similar organizations in the industry
Where are the weak points and what can be done to improve them
The gap between the as-is and the to-be and what has been done to bridge that gap (the implementation roadmap)
Note
Chapter 1, Application Services Governance, of the book Oracle SOA Governance 11g implementation (https://www.packtpub.com/application-development/oracle-soa-governance-11g-implementation) has a section that discusses in detail the elaboration of a maturity assessment.
Reference capability model: A capability model is a vendor-agnostic conceptual architecture that depicts the different components and capabilities that build up a particular platform and/or solution. The more ASG requirements derived from the business, the more robust the capability model will be, as each component in the model should be targeted at addressing a particular requirement rather than just being there for the sake of having a capability without any view on whether the business actually needs it or not. As depicted in the following sample model, capabilities that should normally be considered API Management tools are mobile integration support, cloud and on-premise integration capabilities, in-memory caching, to name a few.
Implementation roadmap: A roadmap is a high-level plan depicting what capabilities will be delivered and when. A roadmap should ideally commence with the elaboration of the strategy (as has been previously recommended), then be processed with a foundation phase. The purpose of the foundation phase is to provide a solid base for the entire ASG implementation by delivering the design-time and runtime artifacts of the framework. The foundation should also be responsible for identifying the teams that will support the ASG initiative going forward and also ideally deliver a limited set of use cases that provide high business value but aren't very complex (quick wins). The subsequent phases consist of the broader rollout of the solution to other departments and/or geographies.
One of the greatest challenges to successfully implement ASG-related solutions has, in fact, nothing to do with the intrinsic complexity behind their technology platforms. It is widely recognized that the real difficulty lies in dealing with people and processes from different parts of business and aligning them with the sole objective to deliver enterprise-wide solutions. This is not only true in the case of SOA and API architectures but also a challenge faced in systems design in general.
Note
The people challenge and the consequence it has for IT has been nicely put in Conway's law:
There is no simple solution or answer to the people challenge in an SOA. However, a good place to start is by defining an accountability framework to bring clarity to the roles required in an ASG initiative and their responsibilities. A very efficient way to represent such a matrix is by creating a RACI model. In a RACI model, all activities and deliverables are listed and mapped against responsible, accountable, consulted, and informed parties. However, before such a model can be created, one must first understand the roles in an SOA and API software development lifecycle and their relevance in an ASG implementation:
C-Level executive sponsors: Having C-level sponsorship (that is, CIO and CTO) in an ASG initiative is imperative for success. Having the right level of sponsorship will ensure that the organizational, behavioral, and cultural changes needed to implement the governance processes and procedures are embraced by the people within the organization and not ignored or rejected.
Functional/Business analyst: This role consists of experts in the organization's business processes. In general terms, he/she is responsible for producing suitable functional requirements such as functional design documents, a future process model, and/or a business rules catalog. The functional analyst should, among other things, engage with the business to ensure that all the functional requirements are well understood by the technical and solution architects to ensure that the requirements are presented in an appropriate format. The role of the analyst in an ASG implementation is to assist in the identification of relevant use cases where SOA and API solutions can add value to the business. This can be accomplished by sustaining a series of workshops with the relevant analysis with the sole purpose of identifying where the main pain points in the business processes are and catalog if potential SOA solutions could be built to address them.
Enterprise architect: An enterprise architect is responsible for ensuring that the solution being delivered by the project not only delivers the desired business goals and can be successfully traced back to its original requirements, but also ensures that the overall solution aligns with the wider enterprise strategies and standards. The role of the enterprise architect in an ASG initiative is to help deduce the correct ASG requirements from the business and IT strategies and any other business domain-specific need.
ASG Solution architect: He/she is responsible for producing solution architectures, capturing the non-functional requirements and also ensuring that the functional requirements are consistent with the solution as defined in the solution architecture. The solution architect may also participate and/or influence the definition of detailed designs, and may take part in the approval or rejection of these documents. The ASG solution architect is the subject matter expert (SME) in the technologies in question. Some of the responsibilities of the solution architect include:
Analyzing project requirements and ensuring that these conform to the expected level of quality
Identifying and cataloging the SOA and API assets and ensuring that these are discoverable and reused when appropriate
Providing technical leadership and guidance to the design and development teams (in development activities as well as support)
In an ASG implementation, the solutions architect is responsible for elaborating the ASG strategy, the accountability framework, design-time artifacts (with support from designers) and the development of runtime artifacts and implementation of the relevant ASG tools
Tip
The role of an ASG solutions architect should be as much about integrating people as it is about integrating systems. Dealing with people from different departments, backgrounds, and agendas is a huge challenge that must not be underestimated. The ASG solutions architect role requires someone that not only has a sound architectural and technological background but also has charisma, interpersonal skills, and can communicate to the business and technical teams equally. In a nutshell, it requires someone that can connect people and is capable of selling solutions which address different points of view.
ASG designer: He/she is responsible for providing suitable detailed designs that successfully deliver all of the desired business and technical functionality. The designer is also responsible for providing further clarification and guidance to the development teams. An ASG designer will also create the XML assets such as WSDLs and schemas, and define the unit test scripts that should be executed by the developer. In an ASG implementation, the designer will support the solutions architect in the delivery of design-time artifacts.
ASG developer: He/she is responsible for building and unit testing SOA services. The developer is also responsible for packaging the code ready for release, and for producing any relevant documentation that is required to support the deployment of the code (such as release notes). The SOA developer may partially contribute to the deployment and monitoring of services. In an ASG implementation, the designer will support the solutions architect in the delivery of run-time artifacts.
ASG testers: Test teams are responsible for defining and executing the test scripts to support different testing stages (for example, system test, system integration test, user acceptance test, performance test, among others). In the context of the ASG implementation, the ASG tester should contribute towards the creation of a testing strategy and subsequently the implementation of the relevant DevOps tooling.
ASG middleware engineer: He/she is responsible for installing and configuring the ASG infrastructure. He/she is also responsible for the deployment of services and APIs between different environments and also monitoring of the SOA and API infrastructures. Once a service or API has gone live, the support specialist is also responsible for monitoring the platform, conducting root cause analysis exercises when needed, and performing bug fixes and regression testing on the code.
DevOps engineer: He/she is responsible for the installation and configuration of the DevOps-related tools (for example, version control systems, continuous integration, and packing tools) and also for helping define DevOps processes and enforcing them in the SOA and API lifecycles.
DevOps manager: Owner of the DevOps solution, his/her role is to define and implement DevOps and also enforce its adoption. The success of this role should be measured by its ability to make the developers' lives easier by automating tasks such as deployments, installation, and configuration, but should also be responsible for bridging the gaps between development and operations teams.
The following RACI model provides a comprehensive view of the accountability matrix when implementing ASG:
|
Deliverable |
Responsible |
Accountable |
Consulted |
Informed |
|---|---|---|---|---|
|
Business strategy |
CEO |
CEO |
CIO/CFO/COO |
Everyone |
|
IT strategy |
CIO |
CIO |
CTO / Enterprise architects |
Everyone |
|
EA strategy |
Enterprise architect |
Chief enterprise architect |
ASG solution architects |
All architecture groups |
|
ASG strategy |
Solution architect |
Lead solution architect |
Enterprise architects |
All architecture groups |
|
ASG RACI |
Solution architects |
Lead solution architect |
Enterprise architects |
ASG architecture groups |
|
ASG design-time |
Solution architects and designers |
Lead solution architect |
Enterprise architects |
ASG groups |
|
ASG runtime |
Developers and middleware engineers |
Solution architects and designers |
ASG groups | |
|
DevOps |
DevOps manager |
DevOps manager |
DevOps engineer |
All architecture groups |
|
ASG tools |
Middleware engineers |
Lead solutions architect |
Solution architects and designers |
ASG groups |
|
DevOps tools |
DevOps engineer |
DevOps manager |
DevOps engineer |
All Architecture groups |
ASG design-time focuses on the inception, discovery, cataloging, harvesting, design and testing of services throughout its lifecycle. The purpose of ASG design-time is to provide mechanisms to ensure that these phases can be achieved as efficiently as possible without compromising on quality and delivering as much business value as possible. One of the key concerns of ASG design-time is to make sure that services and APIs can be properly cataloged and/or harvested with the right level of metadata. It is equally concerned with ensuring that services and APIs can be discovered and reused. For this purpose, ASG leverages the strong community management capabilities that API Management portals usually come with.
The API lifecycle focuses on ASG runtime, as can be seen from the preceding diagram. However, it is important to understand that SOA Governance and API lifecycles overlap and are complimentary methodologies that can be executed in parallel.
A key concern of ASG design-time governance is the ability to deliver a fit for purpose reference architecture and overarching standards. In ASG there is a lot of emphasis on collaboration and communication. It is therefore necessary that any architecture and/or standards produced in support of ASG design-time governance are the result of interactions between different groups and that supporting processes are in place to allow these to be modified and improved with time.
Some of the key deliverables of ASG design-time are:
ASG SDLC: ASG software development lifecycle (SDLC) defines the end-to-end software development lifecycle for the delivery of services and APIs. Development standards should cover in some detail the end-to-end software development lifecycle including all the different assets that should be produced throughout the different phases of the lifecycle. The diagram shown earlier is a lightweight version of an ASG SDLC.
Logical reference architecture: This architecture materializes the conceptual capability model by defining vendor-specific products that can be utilized to deliver the different capabilities required.
Physical reference architecture: This defines a deployment topology for the products within the logical architecture that is capable of addressing all non-functional requirements such as high availability, security, compliance, among others.
ASG design standards: This document describes the service taxonomy and a design patterns language that the designers and architects should consider when producing designs for SOA and API-related solutions.
ASG programming standards: These provide developers with coding best practices and sample recipes to cook common solutions to common problems. These standards also provide guidance on how to leverage reusable code components such as exception handling, deploying, and monitoring frameworks.
ASG information standards: These provide developers with best practices and samples to create data schemas using the notation of choice such as JSON or XSDs.
Security standards: These define guidelines that should be considered when defining and applying security policies for services and APIs. Note that these standards will be dependent on how the runtime security framework is implemented.
Monitoring and SLA standards: These define guidelines on the implementation of sensors and business activity, monitoring dashboards, composites and proxy services. These standards become particularly important at runtime as they enhance the search and monitoring capabilities of SOA instances once a service becomes operational.
Service and APIs catalog: This contains the one and only version of the truth regarding services and APIs available in the enterprise. Building a catalog can be a very complex task and many organizations end up using spreadsheets. ASG opposes the use of such methods and recommends the use of tools such as API catalogs that can be used to dynamically or manually maintain an accurate inventory of all services and APIs available. The focus of this deliverable is to define what this catalog should look like in terms of metadata and make sure that the chosen tool can deliver it.
Service and APIs catalog: ASG promotes the use of tools such as Wikis or content management systems for the publication of ASG-related information. This is a lesson learned from traditional SOA Governance implementations where several documents were usually created only to be outdated and ultimately forgotten. Instead, by using tools such as Wikis, not only will all information be available by just using a browser, but an entire community can contribute towards keeping the content updated.
ASG runtime focuses on the runtime operations lifecycle (promotion, operation, deprecation, and retirement), also including community management, runtime analytics and metering, enforcement of runtime policies, and LEAF (logging, exception handling, and auditing frameworks). Note that, in contrast to ASG design-time, API Management is mainly focused on runtime and therefore, from an ASG standpoint, the framework should be able to address not only traditional SOA runtime governance requirements but extend beyond and support all features expected of an API Management solution.
Runtime operations: This runtime deliverable consists of a tool capable of supporting the runtime lifecycle of services and APIs from the moment they are promoted into production and subsequent stages. It is essential for any tool to be able to create and publish APIs from exposed services, to monitor the runtime characteristics of an API, and to manage the API lifecycle, ultimately to retirement.
Community management: ASG runtime should have the capability to manage a community of known and unknown people (or actors). The solution should be able to support the following personas at a minimum:
API developer: This creates and publishes an API so others can discover it and use it
API consumer: He/she is a registered user that is capable of browsing the API catalog and generating API keys in order to consume APIs
API administrator: He/she is responsible for operating the platform
Runtime analytics and metering: This is the ability to keep track of consumer-specific usage and to be able report on it in real time or historically.
Policy enforcement: This is the ability to enforce policies at runtime. Policies can be about security (authentication and authorization), but also contractual (for example set and enforce restrictions based on usage).
LEAF: Logging, exception handling and auditing frameworks (LEAF) delivers a robust framework that can be leveraged by developers to standardize the way logs are generated, exceptions are handled, and business transactions are monitored:
Deployment framework: Creating a framework of components that standardize the way code is promoted to different environments is extremely important, not only from a configuration management perspective but also from a project point of view. Applications such as Oracle SOA Suite 12c or OSB 12c do come with services exposed by the external systems within a BPEL or a pipeline flow. These external systems are expressed in the form of URLs which change from environment to environment and should therefore not be hardcoded into applications.
Tip
Promoting code using IDE tools such as JDeveloper and/or Eclipse is highly undesirable and not recommended, as this approach requires a high degree of manual intervention, which can introduce human error.
A runtime deployment framework should standardize and centralize the way code is promoted between all environments. Oracle API Gateway and Oracle API Manager come with utilities (for example, the ANT utilities or the WSLT scripts) that can be used and extended in order to create a robust deployment solution.
Exception handling framework: One of the key success factors of an ASG implementation is the ability of the business as usual (BAU) team or the operations team to be able to support the solution once it has gone into production. Although this depends on many factors (for example, infrastructure, networks, storage, external systems, and so on), it is essential that errors are reported to the appropriate teams quickly and efficiently in order to minimize the risk of disruption to a business. The team running productions systems must be able to conduct root cause analysis and search for errors and logs in order to rapidly diagnose errors.
An exception-handling framework is a set of runtime components that are built with the sole purpose of standardizing the way exceptions of all types (business faults, systems faults, remote faults, binding faults, among others) are handled, reported, and diagnosed. By providing a consistent way of handling exceptions in different scenarios and for different message interaction patterns (for example, synchronous request/reply, asynchronous fire and forget, asynchronous call back, and so on) the complexity of supporting services and APIs in production systems can reduce dramatically.
Auditing framework: In distributed environments where transactions can jump across several systems it becomes extremely difficult to keep track not only of the number of business transactions processed but also their success ratios. This lack of visibility creates discomfort among business users and harms the perception business has of ASG-related solutions. Implementing a solution (either commercial off the shelf or built from scratch) that can deliver user-friendly dashboards with statistics on business transaction success ratios and statuses can dramatically increase business confidence.
Development Operations (DevOps) is a relatively new discipline that brings together development and operations throughout the entire ASG lifecycle and therefore avoids the issues that can result from both teams not collaborating. Some of the issues that DevOps aims to address are:
Environmental issues due to a discrepancy in the configuration of the production, test, and development systems
Issues resulting in the manual deployment of artifacts into different environments
Lack of agility as it takes too long to regress test service and APIs
Operational related issues as support teams don't know how to monitor and troubleshoot services and APIs
The key deliverables of DevOps are:
Continuous integration framework: Ensuring that the code stored in the code repositories is always working and testable is one of the golden rules of continuous integration. By implementing tools such as Jenkins (http://jenkins-ci.org/) or Hudson (http://hudson-ci.org/) to support continuous integration efforts in the ASG development lifecycle, the quality of the code, and the speed of delivery can improve dramatically. Having such components in an ASG framework means that projects delivering services and APIs have to align with the practices and guidelines specified, resulting in better quality solutions and a faster delivery.
Testing framework: As the amount of services and APIs increase and their interdependencies grow, the process of unit testing and regression testing can become a challenge. Testing teams are not necessarily subject matter experts in SOA or API development and, therefore, it is imperative that they are able to utilize standardized solutions to test the web services. There are several tools available that can help automate the testing of SOA services.
Provisioning framework: Manually installing and configuring an entire SOA and API platform can be very complex and can take several days. Furthermore, all installations are manually executed, increasing the chance of human error. Furthermore, ideally, the installation should be easily reproducible for creating further environments with identical topologies. Human error and configuration problems can be the cause of severe delays when testing code, as it becomes extremely difficult to identify the root cause of issues. To avoid such issues and the many others that can result in a poor infrastructure configuration management, a framework should be constructed to automate software installation and configuration across all environments. Automating installation and configuration tasks using tools such as Puppet Labs (https://puppetlabs.com/), Chef (https://www.chef.io/chef/) or Ansible (http://www.ansible.com/home) represent a much better option than doing this manually. The investment is minimal compared to the cost of issues that can result due to environment inconsistencies and misconfigurations.
