Process Template

Recently I have developed a document which will be helpful in documenting enterprise process. This document is in MS WORD format. The document is structured in technology independent way. I will certainly welcome any suggestion and changes with respect to technology and platform.

I hope this document will be helpful tool to lot of us to sail through EPM/BPM universe.

The document can be downloaded from Scribd.com.

Anatomy of Service

Service is one of the basic component of SOA. But what is Service and what are the different attributes of a service and how to interact with each other and across different services?

In simplistic terms service is logically grouped set of operations capable of performing related unit of work and operates over a message.

So service is units of processing logic, which are closely related.

Therefore service has two major components:

1. Message: Data required to complete partial or all parts of a unit of work.
2. Operation: The logic required to process message in order to complete a unit of work.




A well-designed service should have following attributes:

1. Service should be reusable
2. Service should exposes a formal interface contract
3. Service should abstract underlying logic
4. Service should be composable
5. Service should be autonomous
6. Service should be stateless
7. Service should be discoverable
8. Service should be available over network

In addition of these, if we consider SOA as confederation of services on more attribute is desired:

9. Services must be loosely coupled.

So how these attributes interact!!



Services may also be viewed from application perspective over which they are written or which they make in some composable fashion. The second perspective is process perspective, which view services from enterprise process perspective.



Application perspective of Service




Process perspective of Service




Hybrid perspective of services

SOA and Business

SOA is for business enablement. It is for business to help manage IT assets more effectively and efficiently.

IT assets in general may be viewed from three perspectives.

1. Enterprise
2. Application
3. Product

Enterprise perspective include IT assets at inter and intra enterprise level. It considers Business value of IT assets on enterprise level; so it’s effects is also enterprise level. This perspective is generally elaborated in terms of inter and intra enterprise integration. This perspective is generally taken care by CIO and CTO level while deciding enterprise wide IT strategy. Enterprise Architects also play major role in shaping this perspective.

Application perspective limits IT assets within the boundaries of application – off the shelf or custom-made. This perspective considers Application or its significant functionality as service. This perspective makes sense to VP Engineering and CTO. This perspective gains significance in case of purchasing new enterprise application.

Product perspective becomes very important for product developing companies. This perspective focuses on the services offered within a product. Proper usage of this perspective helps in construction of robust and flexible product/product family. This perspective assumes significance for Technical Architects.

All three perspectives help business to save on cost and decreasing the recurring expenses in terms of maintenance & support. They also helps in reducing time to market.

SOA also brings technological breakthroughs which helps in cost saving and fasten the introduction of products in market.

1. Increased reusability of components/services.
2. Ease in integration – inter and intra enterprise.
3. Increased pace of development and adjusting IT assets as business evolve.
4. Facilitation of incremental construction (architecture, design & development) of IT assets.
5. Facilitation of geographically dispersed development in varying time zones.
6. Facilitation in outsourcing/off shoring the IT assets – construction and maintenance & support.
7. Facelift of legacy application and integration of them in modern IT landscapes.
8. Increased utilization of human resources.
9. Reduced and managed risk with respect to legacy IT assets and resources.

Service Interface / Contract Definition

Recently I have developed a document which will be helpful in documenting service interface or contact. This document is in MS EXCEL format. The document is structured in technology independent way. I will certainly welcome any suggestion and changes with respect to technology and platform.

I hope this document will be helpful tool to lot of us to sail through SOA universe.

The document can be downloaded from Scribd.com.

Book Review: SOA Governance by Todd Biske (ISBN-13: 978-1847195869)

SOA Governance is one of the most vigorously debated and misunderstood subject in SOA world. Because of inherent complexity of governance and SOA, SOA Governance becomes more complex subject.

Book is written is style of novel coupled with some hard technical talk in end of each chapter. This makes reading of book very relaxing though covering complex subject. Book covers governance issue in such a fluid manner that while reading it seems that one is just browsing a story book. Book starts with basic concept of governance and start peeling layers. It covers architectural, design and project management challenges to have good governance model from project as well as enterprise perspective.

Book also take advantage of case study approach and assume very realistic insurance company with all nuisance of IT Management and Architecture noise.

Book is certainly a good read for its target audiences – Architects, IT & Business management folks and certainly those who are interested in SOA Governance.

Go and grab the book at packtpub.com

Players in SOA universe

In SOA universe there are two types of players. One those who are selling different products under SOA umbrella and second one are pure intellectual efforts who are defining SOA.

There is also third breed which consists of service providers who are mainly focusing on implementation strategy.

Breed one and two has lot of cross pollination to promote their respective product suite. This cross pollination has resulted in plethora of specification and some time at the expense of each other.

So the question arises is this leading to some where or it will turn SOA into one more kind of Lock-in strategy by product companies?

Enterprise Process Management

Enterprise Process Management should be considered as superset of Business Process Management. In current scenario whenever we talk about BPM we always think of business organization. But in real world every organization is not business organization. There are various organizations which have interests beyond business e.g. government organizations, non-government organizations (NGO), philanthropic organizations, etc. So the term BPM is not sufficient to cater their needs. I propose Enterprise Process Management (EPM) because Enterprise covers not only business organization but also lot of other non-business organizations.

Keeping EPM into sight, what is process?

“A process is a coordinated thread of sequential and parallel activities needed to deliver value to stakeholders.”

So what is EPM?

“Enterprise Process Management is assuming controlling position on processes from inception/idea to its delivery, from very first activity to last, and analysis of each and every activity for continuous and step improvement.”

From definition it is very clear that EPM can be visualized from three view points:

1. Inventive viewpoint
2. Execution viewpoint
3. Improvement viewpoint

1. Inventive viewpoint: This viewpoint is affected by strategic and tactical decisions of any enterprise. Strategic and tactical success of any enterprise demands effective and efficient processes with continuous improvement build in. As environment is dynamic and most of the time beyond the control of enterprise, processes must be adaptive in nature and if possible self-aware.



2. Execution viewpoint: This viewpoint focuses on execution part of processes’ lifecycle. Once process is released in any enterprise for execution purposes this viewpoint takes over. In any successful process this viewpoint has longest duration with numerous small changes but insignificant number of step changes.



3. Improvement viewpoint: In this viewpoint focus is on improvement related aspects of process. This viewpoint gets feed from execution and gives to inventive viewpoint. The improvements may be minor chunks (part of continuous improvement) or quantum leap.

Classification of Enterprises Processes

On the basis of complexity, enterprise processes can be divided into three categories:

1. Material Processes
2. Information Processes
3. Human Interaction Processes

1. Material Processes: MPs are primarily concerned to “THINGS”. The purpose of these processes is transforming the material and consume resources. These processes rely heavily on Industrial engineering concepts. These processes are found in primarily in manufacturing, assembly, process industrial environment. Generally speaking these processes are so matured over time period across the industries that they more or less resemble procedure or long running procedures. This means the input and output of these processes can be defined objectively.

2. Information Processes: IPs are primarily concerned to “DATA”. These processes transform data into information or information into more abstract information. These processes rely on Computer science, Software engineering and Information science concepts. These processes are prevalent in service and non-service industrial environment. Again due to tremendous progress in computer science, information science and software engineering these processes resemble to procedures.

3. Human Interaction Processes: HIPs are primarily focus on human cognitive aspects. HIPs are articulation of subjective issues and getting agreement among various stakeholders. HIPs are based on structure of human communication and coordination, which are prevalent in natural languages and cultures. HIPs are highly affected by cognitive abilities of participating actors and surrounding environment. Most of the time HIPs are highly subjective so their automation is a big challenge.

In real life settings, most of the enterprise processes are combination of MP, IP and HIP.

For simplicity let us consider a simple process: Opening a checking bank account by a new customer.

So to open checking bank account, customer has to submit a form to branch where he wishes to have account. The account opening form also need some supporting document like identity card of the prospect along with letter from an existing account holder. Upon submission of completed form and supporting document, a clerk at branch verifies the documents at preliminary level and sends to regional office for detailed verification. If verification result is OK, account is opened, base branch & customer are informed and checkbook, ATM card are dispatched to customer. If verification result is NOT OK, base branch & customer are informed.

In the first glance the process seems to be well structured and simple enough for automation. This process seems to fall in Material and Information Process category. So automation looks easy and within reach. Now let Human aspect enter into the process.

Mr. Clerk who accepts documents from prospective account holders has friendship with Mr. Tony. One fine day Mr. Tony reaches to branch where Mr. Clerk is stationed and submit Account opening form. Mr. Tony does not was not carrying supporting documents, so he talks to Mr. Clerk and makes promise to submit supporting documents on next Monday and asks Mr. Clerk to hold his docs till then. So human relations start tearing a well-structured process.

Since human interactions and communication is unstructured and asymmetric, so automation of such aspects is very difficult.

Enterprise SOA Reference Architecture

Here I propose SOA Enterprise Reference Architecture. This is for enterprise and certainly for an application it should be inspiration.



Reference Architecture is independent of platforms offered by various vendors. It is also possible that none of the vendors in market provide complete solution to realize this reference architecture.

Data Persistent Layer

Data Persistent Layer is primarily consists of Enterprise Data - Master, Transaction and Meta. It may also include Datamart and Data driven reporting infrastructure.

In most of the large enterprises Data Persistent Layer consists of relational (RDBMS), hierarchical (LDAP, IMS), unstructured (Emails, chat sessions, file systems, etc) and any other type. These variations in data characteristics warrant unique data handling and treatment technologies.

Service Providers

This Layer consists of different business applications. These applications may be packaged (e.g. SAP, Reteck, etc) or may be custom developed in various technologies (e.g. Java, .Net, etc). Again application in this layer may be service enabled or legacy once, which do not expose their functionalities as service.

At this layer EAI exist. Some may argue that SOA is one form of integration then how EAI can exist at this layer. I will discuss this argument in one of my future posting but not now.
Services Layer

This layer is consists of multiple sub layers which encapsulate different class of services. It encompasses Connectivity Services, Data Services, Business Activity Services and Business Process Services. Each layer in turn consists of other sub-types of services.

This layer exposes different aspects of business - functional and infrastructural aspects. Connectivity and Data services are consisting of Infrastructural services while Business Activity and Process services expose functional Services.

These services though in layers but still breach layered architecture to expose these services independent of layers and for better performance. These services can also be exposed to multiple ESBs if situation warrants.

1. Connectivity services primarily cater to accessing system level functionalities, any adapter & connector services and partner (B2B) integration services.

2. Data services comprises of services, which caters to Business, Technical and Meta data related services.

3. Business Activity Services consist of Atomic business/functionalities and any custom business logic developed by combing one or more Data & Connectivity services.

4. Business Process Services primarily consist of Process integration, Machine and Human centric services. These services primarily comprise of Business Activity services.

Enterprise Service Bus

ESB provides four functionalities:

Orchestration
Routing
Mediation
Enrichment

Some ESBs may provide one more functionality - Choreography. This functionality is associated with BPM systems and currently none of Commercially available ESB provides the same.

Enrichment may be of two types: Data Transformation and Channel Transformation. Most of the analysts & architects keep Data Transformation & Channel Transformation separate functionality.

Presentation Services

These services support various user interfaces (human or machine centric) and let outer world (intra & inter enterprise) interact with enterprises IT assets. The consumers of presentation services are

Human

• Thin Client

Browser clients: May be based on traditional HTML based or AJAX based

Portable devices: May be based on open standards (WAP) or propitiatory (device or carrier specific)

• Thick Client

Alway Connected: These clients connect to service providers and consume the services in real time. Traditional think client falls in this category

Offline Clients: These clients do not connect to service providers in Always On fashion but an actor can work in offline mode and when ever connectivity available, services consumed. Java WebStart and Adobe AIR are two most widely used technologies in this field.

Voice Based Client

Voice based clients are gaining popularly to reduce load on contact center. To facilitate this type of user interface, IVR plays great role.

Machine

These types of interfaces are not very obvious but very prevent at enterprise level. Application level interaction (Software to software) and interaction with various devices like RFID tags, Bar code readers, etc (Hardware to Software or vice versa) are part of this type of interfaces.

Integration with Outer World

Most of the large enterprises also get integrated with external world. This integration is both ways - information flow is in both directions. To further complicate the scenario, enterprises increasingly using SAAS. This situation requires special attention and tools. In case of integration with business partners B2B Engine plays major role in conjunction with Partner services. SAAS services may be consumed directly or via B2B engine. In case of Human interaction (CRM - Salesforce.com) B2B Engine should be bypassed but for m/c interaction or substantial data transfer (Document management for insurance companies) B2B engine should be engaged.

Security

Security in SOA is one of the biggest concern due to distributed & heterogeneous systems interacting with each other. The security related aspects should be handled at each level of enterprise. In the proposed model security is considered from physical and virtual perspective. It also takes into account security from network and application view.

At monitoring front it take care of Technical - For support staff, Process & Managerial - For business process owners & managers and Executive - for top rung of business leadership requirements.

Governance

Governance is one of the burning issues in SOA environment. To discuss SOA Governance's Technical architecture, it is necessary to understand SOA Governance model.

Following figure depicts the SOA Governance Model.



The picture below intends to depict SOA Governance Technical Architecture.

Service Life Cycle Management

Service Life Cycle Management essentially helps in managing service life cycle from its conception to decommission. This is one of the vital components, which determine the success of SOA initiative in any enterprise and also its long-term viability.

Rule/Policy Repository & Engine

Rule/Policy Repository contains various Business Rules and Governance & Security Policies. The engine part of the same will help in execution of Rules and Policies as and when demanded.

Service Directory

Service directory essentially consists of UDDI, which facilitates consolidated listing of services.

Event Management

To exploit SOA to its full extent event handling plays crucial role. In SOA environment Events can be classified into three categories


1. Generic Event
Event fired/raised in response of event or interaction with other service/s.

2. Time Based Event
Event fired/raised at particular moment of time to perform certain task

3. Heart Beat Event
Event fired/raised by services at regular interval to announce its existence and may be level of throughput it is working.

Generic Event Handling

These are events which fire/raise other events or results due to interaction of services in ecosystem.



1. Event Generator generates events with minimal data set.

2. These events should be formatted as per the event protocol defined in ecosystem, so that theses event are available to entire ecosystem for further processing.

Ø Event Pre-processing: Check that mandatory attributes are present in the incoming events
Ø Event Refinement: Introduce the other system-generated attributes like start time etc.
Ø Situation Refinement: Check the target status on which action would be performed
Ø Impact assessment: Required result assessment after that impact applies on the target

3. Event factory determine the specific type of action, which the given event have.

4. Event listener executes the specific action, which stated at the specific action code. There we have some general action apply to all type of system is present in generic action code and specific to specific action is present in the specific action listener

5. Event listener executes the action and calls the required system to complete it.

6. Some event, which completed their actions, and no further support needed from any other services are terminated

7. Some events which need further processing from some other services or may be generate other event in response of event is send back to the event engine

8. Event in response of event is again run as a new event and again it initiates its processing cycle in the event engine

Time Based Event Handling

Scheduled event delivered to a service at specific time in the future. Event deliveries can be deferred for short periods of time (such as seconds or minutes) or for long stretches of time (for example, hours later for batch processing). Until that delivery time, the Event is essentially invisible until it is delivered, allowing you to schedule work at a particular time in the future.



Heart Beat Event Handling

Heartbeats, a common software construct, verify the continual operation of a specific service. With heartbeats, a targeted service continually broadcasts a signal across its environment. System works normally when client services can detect a targeted service's heartbeat signals

There is number of interdependent services in SOA ecosystem such that, if one failed, others will also fail.

Heartbeat generally propagated across SOA ecosystem using publish/subscribe paradigm.

Transaction Management

Transaction Management is one of the most difficult aspects of any enterprise architecture due to varying level of understanding among stakeholders, various technological platforms involvement and spread of transactions – intra & inter enterprise.

In general any enterprise grade transaction framework follows ACID but due to presence of long running business processes and involvement of asynchronous communication traditional ACID becomes irrelevant. To accommodate long running business processes and asynchronous communication relaxed ACID is used which allows violation of ACID but with capability of correcting the violations in reasonable time period and with little business impact.



SOA capable enterprise grade Transaction Framework should have following features:

· Multi level undo/redo
· Coalescing of transactions
· Automatic aggregation of nested transactions
· Supports batching of transactions (Aggregation)
· Rollback of aggregated transactions for error recovery
· Listener support
· Synchronous and Asynchronous Transaction
· Compensatory Service
· Transaction Propagation

Exception & Error Management

Exception & Error Management is consist of
Exception & Error Handling
Exception & Error Logging
Exception & Error Notification
In any enterprise under SOA ecosystem Exception & Error Management poses following challenges:

Exception & Error Handling

No predefined format for passing exception information among participating services
Web services do not have capability of maintaining stack trace
Lack of common exception vocabulary across enterprise

Exception & Error Logging

Scattered logs across services/applications
Varying formats of logs across services/applications
Toggling logs at run time is service/application dependent

Exception & Error Notification

Every enterprise/LOB has its own unique notification requirement
Toggling Notification at run time is necessity
SLA and exceptions are coupled
To counter these challenges there must be a comprehensive Exception and Error Management framework which can manage at enterprise level and form the foundation for Remediation framework.

On the high level, Exception and Error Management framework’s architecture can be depicted as follows:

Remediation Framework

Remediation framework plays very important role in any SOA ecosystem. It can be though of extension of Error & Exception Management and Transaction Management frameworks. Remediation framework leverages Error & Exception Management Framework by utilizing logged error & exceptions. It also supplement Transaction Management Framework via helping to implement relaxed ACID where human intervention might be needed. Remediation framework also makes whole of SOA ecosystem robust via utilizing compensatory services.

Relationship

In SOA ecosystem Security, Governance, Operations, Quality are closely related. Each face of SOA ecosystem affects other.

Book Review: SOA Approach to Integration XML, Web services, ESB, and BPEL in real-world SOA projects

Book Review: SOA Approach to Integration XML, Web services, ESB, and BPEL in real-world SOA projects by Matjaz, Ramesh, Pooranchandra and Frank (ISBN: 978-1-904811-17-6)

At first glance this would appear to be another "me too" title, and while in some ways that may be true. However there are a couple of things that set this book apart and it can be of value to those experienced in Integration & SOA as well as those new to the subjects.

Those who are new to the book I will recommend read as much as possible and this book must be one those. This book covers basic essentials of Integration, SOA, WebServices and BPEL in very simple and lucid language, which can help any greenhorn.

The most important part of this book is its language. The most glaring lapse in this book is assuming SOA means web services.

It has explained concepts of EAI in very concise & to the point in first chapter. While explaining Integration Infrastructure authors have drawn nice picture, which covers almost all aspects of integration tool stack. But in the section of Integration Technologies section, it fails to mention that MoM, TPM, ORB, App Server and ESB are various generations of the same and no word about “Several hybrid and proprietary products”.

The second chapter covers SOA in technology agnostic as well as Java &. Net specific way. This chapter nicely covers basic concepts of SOA, which include not only ESB but also web services and its Java and .Net implementation. This chapter also tries to cover process-based architecture in the end, which makes things little bit clumsy.

The third chapter covers XML, Schema, XML document security, and XML parsing. This chapter covers a lot but also leaves behind a lot. It totally neglects non Java XML parsers.

The forth chapter seems to be inspired from IBM’s eBusiness patterns. This chapter also touches Web Services Interoperability (WS –I) stuff, which is pretty elaborated.

The fifth chapter covers BPEL and integration with emphasis on process orientation. This chapter is good reference for BPEL and especially Oracle way of BPEL in its Fusion Middleware.

The last chapter covers ESB in detail. The name of the chapter and content of it does not match.

The most wanted but missing part in this book is references for each chapter in particular and for book in general.

In summary I would suggest that this is a good book for anyone involved in SOA and integration project. It will provide a good introduction for those new to the discipline of Integration and SOA. While acting as a useful refresher for those with more experience

go and grab the book at packtpub.com

SOA and Business

SOA is for business enablement. It is for business to help manage IT assets more effectively and efficiently.

IT assets in general may be viewed from three perspectives.

1. Enterprise
2. Application
3. Product

Enterprise perspective include IT assets at inter and intra enterprise level. It considers Business value of IT assets on enterprise level; so it’s effects is also enterprise level. This perspective is generally elaborated in terms of inter and intra enterprise integration. This perspective is generally taken care by CIO and CTO level while deciding enterprise wide IT strategy. Enterprise Architects also play major role in shaping this perspective.

Application perspective limits IT assets within the boundaries of application – off the shelf or custom-made. This perspective considers Application or its significant functionality as service. This perspective makes sense to VP Engineering and CTO. This perspective gains significance in case of purchasing new enterprise application.

Product perspective becomes very important for product developing companies. This perspective focuses on the services offered within a product. Proper usage of this perspective helps in construction of robust and flexible product/product family. This perspective assumes significance for Technical Architects.

All three perspectives help business to save on cost and decreasing the recurring expenses in terms of maintenance & support. They also helps in reducing time to market.

SOA also brings technological breakthroughs which helps in cost saving and fasten the introduction of products in market.

1. Increased reusability of components/services.
2. Ease in integration – inter and intra enterprise.
3. Increased pace of development and adjusting IT assets as business evolve.
4. Facilitation of incremental construction (architecture, design & development) of IT assets.
5. Facilitation of geographically dispersed development in varying time zones.
6. Facilitation in outsourcing/off shoring the IT assets – construction and maintenance & support.
7. Facelift of legacy application and integration of them in modern IT landscapes.
8. Increased utilization of human resources.
9. Reduced and managed risk with respect to legacy IT assets and resources.