One thing that worries me a bit, is that you hardly find any discussion on data and system integrity in an SOA environment. At least, I don't see many. To me, data and system integrity is the most challenging issue we face today and the days to come. We will see more and more combinations of Services, SaaS, legacy apps, which will make a 'standard solution' to this problem even more relevant.
Lots of vendors I talk to tend to minimize the problems of integrity and robustness. They point at their infrastructure and say: well, our infrastructure is WS-Transaction compliant, you will not lose any messages. OK, that may be true, but will it fix failures? You can guess the next comment: with our high availability strategem, we can ensure a 99,99% availability, so you need not to worry.
Despite their reassurances, I still tend to worry: 99,99% availability is not 100%. Things can still (and will!) go wrong.
Let me try to explain the issue as I see it. A very simplistic example:
Service A calls Service B to execute a process. During execution of Service B it calls upon Service C to handle financial details.
Suppose Service B needs to be restored to a certain point in time because of an internal failure (does it really matter what caused the need for in-time recovery? I think it does ...). What does this mean for Service A, B and C? What kind of functionality do we need to have in place to make sure the entire system will not lose its integrity? How do I make sure that Service B 'catches up' with Service A and C? One might argue that due to the statelessness of a service, this shouldn't be a problem, but it is (besides the fact that there's loads of statefull services out there).
This used to be no problem in our legacy application environment. You just rolled-back the whole system and started all over again. However, our boundaries have become much smaller and larger at the same time. It is still a valid approach within a service boundary, but not in an SOA environment (which has no clear boundaries to begin with). Especially as you use services that might not be under your control (SaaS vendors, chainpartners, etc).
My worries come from the fact that most companies I visit, do not have a strategy to maintain this integrity. Mostly, they do not even acknowledge this problem, until they are confronted with it in real life. Suddenly it's become a major problem, because it is very hard to determine what to do, but there's a lot of pressure to fix it right this very minute!
What we need to keep in mind here, is that it is not just a technological problem. It is functional as well. How does the business wants to respond to failing (internal or external) components?
Luckily, having a good middleware infrastructure mitigates the problem somewhat, so it is possible to reduce the problem a lot, but especially in high-volume environments you really need to have a well thought and tried-out strategy in place. There's no falling back to manually fix things when you're processing thousands of transactions a minute.
Showing posts with label soa. Show all posts
Showing posts with label soa. Show all posts
Monday, 15 February 2010
Wednesday, 18 November 2009
SOA Magazine
I'm very proud to announce my first publication in SOA Magazine, on preventing failures in an SOA environment by implementing good governance.
It's not quite a standard article. I wrote it together with Dave Berry but it really originated from a discussion we had on his/this blog and in real life as well. We ended up doing a fictitious interview with an architect. The architects' answers reflect our real-world experiences rather closely.
Enjoy, and let me know what you think of the article!
It's not quite a standard article. I wrote it together with Dave Berry but it really originated from a discussion we had on his/this blog and in real life as well. We ended up doing a fictitious interview with an architect. The architects' answers reflect our real-world experiences rather closely.
Enjoy, and let me know what you think of the article!
Saturday, 24 October 2009
SOA Manifesto presented
Yesterday at the closing of the SOA Symposium in Rotterdam, the SOA Manifestowas presented to the audience. The working groups have worked very hard to finish the Manifesto to be presented here. A - not complete picture (some people had to leave early) - of the working group:
The whole manifest can be found on the SOA Manifestowebsite.
The whole manifest can be found on the SOA Manifestowebsite.
Tuesday, 11 August 2009
Podcast on SOA Governance
A while ago I was at Oracle HQ talking to Dave Berry, product team manager of Oracle Fusion Middleware, about governance. We started out a discussion on our blogs and we felt it would be a nice idea to convert our discussion into an article.
Somehow Bob Rhubart, manager of the Architect Community within Oracle, got wind of our discussion and invited us to give our take on Governance in general. The result is two podcasts on Governance:
* OTN Arch2Arch Podcast: SOA Governance: It's Cultural
* OTN Arch2Arch Podcast: SOA Governance Perspectives.
Let me know what you think of it!
Somehow Bob Rhubart, manager of the Architect Community within Oracle, got wind of our discussion and invited us to give our take on Governance in general. The result is two podcasts on Governance:
* OTN Arch2Arch Podcast: SOA Governance: It's Cultural
* OTN Arch2Arch Podcast: SOA Governance Perspectives.
Let me know what you think of it!
Tuesday, 30 June 2009
Process Engines vs State Engines
Process Engines, usually BPEL engines or Workflow engines, execute proces-steps based on a predefined process. This is a sound practice, however not for long-running processes.
Long-running processes have the nasty habit to change over time. This will affect all active and running processes. Why? Because the process engines usually store the process state and corresponding data in a persistence store. The effect of a change in the process requires you to migrate all those saved instances, unless you are able to leave them running the old version of the process.
A different approach is a State engine, where the next step in the process is not determined by the predefined steps in the process, but on the current state of the subject and the event that has been received. This leads to more decoupling between process and services. However, so far I haven't seen any Process engine that works like this.
In theory there's not much difference between the two approaches. You could state that a process engine uses implicit states as defined in the process, whereas a state engine uses explicit states. The end result could/should be exactly the same. A state engine would give some more flexibility, but also needs more functionality to be able to have an overview and control of the process.
Anyone knows of any tool that works like a state engine? I'd really like to know.
Long-running processes have the nasty habit to change over time. This will affect all active and running processes. Why? Because the process engines usually store the process state and corresponding data in a persistence store. The effect of a change in the process requires you to migrate all those saved instances, unless you are able to leave them running the old version of the process.
A different approach is a State engine, where the next step in the process is not determined by the predefined steps in the process, but on the current state of the subject and the event that has been received. This leads to more decoupling between process and services. However, so far I haven't seen any Process engine that works like this.
In theory there's not much difference between the two approaches. You could state that a process engine uses implicit states as defined in the process, whereas a state engine uses explicit states. The end result could/should be exactly the same. A state engine would give some more flexibility, but also needs more functionality to be able to have an overview and control of the process.
Anyone knows of any tool that works like a state engine? I'd really like to know.
Monday, 22 June 2009
ODTUG Update: SOA Symposium
Yesterday at 8 pm we started the SOA Symposium at ODTUG (yes, Sunday and Father's Day to boot). ODTUG is well known for its technological content, but as SOA becomes more and more mainstream, SOA will become a integral part of ODTUG.
The symposium was organized by three Dutch Oracle Ace Directors: Lonneke Dikmans, Lucas Jellema and myself. The setup we choose was a little different then one would expect from a symposium. From the start, we aimed to get a very interactive symposium where the goal is to meet other SOA adepts and to exchange experiences. We managed to achieve this by doing just 3 presentations (to create a mindset) followed by two workshops and a panel discussion.
The day was divided in two parts. In the morning we approached SOA from a business point of view, in the afternoon we discussed technology. The average knowledge level and experience of the participants was rather high, which made discussions very very interesting. It was good to see that there were a lot of Oracle Ace Directors present.
We've made several very interesting conclusions which will be put on the Oracle Wiki.
The symposium was organized by three Dutch Oracle Ace Directors: Lonneke Dikmans, Lucas Jellema and myself. The setup we choose was a little different then one would expect from a symposium. From the start, we aimed to get a very interactive symposium where the goal is to meet other SOA adepts and to exchange experiences. We managed to achieve this by doing just 3 presentations (to create a mindset) followed by two workshops and a panel discussion.
The day was divided in two parts. In the morning we approached SOA from a business point of view, in the afternoon we discussed technology. The average knowledge level and experience of the participants was rather high, which made discussions very very interesting. It was good to see that there were a lot of Oracle Ace Directors present.
We've made several very interesting conclusions which will be put on the Oracle Wiki.
Monday, 15 June 2009
Accessing Oracle SOA documentation
Ever tried to find that one document about the ESB on OTN? Did you quite remember where you found that BPEL Developers Guide? Well, life's been made easy for you. Marc Kelderman (Oracle Consulting) has made a list of just about all you need to start out with SOA from an Oracle perspective. Go and have a look at Marc Kelderman's shortlist. Way to go, Marc!
Wednesday, 3 June 2009
SOA: Process driven, message driven or event driven?
For a long time I thought that SOA is process driven, meaning that the functional requirements were discovered by working downwards from a business process perspective to a comprehensive set of services to deliver the business value.
However, the more solutions I see, the less I believe it. Saying so, I realize I need to clarify that a bit more. Whenever you're building an SOA from the Business process downwards, it is smart to discover possible candidate services from your existing legacy environments (building upwards) so you can execute a 'meet in the middle approach'. This usually results in orchestrating business processes (with BPEL) and -simplified - composing composite services (based on atomic services from your legacy).
Is that bad? Not in itself, I think. It depends on the type and the number of processes. Suppose you have - on average - 50.000 running instances of a process. If you have to change that BPEL process you defined, what will you do? The first question to answer is: what do I need to do with my current active processes? Can they continue using the existing process or do the need to switch? Does it depend on the state they are in or not?
This problem gets worse the longer a process runs. It increases the chance that a process change will occur during the time it runs. Besides, not only functional changes impacts the running processes, but an update of the underlying BPEL engine may do so as well. Of course, if the number of instances increases, your problem gets worse too. Ever tried to restart a BPEL engine with 150.000 instances?
If it's a short running process, you could stop all incoming transactions for the specific process and wait for the running processes to finish before you upgrade the process to it's new version. Unfortunately life is usually not that simple.
Let's get back to the initial thought: is an SOA process driven or not? If you follow the scenario I sketched above, you might end up with one that is. What you should keep in mind while designing an SOA application, is that you have to look ahead to see what possible situations you need to able to handle. If you know beforehand that you will have a (very) large number of active processes, that they will be long-running and that they will very likely be subject to change, you might want to consider a different approach. Why? Because migrating running processes in a BPEL environment is a very hard thing to do (and expensive too). You will need to make allowances in your architecture to prevent problems.
In that scenario, a more event-driven or message-driven approach may be the answer. By loosely coupling the stages a process will go through, using either events or messages, you are a lot more flexible when it comes to migrating to new versions of a process(step). This will mean that your 'highest level' process will probably not be visible as a process in your BPEL engine, but the underlying steps may be. Again depending on the duration of the processes.
However, the more solutions I see, the less I believe it. Saying so, I realize I need to clarify that a bit more. Whenever you're building an SOA from the Business process downwards, it is smart to discover possible candidate services from your existing legacy environments (building upwards) so you can execute a 'meet in the middle approach'. This usually results in orchestrating business processes (with BPEL) and -simplified - composing composite services (based on atomic services from your legacy).
Is that bad? Not in itself, I think. It depends on the type and the number of processes. Suppose you have - on average - 50.000 running instances of a process. If you have to change that BPEL process you defined, what will you do? The first question to answer is: what do I need to do with my current active processes? Can they continue using the existing process or do the need to switch? Does it depend on the state they are in or not?
This problem gets worse the longer a process runs. It increases the chance that a process change will occur during the time it runs. Besides, not only functional changes impacts the running processes, but an update of the underlying BPEL engine may do so as well. Of course, if the number of instances increases, your problem gets worse too. Ever tried to restart a BPEL engine with 150.000 instances?
If it's a short running process, you could stop all incoming transactions for the specific process and wait for the running processes to finish before you upgrade the process to it's new version. Unfortunately life is usually not that simple.
Let's get back to the initial thought: is an SOA process driven or not? If you follow the scenario I sketched above, you might end up with one that is. What you should keep in mind while designing an SOA application, is that you have to look ahead to see what possible situations you need to able to handle. If you know beforehand that you will have a (very) large number of active processes, that they will be long-running and that they will very likely be subject to change, you might want to consider a different approach. Why? Because migrating running processes in a BPEL environment is a very hard thing to do (and expensive too). You will need to make allowances in your architecture to prevent problems.
In that scenario, a more event-driven or message-driven approach may be the answer. By loosely coupling the stages a process will go through, using either events or messages, you are a lot more flexible when it comes to migrating to new versions of a process(step). This will mean that your 'highest level' process will probably not be visible as a process in your BPEL engine, but the underlying steps may be. Again depending on the duration of the processes.
Monday, 11 May 2009
My Take on Governance: People, Process and Tools
I was reading Dave Berry's blog on how to achieve the ROI on SOA by implementing Oracle SOA Governance tooling. Dave starts with a simple example of how customers - using Oracle BPEL PM - have no clue as on how to achieve reuse. In his blog he states that 'one of the main values of SOA Governance is to deliver savings by enabling this reuse of existing services'. From there on he moves towards governance tooling, enabling visibility, control and monitoring of all SOA artifacts.
My first response was to point out that tooling for reuse is nice, but there's also something about culture. What about the 'not invented here' syndrome? After that, i started thinking (well done, Dave, you got me thinking!) about the elements of SOA Governance. To me governance is about
people, process and tools. People, because it's all about the professionals on the job and the need to govern their actions to make sure the right things happen at the right time. To do that, you need a governance process in place that ensures that all artifacts are reviewed and judged to be either confirming to the architecture, or rightly not so. But, that should be done, before anything get's built! Finally it will really enable to process and the people to have good governance tooling available. For to be sure: we do need visibility, control and monitoring to enable reuse. And we do need reuse to reap the major profits of SOA.
For instance, I know of a customer case where the people involved were SOA adepts. However, they decided not to use an existing service, even though it was accepted by the Architecture Board. The main reasons being, that the service was developed by a different department, it's quality and availability could (maybe) not be guaranteed and they could probably do a better job. Unfortunately, this happened not just once, but four times for the same artifact. The company ended up having 4 almost equal services doing almost the same thing. Being able to govern this process would have saved them 3 times building the same component! Tooling in itself would probably have made this problem more visible, but would not have prevented it.
So what are the typical governance processes you see being advocated either internally or by vendors? Let me know!
My first response was to point out that tooling for reuse is nice, but there's also something about culture. What about the 'not invented here' syndrome? After that, i started thinking (well done, Dave, you got me thinking!) about the elements of SOA Governance. To me governance is about
people, process and tools. People, because it's all about the professionals on the job and the need to govern their actions to make sure the right things happen at the right time. To do that, you need a governance process in place that ensures that all artifacts are reviewed and judged to be either confirming to the architecture, or rightly not so. But, that should be done, before anything get's built! Finally it will really enable to process and the people to have good governance tooling available. For to be sure: we do need visibility, control and monitoring to enable reuse. And we do need reuse to reap the major profits of SOA.For instance, I know of a customer case where the people involved were SOA adepts. However, they decided not to use an existing service, even though it was accepted by the Architecture Board. The main reasons being, that the service was developed by a different department, it's quality and availability could (maybe) not be guaranteed and they could probably do a better job. Unfortunately, this happened not just once, but four times for the same artifact. The company ended up having 4 almost equal services doing almost the same thing. Being able to govern this process would have saved them 3 times building the same component! Tooling in itself would probably have made this problem more visible, but would not have prevented it.
So what are the typical governance processes you see being advocated either internally or by vendors? Let me know!
Oracle Service Bus Explained
There's a lot of confusion towards the positioning of the Oracle Service Bus. In this post I'll try to clear up the issue as best as I can.
First of all, Oracle has already shown a convergence of BEA and Oracle FMW. This will continue even stronger in the upcoming releases, 11g in particular. The strategic platform as it will be introduced with 11g will consist of two main components: Mediator and Oracle Service Bus.
The mediator is an intra-composite mediation component within an application. It is responsible for brokering communications between components that make up a composite (conform Service Component Architecture - SCA). It will enable transformation, routing, event delivery and payload validation. The mediator is almost exclusively based on Oracle ESB (yes, the old Oracle Enterprise Service Bus).
The Oracle Service Bus (OSB) provides service bus capabilities for the entire company, again including standard functionality as transformation, routing, event delivery and payload validation. It's main function is to decouple intra-application communication from inter-application. Endpoint changes will not affect the internals of composite applications. The OSB is based on the Aqualogic Service Bus, augmented with key features from the Oracle ESB, especially JCA adapters, DVM, X-ref and JDev based design-time.
In the long run, I expect the distinction between these implementations to disappear, but I do like the current setup, as it differs between development of application and integration. I wouldn't be surprised to see this pattern appear more.
First of all, Oracle has already shown a convergence of BEA and Oracle FMW. This will continue even stronger in the upcoming releases, 11g in particular. The strategic platform as it will be introduced with 11g will consist of two main components: Mediator and Oracle Service Bus.
The mediator is an intra-composite mediation component within an application. It is responsible for brokering communications between components that make up a composite (conform Service Component Architecture - SCA). It will enable transformation, routing, event delivery and payload validation. The mediator is almost exclusively based on Oracle ESB (yes, the old Oracle Enterprise Service Bus).
The Oracle Service Bus (OSB) provides service bus capabilities for the entire company, again including standard functionality as transformation, routing, event delivery and payload validation. It's main function is to decouple intra-application communication from inter-application. Endpoint changes will not affect the internals of composite applications. The OSB is based on the Aqualogic Service Bus, augmented with key features from the Oracle ESB, especially JCA adapters, DVM, X-ref and JDev based design-time.
In the long run, I expect the distinction between these implementations to disappear, but I do like the current setup, as it differs between development of application and integration. I wouldn't be surprised to see this pattern appear more.
Thursday, 9 April 2009
ODTUG SOA/BPM Symposium
Ever hear of the ODTUG Kaleidoscope? It's the yearly conference that's organized by the Oracle Developers Technical User Group (ODTUG). This year there will be a number of symposiums at the day before ODTUG starts.
One of them is the SOA & BPM Symposium, that's organized by Lonneke Dikmans, Lucas Jellema and myself. It will not be your average symposium. It will be a very interactive meeting of minds. The symposium is about working together, exchanging experiences and trying to determine the best, most practical way, to go about implementing an SOA.
So, if you feel you have anything useful to bring to the table or you want to hear what recognized specialists in the industry do and think, why not join us at the ODTUG SOA/BPM Symposium?
This symposium is organized by a large number of Oracle ACE Directors. Expect a large number of them to be there and participate actively! Come and join us!
One of them is the SOA & BPM Symposium, that's organized by Lonneke Dikmans, Lucas Jellema and myself. It will not be your average symposium. It will be a very interactive meeting of minds. The symposium is about working together, exchanging experiences and trying to determine the best, most practical way, to go about implementing an SOA.
So, if you feel you have anything useful to bring to the table or you want to hear what recognized specialists in the industry do and think, why not join us at the ODTUG SOA/BPM Symposium?
This symposium is organized by a large number of Oracle ACE Directors. Expect a large number of them to be there and participate actively! Come and join us!
Subscribe to:
Posts (Atom)