It is truly amazing that it has been over 5 years since my last blog post. In that time, architecture, application development and integration has changed immensely.
Service-based integration has gone through this transition:
- ESB-centric SOAP-based SOA has died, reimagined with a focus on beautiful RESTful APIs implemented that microservices that finally realise the principle of service autonomy.
- With the rise of mobile, B2B gateways gave way (through evolution or revolution) to APIs gateways – which started out light but are starting to smell like ESBs, so ..
- in the spirit of smart endpoints and dumb pipes, the next big thing is the service mesh – so that’s where my blogging journey will probably begin. To whet your appetite, check out this really interesting talk on Istio and Envoy, which included a passing reference to a project that I find particularly interesting .. Open Policy Agent.
In the constant quest to build more resilient systems, reactive architectures have led us away from the mindset where we rely on strong consistency and synchronous request reply interaction. There is a resurgence in the use of messaging as a pattern, and ideas that were previously considered heresy such as eventual consistency (“it’ll be right” is ok for the enterprise!) and event sourcing to build domain-specific views of reality are mainstream.
Data integration has shifted gears even more, moving from batch-oriented ETL to real-time data streaming. Just to make it more confusing, we see the convergence between data streaming and messaging in platforms like Kafka and Amazon Kinesis. Is it a message broker? An event store? Why not both?
finally .. the cloud is no longer just a playpen for startups – it is mainstream in the enterprise and has moved infrastructure into the hands of the developer – which means that in addition to knowing how to build apps in their chosen functional or technical domain, developers, engineers, architects have a whole new domain to learn.
oh, and let’s not forget containers .. and AI .. machine learning .. virtual assistants …
Anyway, this is just a finger-stretching exercise to check that there’s still a blogger in the house. The next instalment will a) hopefully not be in 5 years and b) dive a little deeper into one of the above topics.
Until next time …
A recent post on LinkedIn got me thinking about my use of namespaces to signify that two XML structures are compatible, and how I manage schema compatibility in general.
A common practice (and my current preferred practice) is to use a namespace to denote messages that are structurally compatible. The key driver for this approach was Thomas Erl’s ‘Standardised Service Contract’ principle of service design, and a consequence of successful application of this approach is that you will have little or no message transformation happening in your solution. Versions of the schema that have the same major version are compatible. Document instances for version 2.8 of the service will validate against the version 2.1 schemas – and vice versa. Because consumers and providers that are certified to a given major version all understand the messages, there is no work for integration developers so they move on to greener pastures where ..
.. a different line of thought prevails. Major and minor versioning is still used, however in this case the major version denotes semantic compatibility. In this world, we allow schemas to be refactored over time as our understanding of the domain grows, and judiciously use the power of integration tools to isolate users of older compatible versions from the effects of change.
So how to choose?
- Go for structural compatibility + transformation avoidance if you have clever schema designers and a small or tightly-controlled IT application portfolio that will accomodate the occasional need to roll out wholesale changes as you fine-tune your schemas for more backward compatibility and less transformation! If you are part of an IT shop that runs a well-oiled release train this could be the choice for you – particularly if there is lots of automated testing that can be used to identify impact of any wholesale schema changes in minutes
- Go for semantic compatibility + contained transformation if you are in an environment where there is a lot more formality and cost around getting changes done to the applications that provide the services. In this case, you may choose to set up a dedicated integration team or better still, just add the transformation/integration tools to the toolkit of the developers who inherit the apps when the vendors leave and provide them with guidance on when to use them – or not.
As organisations grow, funding reduces and the need to do localised changes quickly increases, they may gravitate towards the latter approach so it’s probably best to plan to end up here from the outset, even if initially you mandate a “no transformation” rule.
Heresy? Anarchy? Job creation? or just plain pragmatic? The jury is still out on this one….
After a few years of pushing into enterprise architecture I have taken a break and gone back to a solution/delivery focussed role. This was a move made with some trepidation but 5-6 weeks in, I have to say that it is good to be back in solution mode, working to create a practical rendition of someone else’s enterprise architecture vision. What better way to inject some reality into the Powerpoint-coloured glasses 🙂 Interestingly, even though I am officially “just” a delivery guy now, quite a few of the things I am having to do will have enterprise reach if successful. So watch this space for insights from “the other side”. Topics may include SOA in a COTS environment, data governance and possibly even some stuff on the middleware of the day (see my tag cloud for hints) if I manage to get close enough to it.
Over the last year or so, I have been leading an SOA initiative which by all accounts has gone relatively well, with one downside. All the developers and architects will give it a qualified thumbs up (hopefully not just because it’s good resume fodder), but the business analysts and business owners of the applications that are enabled by our beautiful service catalog, have no appreciation of what the services are and how they affect them. The only exception from this is services that we expose to our customers as web services.
So how do we ‘take SOA to the business’ – and more importantly, why should we?
The recent release of TOGAF 9, which includes a chapter on SOA has made it clearer. Basically, we have been doing developer-led SOA (or for the sake of my ego, architect-led SOA), building little-S services that are important building blocks for the techies, but the business couldn’t care less about. Ironically, the highly resuseable services that the developers love are probably the ones that are least meaningful to the business (future post on this). In order to ‘take SOA to the business’, we need to think of the entire business as a set of capabilities that we provide to customers through a combination of people, process and technology. These capabilities are big-S services, and are recognisable elements of the business value chain (e.g. assessing a credit card application). Their work has a quantifiable real world effect on the bottom line (as opposed to a technical real world effect such a database table being updated), and similarly their SLAs on performance, availability etc. are traceable to a commitment to a real customer or other business stakeholder.
But wait, there’s more.
ITIL v3 gives us a service-centric framework for managing the lifecycle of IT services, i.e. capabilities provided by IT as the technology contribution to big-S services, and is already well established at the ‘lights-on’ end of the service lifecycle. In my opinion there is no reason why we couldn’t use ITIL as an overarching framework to manage the lifecycle of all big-S services, and then overlay whatever else we need at the lower-level to manage the lifecycle of the little-S services that contribute to a big-S service. As an examlpe, this is where we would see the intersection between the service registry/repository and the CMDB.
So Bill (are there people in my service), I think I might finally be getting it. Now I’m off to re-read the debate on B-SOA vs T-SOA 🙂
ps – if you have a southern drawl, replace big-S service with ‘Business service’ and little-S service with ‘Technical service’.