Recently in Event driven autonomic management Category

(Part I - Preliminaries, Part II - The Long Kiss Goodnight)

<sigh> Sorry. I tend to talk more about the surrounding atmosphere than the thing itself. In this post I hope to remain focussed and actually discuss the actual meat of the architecture and the skeleton upon which it is based. Apologies for not providing the color and fluff of life that actually surrounded the process ;)

Point 1 Humans are good at making declarative statements and woefully incompetent at micromanagement.

This is one of those points that one shouldn't have to make, but the simple fact that bears repeating is that Humans are really good at figuring out what should be done, but really shitty at actually - you know - doing what they think should be done. In keeping with the spirit of the theme of this post, I leave it to the reader to reflect upon the profound reality that it is far easier to see the goal than the path to it. The profound insight I have (not singular, as I'm sure you're aware) is that a management system which caters to the micro-manger who actually is competent at orchestrating a complex series of transformations under chaotic conditions are few and far between - so rare as to be non-existent to several orders of approximation (or so expensive, which amounts to the same thing).

The take away is simply that any system which depends on a human to do the actual work is simply not going to work - by definition. Seeing as how this is one of my premises, it's not something I can really argue. It's a premise derived by years of observations of not just other humans but also myself. Again, I'm not making the claim that there are not humans who are absolutely brilliant at micro-managing large scale distributed systems - let's be crystal clear on that point. No. My point is simply that these people are incredibly rare and you - the actual person paying the price - will have to pay through the nose if you find such a person. And, quite frankly, the chances of you actually finding such a person is so miniscule as to be almost unmeasurable. Most likely, what you'll do is find someone who claims to be such a person or someone whom someone you trust claims to be such a person. And the odds are overwhelmingly that you are just a complete maroon and have been hoodwinked into paying a lot of money for a cheap imitation of such a being. Get used to it. It's just a simple fact of reality.

(Part I - Preliminaries
Part III - The First Cut is the Deepest)

From my perspective, one of the major pitfalls in any project which starts out to produce a management infrastructure is that the project almost immediately starts focussing on the API layer rather than the defining the large scale system behavior. In many ways, this is completely understandable, given that the API has the most immediate impact on the first users of the system - i.e. those hapless fools that form the brigade of developers who have to integrate their systems into your management infrastructure. Given that in most large organizations, APIs become the mechanism that groups use to mediate their interaction - not just at the Java level, but in a visceral sense that governs the actual political interaction between the groups. Somewhat because APIs are something concrete and form a nucleus around which people can argue concretely about. But mostly it's because most people are rather ignorant of how systems actually interact, but one thing they do know is that there are APIs and consequently these concrete manifestations of handles that can be universally understood become the battleground upon which system integration takes place. Or, put another way, APIs are the lowest common denominator that even managers can understand, consequently they are the only focus of pretty much every large scale project.

But the problem with this focus is that an API doesn't define a system, rather it's the other way around. The way I think about it is that the APIs of a system are like the inner core of a sphere. Defining the surface of the system - i.e. what the system "looks" like - will provide enormous leverage on the internals of that system. And this leverage will simply force things into place - meaning that the reason the API exists is because it is literally the inevitable result of the forces that keep the system together.

But I digress.

Update: Part II - the long kiss goodnight, Part III - The First Cut is the Deepest

This is the first in a series of posts documenting the research I've been doing into a different way of thinking about system management infrastructure. For quite some time, I've been obsessed with the idea of how to simply and effectively manage large scale systems. Throughout this obsession, I've travelled down various roads and found myself in several box canyons along the way. I've tried out a lot of different strategies and have finally settled into something which provides the kind of framework I've been looking for which I haven't found replicated anywhere.

Note that I'm certainly not making the claim that it is "Teh Best" management infrastructure. Rather, what I'm making the claim is that it's the most interesting management architecture to me. As anyone who knows me can testify, I have rather peculiar tastes and I am a strange bird at times. So fair warning, eh?

In any event, what I plan to do is to provide a fairly deep dive into the architecture that I've come up with. In the standard tradition of all literature scientific and technical, it will be presented in precisely the opposite order in which I actually came up with things - i.e. from the top down, in a semi coherent form that makes sense. Lord knows that actual discoveries and explorations are more a matter of luck in which you discover something and then spend an inordinate amount of time tracking down why the heck you managed to stumble upon it and where it fits in the larger picture of things that you're trying to map out. I've always found this cognitive dissonance amusing, myself, and hope you won't mind to much when I veer off into seemingly irrelevant paths rather than sticking to the point at hand.

If you're one of those people who can't wait until the end of the story to find out what's going on, by all means download the PDF of my talk on the subject at last year's Spring Experience entitled Digging the Trenches on the Ninth Level. If you're not familiar with Dante's Divine Comedy, then you won't get the joke. But suffice it to say that I'm a big believer in the principle that every time you solve a problem, you discover ten more problems that you didn't know you had.

A perfect example of this sometimes perverse law is something as simple as email. Email solved a lot of problems that a modern economy and social population have, but in doing so it created a lot more. Without email, we would never have been subjected to the sublime beauty of penile extension spam nor would your grandmother be subjected to the horror of id phishing which you discover has snagged her bank account and drained all her life's savings leaving you with a predicament that makes you wonder what all this progress was supposed to do in the first place.

Likewise, I firmly believe that in solving the problems I believe have been addressed by OSGi, Spring DM and management architectures like mine, we've inadvertently unleashed new levels of horror that will ensure future generations will curse our names as they suffer from the fall out and live the unspeakable abominations unleashed from these "solutions" and witness them unfold in ways that we couldn't possibly imagine.

So, with that cheery panorama as the back drop, I'll end this introductory post and start working on the next post, which provides high level overview and ten dollar tour of the sewers that I've been digging for your benefit on the ninth level of hell.

Remember. I dig because I care. After all, you do want that frozen crap to be routed somewhere and dealt with, don't you?