<sigh> Sorry. I tend to talk more about the surrounding atmosphere than the thing itself. In this post I hope to remain focussed and actually discuss the actual meat of the architecture and the skeleton upon which it is based. Apologies for not providing the color and fluff of life that actually surrounded the process ;)

Point 1 Humans are good at making declarative statements and woefully incompetent at micromanagement.

This is one of those points that one shouldn't have to make, but the simple fact that bears repeating is that Humans are really good at figuring out what should be done, but really shitty at actually - you know - doing what they think should be done. In keeping with the spirit of the theme of this post, I leave it to the reader to reflect upon the profound reality that it is far easier to see the goal than the path to it. The profound insight I have (not singular, as I'm sure you're aware) is that a management system which caters to the micro-manger who actually is competent at orchestrating a complex series of transformations under chaotic conditions are few and far between - so rare as to be non-existent to several orders of approximation (or so expensive, which amounts to the same thing).

The take away is simply that any system which depends on a human to do the actual work is simply not going to work - by definition. Seeing as how this is one of my premises, it's not something I can really argue. It's a premise derived by years of observations of not just other humans but also myself. Again, I'm not making the claim that there are not humans who are absolutely brilliant at micro-managing large scale distributed systems - let's be crystal clear on that point. No. My point is simply that these people are incredibly rare and you - the actual person paying the price - will have to pay through the nose if you find such a person. And, quite frankly, the chances of you actually finding such a person is so miniscule as to be almost unmeasurable. Most likely, what you'll do is find someone who claims to be such a person or someone whom someone you trust claims to be such a person. And the odds are overwhelmingly that you are just a complete maroon and have been hoodwinked into paying a lot of money for a cheap imitation of such a being. Get used to it. It's just a simple fact of reality.

Again, this is simply one of things that shouldn't have to be mentioned, but we live in the age when the Uber Web2.0 crowd can't seem to scale their fantastic money making ideas and blame the language for their failings as architects. So it's a fact worth repeating: anything centralized is limited in its ability to scale. Given the realities that most things don't need to scale (i.e. no one is going to use your stupid Web2.0 application anyway), it's simply no surprise that this seemingly basic fact of reality remains the province of high priests and those who walk in the light (fun fact: "He who walks in the light" is the literal translation of my name).

From this assertion, we can derive that any architecture which relies upon centralized control is an architecture which simply won't scale - QED. Again, seeing as how many things don't actually need to scale, this really isn't a problem that comes up in the bulk of most people's experience. However, when it actually does come up - i.e. when you have an actual scalability problem - it bites you so hard in the ass that it's almost like a near Death experience.

And look around you. Find a single management architecture which does not reduce to a centralized, "spider in the middle of the web" architecture. Go ahead. I'll wait. Shit, I'll give you a year to come up with 3 of them. I doubt you can name 2. Personally, I know of at least four, but that's just me. Two of these are academic, one is actually released into the wild as open source and then there's mine. But hey. I live a sheltered life. Perhaps decentralized configuration management and provisioning architectures are common as fleas in the De La Pulgas land grant here in the SF Bay area...

Point 3 Management systems should be pattern driven via events rather than command driven

This really follows trivially from the first two points. A management system which is managing a large scale distributed system (or, I would claim, even a small scale distributed system) really is a goal driven system which reacts to events rather than smugly issues commands which are largely irrelevant at the time they were issued. The mistake I believe a lot - hell, all - management systems make is that they have the hubris in believing that that the commands they issue are a) relevant and b) will be followed.

Distributed systems are exceedingly complex environments in which the main goal is simply to adapt to the current environment. Not to be redundant, but this hearkens back to my very first point in that you should declare what you want to come about, not decree by fiat what commands will be followed. Think about it. The moment that the command has passed your conscious thought process and has been (successfully!) entered into the management system, the massively distributed system you're trying to tell what to do has changed - almost always dramatically - under your feet and is no longer the same system that you were dealing with when you issued your command. Sorry, but you're going to have to let that sink in a bit because it's a really important point: the system that is reacting to your commands is not the system you gave the command to. It's just a simple fact of physics - something that Einstein figured out almost 100 years ago working as a patent clerk in some 2nd rate country.

So these three points are the three driving points in what I call Event Driven Autonomic Management. Maybe it's old hat to such wizened sages as those who read my blog, but it's hard won knowledge spanning several decades for me. In my next post on the subject I'll take these three principles and lay out a rather simple architecture which satisfies these principles and provides an autonomic, goal seeking system which reacts to high level commands by asymptotically approaching the high level goals on wildly dynamic and unruly systems which - I claim - is the reality of modern distributed management and provisioning systems we currently face.

When you're trying to build a massively multiplayer online gaming platform (MMOG), probably the most important part of the system is scalability. After all, if it doesn't scale, it's simply a multiplayer online gaming platform - without the "massive". While it almost seems embarrassing to point this out, it's extremely interesting to note that there have recently been a lot of discussion about scalability of online systems - in particular, the Web 2.0 applications. I won't point to these discussions, but suffice it to say that I find it terribly amusing to hear the various forms of the argument that you can worry about scalability later - i.e. it's not something that has to be designed in from the start. (Arguments of the form "don't worry about scalability because no one is going to use your application anyway" are perfectly fine, however). As the history of MMOG has shown, the application's architecture has a huge impact on the ability to scale. As many gaming platforms have discovered, scalability isn't something you can simply "add on" after you "get things right". Anyone who thinks that this doesn't apply to other network application architectures amuse me to no end, given as if they actually produce something of value, it will fall over when it hits the natural scalability limit of their crappy architecture.

In any event, there's a couple of basic problems with MMOG that limit scalability. The first has to do with what is known as "Area Of Interest". The idea here is familiar enough to anyone who has done any distributed communication in that the gaming platform doesn't want to find itself in an N² connection topology. In MMOG, the entities (gamer avatars, NPC, etc) have to communicate with other entities in the game. If you can't find a way to limit the communication to the entities in the area of interest - i.e. the other entities that the entity in question is limited to communicating with - then you have a huge scalability issue due to sending messages to entities that simply don't care about the communication because it can't possibly affect them. This not only wastes bandwidth and precious OS network resources but causes a host of other issues having to do with the time ordering of distributed events and filtering our events that aren't relevant. It's a mess.

As you might expect, there's been a lot of active research in this area. I've spent a non trivial amount of time over the last six months or so pouring through the research and seeing if I could find anything that wasn't lame that worked the way I thought it would. I won't review the literature here, as the papers I'm going to site already do a good job of reviewing the various ways of trying to deal with interest management in MMOG.

The second problem that I mentioned - that of load balancing - is closely related to the problem of interest management. The idea here is that you need to split up your processing load across your population - dynamically - so that you aren't trying to host 100,000 entities on one CPU and distributed the rest of the 50,000 entities across 100 CPUs due to flocking. It's a dynamic problem due to the movement of the entities which changes the interaction patterns as time progresses.

In any event, what I've been playing with over the past week or so is a rather elegant and butt simple mechanism for solving both issues - i.e. interest management and load balancing. The idea is very simple, being based on the use of Voronoi division of the virtual world to create a P2P overlay network between the various entities. You can find all the papers here at their site, a good intro paper is VON:AScalablePeer-to-PeerNetworkforVirtualEnvironments.

While I still have a ways to go on the project, what I have done is taken their 2006 Java demo code and refactored the hell out of it. I mean, geebus. C++ programmers seriously don't know much about OO programming. I mean, it's not just a "programming in another language" issue - I've seen their C++ code as well. Basically I just think that C++ doesn't encourage OO and consequently not many of the people who learn OO via C++ actually end up learning OO. Having said that, I do note that they do understand the interest management problem and have come up with a very excellent solution.

A key aspect of the code is the calculation of the Voronoi domains and this is done with a Java translation of the de facto algorithm: Steve Fortune's algorithm in C++. This is pretty much the only Java translation of this algorithm I've been able to find, so that fact alone allows me to completely forget and forgive the absolutely crappy OO design of the system. I've cleaned up that code as well, but haven't gone into a big refactoring frenzy on its ass because it works and quite frankly I'll hire someone to do this if it ever becomes a problem - there were just certain things I had to do to the Java implemenation and there were other certain things that I couldn't - quite frankly - allow to exist in source that I actually make use of... :)

In any event, being an LGPL'd source code, I've also made my results public domain using the LGPL license as well. You can download both the source and the binaries here. This includes, naturally, the Java version of Steve Fortune's Voronoi domain generator.

Below is a nice Java applet that I also rewrote which they provided as a demo of how the system works. My changes are to make the demo deterministic through a constant seed of the random generator and to make the simulated behavior a bit more realistic than their previous version had done.

There's obviously some problems with this initial algorithm that they seem to have taken care of in latter versions of the protocol, and I'm actually adding tests to the system so I can verify the behavior. But it's a fantastic start at a system which promises to solve two extremely nasty problems in MMOG and allow me to continue on with my Third Space project.

Java applet demo  (compiled with Java 5.0)

click on the applet then press 'ENTER' to start...   (other key controls)

From my perspective, one of the major pitfalls in any project which starts out to produce a management infrastructure is that the project almost immediately starts focussing on the API layer rather than the defining the large scale system behavior. In many ways, this is completely understandable, given that the API has the most immediate impact on the first users of the system - i.e. those hapless fools that form the brigade of developers who have to integrate their systems into your management infrastructure. Given that in most large organizations, APIs become the mechanism that groups use to mediate their interaction - not just at the Java level, but in a visceral sense that governs the actual political interaction between the groups. Somewhat because APIs are something concrete and form a nucleus around which people can argue concretely about. But mostly it's because most people are rather ignorant of how systems actually interact, but one thing they do know is that there are APIs and consequently these concrete manifestations of handles that can be universally understood become the battleground upon which system integration takes place. Or, put another way, APIs are the lowest common denominator that even managers can understand, consequently they are the only focus of pretty much every large scale project.

But the problem with this focus is that an API doesn't define a system, rather it's the other way around. The way I think about it is that the APIs of a system are like the inner core of a sphere. Defining the surface of the system - i.e. what the system "looks" like - will provide enormous leverage on the internals of that system. And this leverage will simply force things into place - meaning that the reason the API exists is because it is literally the inevitable result of the forces that keep the system together.

But I digress.

So, what is the overall properties of the system we're trying to model here? What is it that we're trying to accomplish in the dynamic sense of system definition?

Luckily, the problem of management is pretty common so there's lots of examples of real world systems we can examine to see what's up. One resource I've found amazingly useful is the open source information repository found in infrastructures.org. Oddly, I've found that when I point this treasure trove out to people their eyes just kind of glaze over. Over the years I've come to judge the information content quality of a site (from my context definition of "good") by how much a manager or high level project coordinator's eyes defocus when I try to get them interested in the particular mine of information. Infrastructures.org is a sight which literally will cause any manager to start pulling out their phone and start texting random people in a desperate attempt to look busy. Even better, the "big picture" personnel will literally start walking out of your office when you start diving into the site itself, exploring the various nuggets of collective wisdom. So you know that it has to be a good site. Check it out, as it'll be well worth your time, if you're interested in this stuff.

The purpose of a management infrastructure is to automate the process of change. One of the desired effects of a successful system is that the investment that you have to make in terms of human labor goes down - i.e. you should have to dedicate far less people and have them spend far less time dealing with the system. I know this sounds rather obvious when you see it here in the ASCII, but if you simply look at the available systems out there, you find that - perversely - it seems like precisely the opposite has happened. Rather than reducing the amount of time someone has to dedicate to the system, what seems to happen is that the time required now increases polynomialy as more and more "functionality" is added to the system. Rather than simplifying the problem, the complexity grows with each and every feature as things interact in complicated and unforeseen ways. Instead of reducing the complexity of a very complex problem, the complexity increases.

Now, the absolutely stunning thing to me is that end users and administrators don't simply revolt. But I guess there's a certain macho aspect to doing system administration. An aura of "preisthood" and a realm of complexity that marks one's manhood. Or it could be that Asperger syndrome is far more common amongst IT personell and they simply don't notice the complexity because they are supermen and shrug off complexity like a duck sheds water.

But where you can't ignore these issues is on the bottom line of your expenses. People cost money. And smart people who can herd the cats that you have in your data center cost a lot of money. So any time you have to inject a human into any process, you are spending a lot of money. Consequently, anything that removes humans from the process and allows you to leverage the smart ones you have more will pay you back handsomely.

So what we are looking for, from the high level view of the system, is something that drastically reduces the complexity in dealing with the incredibly complex process of managing a living and breathing system of perhaps 10's of thousands of processes and all of their complicated interacting pieces.

Another mistake that I quite frequently find in the way projects approach system management is that they somehow believe that they have to solve the entire problem or they can solve none of the problem. So, for example, you see people literally start with bare metal provisioning and work their way up from there. Now I'm certainly not saying that we don't need something - or some things - which cover the entire suite of issues, soup to nuts. But the problem with focussing on everything is that the problem really is quite stunningly huge.

It never ceases to amaze me in my interactions with smart people who are doing cool things, in that they simply don't seem to comprehend that there is great value in not setting the initial bar to a height somewhere around the orbit of Pluto. I can't tell you how many times I've had the conversation where I'm talking with a colleague about, for example, OSGi. When I said that he should take baby steps and simply make the system we were discussing a single, large bundle he kept asking what was the point. He wanted to immediately start dissecting the large system into a highly modular system with a correct factoring of interacting services.

When asked the question "How do you eat an elephant", the correct answer is "bite by bite".

It really is lunacy to think you can wolf down an entire elephant in one bite. And if you take it bite by bite, you'll find that each bite is actually useful in and of itself. In the OSGi example with my colleague, I pointed out that simply having the system as a single bundle was actually a huge step forward. By having the system as a bundle, it was now something that came under the same provisioning mechanisms that all OSGi bundles are subject to. For example, I can start/stop/load/unload the bundle - something that we literally cannot do with the system in its current form. Further, we can now deploy the bundle to any OSGi container by referring to the location URL for that bundle. This means we now have a pull model for deployment which means we don't have to worry about provisioning the physical directory of the running process which would host that process. These are non-trivial things.

And then look at where you now are with respect to the ultimate goal of having a well factored, modularized system composed of interacting services. If you now have the system running as an OSGi bundle, the system is now operating in an OSGi environment. Even if you do nothing with the system at this point, and merely leave it as a single bundle, any new functionality you add to this system - and believe me, this system is not static and is constantly having new functionality added to it - can now take full advantage of being in an OSGi framework. This means that you can stop using Singletons for your services and simply use OSGi services as god intended. Further, you can now provide the new functionality as OSGi bundles rather than pounding the original system, pushing the functionality into it as you normally would.

Taking a bite of the elephant and chewing it before you swallow and take the next bite has advantages even if the elephant is still there.

Again, sorry for the diversion, but the point was that by trying to solve the entire system management problem, we almost always paralyze ourselves into non-action because the problem is so huge that we don't know where to start. Worse, because we're trying to do everything we find that we do nothing well at all. The resulting system then ends up increasing the complexity because all of the parts of it don't fit well together, haven't gotten the design attention that they need and are poorly implemented because of the sheer size of the work to be done.

So that's why I focussed on a limited domain in my research. As I think you'll see, simply because I've limited the domain I'm applying the solution to, I haven't actually limited the potential scope of the solutions that can be solved by the system. The system is general enough to do a whole host of things. However, but trying to solve specific problems in a limited domain, I've created an environment which can grow to solve large sectors of the problem in an organic and managed fashion.

The trick is to create something that can easily fit into existing mechanisms, procedures and tools that administrators already use to manage these complex systems. The idea is to take over some of the complexity of the existing systems and reduce that complexity without interfering with the rest of the system. So, while a good deal of the total scope of the global problem still remains, we have still made things better by reducing the overall complexity by taking a couple of bites out of the elephant.

Even better, we end up with a process which can be applied to the rest of the system (or a large sector of it) in a systematic fashion to solve the complexity remaining.

Bite by bite.

The talk is about OSGi and how the next generation of application server platforms will simply do away with the cumbersome and rather dated component models that we all know and hate in favor of the vastly superior OSGi platform. Or that's the theory at least. Only time will tell if I'm correct or just another mad hatter sniffing too much mercury outgassing from the various toys littering his office.

In addition, I also lay out the management architecture I've been experimenting with for the past year. Obviously, it uses OSGi as its base, but OSGi - by itself - isn't sufficient to provide the kind of management infrastructure you need to manage large numbers of processes. I call this management architecture - for lack of a better name - Event Driven Autonomic Management. I'll be kicking off a series of posts going into far more detail on this architecture as a means of documenting the research I've been doing.

Think of it as therapy, as talking about it on this blog - posting, so to speak, to the wind about concepts and issues that no one else seems to find terribly interesting or useful. You can tell I'm a great hit at parties, can't you?