Tensegrity

The Future is Composite

2010-01-27T03:47:33Z

Materials science has always fascinated me. How things actually work, on a mechanical level, when it really, really matters. I mean, it's one thing to simply build in the abstract, where you're working with heuristics. Big Picture™ stuff where tolerance isn't really an issue. Miss by a factor of two and everyone thinks you're a fucking Einstein. A Wilbur Wright. Edison and shit.

But even at a level we can directly experience without instrumentation (all we need is a good sense of touch or the benefit of good eyesight - with or without lenses), the actual honest to "Bob" mechanical structure of whatever reality is there really becomes the sole determination of what is real NOW, what just HAPPENED and what is INEVITABLE. The smaller and smaller we go, the less and less freedom we appear to have - the more DETERMINISTIC things become.

Or so we naively thought, of course. The reality we enjoyed before Bohm kicked Einstein's ass by bludgeoning him with sub Newtonian probability non linearity even his prodigious intellect couldn't model. But such is the mystery of reality that we need not bring up further in the course of this rant....

As I was saying, how things work at the mechanical level we intuitively understand has understandably fascinated me. And one of the more fascinating things that I've wondered about is the whole field of composites. A composite sounds like an exotic, Science Fiction™ material, but its really not. Take some wood and shred it. Pour some epoxy on the fibers and let it dry. Viola! You have a Composite material.

The great thing about composites is, from one POV, is that they have inevitably have properties that often far surpass the linear combination of the properties of the materials that form the composite.

Back in the day, my operating reality was assuming that things like a pure titanium crystal would be your penultimate material of choice for extreme material coolness. For example, they grow - yes, grow - jet engine turbine blades out of solid crystals of titanium allows. Stunning. And no doubt a feat of precision engineering seldom found in nature.

But think about it. We can build composites from materials - such as wood and some oily shit. And even if we do things in a slipshod manner, we find that we can invariably produce amazing stuff. In fact, the very "imperfections" and randomness in the composite material is where the strength arises.

And then there are alloys of different materials, where the different sized atoms form quasi periodic imperfections in an otherwise homgenous lattice - e.g. graphene doped with some random element. Suddenly very cool and amazing properties spring from such unlikely combinations and seemingly haphazard arrangments of primitive chunks of base reality. Amazing when you think about it.

Lately, I've been pondering the wonders of three dimensional printers - i.e. 3D replicators. These nifty little machines will essentially "print" a solid object replicating pretty much anything you can model in 3D space. Low end models like the one I built extrude plastic and consequently don't produce parts you can use - say - in making a car.

But that isn't a limitation that is currently shared by the state of the art in 3D printing. Actually, it's not a limitation that's really been around for quite a while. Almost immediately, it seems, 3D printers appeared which use metal powder. Each layer of the object is produced by laying down a thin layer of metal and using a laser to sinter the powder to order. What's produced is a loose lattice of metallic grains that are kind of "welded" together. What you do when that is finished is that you dunk the object in some molten metal which fills the gaps and viola you have a pretty solid chunk of metal that you custom printed in the 3D shape you desired. Tough. Industrial. And locally created from essentially raw materials, rather than some huge big ass industrial plant with union laborers and drone managers.

Turns out you can also do much the same thing and create even more exotic composites, such as ceramics and glass. The amazing thing, of course, is that you don't even need a frickin' laser to produce these ceramics. You just fire the objects in a kiln after you've glued the matrix together. Pretty cool.

In any event, what got me thinking about this was the fact that a lot of this science fictiony stuff is possible only because it's actually a good idea to build materials with diverse components. It turns out that it's actually advantageous to have structures built using the fundamental principle of diversity.

Which brings me to the point that I actually was mulling about when I decided to write up this post. People whine about diversity for diversity's sake and wonder why on earth liberals and progressives think it's a good idea - intrinsically good idea. Your average independent will scratch his head and wonder what the big fuss is. Your right of center type will simply know, instinctively, that it's a bad idea. Your libertarian nut jobs will wonder what the hell is wrong with the moron who even contemplates such ideas.

But diversity in systems can result in amazing emergent properties that are quite valuable. Properties that simply aren't obtainable in monocultures. We see this in complicated macro scale systems like biological immune systems. Inter species systems where a diversity of genes as well as large scale actors are seen as essential to the overall health and evolutionary fitness of these ecosystems.

It's not a hard jump to the conclusion that diversity in the workforce, diversity in our society, diversity in our friends is not a terribly bad idea that results in emergent properties that are quite valuable to the systems that make use of these human cultural constructs. Enumerating these properties isn't exactly easy, for much the same reason that it is hard to explain to someone that adding Chromium and Nickel to Iron produces a material which has an order of magnitude more tensile strength than the sum of the tensile strength of these three elements alone. It's these synergetic effects - emergent properties of systems - that provide tremendous value. But trying to understand how they come about, much less explaining to someone else as to why it works the way it does... well, that is actually pretty hard to do.

But valuable it is. And even if the right wing loons and even the more moderate folks who operate purely out of "common" sense can't understand the reasoning, diversity is still valuable. Intrinsically. And not being able to understand why that is, is most likely not a proof of its lack of intrinsic value.

The Colour Out of Space

2010-01-18T01:17:22Z

A while back, our local godless communist public radio station (KQED) held its regularly scheduled talk show in the 9-10 am slot, Forum. The topic was the appointment of the new "Security Czar" of the intertubes for the Obama administration, and they had a number of "computer security experts" to discuss it. Besides talking about the politics, the show revolved around the issue of the threats faced by pretty much everyone because we are now fully dependent upon the intertubes and computer technology in general for pretty much everything.

Being who I am, I just had to capitalize on the opportunity to talk to these experts on this public forum and ask them a couple of questions I had on the subject at hand. Sadly, due to my limitations as a questioner I didn't quite make my point clear in the questions I was allowed to ask.

(Thanks to the wonders of the intertubes, you can find the MP3 of the episode here and I come in at the 33 minute mark)

However, I don't think it was simply my lack of skills that was the source of the misunderstanding. The fundamental issue that the experts were discussing was the security of our networks and the security of our computerized systems in general. The questions over how we protect ourselves in this digital age and why in heaven's name we simple aren't doing so in the 21st century.
]]>
And this need for surveillance and interception as well as the cracking of encrypted contents of suspected terrorists' and criminals' data causes a rather big problem for us, as citizens. Because these needs, this puts the government in direct conflict with the security needs of its citizens who live in the digital wild, wild west. Granted this thought of mine is hardly original, but I would have liked to have these computer security experts discuss this elephant that is taking up all the room on the couch as they discuss the security issues we currently face.

Basically, the government would like to have back doors into everything that they can open at will to do what they need. Now for the moment, leave aside the obvious and quite serious issues with the potential for abuse of these back doors. The issue I was trying to get these experts to discuss was the fact that because of these needs and desires, the government was conflicted in its efforts to help keep our networks and computer systems secure. And this conflict isn't just some theoretical ethical and moral issue... rather it was a fundamental impediment to actually accomplishing the goals of these security experts.

After all, if we truly had secure networks, communications and computer systems, then the government would find that itself in a hell of a pickle as they found themselves locked out and thwarted by these security systems - there would be no backdoors. Consequently, the best interests of the government - itself - was working against the very obvious best interests of its citizens. One hell of a pickle, even for those who consider themselves on the side of "good." And consequently, an issue that security experts, themselves, have a hard time dealing with effectively - or honestly, in my humble opinion. After all, one of the guests on the program had worked on the Clipper chip, himself.

The reason why I was reminded of this exchange though, was the recent news of the breach of Google by China. In The Great Google Coverup, Douglas Rushkoff ends with a rather entertaining thought about the incident

At least a conspiracy theory, in which Google willingly gave the Chinese authority over its clients' communications, offers the comfort of there being some human agency in all this. Just as we prefer to find out that a single pilot was drunk than that there's a problem with every plane in the sky, it is easier to contend with the notion that Google's young executives made a stupid decision by engaging with dictators than to consider the alternative: that the cloud being entrusted with an increasing amount of our banking, business, and everything else, is the for the taking.

It's incidents like this that always leave me wondering what it must be to live in a world where people actually believe and trust the companies that do this "cloud" shit. No, I'm not ranting about the "cloud" as a concept... rather I'm ranting about the trust that people place in "the cloud."

Part of this perception is undoubtedly due to the fact that I'm a pre-Millennial. Technically a member of the Baby Boomer generation, but only because I slipped under the net by a year. I grew up in the time before the intertubes when physical reality provided what we now know as the illusion of privacy. And given the way the Millennials seem to have adjusted to the new realities of the 21st century, perhaps all I'm talking about here is simply a quaint notion about a world long gone and which never really existed in any true sense of how we understand things.

But regardless of whether we have any meaningful sense of privacy any more, we will obviously always have issues with transactional integrity: in its basest form, I'm simply mean accounting - financial accounting, obviously. Our economy is built on digital transactions and without security protecting those transactions, we simply don't have an economy. The "cloud" and the intertubes in general, are becoming a bigger and bigger piece of that economy - becoming the foundations of that economy. And if those foundations aren't secure, then we're basically fucked. Three ways to Sunday, to coin a phrase.

And it doesn't take a security expert to see this. After all, it doesn't take a civil engineer to see that a bridge is ready to tumble over. Basically security technologies and infrastructure are the manifestation of trust. And without trust, there is no economy - certainly no modern economy. Everything crumbles to dust as soon as everyone realizes that not only does the Emperor have no clothes but they, themselves - the observers and participants - are standing around naked as well. Naked. Exposed. Supremely vulnerable.

A while back, Google's CEO Eric Schmidt ruffled more than a few feathers when he said

"If you have something that you don't want anyone to know, maybe you shouldn't be doing it in the first place."

The cynic in me would like to believe in the illusion that Google's action over China is all about trying to regain their image as "the good guys" - the one's who do no evil - and that there isn't something far more fucked up going on. However, like Douglas Rushkoff, I think we're just seeing the reactions on the surface of the water to events far below it. Ugly realities that, like the unnamed surveyor in the H.P. Lovecraft story, we've yet to realize the true horror of.

In my more cynical moods, I console myself with the fact that at least Google will be one of the first backed against the wall when everything goes pear shaped. Petty, I know. But sometimes pettiness is all we peons have.

Open Sourcing of 3rd Space

2009-11-26T00:06:36Z

Rather than trying to deal with any licensing issues with those who wanted to use or play with the work I've done under the umbrella of the 3rd Space project, I've decided to open source the system. The source is hosted on my SVN repository, which is:

http://svn.tensegrity.hellblazer.com/3Space

I have bugzilla and Trac enabled, but I haven't done much of the necessary work to make those usable yet.

Right now, the system isn't all that impressive from a full featured simulation environment perspective. I do have a lot of the basics covered and have a good idea of where I'm headed, but there still an enormous amount of work to do, obviously.

Still, the system is quite different from any of the other virtual worlds or simulation environments in that the underlying framework is an innovative system which provides an event driven simulation framework using Java as the language you write the simulations in. Naturally, you can also use any of the scripting languages supported by the JVM (not to mention, languages which use the JVM, such as Scala, etc), so it's actually pretty interesting from that perspective as well.

I've also fully implemented the 2 dimensional Voronoi based Area of Interest management which I think is actually quite innovative. Currently, the state of the art in this arena is message based pub/sub systems which simply don't scale at all, or some Frankenstein's monster based on what amounts to a QuadTree structure. Not optimal. Naturally, the AOI management is all event driven, being implemented as part of the aforementioned simulation framework. I have the 3 dimensional Voronoi Sphere of Interest management in the works. Once that is in place, things will really start rocking.

You'll find some interesting things in the source tree, such as a pretty darn sophisticated Data Flow Analysis framework that I found. I don't actually use it in the simulation framework's transformation logic, but it's something that will likely have to be pulled out to solve some of the more nastier analysis issues that will inevitably have to be solved in that arena.

You'll also find a Composite Object framework that I put together, which I call Janus. The composite patter is an important pattern in the simulation / virtual worlds area and I couldn't find anything out there remotely useful. Yes, I've looked at the available frameworks, but they were either too simple to do anything useful, or literally tried to boil the ocean in the attempt to solve all problems with their framework (I'm looking at you, Qi4J). It's small and extremely tight, leveraging ASM for the bytecode transformations to implement the composite, rather than creating a maze of twisty proxies, all alike. Again, I haven't used it in anger yet, so who knows if it's really useful. But it seems promising.

In any event, be aware that the code is licensed under the Affero GPL V3. I am fully aware that this is the "yes, I'm a complete asshole" license. Oh well. Please note that I'm a rapacious capitalist at heart and believe in making money off of stuff. But I also believe in the open source model. I figure if you want to contribute back your mods, then that's a decent enough payment. The Affero GPL does a pretty good job of modeling the balance I want to strike with the release of the source.

As always, this is all "as is" and without warranty. I don't exist to fix any bugs you might find and I may not even respond to your emails. But if you're interested in working on some stuff that I think is pretty darn interesting, then this might be something you find useful. Let me know.

I'll definitely start putting in the work to fill out the Trac WIKI with some more useful information and start detailing what the sub projects are, documenting the various frameworks and start laying out the road map I have in my mind.

In any event, I hope you find the code useful and interesting.

Third Space postings on Tensegrity

When you shake your ass, they notice fast - and some mistakes were built to last

2009-11-18T00:16:06Z

One of the more interesting books I've read in the past couple of years is No One Makes You Shop At Wal-Mart. One of the take-aways from the book is the unmistakable conclusion that the idea of "revealed preferences" is simply wrong. You see this kind of short cut thinking all the time in that people basically believe that the choices that someone has made reveals their preferences. So, if you see that USA Today is the best selling newspaper, the theory of revealed preferences tells us that what this means is that news consumers really, really do prefer USA Today.

What you learn from the work of Tom Slee is that this thinking is simply bullshit. As anyone with a pulse can tell you, life is filled situations that amount to versions of the Prisoner's Dilemma where in your best strategy is often counter to your best interests. The long and short of the argument is that, inevitably, you're choices suck and interpretations based on the results of your decisions hardly reflect the trade offs that are involved in the process of making that decision.

A good book and not all that long. Definitely recommended, if you're interested in that sort of thing.

One of the reasons that Slee's book interests me is its application to my chosen domain of software. One way to look at the Slee's thesis is through the lens of the Innovator's Dilemma. In the software industry, one sees this reflected in the idea that what a customer wants is more of what they've been buying in the past, only "better" - e.g. cheaper, faster, whatever... A decent way to understand this is the transition that took place in the transportation when the automobile came into fore. If you were had asked the customers of the horse drawn transportation (i.e. the "legacy" industry) what they wanted, the customers would have stated "faster horses".

And so it goes in this industry we love so much. What do customers want? More Of The Same! Faster Horses™!

Given that I work in the legacy industry, I'm certainly understand that there's a lot of stuff out there that is way over hyped and fizzles faster than it became fashionable. But this notion of basing your strategy on the revealed preferences of your customers, rather than understanding what their actual problems are, is something that definitely keeps me up at night. The idea that someone's current investment reveals their preferences for the "way things are done" seems to be one that's based on manifestly shaky ground.

And unlike - say - the auto industry, where the dinosaurs of the automotive industry were caught flat footed after years of ignoring all the warning signs, it's fairly clear that changes in the more ephemeral industries such as ours - industries where changes can happen far more rapidly than the buying cycle of durable goods - can happen in a moment's notice and fundamental changes in technology - such as the rise of the internet, for example - can completely catch dominate players flat footed and they find themselves in the "me, too" category of innovation. Playing catch up in the also ran category.

Or selling Horse Drawn Carriages in the age of Henry Ford.

Deep thought

2009-11-03T16:42:04Z

It's the definition of passive-aggression and really quite unseemly, to set out to provoke people, and then when they react passionately and defensively, to criticise them for not holding to your standards of a calm and rational debate.

Just sayin'

All we need to do is take these lies and make them true (somehow)

2009-10-19T23:18:24Z

Having worked in OSGi for quite a while, the most frequently asked question I get on the technology is "what's the value proposition." Being a rapacious capitalist at heart, I think this is an eminently fair query from anyone looking at this OSGi stuff and scratching their head and wondering why the heck they would even want to consider using this technology in their systems. There's a non-zero and usually non-trivial cost associated with changing technology - a cost which usually grows in direct proportion to how "low" the technology is on whatever technology stack you are deploying. OSGi is pretty low on that technology stack, so it has the potential to be very disruptive and hence very costly to an organization which adopts the technology. Surely the benefit of making the switch should be at least proportional to the costs and a prudent business would like to understand what they're going to get for their trouble, and more importantly, their hard earned money.

To answer this question, what I first ask is that people think about their current environment. Assuming you're not a startup - a domain which I'm not considering in this post - then you are undoubtedly dealing with a mature system which is now or quickly will resemble Frankenstein's monster more than anything else. If your system is successful to any degree - and if it isn't, then we aren't really having this conversation - what you find is that your system is a victim of its own success. Grizzled veterans remember the "good old days" when builds would take less than an hour and everyone could sit in a room and share a common understanding of what the hell was going on in this money maker of yours.

Sadly, one day you look up and find that no one knows what the hell is going on anymore. Build times - perhaps one of the more visceral measurements of complexity we have - have jumped dramatically. These days, people fire off builds and then go on lunch breaks. Worse, your projections are that in just a short time in the future, the nightly "integration" builds you kick off will still be running well after your developers have shown up for work. It's at this point that one panics and decides that dramatic action is required. Something MUST be done. Well, as long as that something doesn't require any change to what you're currently doing - i.e. one starts searching for a silver bullet which will slay this beast of chaos that you've collectively created and return your life back to the way things used to be. Before "IT" happened.

]]>
Naturally, I think that OSGi has something to do with the solution to this problem. It's not the only thing that OSGi offers, but it's one of the aspects of the system that provides the most understandable benefit that is easy to explain using the problems that we all can relate to.

On the right, I have a simple graph with two lines. The Y axis represents "complexity" and the X axis is time. The astute observer will notice that one line is linear and the other is non linear. My contention is that the linear line represents the behavior of systems when developed using OSGi, strictly from a modularity perspective. The non-linear line represents systems when developed using what I'll label as "traditional" technology.

One of the interesting things to note is that the initial "cost" in complexity is actually higher in the beginning of the system's lifetime for the OSGi based system. The reason why is that modularity does have a certain fixed cost which cannot be simply erased by waving hands. Understanding the basics of any modular system requires some up front investment in time, training and build infrastructure. Some thought needs to be put into the way things are done. Developers need to be familiar with this technology and understand the processes that are in place to maintain the system. This is what I call the "cognitive burden" of OSGi.

In contrast, you'll note that the so called "cognitive burden" of traditional, non-OSGi technology is rather low, and continues to be lower than what is required of the developer who works with OSGi for quite some time of the system's lifespan. What this means, in effect, is that it's pretty easy to get started in the traditional mechanisms, but it takes a bit of work when you want to use OSGi.

However, what ultimately happens to any successful system is that the complexity starts to go through a non-linear transition. As I mentioned above, build times start to sky rocket. Tests take forever, dogs and cats start living together. Total anarchy appears on the horizon and threatens to drink your milkshake - in more ways than one.

Basically, what happens is easily explained by the geometric nature of connections. For small numbers, things look pretty good. But as the number of subsystems and people become involved, these connections grow geometrically and that starts to suck pretty hard when you're well into the knee of the curve. It's at this point that you start looking around for something - anything - to get a handle on the situation and return things back to the friendly, cuddly system you used to know.

The value proposition for OSGi is that the time spent beyond the hard knee of your curve is where you get your major benefit as a business. What invariably happens when your system reaches this knee and it starts punching you in the gut is that organizations start implementing more oppressive bureaucracies to manage the geometric explosion. Committees, meetings, high level pow wows - human systems start being brought to bear on the exploding complexity of a system that used to be so well behaved.

So, the value proposition of OSGi is that it provides the mechanisms which can control this complexity. The simplest being the module meta system which defines the dependencies between cooperating components. And whether you agree that OSGi provides the answer, or some other module system provides the answer instead of OSGi, I would claim that this answer has to be at least as good as OSGi.

The ability of OSGi to handle complex systems as a set of interdependent modules is kind of like toilet paper: Sooner or later, you're going to want to use it. Naturally, as software developers, we simply cannot reuse any existing system such as OSGi and consequently there will be a lot of stupendously successful efforts which essentially recapitulate much of what OSGi already provides today. Rather than seeing this as a fundamental problem with OSGi, I see this as merely one of the best forms of flattery - my mom always told me that Imitation Is The Best Form Of Flattery, after all...

But whether you think there is something - perhaps proprietary, perhaps a secret plan produced by your favorite vendor - that will supplant OSGi, the fact remains that what it represents, both at the simple module level and the far more useful service level (a topic of exploration of another post), what will eventually become a common, accepted fact of software development. Something that all the XP and Agile types will have claimed to have known for centuries. Why? Because it solves the hard problem of geometrically escalating complexity. And solves it well.

Yea, it's going to be something more that "gets in your way" at the beginning. But only people who develop demos actually program in isolation - and then throw the system away after the keynote - or those who develop exclusively in "start up mode" will be primarily concerned with how fast they can develop "teh quick and dirty". In the real world, systems accrete complexity over time. In the real world, there are multiple existing systems that a "new" piece of software has to integrate with to work. In the real world, new software has massive dependencies on pre-existing systems - systems that, when you inquire about them, you find that anyone who knows about them has apparently "died".

Unfortunately, our industry pretty much seems to ignores stuff like this. No one likes having to do extra work and sadly the people who direct most of the infrastructure development simply don't build a lot of applications that need to be maintained by a moderately skilled workforce using the infrastructure technology they're developing.

Demo systems are invariably thrown away rather than carefully nurtured like the cash cows that are the reason why you're developing for whatever god forsaken system you're developing/managing as part of your daily job. Sadly, much of the razzle-dazzle is focused on satisfying what amounts to a sugar rush. A quick fix that satisfies and what ever consequences of this exercise are simply forgotten after the targeted purpose of the demo is completed. The result is that all the focus is on the development of what amounts to green field systems which is an experience that has little to do with the day to day life of mining a large, successful application that has scores, if not hundreds, of developers toiling away tirelessly at its impenetrable hide. And I have yet to even mention the vast fields of integration testers and system management entities (they aren't human, you know) required to actually turn those bits into dollars.

So what is the value proposition of OSGi? In a nutshell, it's a technology that pays off in the medium to long term, for systems that are successful and have more than a handful of developers. That's a hard sell for the attention deficit set, but something that is all too familiar to those who are tearing their hair out, wondering why their gentle purring kitten has become a mutant monster three stories tall, threatening to destroy their cash cow and expected retirement plan.

We, as vendors, have the job of reducing this "cost" of OSGi. Part of this is through tooling such as IDE integration and such. Part of this is through education and evangelicalism - if you have been exposed to the basics of OSGi, it's not that scary when you actually try to use it, after all. Part of it is the development of frameworks which lower the "cost of entry" into OSGi, such as the Blueprint specification. However, for the foreseeable future, there will always be a non-zero cost associated with OSGi - just as there is with more traditional technologies. With time, this cost will become institutionalized and part of the expected curriculum and best practice within the industry.

But for now, I have to write long winded blog posts on the issue, hoping that some shred of lucidity comes across in the ramblings...

Snausages

2009-05-18T15:12:15Z

So now I'm the guy wielding the process stick. Stranger is the wonderment by many that I am "a process guy". I guess it just goes to show one that you're never quite able to see yourself as others see you. Certainly, I'm a "process guy". Process is how you deal with the inevitable problems with groups of people trying to do things. It's called "protocol". Considering that all my professional life has been focused around the problem of consensus in distributed systems and the engineering of distributed protocols, I do find it slightly bewildering that people are surprised when I bring up process and my belief in good process.

One of the more painful lessons that I learned in business is that good contracts keep good friends as good friends. I had to learn the hard way how stressful it is on even good, solid relationships between very smart and honest people when misunderstandings occur about things that mean something to them - i.e. MONEY, or more generally BUSINESS. When things are just left up to good intentions and the belief that "we can all just work this out like normal people", I invariably find the trail of debris in broken friendships and business partnerships.

]]> Now, in this particular case, I've been accused of "not reading" the plain and obvious and that I simply should have done my homework. The point in question is the interpretation of the following from the process documents:

5.6.2 Actions
The SDE (Specification Document Editor) shall create the appropriate documents based on the content of one or more RFCs. Under ideal circumstances the formulation of the Specification would be a mechanical process but it is expected that the SDE will uncover inconsistencies or other issues in the RFC(s) which require clarification by the appropriate EG. In this case the SDE shall liaise with the appropriate EGs directly to resolve the issue. The SDE shall have at least one and preferably several review cycles with the appropriate EGs to ensure accuracy prior to completion of the Specification.

Certainly, this is something that I completely understood. It is unfathomable to me that the SDE wouldn't uncover inconsistencies and have to come back for "clarification by the appropriate EG". However, this is not - imho - what actually happened. What happened was that the SDE came back with the recommendation that we fundamentally change the specification. This is not "clarification", in the way I understand the term "clarification". Clearly (to coin a phrase), clarification means to make more clear. It does not mean to "change the specification in fundamental ways" because there are time issues which force us to make difficult decisions.

A specification, particularly the final RFC, represents a compromise amongst many competing business interests. If you want to change that compromise, you're invariably going to work against at least one of those interests and thus destroy the compromise. That is not the process of clarification. Now, perhaps I have completely misunderstood the use of the word "clarification" in section 5.6.2, but then that just shows that the process itself needs - ahem - clarification.

Further, one of the deep issues with the process as practiced, has been the underlying assumption that the only vote we would get would be on the entire specification, not just the particular specification under question. This was not just my confusion, but seemed to be a shared understanding by many - on both sides of the issue. Certainly, the way it was repeatedly represented to me in my queries was that the only chance we would have for another vote would be on the full specification which contained the array of individual specifications. After much investigation did it turn out that this was not a correct interpretation, but the odd thing was that it was somehow considered to be the "nuclear option" - that is, actually taking a vote on the SDE's rewriting of the specification was considered to be in poor form and something that would cause deep divisions.

Now, of course, I could be confused about this as well. Certainly, at this point, I'm wondering what all the fuss would be about if, in fact, that a vote following the SDE rewrite of an individual specification was actually spelled out and understood as part of the process - certainly that wouldn't seem to be a "nuclear option", nor would it seem that standard operating procedure should be a source of divisions within the group. So, color me confused yet again.

In any event, I think it is still necessary to understand exactly what the SDE's limits on "clarification" are. If "clarification" means making fundamental changes to the specification, then I don't think it should be all that controversial to have a good discussion and another vote on the "clarification". It's a fundamental change to the compromise and we should be required to see if the compromise still holds.

But then, I'm just a process guy. What else would I think?

Slouching Towards Bethlehem

2009-05-13T16:20:19Z

I am of the opinion that no one actually sets out to do stupid things. Rather, stupid things happen almost invariably for the best of intentions. Thus the phrase "The path to hell is paved with the best of intentions".

And so it goes with standards. In the past week, I've been privileged to witness two crystal clear examples of first class paving of the path straight to the gates of hell. The root of both of these examples is the simple fact that the specification organization doesn't have a process in place for dealing with the changing of a specification after it has been accepted as final. Basically, in this organization, the "final" version of the spec is presented to the members, and the members vote on whether to accept it. Sounds simple, right?

Well, the devil is always in the details and due to the way these things are scheduled, the reference implementations and conformance tests for these specifications aren't finished at the time that the members vote on the "final" version of the specification. Consequently, if anything shows up in either the creation of the RI, or in the creation of the conformance test which is to verify that the behavior in accordance with the specification, there's a serious problem that needs to be resolved.

In the two cases I've been privileged to see this week, the first was an issue with time. As with all things, resources are limited, and certain resources are scarcer than the sympathy in a banker's cold, dead heart. Consequently, when the time pressure to produce something gets unbearable as the deadline approaches, the reality will set in and like survivors on a sinking lifeboat, everyone starts looking for stuff to throw overboard. And because this process is done under pressure, there's not an awful lot of thought and strategy put into the choice of what is being chucked overboard.
]]>
So, when I asked for an explanation for why the feature was considered to be too hard, the response I got back was a lot of hand waving. I mean, literally hand waving. Now, I certainly understand time constraints, and if the issue is just one of time constraints and this was considered to be one of the things that we decide to chuck to meet these time constraints, then that's a perfectly valid reason. But for some reason, humans aren't willing to be that simple and straight forward. We have to rationalize our actions so that they make sense. And that process of rationalization can get pretty darn ugly.

Which is, of course, why things like "process" exists. Process allows us to formalize the process so that there is not need to provide a rationalization for a decision which was made for other reasons. It's the mechanism which boils away the dross so that only the underlying issues are presented so that the system can make a good decision without being screwed over by our collective human nature.

Instead, I was presented with the simply hilarious process of persons trying to convince me that things as they were constructed were "fundamentally" broken. Now, back in the day, I used to do quite a bit of formalized debate, so I know a bit about argument structure. So, when someone makes claim X, there's this little thing we used to call "Burden of Proof". Basically, when you make a claim, you have to back it up. The failure to understand the burden of proof, and where this burden lay is the root of a lot of very tragic mistakes.

For example, back in the early years of this new century, the claim was made that Iraq had weapons of mass destruction. That was a claim. The burden of proof should have lain with the people making the claim to prove their case. Instead, what we witnessed was the farcical situation where the burden of proof was shifted to those who questioned the assertion that Iraq had weapons of mass destruction. The result was predictable and quite tragic. We'll be paying the wages of this insanity for generations to come.

And while not nearly even in the same order of magnitude of import as where the burden of proof lay with those claiming someone has weapons of mass destruction, the same basic stupidity and lack of logical reasoning was on brilliant display when I was flatly told that I had not "proved" my case that the feature being claimed as fundamentally flawed was not, in fact, fundamentally flawed. What happened instead was merely that some issues were presented and the mere claim that these were fundamental was presented. That's it. I asked why these were fundamental flaws, and further asked for examples as to what couldn't be done because of these flaws which were fundamental to the issue.

Silence.

And thus it became quite clear that there wasn't any argument. There was simply the need to rationalize a decision that was already made.

The second issue which I've been privileged to witness this week is somewhat more troubling. Granted, having no argument and relying on faulty logic is troubling, but the actual underlying reasoning had some merit, so even though it was done for the wrong reasons, the result wasn't totally tragic. Not so with this issue.

In this case, the problem is that there is a fundamental problem with the specification. There is no "there" there. And consequently, when you try to make a conformance test for the behavior, there is nothing you can check.

This is, naturally, a troubling situation. And rather than step back and question the fundamental issue at stake: i.e. maybe you aren't specifying anything at all in the first place - the reaction was to make an optional service mandatory so there would be something to test.

Madness.

In the first place, the optional service is purely an internal API. That is, it has absolutely no user visible parts. Consequently, the API is there for the implementors of the specification to use. Now, there's a zillion different implementations of this spec which don't make use of this API in their internal implementation. I have a couple already working here in House Harkonnen labs, so I have proof positive. Further, no one actually disputes the fact that one doesn't have to use this API to make an implementation of the spec - so there isn't even an argument about whether the API is necessary - the fact that it isn't isn't contested.

And so, because there has to be something to test, we will be making a change to the specification - after we have voted on the final specification itself.

Madness.

And this is how things go badly wrong. It isn't because someone sets out to do something stupid. Rather, stupid things happen for the best intentions. Everyone is working overtime to make that eight lane hiway to hell as smooth as the best German autobahn so we experience no traffic and the smoothest ride ever.

Right up to the gates of hell.

Update: a response to their response to my response

And Then Hemingway Punched Me In The Mouth

2009-04-26T00:43:10Z

From my own point of view, I continually say lines from movies, expecting people to understand their applicability to the current context, but given that I often find myself looking at a lot of confused, blank stares when I say these things without explanation. Consequently, I show the image on the right as context. It's a scene from the movie Groundhog Day. In this scene, Bill Murray's character is going to kill himself for the ~~umpteenth zillion~~ first time by driving a truck off the cliff of a quarry. As you can see, the groundhog is actually driving the car as the police are chasing them at high speed. The line in question - i.e the line I would say which would then draw blank stares - is what Bill Murray's character is saying to the groundhog at this precise moment in the scene:

"Don't drive angry! You should never drive when you're angry"

And that's pretty much what I would say about tweeting when you're angry: don't do it.
]]>
In any event, the problem with the 21st century's super efficient social networking software is that that they really are super efficient. Within milliseconds of tweeting angry, not only did I receive tons of replies, but I also saw multiple retweets. And so not only had I spread my perhaps ill advised tweet (really, if you haven't seen it, it was not horrible, just perhaps ill advised) amongst my own connections, I found it amplified on some super connected networks that some of those in my own networks maintain like large herds of psychic cattle. Amplified beyond belief.

And so I quickly found myself deluged with dozens and dozens of emails inquiring, offering and wondering. Amazing.

Now, the good news about all this is that I found out that I have a lot of very good friends out there who care a lot about what happens to me. I found out I have even more good acquaintances and business associates who also care about what happens to me. And people who I haven't really even met before - friends of friends and people I just know through email or remote collaboration on various projects - who also care about what happens to me. And people who have commercial interest in what happens to me.

So, that's all great. Overwhelming, really. And that's part of the problem. See, back in the last century - heck, back at the beginning of this century - it would have taken at least a week for such information to make the rounds. And before such things as super efficient social networking software, probably very few people would have heard me say something like that anyway.

Consequently, the amplification and feedback is also a bit hard to deal with. When you are in a mood and doing something that may or may not turn out to be rash, you probably don't want to have a super efficient social networking infrastructure laying around like a nuclear bomb with a hair trigger. It's just f'ing dangerous.

Now, granted I'm no Julia Allison, Dave Winer, or "Bob" forbid Robert Scoble. But all of this has been a very interesting lesson to me on the two edged sword that this century's software technology has casually made available. All I can say is thank "Bob" there aren't incriminating pictures involved.

Oh, and the title to this post? It's another one of those lines that I say every once and a while which garner blank stares from those around me. It's a line from my favorite Woody Allen bit, The Lost Generation:

I mentioned before that I was in Europe. It's not the first time that I was in Europe, I was in Europe many years ago with Ernest Hemingway. Hemingway had just written his first novel, and Gertrude Stein and I read it, and we said that is was a good novel, but not a great one, and that it needed some work, but it could be a fine book. And we laughed over it. Hemingway punched me in the mouth.
That winter Picasso lived on the Rue d'Barque, and he had just painted a picture of a naked dental hygenist in the middle of the Gobi Desert. Gertrude Stein said it was a good picture, but not a great one, and I said it could be a fine picture. We laughed over it and Hemingway punched me in the mouth.
Francis Scott and Zelda Fitzgerald came home from their wild new years eve party. It was April. Scott had just written Great Expectations, and Gertrude Stein and I read it, and we said it was a good book, but there was no need to have written it, 'cause Charles Dickens had already written it. We laughed over it, and Hemingway punched me in the mouth.
That winter we went to Spain to see Manolete fight, and he was... looked to be eighteen, and Gertrude Stein said no, he was nineteen, but that he only looked eighteen, and I said sometimes a boy of eighteen will look nineteen, whereas other times a nineteen year old can easily look eighteen. That's the way it is with a true Spaniard. We laughed over that and Gertrude Stein punched me in the mouth.

OSGi RFP 122 - The OSGi Bundle Repository

2009-03-30T16:02:54Z

This week at EclipseCon, I discovered that I had inadvertently opened a can of worms and found the entire Landsraad of the Open Source community arrayed against me. My crime? Apparently it's simply that I've had the audacity to pick up an OSGi specification that has been in existence - and in the public domain - since 2005 (i.e. OSGi RFC 112, the OSGi Bundle Repository) and attempt to work out the issues with that specification so that we can finally formally release it as part of the OSGi specification.

Much of the suffering I was dealt was due to serious misunderstandings of those involved with the process that is currently being played out. Some of this misunderstanding is due to ignorance of the OSGi process - and note that I use the term "ignorance" not as a pejorative, rather simply as a statement of fact. Those who aren't involved in the OSGi process, nor familiar with the way that specifications in general are produced can sometimes be left bewildered by the array of TLAs and the process by which consensus is reached. Certainly in the Open Source world, sometimes things are done quite differently than they are done in standards bodies (note: I'm not saying that this is universal, only making the point that standards bodies are their own beasts and Open Source communities rarely conform to such formalized systems).

So let me make a couple of points, and talk about how I'm going to carry out the process of specification of the OSGi Bundle Repository to ensure that the world outside of OSGi can participate in this process.
]]> First, I would appreciate actual technical conversation in a forum, rather than back door politicking and pressure tactics. One of the things that I've found enormous value in the OSGi is the fact that all the specifications and discussions I've been a part of during my tenure in the organization have all been above board and professional. What I mean is that we air our disagreements out in the open and discuss them on the technical merits. I have had serious disagreements with others on any number of matters in the OSGi and I have yet to have people not discuss the issues like professionals.

Second, I would like to explain the process of this specification as it has been quite unusual. The OSGi Bundle Repository (OBR) first started out its life as the OSCAR bundle repository. OSCAR was the predecessor to the Apache Felix OSGi framework, headed by Richard Hall. As far as I can tell, the OSCAR Bundle Repository was first presented to the world in 2004, so it's been around for quite a while. In 2005, an OSGi RFC was produced - RFC 112, and it was renamed the OSGi Bundle Repository. Now, for those not familiar with the way specifications are produced in OSGi, it is unusual that the OBR became a specification without ever going through the requirements process (in OSGi parlance, the production of an RFP). RFC 112 sat on the shelf gathering a bit of dust, although it was still in wide use, until late 2008 when I decided to pick up the ball and try to finish the outstanding work still required to make the OBR part of the released OSGi specification.

Prior to the OSGi F2F meeting in Feburary of 2009, I worked with Richard Hall and Peter Kriens to work out the remaining issues tha they had left unanswered in their work on the specification. I also enlisted the help of the folks at Paremus, who had also been making use of the OBR in their excellent work on their Sigil project. After rendevousing a couple times, I had filed a number of bugs against the spec regarding what I thought were the remaining issues that needed to be pinned down, so that we could discuss them at the next F2F.

Little did I know what I was walking into, being ignorant of the various political activity that festers around such things. At the February OSGi F2F, it became quite clear that we needed to go back to the requirements phase of the OSGi process so that we could get clear consensus as to what we were trying ot accomplish before proceeding with the specification phase.

Now, at this point I'd like to stress that this is no trivial thing. What it means is that I have effectively given up the RFC "blessing" that RFC 112 had - regardless of how unusual it had acquired that blessing. What this means is that by going back to the RFP stage, I have thrown this process back on the mercy of the OSGi process for accepting a requirements document. For those of you who are not members of the OSGi, what this means is that I have to now produce an RFP - i.e. RFP 122 - and get consensus within the OSGi organization on its contents. After this is done, the RFP is then brought up for a vote. And what this means is that the majority of voting members has to approve the RFP before it can then proceed back to the RFC stage.

Which brings me to my third point, that I have not been trying to ramrod the OBR through the process as some have misperceived. Rather, I have done precisely the opposite. If I was actually trying to ram this through the process, I would not have voluntarily reverted the process back to the RFP stage. Instead, I would have taken advantage of the RFC status of the OBR and simply ignored all the issues that were brought up at the F2F. Instead, I decided that the concerns that people had raised needed to be addressed in the appropriate forum, and process - i.e. an RFP was needed.

Having said that, I am not going to take the slow route with this process. I'm going to press the issue and demand that those who have an interest in defining the OBR actively participate. As I pointed out previously, the OBR has been around since 2004 and is in active use, both in the open source community as well as in private industry. Further, the actual specification, RFC 112, has been in the public domain since 2005 as well - something highly unusual for an OSGi specification, which is normally closed intellectual property of the OSGi until it becomes a released specification.

Consequently, I don't really take well to the idea that people will "get around" to dealing with the specification. It's been around forever, and it's not like this appeared out of the blue. Several implementations exist and are in active use. If you're interested in the specification, then you should make it priority to deal with, just as I have made it a priority to deal with. I, like many others, have a lot of stuff on my plate and a job I do every day. This specification is important to me and I intend to make sure that it becomes a released standard and will work hard to ensure that it will become one.

Finally, my fourth point is that the process of technical discussion about a standard is inherently going to entail frank discussion about technological issues. What this means is that everyone is going to tell you that your baby is not only ugly, but should have never been born in the first place. Further, now that your baby is born, you really should hide it under a blanket so it doesn't scare the rest of us with its hideous deformities that you find charming and cute. This is the nature of technology, in that we all get personally attached and invested in whatever it is we work on. Certainly, I am no different. My children are just as ugly, misshapen and horribly inadequate as everyone else's.

Having said that, I want to make it clear that I will brook no arguments about the superiority of "open source" work over other work. I would only point out that OBR has predated a lot of the technology that is being presented as "TEH ONE" and has just as much a claim to the mantle of open source due to the seminal work of Richard Hall and Peter Kriens as anything else. The fact that this is being done in the OSGi standards body is because of the desire to make this an OSGi standard. If you don't care about standards, then the process I'm presenting here won't matter to you. Like all standards in OSGi which are not concerned with the Core Framework, it will be an optional standard - just like the Initial Provisioning Spec, the Configuration Administration Service, etc. No one is going to force you to make any use of the OBR if you don't like it, think it smells or is tainted by the very involvement of House Harkonnen in its creation.

In conclusion, I would like to point out that what I'm doing is rather unusual for the OSGi in that we're reaching out to the open source community and those interested in this standard and ensuring a high quality product. This does not mean that the open source community gets to vote on the OSGi standard - that is, after all - something that the OSGi controls. However, this does mean you get to bitch to me directly, rather than having to sit in a corner, sulking that your concerns were not addressed, or screaming at the cage door flinging poo at me through the bars.

As such, I'm going host a copy of the current state of the RFP on this site: OSGi RFP 122. As the specification progresses, I'll be updating the link with the current state. Right now, this link refers to the state of the requirements after the last internal conference call we had. I encourage you to read it and send me any comments you may have. My email is in the document.

I will also continue to blog about the issue as it develops. I currently need to blog about the latest discussions I had at EclipseCon on the requirements for the OBR and my thoughts on the matters at hand. However, this post is already way, way too long and I need to get to that overflowing in box of things I have left to do. Hopefully we can, together, produce an excellent spec that the entire community can be proud of.

Thoughts on IBM Acquiring Sun

2009-03-21T22:16:40Z

My first thought was that it seems like a lot of money to ensure that OSGi is used as the module system for Java.
]]>

hahahahahahahahahahahahahahahahahahahahahahahahahahahahhahahahahahahahahaha
hahahahahahahahahahahahahahahahahahahahahahahahahahahhahahahahahahahahahaha
hahahahahahahahahahahahahahahahahahahahahahahahahahhahahahahahahahahahahaha
hahahahahahahahahahahahahahahahahahahahahahahahahhahahahahahahahahahahahaha
hahahahahahahahahahahahahahahahahahahahahahahahhahahahahahahahahahahahahaha
hahahahahahahahahahahahahahahahahahahahahahahhahahahahahahahahahahahahahaha
hahahahahahahahahahahahahahahahahahahahahahhahahahahahahahahahahahahahahaha
hahahahahahahahahahahahahahahahahahahahahhahahahahahahahahahahahahahahahaha
hahahahahahahahahahahahahahahahahahahahhahahahahahahahahahahahahahahahahaha
hahahahahahahahahahahahahahahahahahahhahahahahahahahahahahahahahahahahahaha
hahahahahahahahahahahahahahahahahahhahahahahahahahahahahahahahahahahahahaha
hahahahahahahahahahahahahahahahahhahahahahahahahahahahahahahahahahahahahaha
hahahahahahahahahahahahahahahahhahahahahahahahahahahahahahahahahahahahahaha
hahahahahahahahahahahahahahahhahahahahahahahahahahahahahahahahahahahahahaha
hahahahahahahahahahahahahahhahahahahahahahahahahahahahahahahahahahahahahaha
hahahahahahahahahahahahahhahahahahahahahahahahahahahahahahahahahahahahahaha
hahahahahahahahahahahahhahahahahahahahahahahahahahahahahahahahahahahahahaha
hahahahahahahahahahahhahahahahahahahahahahahahahahahahahahahahahahahahahaha
hahahahahahahahahahhahahahahahahahahahahahahahahahahahahahahahahahahahahaha
hahahahahahahahahhahahahahahahahahahahahahahahahahahahahahahahahahahahahaha
hahahahahahahahhahahahahahahahahahahahahahahahahahahahahahahahahahahahahaha
hahahahahahahhahahahahahahahahahahahahahahahahahahahahahahahahahahahahahaha
hahahahahahhahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahaha
hahahahahhahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahaha
hahahahhahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahaha
hahahhahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahaha
hahhahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahaha
hhahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahah
hahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahha
hahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahha
hahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahha
hahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahha
hahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahha
hahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahha
hahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahha
hahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahha
hahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahha
hahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahha
hahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahha

Spice is Not a Recreational Drug

2008-12-03T15:41:47Z

Well, it seems our friends at Sun have decided that their Spice^tm Enhanced brains are completely sufficient to create an entirely new - but far simpler, mind you - module system for the JDK. Mark "I'm just a simple Guild Navigator" Reinhold has spent a number of blog posts doing the electronic equivalent of the Dance of the Seven Veils, culminating with revealing the aptly named Project Jigsaw.

First off, what Spice^tm Enhanced marketing mind came up with the name Jigsaw? I mean, really. Who thought it would be a good idea to associate a "simpler" module system with a 1500 piece puzzle set that is composed of pieces that look terrifyingly all similar and takes multiple people multiple months to put it together?

Marketing brilliance!

But this, of course, is really just the beginning of the insanity.

]]> that shall not be revealed^tm. Whatever.

But what I can reveal is my reactions to the great and hermetic secrets that Mark Reinhold, himself, presented to the collected assembly in the offices Barron Harkonnen provides his vassals for such meetings. And my reaction is summed up in the phrase, common in the country of my birth:

Don't piss on my leg and tell me that it's raining

Over the years I've come to rely on a single heuristic - a principle, really - that has kept me from being suckered by time wasting morons and their projects doomed to failure. The heuristic is simple:

Good ideas don't need a lot of lies told about them to gain public acceptance

Now, perhaps the word "lie" holds too much emotional impact, so I will not characterize anything that Sun has told me on the subject of modularization as a "lie". However, the "facts" and presented evidence from Sun on the subject has clearly skirted the fine line between dissembling and misleading.

For example, in the discussion with House Harkonnen, Mark presented a series of what he referred to as "requirements" for the module system, which in satisfying he was reluctantly led to the conclusion that OSGi would simply not work. I put the word "requirements" in quotation marks because it was quite clear that the list that Reinhold presented was not in any way properly characterized as a list of "requirements". Rather, what Reinhold presented was simply a list of solutions.

Now, living in the valley as I do - and perhaps more to the point, being a resident of the Emerald City of the valley - I'm quite used to people presenting solutions as requirements. In this industry, it's hard to find anyone that simply doesn't think about the actual problem, but instead thinks about their shiny solution and presents the solution as the requirements of the problem. Fair enough.

But what Reinhold presented wasn't simply a list of solutions. It was a list of solutions that appeared to be specifically designed to eliminate OSGi as a solution to Sun's desire to modularize the JRE. Now, I'm certainly not one to believe that OSGi is the be-all and end-all glorious solution which cannot possibly be changed in any way. There's a great deal of very cool things that could be done with OSGi in this space that would result in changes that would be very good for OSGi. But the list Mark Reinhold presented was not among them.

Pretty much every single "requirment" that Reinhold presented (and Mark, if your reading this and want to release this from NDA so we can actually debate it in the open, I'd be thankful) was immediately solvable in OSGi. A rather irritating pattern of the presentation was the phrase "I'm not an OSGi expert" which was then followed by making a declarative statement about OSGi and how it wasn't up to solving the particular "requirement" gathered by the Third Stage Guild Navigators of Sun. Now, maybe I'm old fashioned, but I think that if you aren't an expert in a particular subject, then you probably shouldn't be making declarative statements about what can and cannot be done in that area - especially if you're affirming that you aren't familiar with the subject at hand.

But I guess the Spice^tm Enhanced minds at Sun can break those rules because they can see far into the future and navigate the threads of futures past and come to concrete conclusions based on their opinions.

So, let's just end this post by reviewing some simple facts. Sun is abandoning JSR 277, the Java Modularization Specification. A specification that received universal scorn because Sun wasn't paying attention to OSGi, I might add. A specification that was widely regarded as an "Not Invented Here" response by Sun to pee on the Modularity real estate because they believed that they could do a better job than OSGi. So now, having failed in that attempt, they're taking their ball and going home to the "Open" (snort!) JDK where they conveiently own the entire governance such that they don't have to listen to anyone else in creating their uber module system.

Further, they're going to create a bait and switch for the rubes in the Java community by foccussing their modularization efforts in JSR 294. What this means, of course, is that while Sun is busily creating, marketing and pushing the module system of JigSaw, they hope to keep everyone occupied with the bright shiny baubles they'll dangle in 294 where committees of people who haven't actually used a module system such as OSGi (name another one!) will fight over how many baubles they can lay claim to.

Oh, this reminds me of another bit of dissembling from Mark Reinhold regarding JigSaw. Repeatedly, Reinhold characterized and stressed that JigSaw was an "internal" module system, not really meant to be used in the wild - in "applications". The impression that Mark was trying to leave with us was that this was just an implmentation artifact that would be largely hidden and unseen by the mass of Java programmers out in the real world. When I finally had enough of this horsepucky, I pointed out that this was clearly bullshit and that they were, in fact, going to be pushing this as a modular solution that applications can use. Mark's dissembling on this issue was literally so bad that one of the gentlemen from Sun - I forget his name at the moment - literally stopped Reinhold and told him that he was being disingenuous in characterizing JigSaw as purely an "internal" system - that it clearly was going to be pushed and used outside of the JRE, in application land.

Again, you don't need a lot of lies to gain acceptance for good ideas.

So, it'll be interesting to see what becomes of this great effort on Sun's part. It's pretty clear to me that Mark Reinhold, at least, certainly has a chip on his shoulder regarding OSGi - I believe Mark stated that he wasn't going to go to OSGi, hat in hand, and negotiate this stuff. Whether my reading of Mark's attitude is correct and whether this attitude reflects a general feeling at Sun or not, the problem with this whole "module" fiasco has been simply the inability of Sun to recognize that there already is an existing and quite mature module system for Java in OSGi.

And so we're left with the hilarious practice of living with the Second System effect, where Sun - while not actually having actually used the technology in question - is going to produce the "real" version of that technology using their Spice^tm Enhanced brains... It's not necessarily a form of insanity endemic to Sun, per se. Certainly this "not invented here" syndrome is prevalent in our industry and certainly is well ensconced here in the Emerald City where I spend my work hours slaving for the Barron. But really... isn't it time that this stupid shit stop? I mean, does Sun - and Reinhold in particular - really think that they are so frickin' brilliant that they are going to produce something that is even measurably better - and of measurably higher quality - than OSGi? I guess they do.

The unfortunate effect is that, due to this hubris, we're now going to have to deal with the fallout.

It's kind of like sending 200,000 troops to Kuwait: Facts on the ground. After all, we have this existing module system now, what makes you think that Sun will now have any incentive to make it compatible with OSGi at all? If they wanted to do so, they could have done so in the first place. The Apache Harmony project is proof positive that OSGi can be used to modularize the JRE.

If you start off by claiming that you're not going to negotiate with OSGi about any of this, then it shouldn't really suprirse anyone at Sun to see people rolling on the floor, laughing at their assurances that they're going to work really, really hard to make the system compatible with OSGi. I mean, really Mark. Pull the other one!

Distributed OSGi - On The Scent Of Red Herrings

2008-06-16T23:13:10Z

So, this morning my alarm clock was inexplicably set to the central time zone and consequently I arose 2 hours before my regularly scheduled time. Unable or unwilling to go back to bed after my morning "get ready for work" rituals, I settled down to read a few articles from my RSS feeds. The first one I read was a rather odd post on Distributed OSGi - Tilting at Windmills by Roger Voss. Laughing a bit as I read the opening paragraph, I didn't think much more about the post as it really had nothing to do with what the OSGi RFP 119 "Distributed OSGi" really is about. Fortunately or unfortunately, several people started peppering me with tweets, IMs and emails asking if I had read the post in question and what were my thoughts about it. So, here they are.

Basically, I'm not really going to defend the OSGi EEG RFC 119, "Distributed OSGi" as my own interest in the matter rested largely with the acquisition of the so called "registry hooks" which allow infrastructure developers such as myself to hook into the queries of the OSGi service registry and do cool things like manifest services on demand. Once this capability was present and decoupled from the 119 RFC, I felt I had all the tools I needed to do any damn thing I wanted to, regardless of what the 119 RFC was doing. (for background as to how I fell in love with the idea of the registry hooks, see my posts on remote OSGi, which predated the RFC 119 here and here and a post during the formulation of the RFP which led to RFC 119 here)

]]> Oddly, I happen to agree with the basics of what Roger Voss has to say in his post which really doesn't have much to do with Distributed OSGi qua RFC 119. I mean, I was doing orthogonal persistence and distribution before Microsoft even invented the term Object and when the OMG was a toddler. All the issues that have since become well known regarding distributed computing, RPC and "transparence" are well known to me and something that I internalized quite a long while ago.

There's something that I've tried to get across in the various forums I've been a part of - SCA and now the OSGi EEG - but the point doesn't seem to get across too well and that is this:

Fine. You don't like transparent, RMI style distribution. You think there's a huge difference between 'remote' communication and 'local' communication that you believe should be ensconced in the very bowels of a 'distributed' interface. Fine. But what you seem to be completely missing is that the API presented by the 'service' is a Java object. It has Java semantics. If you want an exception thrown to indicate remote communication issues, then you had damn well put it in the interface specification. If you want to expose the wonderful rawness of the asynchronous communication patterns underlying all distributed communications, then by all means encode it into the interface of your Java object.

The point being simply that you have a Java object there. You don't like "transparency". Then you can pretty much define any interface you damn well like. My only request is that you don't "force" it on the rest of us.

Look. OSGi services aren't anything magical at all. One thing that I think people who haven't used OSGi at all are extremely confused about is that OSGi services are simply Java objects. They are not proxied - meaning that you have access to the actual, honest to "Bob" Java object that is the service. They do not have to implement an interface - meaning that there is literally no restrictions on what you can publish as a service.

And so all this whining and moaning about how we don't need yet another failed "transparent", "RPC-based" distributed object model is really just whining and moaning without understanding what is going on. RFC 119 (so far!) doesn't say jack about what the actual model for the distributed object model actually is. Yes, there's a lot of people working on the spec who honest to "Bob" believe that they really, really do have the uber distributed computing model sewn up and that's what all the really cool (and productive) programmers will be using next year. But RFC 119 doesn't promulgate any such model.

And that's the way I hope it stays. There's always a lot of loose talk about defining an "asynchronous" model or another god forsaken web services model, but these are and should remain orthogonal to the oddly named RFC 119, "Distributed OSGi" specification. My personal opinion is that the days of communist inspired, five year centrally planned API specifications by standards organizations are long, long gone and will be quite infrequent in the future. POJOs rule and there's really absolutely no reason to specify much beyond what is already specified in RFC 119 (there's already, imho, too much specified in RFC 119, but that is purely my personal opinion and in any event, it's done in such a way as I don't have to pay attention to it if I don't want to).

However, what is important is the way services are used in OSGi and the way the service registry hooks allow some highly sophisticated and quite magical (if I may be so bold as to use that term) capabilities in OSGi that really are quite handy. Regardless of whether you have Ajax communication, JMS messaging, god's own event oriented communications APIs or (shudder) straight up RMI "transparent" communication, the facilities that RFC 119 relies on (and, of course RFC 119 itself) will make not only the development, but the actual interaction with your services from the end programmer's perspective a fantastic experience.

Focussing on the API of some mythical dream of transparent, RPC style (or even event oriented) distributed communication is completely missing the point of what is happening with OSGi and distributed communications - and RFC 119, "Distributed OSGi", in particular.

My Dinner With Andre

2008-06-04T13:42:19Z

Well, after my last blog entry regarding Sun and JSR 277, I received a visit from the enforcers of that fine establishment. Wearing heavy overcoats on a nice sunny day, the elevator door opened and they came rushing out. Like a scene from the Matrix, it seemingly took them forever to pull out their impossibly huge automatic weapons they had hidden under the black leather trench coats. In only several hundred milliseconds, Sun's professional enforcers unloaded 20 or 30 pounds of lead in the form of hundreds upon hundreds of rounds of high powered ammunition into what used to be my body, splattering my physical essence across the interior of the reception area of building 200 on the Geidi Prime campus.

Or not. Rather, what did happen was Alex and Stanley showed up at my office escorted by a fellow member of House Harkonnen and after a slightly awkward moment with Stanley (I had, after all, just blasted across the internet - with the awesome power of my blog - stating that he was an idiot), Stanley and my compatriot excused themselves to work on JSR 277 related specification business leaving Alex Buckley and I in my office, as I watched him finger a menacing looking lead pipe that he pulled from his brief case.

Or not. Rather, we sat down and talked comfortably for quite some time about about everything but the issues surrounding JSR 277 and the reason why Alex even knew of my existence. Eventually, however, we directed the conversation towards the actual issues we really needed to discuss and left the safe domain of my Dr. Who toy collection and the pleasantries of traveling through Heathrow in the U.K.

]]> Now, Alex Buckley is obviously one of the top Spice^tm enhanced brains working at Sun. Very easy to talk to and generally a no nonsense individual who clearly wants things to work out for the best of both for Sun and the OSGi community. I won't actually go into any of the discussions about technical particulars that we discussed about such subjects as versioning formats, meta data information locations and such. These things are important, but there's a point that is, I think, far more important than any of the technical details which will have to be worked out and that is how this process is actually working out.

We as technical people invariably have this rather blinkered view of reality in that we see the "real" problems as being the "hard" issues of the technology in question. As I mentioned, the issues of how many indexes and what format the version strings will have in the module metadata; whether we should reuse the META-DATA/MANIFEST.MF entries like OSGi does, and whether that is even sufficient for the super complex, pan dimensional needs that the Spice^tm enhanced brains from Sun foresee as requirements due to their ability to look far beyond the present and into the bright future where we'll all need shades just to keep our eyes from being seared because it's such a hot technology. No, these issues are all things that I do believe that having the Third Stage Guild navigators from Sun locked in a room with the mere mortals of the rest of the industry could figure out in a reasonable amount of time.

The problem is, of course, that there really isn't any functioning expert group, nor is there any serious interaction between Sun, the JSR 277 expert group and the OSGi organization.

Now, while it's somewhat flattering to have Alex show up in my office and dedicate a non trivial amount of his time discussing the technical issues with me, the problem is that I'm not the OSGi organization. True, I'm a member of the Core Platform Expert Group and the Enterprise Expert Group. But in reality, it's just me and this wasn't even a working session where we were expected to produce anything other than a polite blog entry such as this explaining how wonderful things really are when you look past all the politics with JSR 277.

Indeed, the problem - from my humble and obviously biased POV - is that Alex is having a substantial number of bilateral talks with various interested parties of this rather serious issue. That is, he's talking directly with Peter Kriens, Rod Johnson, Richard Hall, Glyn Normington, Bryan Atsatt and god only knows who from Google and the zillion other people from companies who represent various parts of this grand collective we affectionately know as the Spice Industry^tm.

And there in lies the nub of the problem. As has been said by many before me, "He who controls the Spice^tm, controls the universe". And what is quite clear from the conversation with Alex is that Sun believes it controls the Spice^tm. And it's quite hard to argue with this fact. One can look at the dead bodies strewn along the path of any number of previous JSRs and see how Sun dealt with technologies that predated the JSR in question, inspired the JSR in question and were gutted and left to die in the hot sun by the JSR in question. Clearly, one can look at the writing on the wall and see much the same in the near future of OSGi. Once something is in the JRE, any technology competing with it is toast. End of story.

Consequently, Sun does not want to even give the impression that they are "negotiating" with OSGi, nor working with them in any form other than talking with the various members of this organization. Whatever one may want to do as a spec lead, it's really the higher ups in the hierarchy of Sun that ultimately define the boundaries of interactions and it's rather clear that those boundaries do not include actual interaction with OSGi qua OSGi. Thus, we're left with piece meal, bilateral negotiations between individuals which may work to improve things around the edges, but won't even come close to dealing with the weighty issues that are at the heart of the matter.

And so, while I think one has reason to be optimistic about things from a technical perspective, I think that optimism has to be seen in the perspective of the political realities. The political realities of the situation are such that Sun will not give one iota and cross the line to deal with OSGi. They simply can't do this politically. Rather, we will continue to hear whining about how OSGi will not cross the line in the form of Peter Kriens and BJ Hargrave and fully work with the 277 expert group - an expert group which, from what anyone can tell isn't really a functioning expert group any more, given how it's gone completely dark and all the actual work is happening in the Java Modules project without much input from anyone else.

Consequently, however little light there may be between the technical solutions that may possibly be out there, I think the simple fact that Sun will continue to dig in their heels and hold onto control over the Spice^tm, which will essentially doom the entire project. No, I'm not talking about dooming JSR 277. It's quite obvious that Sun, controlling the JRE as they do, will be able to push through whatever they want despite any grumbling from the peanut gallery and the awesome push back all three readers of this blog will be able to generate. Rather, what will be doomed is OSGi as we know it because Sun will basically add OSGi to the long list of dead bodies of other projects that JSRs have killed in the past. It won't be done purposefully and won't be done because of the ego of individuals as has been suggested was the case previously. Rather it will be done because of the egos of organizations who are unwilling or unable to come to the table and hammer these admittedly highly tractable technical issues out as a team - as peers.

Granted, I hope I'm wrong and lord knows I'm quite the pessimist on such matters. But one doesn't have to put too pessimistic of an interpretation of the past wrt JSRs and the fate of external technologies that inspired them to see the progression into the future. Whatever technical issues there may be are indeed solvable and perhaps trivially so.

But the politics - ah, the politics. The politics will provide the mechanism to compensate for what individual egos perhaps accomplished in the past to screw things over. The very fact that the JSR 277 expert group is essentially non functioning should be a warning sign that shouldn't be ignored. The very fact that Alex is involved in complicated, tedious and no doubt highly time consuming multiplicity of bilateral negotiations should also be a warning sign that shouldn't be ignored.

Often, even with only the good intentions of everyone involved, things still go very badly. Often, despite everyone wanting things to work out, the group interactions prevent these good intentions from being realized. Perhaps JSR 277 will be the exception to the rule and we'll all get not only all the Ponies^tm we want but we'll each be given our own private Leprechaun with a personalized pot of gold to boot.

But count me extremely skeptical.

I'm sure that OSGi will be able to adapt and deal with whatever falls out. And perhaps we'll all be better off after we pass through the fires to come. Who knows. Stranger things have actually happened. But it's not looking like it's going to be a fun couple of years which is the way it should - in anyone's approximation of a realistic world - have worked out.

the sleeper must awaken

Stanley Ho is Insane

2008-05-30T14:43:04Z

Stanley Ho shows up for another meeting of the JSR 277 Java Modules Specification with a brand new versioning scheme

Dear god, why are we subjected to such hubris?

Stanley Ho, working on the Java Module System (formerly known as JSR 277) has decided that Sun is so smart that they need to invent yet another versioning system. Now, maybe Stanley Ho really is a super smart dude who makes me look like I was a fifth rate, multiple generation inbred dufus from the white trash mountains of Virginny (many, many people have actually pointed that I actually am a fifth rate, multiple generation inbred dufus from the white trash mountains of Virginny, but I digress), but it seems to me that it is self evident that versioning is what we in the business know as a HARD PROBLEM^tm. And to think that someone even as super intelligent as those people at Sun are known to be will come up with a scheme that doesn't have any unintended consequences that were not foreseen by even their Spice^tm enhanced brains is simply something that I think you'll find many in this industry simply can't conceive.

There will be bugs in what Stanley has designed. There will be unintended consequences. It will only be after years of serious use and abuse by many hundreds of thousands of developers in real world situations that even the Spice^tm enhanced mind of Stanley Ho hasn't dreamed of will the full extent of his hubris become clear.

One might ask the question: "Why, Stanley, didn't you use the OSGi versioning scheme?".

This shit is hard. This path is fraught with peril. In the 21st century, we simply don't go around inventing shit for the hell of it simply because we can. We reuse existing things even though they might not be the correct shade of purple we were dying to have. Simply because the corners are a bit worn.

The absolute last thing we need from Sun in the Java Modules is Yet Another Brilliant Sun Shiny Invention that we'll have to suffer though for years until all the bullshit is extracted from the hubris that went into creating.

Geebus. Please. Dear God. Please.

Stop the madness. Someone please hold an intervention with these people. The industry simply cannot stand to have this crap thrown at them, especially in something as fundamental and as far reaching as the frickin' base Java module system.

See Peter Krien's rant on the hubris that is Stanley Ho.

Sometimes a turd is just a turd