How to make enterprise software Nimble

Chelonia mydas is going for the air by Mila ZinkovaWith the release of the OSGi Enterprise Specification a whole new set of developers will be trying to figure just what the hell you can do with this OSGi technology anyway?

Those who have started to explore it will have found that surprisingly OSGi doesn’t tend (initially) to make development easier, it also doesn’t tend (initially) to reduce the amount of code you need to write. So what then is it good for?

The key feature of OSGi is that it allows for proper modularization of Java software. There’s a lot of good information out there on the interweb about why modularization is important – but the key message is that modularization allows you to decide what are the building blocks of your application. Not at the nuts and bolts level that is Object Orientation and not at the red car or blue car level that is SOA, but at the intermediate level of engines, tires and gearboxes – where most engineers really like to play…

OSGi makes the task of building sets of applications easier. It is not about building one application but about building applications from applications, software reuse, yada yada, or as I like to think of it – turtles all the way down.

All this sounds great, but in order to build good modular software it is important to decide what are the public parts of your architecture, what are the bits you’d rather keep private and which parts you want to inherit from others. This process is achieved in OSGi via importing and exporting packages between bundles. It is also highly advantageous to use the OSGi services model which allows you to neatly hide implementation specific code behind interfaces and plug in and out different providers at runtime.

The problem now becomes that as you depend on more and more bundles the task of managing that list of bundles becomes more and more onerous. Most peoples first experiences with OSGi (mine included) is a frustrating iteration of trying to start a given bundle but getting a loud exception message saying that a given bundle cannot be resolved because some of its dependencies are missing. You then add bundles that satisfy these dependencies but find that these dependencies are transitive, so satisfying the top level dependencies just leads to another deeper search for the bundles that satisfy those dependencies. You finally get all the required bundles installed and activate your top level bundle and click on the web browser or ui and nothing happens. Silly you, you have forgotten to activate the bundles that provided supporting services for your application.

The benefits of OSGi are obsured by the complex task of managing both the api level and service level dependencies. Which bundles do I need installed? Which bundles do I need to activate? If I remove this bundle what are the consequences? If I stop my application what else can be tidied up? All these questions are exactly those we started asking ourselves when we started writing OSGi applications and Nimble is the logical conclusion of this work.

The interesting thing is that Nimble is in fact built upon the very modularity information that is so complex for developers to navigate, all of this information is actually perfect for a computer to make reasoned judgements about. Given a set of bundles Nimble can very rapidly traverse the set of bundles that need to be installed and activated to make your application work, and when you’re done it will tidy it all away again.

Sample JEE application

To see how this works in practice let’s see how to build a traditional three tier application comprised of a web tier, a business tier and a data access tier. For this example we’re going to reuse the AriesTrader sample provided as part of the Aries project. This sample uses six different parts of the OSGi Enterprise specification.

  • Blueprint (Spring Beans)
  • JDBC
  • JNDI
  • JPA
  • JTA
  • Web Applications

When fully installed this application is comprised of more than fifty individual bundles which could be a real headache to download, install and activate manually. However using Nimble this process is easily managed using it’s advanced resolver capabilities and scripting support.

So running the sample…

Before proceeding you should check that your version of Nimble is up to date. You can do this using the “version” command from the cli. This example requires that you have at least version 1.0.14 installed. For information on upgrading Nimble refer to our online documentation.

As Aries is still in the incubation phase at Apache there are not yet any released artifacts we can build against, so if you want to run these samples you will need to check out the source code from svn and build it yourself – there are instructions on how to do this here. To run the Aries trader sample we also need to download and install derby, again instructions can be found here.

Having completed these preliminaries I will create the AriesTrader database which the sample application uses to store its data.

$ mkdir /tmp/aries
$ cd /tmp/aries
$ export CLASSPATH=$DERBY_INSTALL/lib/derby.jar:/opt/lib/derby/lib/derbytools.jar
$ java org.apache.derby.tools.ij $ARIES_SOURCE/samples/ariestrader-sample/assemblies/equinox-test-harness/target/tradeDB.sql

Here I’ve created a sample directory on my machine and created the empty database in that directory.

In fact the aries trader sample is pretty simple and assumes the database is located on the same machine and the sample needs to be run from that directory as it assumes the database is located relative to the Java process. I guess it’s important to say that this is a limitation of the sample and not JDBC/JPA in OSGi in general!! But for this example be sure to run Nimble from the same directory that you created the database in – in my case /tmp/aries/.

Let’s now run the aries trader example using a OSGi shell script. To do this launch Nimble via the Posh shell executable located in the bin directory of the Nimble installation and then run the aries-trader script in the console.

[dave@feynman aries]$ $NIMBLE_HOME/bin/posh
[feynman.local.0]% pwd
file:/private/tmp/aries/
[feynman.local.0]% sh http://chronological-thought.googlecode.com/svn/trunk/nimble-examples/aries-trader.osh
Internal Paremus Nimble Developer License,
expires Fri Dec 31 23:59:59 GMT 2010.
________________________________________
Welcome to Paremus Nimble!
Type 'help' for help.
----------------------------------
- AriesTrader sample application -
----------------------------------
# Enter aries source location:
/Users/dave/development/apache/aries
Indexing aries:0 -> file:/Users/dave/development/apache/aries/ -> file:/private/tmp/aries/aries.xml
  include=.*target.*jar
  exclude=null
  base=file:/Users/dave/development/apache/aries/
  load=false
Indexed 32 resources
Indexed 64 resources
Indexed 96 resources
Indexed 128 resources
Indexed 160 resources
Indexed 192 resources
Indexed 224 resources
indexed aries:0
Connect to AriesTrader at http://localhost:8080/ariestrader

The first time you run this script it will prompt you for the location of the Aries source folder and will index the osgi bundles you built earlier.

You should be able to verify that the AriesTrader sample is running by browsing to the AriesTrader web application, the address of which is echoed at the end of the aries-trader.osh script. If you want to play around with the AriesTrader application you can populate the database by navigating to the “Configuration” tab on the aries trader web portal and clicking “(Re)-populate AriesTrader Database”. You should see the database data being populated and you can then log in via the “Trading & Portfolios” tab.

A look under the hood

So what’s going on in this example from a Nimble and OSGi perspective? Let’s look at the contents of the “aries-trader.osh” file. To do this we can use the built in “more” command in Nimble:

[feynman.local.0]% more http://chronological-thought.googlecode.com/svn/trunk/nimble-examples/aries-trader.osh

This pages through the contents of the file allowing us to see the contents just like a standard unix more command, one thing to note is that Nimble uses java.net.URI objects in most places where unix would use a file reference so we can more the file directly from the web. So the contents of the file (in bite size chunks)…

tofile = {
 path = $it schemeSpecificPart;
 new File $path;
};
activate = {
  nim add osgi.active.bundle/$it;
};

The first thing I’ve done is declare a couple of closures “tofile” and “activate” that allow me to tidy up boiler plate code that would otherwise be repeated in the rest of the script.

The “tofile” closure converts a java.net.URI object to a java.io.File here you can see that Nimble is able to make reflective calls on Java objects – get methods are camel cased and the return value of the closure is the last object returned.

The “activate” closure just passes the parameter parsed to the “nim add” command which we saw in previous posts to add the specified osgi bundle to this OSGi framework in the active state. I’ve then created some banner code that makes the script a little more user friendly:

echo ----------------------------------;
echo - AriesTrader sample application -;
echo ----------------------------------;

Now we start the work of the script proper. Nimble figures out the bundles that need installing or activating based on indexes, so in order to load the aries bundles we need to generate an index file that describes their requirements and capabilities. The following section of the aries-trader.osh script shows how I created this index:

aries = tofile ($PWD resolve aries.xml);
if { not { $aries exists } } {
  echo # Enter aries source location:;
  read sourceDir;
  source = new URI $sourceDir/;
  pom = tofile ($source resolve pom.xml);
  if { not { $pom exists } } {
    echo Expected aries source directory;
    exit;
  };
  nim index -viI .*target.*jar aries $source $aries;
};

Firstly I check if a file aries.xml exists in the directory where I’ve run this script, if it does then no action needs to be taken, if not then I need to generate an index. To generate the index I prompt the user to tell me the location of their Aries source folder and make a trivial check that it is in fact the Aries source folder by checking if there is a pom.xml file present in that directory. If all is well I then call the nim index command with a number of options that tell Nimble to generate an index called “aries” in this directory based on files that match the regular expression .*target.*jar from the specified source folder. The result being we get an index file aries.xml of all artifacts built by the maven build. We now need to load the indexes.

nim repos -l $aries;
nim repos -l ($0 resolve aries-rules.xml);
nim repos -l springdm;

These three commands add the indexes that Nimble will use to build our application. The first entry loads the aries.xml index we generated above, the second adds a second aries index file that I have pregenerated (more on this in a second) and the third adds an index that points at the Spring Enterprise Bundle repository based on a well known alias name “springdm”. Ok so hang on a minute where does this aries-rules.xml file come from? Let’s take a look at it via Nimble:

[feynman.local.0]% more http://chronological-thought.googlecode.com/svn/trunk/nimble-examples/aries-rules.xml
<repository name="aries-rules:0" xmlns="http://schema.paremus.com/nri/1">
 <rule name="aries.osgi.service/blueprint:0.1.0.incubating.SNAPSHOT">
  <req name="builder/primordial.builder" />
  <req name="osgi.active.bundle/org.apache.aries.blueprint:0.1.0.incubating.SNAPSHOT"/>
  <req name="osgi.active.bundle/org.apache.aries.transaction.blueprint:0.1.0.incubating.SNAPSHOT"/>
  <req name="osgi.active.bundle/org.apache.aries.jpa.blueprint.aries:0.1.0.incubating.SNAPSHOT"/>
  <cap name="aries.osgi.service/blueprint:0.1.0.incubating.SNAPSHOT" />
  <cap name="osgi.service/blueprint.extender:0.0.0" />
 </rule>
 <rule name="aries.osgi.service/jta:0.1.0.incubating.SNAPSHOT">
  <req name="builder/primordial.builder" />
  <req name="osgi.active.bundle/org.apache.aries.transaction.manager:0.1.0.incubating.SNAPSHOT"/>
  <cap name="aries.osgi.service/jta:0.1.0.incubating.SNAPSHOT" />
  <cap name="osgi.service/jta.provider:0.0.0" />
 </rule>
 <rule name="aries.osgi.service/jndi:0.1.0.incubating.SNAPSHOT">
  <req name="builder/primordial.builder" />
  <req name="osgi.active.bundle/org.apache.aries.jndi:0.1.0.incubating.SNAPSHOT"/>
  <cap name="aries.osgi.service/jndi:0.1.0.incubating.SNAPSHOT" />
  <cap name="osgi.service/jndi.provider:0.0.0" />
 </rule>
 <rule name="aries.osgi.service/jpa:0.1.0.incubating.SNAPSHOT">
  <req name="builder/primordial.builder" />
  <req name="osgi.active.bundle/org.apache.aries.jpa.container:0.1.0.incubating.SNAPSHOT"/>
  <req name="osgi.active.bundle/org.apache.aries.jpa.container.context:0.1.0.incubating.SNAPSHOT"/>
  <req name="osgi.active.bundle/org.apache.openjpa:2.0.0.beta2"/>
  <cap name="aries.osgi.service/jpa.provider:0.1.0.incubating.SNAPSHOT" />
  <cap name="osgi.service/jpa.provider:1.0.0" />
 </rule>
</repository>

At the time of writing this file is hand generated, it specifies a number of capabilities that match requirements that are automatically detected from client bundles during the index step above. The capabilities are: osgi.service/jpa.provider, osgi.service/jndi.provider, osgi.service/jta.provider and osgi.service/blueprint.extender. When a bundle is activated that requires one of these capabilities then Nimble traverses its indexes looking for rules that provide these capabilities and resolves the rule plus its requirements. In this case we can see that when a bundle requires the osgi.service/blueprint.extender capability Nimble will install and activate three supporting bundles org.apache.aries.blueprint, org.apache.aries.transaction.blueprint and org.apache.aries.jpa.blueprint.aries.

Here an interesting point is that these osgi.service/xxx.yyy are indirect links to the supporting bundles. In fact if I want to switch to a different provider – say Eclipse Gemini then it is just a case of indexing these bundles instead and Nimble will resolve the bundle dependencies using a different set of providers. I’ll try to do this in a later post when the artifacts are available – stay tuned ;)

The final section of the script loads the aries trader sample bundles:

// trader bundles
activate org.apache.aries.samples.ariestrader.util;
activate org.apache.aries.samples.ariestrader.core;
activate org.apache.aries.samples.ariestrader.beans;
activate org.apache.aries.samples.ariestrader.derby.ds;
activate org.apache.aries.samples.ariestrader.entities;
activate org.apache.aries.samples.ariestrader.persist.jdbc;
activate org.apache.aries.samples.ariestrader.persist.jpa.am;
activate org.apache.aries.samples.ariestrader.persist.jpa.cm;
activate org.apache.aries.samples.ariestrader.web;

Here I’m really focusing on the nine bundles I really need for my application, all of the other bundle dependencies (all forty six of them – which you can see for yourself if you type the lsb command at the Nimble console) are automatically provisioned and configured by Nimble. In fact it is a trivial task to change this script to load the Aries blog sample – an exercise I’ll leave till another post.

So in summary Nimble has allowed us to quickly build up an application with real world components like web services, transactions and databases. We focused simply on top level bundles we require and Nimble resolves and deploys all supporting bundles based on index information that is very easy to generate and reuse between applications.

What’s more for the metathesiophobes (comments on a post card if you know of an actual fear related to dynamic resolution) the resolution path for the dynamically deployed bundles is minutely controllable via the indexes that you add to Nimble – so if you want to prevent certain artifacts from being resolved against because of known bugs etc – simply exclude them from your indexes.

Update: Seems I’m not the only one to have made the turtles reference – Kirk Knoernschild makes the same analogy, probably inadvertently borrowed the same metaphor as I do read his excellent blogs.

  • Print
  • LinkedIn
  • Facebook
  • del.icio.us
  • DZone
  • Google Bookmarks
  • Yahoo! Bookmarks

OSGi Dependencies: Heaven and Hell

The road to hell is paved with unsatisfied dependencies

OSGi bundles don’t share code arbitrarily like ordinary jars. Instead they explicitly export the packages they want to share and import the packages they need. Before you can use code from a bundle, it has to be resolved by the OSGi framework. Resolution involves matching each package a bundle imports with a package exported by some provider bundle. The provider bundle also has to be resolved, so the whole process is recursive.

Manually working out which bundles you need to install before the bundle you are actually interested in will resolve can be a long winded process in which you install the bundle you want, try to resolve it, fail, add more bundles in response to the error messages, fail, tear out some hair, add more bundles, and repeat, until finally you get something that works. At this point you can look at the resolved set of bundles and make a note of them and the order you installed them in to make it work. Even then the result is quite brittle because each resolution is sensitive to the state of the framework and to bundle versions. To cut a long story short, doing things this way leads straight to the OSGi dungeon in dependency hell.

Of course, it doesn’t have to be this way. We can automate the resolution process just like Apt and Yum automate Linux installs. This is where standalone resolvers like Nimble come in. They use repository metadata to figure out and install sets of bundles that will resolve successfully, relieving developers of this burden.

Writing a good resolver turns out to be a real challenge. In fact the problem is NP Complete, and naive, brute force implementations can easily get lost in resolution searches that take weeks or more (I’m not joking). More sophisticated solutions, using powerful heuristics, can do the same job in an instant.

Fortunately, the time spent developing a powerful resolver is well worth it because, as well as solving the original problem, it can be re-targeted to address other dependency concerns.

Active bundle dependencies

Nimble’s resolver is the third generation of Paremus’ resolver technology. One of its design goals is to move beyond resolution of bundle imports and exports, and enable automatic resolution for a variety of different dependency types.

To see how useful this can be, let’s take a look at ACTIVE bundle dependencies. These are bundle dependencies that only apply to bundles in their ACTIVE state.

For example, suppose you have a bundle, HelloDS, that makes use of Declarative Services (DS), and a DS runtime bundle. In its RESOLVED state HelloDS may well have no dependency on the DS runtime bundle whatsoever. However, once you activate HelloDS, it wont work properly unless the DS runtime bundle has also been installed and activated. So, in its ACTIVE state, the HelloDS bundle depends on the DS runtime provided by ACTIVE state of the DS runtime bundle. Note also that the ACTIVE state of any bundle depends on the RESOLVED state of the same bundle. This is all illustrated below.

When you ask Nimble to index a bundle repository it analyses the bundles, detects a wide variety of dependency types, and associates these with the appropriate bundle states. This gives Nimble its unique ability to resolve ACTIVE bundle dependencies, such as the need for the Spring DM, Declarative Services or iPOJO runtimes. It is also the basis of Nimble’s ability to install console commands on demand the first time you use them.

To make this concrete I’m going to work through a simple example using a HelloDS bundle, just like the one I mentioned above. For something more sophisticated, take a look a David Savage’s recent post on Zen and the Art of OSGi Dependency Resolution.

First of all let’s launch Nimble.

$ posh
Internal Paremus Nimble Developer License,
expires Fri Dec 31 23:59:59 GMT 2010.
________________________________________
Welcome to Paremus Nimble!
Type 'help' for help.
[Pequod.local.0]%

Throughout this demo you may see some bundles and other repository files downloading – I’ve omitted these lines for clarity.

Before going any further you should check that you have the latest version of Nimble, and the latest Paremus-hosted repository indexes by typing

% version

If you need an update this will tell what to type to get it.

Next we’ll load the examples repository that contains the HelloDS bundle, and look at the state of the base container.

% repos -l examples
% lsb
0  ACTIVE    org.eclipse.osgi:3.5.1.R35x_v20090827
1  ACTIVE    com.paremus.posh.runtime:1.0.11
2  ACTIVE    com.paremus.posh.shell:1.0.11
3  ACTIVE    com.paremus.posh.readline:1.0.11
4  RESOLVED  com.paremus.util.types:1.0.11
5  ACTIVE    com.paremus.nimble.core:1.0.11
6  ACTIVE    com.paremus.nimble.repos:1.0.11
7  ACTIVE    com.paremus.nimble.cli:1.0.11
8  ACTIVE    com.paremus.util.cmds:1.0.11

Now let’s ask Nimble to RESOLVE the HelloDS bundle, then list the installed bundles:

% add HelloDS@resolved
[Pequod.local.0]% lsb
0  ACTIVE    org.eclipse.osgi:3.5.1.R35x_v20090827
1  ACTIVE    com.paremus.posh.runtime:1.0.11
2  ACTIVE    com.paremus.posh.shell:1.0.11
3  ACTIVE    com.paremus.posh.readline:1.0.11
4  RESOLVED  com.paremus.util.types:1.0.11
5  ACTIVE    com.paremus.nimble.core:1.0.11
6  ACTIVE    com.paremus.nimble.repos:1.0.11
7  ACTIVE    com.paremus.nimble.cli:1.0.11
8  ACTIVE    com.paremus.util.cmds:1.0.11
9  RESOLVED  javax.servlet:2.5.0.v200806031605
10  RESOLVED  osgi.cmpn:4.2.0.200908310645
11  RESOLVED  HelloDS:1.0.0

Here bundle 11 is the one we asked to resolve, and bundles 9 and 10 were brought in to support its resolution. Note that there’s no sign of the DS runtime bundle, because it’s not needed to satisfy any package level dependencies.

Nimble is a full lifecycle resolver, i.e. it doesn’t just install bundles, it can also clean up reliably, so now that we’re done, we can get back to where we started by simply removing the resolved state HelloDS bundle.

% remove HelloDS@resolved
[Pequod.local.0]% lsb
0  ACTIVE    org.eclipse.osgi:3.5.1.R35x_v20090827
1  ACTIVE    com.paremus.posh.runtime:1.0.11
2  ACTIVE    com.paremus.posh.shell:1.0.11
3  ACTIVE    com.paremus.posh.readline:1.0.11
4  RESOLVED  com.paremus.util.types:1.0.11
5  ACTIVE    com.paremus.nimble.core:1.0.11
6  ACTIVE    com.paremus.nimble.repos:1.0.11
7  ACTIVE    com.paremus.nimble.cli:1.0.11
8  ACTIVE    com.paremus.util.cmds:1.0.11

Now let’s add the HelloDS bundle again, this time in the ACTIVE state.

% add HelloDS@active
Hello Declarative Services
% lsb
0  ACTIVE    org.eclipse.osgi:3.5.1.R35x_v20090827
1  ACTIVE    com.paremus.posh.runtime:1.0.11
2  ACTIVE    com.paremus.posh.shell:1.0.11
3  ACTIVE    com.paremus.posh.readline:1.0.11
4  RESOLVED  com.paremus.util.types:1.0.11
5  ACTIVE    com.paremus.nimble.core:1.0.11
6  ACTIVE    com.paremus.nimble.repos:1.0.11
7  ACTIVE    com.paremus.nimble.cli:1.0.11
8  ACTIVE    com.paremus.util.cmds:1.0.11
12  RESOLVED  javax.servlet:2.5.0.v200806031605
13  RESOLVED  osgi.cmpn:4.2.0.200908310645
14  ACTIVE    HelloDS:1.0.0
15  ACTIVE    org.eclipse.equinox.util:1.0.100.v20090520-1800
16  ACTIVE    org.eclipse.equinox.ds:1.1.1.R35x_v20090806

This time the HelloDS bundle has been activated. Not only that, but its ACTIVE state dependency on the DS extender has been met by bundle 16, org.eclipse.equinox.ds, which has also been activated. Note the “Hello Declarative Services” message that tells us everything is working.

Careful readers might be wondering why bundle 15 is also ACTIVE. This is a consequence of Eclipse’s treatment of bundle activation policies. Nimble didn’t activate it, but it does understand and track what’s going on.

As before we can get back to where we started by simply asking Nimble to remove the active state bundle

% remove HelloDS@active
Goodbye Declarative Services
[Pequod.local.0]% lsb
0  ACTIVE    org.eclipse.osgi:3.5.1.R35x_v20090827
1  ACTIVE    com.paremus.posh.runtime:1.0.11
2  ACTIVE    com.paremus.posh.shell:1.0.11
3  ACTIVE    com.paremus.posh.readline:1.0.11
4  RESOLVED  com.paremus.util.types:1.0.11
5  ACTIVE    com.paremus.nimble.core:1.0.11
6  ACTIVE    com.paremus.nimble.repos:1.0.11
7  ACTIVE    com.paremus.nimble.cli:1.0.11
8  ACTIVE    com.paremus.util.cmds:1.0.11

If you were using a resolver that wasn’t aware of ACTIVE bundle dependencies then you’d have to manage all of this yourself. With Nimble you just ask for what you need and let the resolver take care of the rest. This speeds up the development lifecycle and keeps redundant clutter out of your scripts.

There’s a lot more to Nimble’s resolver than what we’ve covered here. Its other features include policy based attachment of fragments and policy based resolution of optional dependencies. I’ll cover these and more in later posts.

Nimble is all about Making Modularity Manageable. If that’s what you want to do then try Nimble now.

  • Print
  • LinkedIn
  • Facebook
  • del.icio.us
  • DZone
  • Google Bookmarks
  • Yahoo! Bookmarks

Zen and the Art of OSGi Dependency Resolution

In a previous post I showed how it is possible to use Nimble to deploy the Spring Simple WebApp using a trivial command:

posh -kc "repos -l springdm;add org.springframework.osgi.samples.simplewebapp@active"
open http://localhost:8080/simple-web-app/

This actually relies on some pretty advanced features of the Nimble Resolver that allow us to perform some interesting gymnastic tricks with regard to OSGi dependency resolution. The previous post glossed over these details but in this post I’ll delve a little deeper into what’s going on underneath the hood.

To understand what’s going on in this example it’s worth looking at a greatly simplified model of the bundles deployed to make this application work.

In this simplified view we can see that there are three logical components that make up this example. The web extender provides the underlying servlet container and watches for web applications deployed within the OSGi framework. Our simple web app contains a set of Servlets and JSPs that define our business application and finally the Servlet API bundle provides shared classes to enable both the web extender and our simple web app to inter-operate.

The important point to realize from an OSGi perspective is that in this simple example in order to resolve our simple web app we only need to install our bundle and the API bundle. However this still does not get us to a state where we can actually use our web app. In order to use our application we need to do two extra things, we need to activate our web application bundle and we need to install and activate the web extender bundle. Nimble gives you the ability to automatically do these things and what’s more clean up when you are finished.

Here the clean up is important! In long running environments the tendency is for garbage to accrete over time which eventually grinds the system to a halt. OSGi actually provides a lot of the meta data required to understand on a modular level what bundles are required to satisfy any given resolution state. This is one of the first things we implemented back in the day when we first started building OSGi applications here at Paremus. Given the set of bundles installed in a framework it is possible to calculate which bundles are actively wired at any moment and from this we can see which bundles are no longer needed and automatically uninstall them. This solves a lot of the problems related to the standard DLL hell question when uninstalling applications – “Do you wish to uninstall this shared library?” to which I usually think “I don’t know – you tell me – is it used by anyone?”.

With Nimble we have extended the dependency model to cope with not just resolution time dependencies, but also activation time dependencies. In order to activate the simple web app we simply ask Nimble to add the bundle in an active state. Nimble then performs a resolution against its repository meta data. The meta data tells us that to usefully activate a web application we need not only a resolved API bundle, but also an active web container bundle. What’s more Nimble can garbage collect these entities when the active bundle is removed.

I’ll leave it for another post to describe how this works in detail, for now let’s just look at how this works in our real world example. Let’s launch an interactive session:

$ /opt/nimble/bin/posh
Internal Paremus Nimble Developer License,
expires Fri Dec 31 23:59:59 GMT 2010.
Welcome to Paremus Nimble!
Type 'help' for help.
[feynman.local.0]%

At this point we have an empty container that contains the minimal set of bundles needed to support Nimble. We can see how this looks by using the “lsb” command to list the installed bundles:

[feynman.local.0]% lsb
0  ACTIVE    org.eclipse.osgi:3.5.1.R35x_v20090827
1  ACTIVE    com.paremus.posh.runtime:1.0.10
2  ACTIVE    com.paremus.posh.shell:1.0.10
3  ACTIVE    com.paremus.posh.readline:1.0.10
4  RESOLVED  com.paremus.util.types:1.0.10
5  ACTIVE    com.paremus.nimble.core:1.0.10
6  ACTIVE    com.paremus.nimble.repos:1.0.10
7  ACTIVE    com.paremus.nimble.cli:1.0.10
8  ACTIVE    com.paremus.util.cmds:1.0.10

To install the simple web app the first thing we need to do is to load the springdm repository index:

[feynman.local.1]% repos -l springdm

This index file specifies the meta information that allows Nimble to resolve the set of bundles that are required to close the dependency graph for bundles in the resolved or activated states. In this case we’ve used a well known name which is an alias for the following file:

http://repository.paremus.com/springdm-1.2/index-nim.xml

I’ll also leave it to another blog post to show how you can create your own indexes (or the impatient can find out here).

Now to install the simple web app. Firstly let’s look at the trivial case where we simply want to install our simple web app in the resolved state.

[feynman.local.1]% add org.springframework.osgi.samples.simplewebapp@resolved

This one line command prompts Nimble to close the set of dependencies needed to resolve the simple web application which results in the following bundles being installed in the framework:

[feynman.local.0]% lsb
0  ACTIVE    org.eclipse.osgi:3.5.1.R35x_v20090827
1  ACTIVE    com.paremus.posh.runtime:1.0.10
2  ACTIVE    com.paremus.posh.shell:1.0.10
3  ACTIVE    com.paremus.posh.readline:1.0.10
4  RESOLVED  com.paremus.util.types:1.0.10
5  ACTIVE    com.paremus.nimble.core:1.0.10
6  ACTIVE    com.paremus.nimble.repos:1.0.10
7  ACTIVE    com.paremus.nimble.cli:1.0.10
8  ACTIVE    com.paremus.util.cmds:1.0.10
9  RESOLVED  javax.servlet:2.5.0.v200806031605
10  RESOLVED  org.springframework.osgi.jsp-api.osgi:2.0.0.SNAPSHOT
11  RESOLVED  org.springframework.osgi.jstl.osgi:1.1.2.SNAPSHOT
12  RESOLVED  org.springframework.osgi.samples.simplewebapp:0.0.0

Bundles 9, 10, 11 and 12 have been installed and resolved in the framework. This is not a functional application as it will not provide any actual services, but it shows the direct dependencies of the bundle can be installed within the framework. In this state we can load classes from the simple web app. If we now remove the resolved application the root web app bundle and all dependent bundles are garbage collected:

[feynman.local.1]% remove org.springframework.osgi.samples.simplewebapp@resolved
[feynman.local.0]% lsb
0  ACTIVE    org.eclipse.osgi:3.5.1.R35x_v20090827
1  ACTIVE    com.paremus.posh.runtime:1.0.10
2  ACTIVE    com.paremus.posh.shell:1.0.10
3  ACTIVE    com.paremus.posh.readline:1.0.10
4  RESOLVED  com.paremus.util.types:1.0.10
5  ACTIVE    com.paremus.nimble.core:1.0.10
6  ACTIVE    com.paremus.nimble.repos:1.0.10
7  ACTIVE    com.paremus.nimble.cli:1.0.10
8  ACTIVE    com.paremus.util.cmds:1.0.10

Now let’s change the scenario and add the web app as active:

[feynman.local.1]% add org.springframework.osgi.samples.simplewebapp@active
[feynman.local.1]% lsb
0  ACTIVE    org.eclipse.osgi:3.5.1.R35x_v20090827
1  ACTIVE    com.paremus.posh.runtime:1.0.10
2  ACTIVE    com.paremus.posh.shell:1.0.10
3  ACTIVE    com.paremus.posh.readline:1.0.10
4  RESOLVED  com.paremus.util.types:1.0.10
5  ACTIVE    com.paremus.nimble.core:1.0.10
6  ACTIVE    com.paremus.nimble.repos:1.0.10
7  ACTIVE    com.paremus.nimble.cli:1.0.10
8  ACTIVE    com.paremus.util.cmds:1.0.10
13  RESOLVED  javax.servlet:2.5.0.v200806031605
14  RESOLVED  com.springsource.slf4j.api:1.5.6
15  RESOLVED  com.springsource.slf4j.nop:1.5.6
16  RESOLVED  com.springsource.net.sf.cglib:2.1.3
17  RESOLVED  com.springsource.edu.emory.mathcs.backport:3.1.0
18  RESOLVED  org.springframework.osgi.log4j.osgi:1.2.15.SNAPSHOT
19  RESOLVED  com.springsource.org.aopalliance:1.0.0
20  RESOLVED  org.springframework.osgi.jsp-api.osgi:2.0.0.SNAPSHOT
21  RESOLVED  com.springsource.slf4j.org.apache.commons.logging:1.5.6
22  RESOLVED  osgi.cmpn:4.2.0.200908310645
23  RESOLVED  org.mortbay.jetty.util:6.1.15.v200905182336
24  RESOLVED  org.springframework.osgi.jstl.osgi:1.1.2.SNAPSHOT
25  RESOLVED  org.springframework.core:2.5.6.A
26  RESOLVED  org.springframework.osgi.commons-el.osgi:1.0.0.SNAPSHOT
27  ACTIVE    org.springframework.osgi.samples.simplewebapp:0.0.0
28  RESOLVED  org.springframework.beans:2.5.6.A
29  RESOLVED  org.springframework.osgi.io:1.2.0
30  RESOLVED  org.springframework.osgi.jasper.osgi:5.5.23.SNAPSHOT
31  RESOLVED  org.springframework.aop:2.5.6.A
32  RESOLVED  org.springframework.osgi.catalina.osgi:5.5.23.SNAPSHOT
33  RESOLVED  org.mortbay.jetty.server:6.1.15.v200905151201
34  RESOLVED  org.springframework.context:2.5.6.A
35  ACTIVE    org.springframework.osgi.catalina.start.osgi:1.0.0
36  RESOLVED  org.springframework.osgi.core:1.2.0
37  RESOLVED  org.springframework.web:2.5.6.A
38  RESOLVED  org.springframework.osgi.web:1.2.0
39  ACTIVE    org.springframework.osgi.web.extender:1.2.0

Here we see that a lot more bundles have been installed and critically bundle 35 and 39 have been activated alongside bundle 27.

If you’re running this yourself you can now browse the web app at:

http://localhost:8080/simple-web-app/

To complete the picture if we remove the active simple web app the bundle and all its dependencies are garbage collected and we return to the “resting state”:

[feynman.local.1]% remove org.springframework.osgi.samples.simplewebapp@active
[feynman.local.1]% lsb
0  ACTIVE    org.eclipse.osgi:3.5.1.R35x_v20090827
1  ACTIVE    com.paremus.posh.runtime:1.0.10
2  ACTIVE    com.paremus.posh.shell:1.0.10
3  ACTIVE    com.paremus.posh.readline:1.0.10
4  RESOLVED  com.paremus.util.types:1.0.10
5  ACTIVE    com.paremus.nimble.core:1.0.10
6  ACTIVE    com.paremus.nimble.repos:1.0.10
7  ACTIVE    com.paremus.nimble.cli:1.0.10
8  ACTIVE    com.paremus.util.cmds:1.0.10

This functionality greatly simplifies day to day development in that you no longer need to actively manage your bundle dependencies and instead can get on with the actual task of developing your application. The garbage collection capabilities also make this suitable for long running server environments. In conclusion Nimble provides you with a really simple and easy to use way to manage not just module level dependencies but also active bundle dependencies.

  • Print
  • LinkedIn
  • Facebook
  • del.icio.us
  • DZone
  • Google Bookmarks
  • Yahoo! Bookmarks

Nimble at OSGi DevCon

I'm speaking at EclipseCon 2010I’ll be discussing Nimble at the OSGi DevCon part of this year’s EclipseCon in a talk called Next generation OSGi shells.

If the promise of OSGi’s modular approach appeals to you, but you’ve found that day to day use of OSGi feels heavyweight and awkward, then you have my sympathy and understanding. Until recently working with an OSGi framework has felt the way that working with Unix would without a shell like Bash. It doesn’t have to be that way: a new generation of OSGi shells that enables a more agile and productive OSGi experience is now emerging.

Please come along to the talk where, using Nimble as an example, I’ll show you how much better a powerful, scriptable, shell with an integrated dependency resolver can make working with OSGi.

  • Print
  • LinkedIn
  • Facebook
  • del.icio.us
  • DZone
  • Google Bookmarks
  • Yahoo! Bookmarks

Nimble interview with DZone

I recently gave a brief interview about Nimble to Mitch Pronschinske from DZone. I think it does a good job of explaining what Nimble is all about and where it’s going.

  • Print
  • LinkedIn
  • Facebook
  • del.icio.us
  • DZone
  • Google Bookmarks
  • Yahoo! Bookmarks

Nimble’s OSGi Config Admin Client

If you’ve adopted OSGi and are using it to factor your applications into modular dynamic systems, then you are probably interested in configuring them in a similarly modular and dynamic way. The OSGi Configuration Admin service is a widely used standard for doing just this.

This post introduces Nimble’s command line interface to the Config Admin service and shows how it can be used to configure an OSGi HTTP Service implementation.

To get started you’ll need to download Nimble (it’s only 55k). On Linux/OS X you can start it using:

$ bin/posh

For windows use:

% bin\posh.bat

This introduction was tested against Nimble 1.0.8. To make sure you’re up to date just type:

% version

at the Nimble prompt, and follow the instructions to get Nimble to self-update if necessary.

The Config Admin service manages property maps identified by Persistent IDs, or PIDs. These come in two varieties, singleton PIDs and factory PIDs. Singleton PIDs are used to configure a single underlying entity, often some sort of service, whereas factory PIDs are used to create and configure multiple instances of some configuration target. Under the covers these PIDs are handled by implementations of ManagedService and ManagedServiceFactory. For more details take a look at this book, or at the spec itself, which like most OSGi specs, is surprisingly readable.

The three commands Nimble provides to help you interact with the Config Admin service are:

  • pids: this lists registered PIDs and their property maps
  • setpid: this initializes or updates a PID’s properties
  • delpids: this can be used to delete one or more PIDs

For detailed usage information invoke the commands with -?.

Let’s work through an OSGi HTTP Service based example. To provision the default HTTP and Config Admin service implementations run:

% add osgi.service/http
% add osgi.service/cm

If you want to see the bundles that this installed just type lsb.

Next we can use the pids command to find out what’s available to configure:

% pids -r
15 org.eclipse.equinox.http.jetty:2.0.0.v20090520-1800
15 F: org.eclipse.equinox.http.jetty.config

All this does is list the PIDs and factory PIDs of the ManagedService and ManagedServiceFactory implementations in the OSGi service registry. In this case there’s only one: org.eclipse.equinox.http.jetty.config, a factory PID, published by the Equinox’s Jetty based implementation of the OSGi HTTP service. This is the one we’ll be using.

Note that as with all Nimble commands pids is installed on demand. You can unload it using load -u pids.

The property which controls the HTTP port is called http.port, so we can create a new HTTP listener at port 18080 by using setpid as follows:

% setpid -f org.eclipse.equinox.http.jetty.config http.port=18080

To confirm that the PID has been set let’s list all PIDs using the pids command:

% pids
org.eclipse.equinox.http.jetty.config/org.eclipse.equinox.http.jetty.config-1263568645437-0 {http.port=1234}

The format here is factory-pid/instance-pid {properties}

To test the service I’m going to use the registerServlet bundle from one of the code snippets on Peter Krien’s site. This registers a simple Hello World servlet with the HTTP service. This bundle is in Nimble’s examples repository. Let’s load this repository and ask Nimble to provision and start the registerServlet bundle.

% repos -l examples
% add osgi.active.bundle/aQute.registerservlet

To see the result go to http://localhost:18080/hello

You can change the port by updating the config as follows. Note that the key used here is the instance PID which may be different for you.

% setpid org.eclipse.equinox.http.jetty.config-1263569144831-0 http.port=18081

The servlet will now be listening at http://localhost:18081/hello. To remove the listener altogether simply remove the PID using:

% delpids org.eclipse.equinox.http.jetty.config-1263569144831-0

You can use pids again to confirm that the PID has been removed, and visit http://localhost:18081/hello to verify that the listener has been removed.

I should end by noting that the PID used here is specific to Equinox’s Jetty based implementation of the OSGi HTTP service. Most implementations of this service are configurable using the Config Admin service, but there aren’t any standards for this.

  • Print
  • LinkedIn
  • Facebook
  • del.icio.us
  • DZone
  • Google Bookmarks
  • Yahoo! Bookmarks

Nimble Post Round-Up

David Savage has written some great introductory Nimble posts. In case you missed them here’s a quick round-up:

  • Print
  • LinkedIn
  • Facebook
  • del.icio.us
  • DZone
  • Google Bookmarks
  • Yahoo! Bookmarks

Jlook: yet another manifest grep tool?

Anyone who has worked with OSGi will be familiar with failed bundle resolutions, such as this:

$ java -jar bin/felix.jar 

Welcome to Felix
================

-> install file:/Users/derek/devl/lib/nimble/com.paremus.util.jlook-1.0.5.SNAPSHOT.jar
Bundle ID: 5
-> start 5
org.osgi.framework.BundleException: Unresolved constraint in bundle com.paremus.util.jlook [5]: package; (&(package=org.osgi.service.command)(version>=1.4.0))
->

In this case the cause of the problem is easily attributed to the attempted start of the jlook bundle. The resolution is not so obvious, unless you are familiar with the unresolved package and know how to resolve it.

Often developers will resort to installing likely bundles to see if they resolve the problem, or perhaps try to identify likely bundles by examining their manifests.

In an ideal world, your universe of available bundles would all be indexed, and rather than attempting to start them using  a low-level tool such as the Felix ‘start’ command, you would use a more powerful OSGi resolver, such as Nimble, which would automatically resolve missing dependencies.

However, there are still times when it is useful to be able to search your local file system for appropriate bundles, without having to configure any resolver.

I wanted a simple command-line tool that could search bundle manifests (and contents) using regular expressions, so I created ‘jlook’:

$ jlook --help
jlook searches archive files and prints lines that match PATTERN in the
contents or manifest. It also searches source files and can recursively
search embedded archives.

Usage: jlook [-OPTION].. PATTERN [PATH | CLASSPATH]..
[truncated -- full usage]

So to find the missing dependency above, I could use:

$ jlook Export-Package:.*@org.osgi.service.command
lib/nimble/com.paremus.posh.runtime-1.0.5.SNAPSHOT.jar:
 Export-Package: com.paremus.posh.api;uses:="org.osgi.service.command";version="1.0.5.SNAPSHOT"
 org.osgi.service.command;version="1.4.2"
 org.osgi.service.threadio;version="1.4.2"
1 match 1 path

This recursively searches all jars below the current directory, looking for any that export the missing package.

The ‘@’ in the PATTERN, is used to force it to match at the beginning, or after a comma, as if we look at the full Export-Package header of this bundle, we see that the target package is not the first export:

$ jlook Export-Package: lib/nimble/com.paremus.posh.runtime-1.0.5.SNAPSHOT.jar
lib/nimble/com.paremus.posh.runtime-1.0.5.SNAPSHOT.jar:
 Export-Package: com.paremus.posh.api;uses:="org.osgi.service.command";version="1.0.5.SNAPSHOT",org.osgi.service.command;version="1.4.2",org.osgi.service.threadio;version="1.4.2"
1 match 1 path

Back to felix, we can try installing this bundle:

-> install file:/Users/derek/devl/lib/nimble/com.paremus.posh.runtime-1.0.5.SNAPSHOT.jar
Bundle ID: 6
-> start 5
org.osgi.framework.BundleException: Unresolved constraint in bundle com.paremus.util.jlook [5]: package; (package=scala)

Unsurprisingly, this just results in the next unresolved package to diagnose :-(

The observant amongst you will have noticed that the next unresolved package is ‘scala’, indicating that ‘jlook’ itself has a dependency on ‘scala’. This is not surprising, as I chose to write ‘jlook’ in Scala, so that I could learn a new language.

This blog is not really about the ‘jlook’ utility itself, but rather about how to deploy responsive Java command-line utilities.

(However, ‘jlook’ is included in the Nimble download if you think you may find it useful; type ‘jlook –help’ for full usage information).

I initially packaged ‘jlook’ as a simple scala script, prefixed with ‘#!/usr/bin/env scala’. This meant it could be deployed with the ease of copying a shell script, but only if the recipient had scala installed.

It was also quite slow, because a JVM had to be created to run ‘scala’ which then had to compile the script, before it could be executed. Start-up time was several seconds, which is unacceptable for a utility that typically runs in less that 1 second.

I next packaged ‘jlook’ as an executable jar. This performed faster, but either had a dependency on the scala library jar, or I had to embed the scala library greatly increasing the size of the utility.

To reduce the size of the jlook jar, I tried using pack200 to unpack the scala library. This worked OK, but the additional execution time for the unpack was significant.

I therefore tried a different approach. I converted jlook.jar into regular OSGi bundle and registered a simple RFC-147 command-line interface and loaded it into Nimble:

$ posh
________________________________________
Welcome to Paremus Nimble!
Type 'help' for help.
[denzil.0]% jlook Export-Package:.*@org.osgi.service.command
*nimble-ext/scala.library_2.7.5.final.jar                         00:06   4.7Mb
*nimble/com.paremus.util.jlook-1.0.4.jar                          00:00    61Kb
lib/nimble/com.paremus.posh.runtime-1.0.5.SNAPSHOT.jar:
 Export-Package: com.paremus.posh.api;uses:="org.osgi.service.command";version="1.0.5.SNAPSHOT"
 org.osgi.service.command;version="1.4.2"
 org.osgi.service.threadio;version="1.4.2"
1 match 1 path

Notice that Nimble automatically resolves and downloads the scala library dependency.

However, this is less convenient, and probably slower, to use than a simple executable jar:

$ java -jar jlook.jar

Happily, both these issues can easily be resolved:

Posh, like bash, can execute commands given after ‘-c’ on the command line:

$ posh -c jlook Export-Package:.*@org.osgi.service.command

and this could be encapsulated in a ‘jlook’ script:

#!/usr/bin/env posh
jlook $*

This still hasn’t resolved the issue concerning the start-up time for the JVM and loading the Scala library, compared to the execution time of the utility.

Those familiar with Scala will know that the scala compiler (scalac) also suffers from start-up delay due to JVM loading and a solution is provided in the form of the fast scala compiler (fsc).

The first time ‘fsc’ is invoked it starts a JVM in the background. Subsequent invocations of ‘fsc’ simply communicate with the already loaded JVM, and thus run without any start-up delay.

We can use the same trick with a background Nimble instace running an ssh daemon and using an ssh client to execute commands in it.

First you need to create ~/.posh/authorized_keys so that ssh clients can connect without requiring a password, as described here.

Then start a background Nimble instance; we use –instance=-1 so we know which port sshd will listen on:

$ posh --instance=-1 -sc sshd &
[1] 3511
sshd: listening on port 8902

(The -s option to posh tells it not to shutdown after running the specified command).

Now you can invoke jlook (or any other OSGi-based command) using ssh:

$ ssh -p8902 localhost jlook Export-Package:.*@org.osgi.service.command
lib/nimble/com.paremus.posh.runtime-1.0.5.SNAPSHOT.jar:
 Export-Package: com.paremus.posh.api;uses:="org.osgi.service.command";version="1.0.5.SNAPSHOT"
 org.osgi.service.command;version="1.4.2"
 org.osgi.service.threadio;version="1.4.2"
1 match 1 path

Nimble contains a trivial script ‘bin/fpc’ for ‘fast posh client’, which simply invokes ssh, so the above could be also run using:

$ fpc jlook Export-Package:.*@org.osgi.service.command

This is not perfect, as it does not pass the current directory into the running Nimble instance, but it does run in about one third the time of the original executable jar solution:

$ time fpc jlook Export-Package:.*@org.osgi.service.command
real    0m0.125s
user    0m0.010s
sys     0m0.008s

$ time jlook Export-Package:.*@org.osgi.service.command

real    0m0.358s
user    0m0.302s
sys     0m0.041s

Read the rest of this entry »

  • Print
  • LinkedIn
  • Facebook
  • del.icio.us
  • DZone
  • Google Bookmarks
  • Yahoo! Bookmarks

Announcing Paremus Nimble 1.0

Welcome to the Paremus Nimble Blog.

We’re pleased to announce that Paremus Nimble 1.0 is now available for Download.

Paremus Nimble combines a feature rich OSGi shell with a powerful dependency resolver. We think it’s the most productive way to manage and interact with OSGi Frameworks that’s available today.

Whether you’re new to OSGi, an experienced OSGi developer, or an administrator charged with deploying and managing OSGi applications, Paremus Nimble will help you get things done.

We’ll be using this blog to provide hints and tips about getting the most out of Paremus Nimble, and to announce new features, so please subscribe to our feed if you want to stay informed.

You can get started straight away by downloading Paremus Nimble, reading the documentation and working through the tutorials.

We hope you find it useful. Please let us know what you think.

  • Print
  • LinkedIn
  • Facebook
  • del.icio.us
  • DZone
  • Google Bookmarks
  • Yahoo! Bookmarks