Sunday, December 31, 2006

Code in multiple modules

I am currently working to overcome a limitation that Cohatoe has in its current state: what if your Haskell code makes up more than one module? In this case, you have multiple object files, and of course you must provide all of them to Cohatoe. But you can only declare one of them - so how will the others be available at runtime? (Needless to say, you certainly don't want to have to declare each of them separately, too ;-)

hs-plugins, which is used by Cohatoe on the Haskell side to load the object code, is able to automatically preload module dependencies. Therefore, as long as the module dependencies (that is, their object files) are available to it on some search path, this should be fine. However, the path to the object files to be loaded is determined by Cohatoe only when the object code is registered. They are presumably (but not necessarily, see below) in some subfolder of the plugins/ folder in the Eclipse installation. Therefore, some more logic is needed in Cohatoe to determine these paths and send them to hs-plugins.

That's not too difficult, but here are some complications:

  • an Eclipse plugin may run out of a .jar archive
  • you might want to have several different folders containing object code
  • you might want to have a lot of object files and they should be packaged into a single archive file
The first point is the most pressing. When Eclipse plugins are exported and deployed to an end user's Eclipse installation, they mostly live in .jar archives. The object files will then be packed into the archives and can't be automatically found by Cohatoe or hs-plugins. For the .o files that have been declared in the plugin.xml, and their corresponding .hi files, Cohatoe uses a trick before sending their path to hs-plugins: it tells the Eclipse runtime that it needs them as 'local file'. The Eclipse runtime then checks if the plugin is currently running out of a file system folder (which is the case both at debug time when the plugin is physically a plugin project in the workspace, and for deployed plugins when they are deployed with the unpack="true" flag in the feature.xml of the feature that provides the plugin). If so, it gives us the file system location. If not, this means the plugin runs out of a jar file, and then the Eclipse runtime would extract it into a cache location on the file system and give us the path to that location.

Now if we make it a convention in Cohatoe that all object files in the same folder as a declared one are also automatically available to hs-plugins, it is not sufficient to just send the path of the folder containing the declared object files, because that folder is not in all cases the folder which hs-plugins actually uses. We have to make the 'as local' conversion for that folder too, and not just for it, but also for all its contents.

The other two points listed above are actually new features that I could add while I'm at it anyway. hs-plugins provides also a functionality to load libraries of object code.

I think I will add, in one of the next versions of Cohatoe, a possibility to declare such packages, which can be used as dependencies. I'd like to give that some thought, though, because it seems to me that just giving a file list is a somewhat short way to address the problem. If you could say
<extension point="de.leiffrenzel.cohatoe.server.core.haskellFunctions">
<haskellfunction interface="..."
implementation="..."
objectcode="...">
<objectcodepackage path="..." />
</haskellfunction>
</extension>

that would solve the problem, but it would force you to provide the binaries for the package in every Eclipse plugin that uses them. I would be interested in having some mechanism to address this, so that you can contribute a package independently, and just refer to it. It would then look like this:
<extension point="de.leiffrenzel.cohatoe.server.core.haskellFunctions">
<haskellfunction interface="..."
implementation="..."
objectcode="...">
<depends id="myLib" />
</haskellfunction>
<objectcodepackage id="myLib" path="..." />
</extension>
The advantage of this would of course be that any haskellFunction declaration could just declare a package as a dependency by its id, even if it was actually provided by a different Eclipse plugin.

Friday, December 29, 2006

How to tell what's going on

In my last two posts (Cohatoe example walkthrough, part I and part II) I have explained the basics of how to contribute some Haskell code in an Eclipse plugin so that it can be called from the Java code in that Eclipse plugin (and other Eclipse plugins that depend on it). If you have experimented a little with this, then you might have encountered situations in which something went wrong and you could not immediately see what - so what can you do for diagnostics?

One helpful instrument is Eclipse's built-in tracing. It is not a full-fledged logging framework, but it can come in helpful. For Cohatoe, I have implemented tracing for the inits and all calls to the Cohatoe server singleton. The server singleton basically manages all registered Haskell functions and delegates calls to them to the native object code. If you switch on the tracing for the Cohatoe server, you will be able to detect problems of three different sorts:

  • declaration problems in the plugin manifest file, plugin.xml (e.g. you have a typo in the declaration of your object files)
  • initialization problems that occcur when the server is started
  • problems during calls to the native code (e.g. load errors that occured when object code could not be loaded or exceptions on the Haskell side, like calls to the Prelude.error function)
Especially the latter sort of problem is still very much work in progress. In later versions of Cohatoe, exceptions that are fired on the Haskell side will be transferred to the Java side and re-raised there, where the caller to the evaluate() method on the server singleton will be responsible for handling them. (Of course, it is even better the Haskell code would already handle them itself :-). But that mechanism is not yet in place as of Cohatoe 0.1, therefore you will only see problems on the Haskell side when you switch on tracing.

And here's how you do it: Assuming that you have an Eclipse plugin project, and you have at least once run it as an Eclipse application from the Run or Debug menu, then you have a so-called launch configuration for that launch. Whether tracing is enabled is a property of that launch configuration, and you can switch it on or off on the launch configuration dialog. Select Run > Run ... from the main menu. On the dialog that appears, select the Eclipse application launch configuration for your launch. Here is how the dialog looks like in my examples workspace:


Switch to the Tracing tab. Here you can see the tracing options from all plugins which support tracing. In order to enable a particular tracing option, you have first to globally enable the tracing functionality - that's the Enable tracing for the selected plugins checkbox right on top of the Tracing tab. Then you must select the plugins themselves, in our case that is the Cohatoe server plugin, which appears in the list with its plugin id, namely: de.leiffrenzel.cohatoe.server.core. Finally, check the particular tracing option that you want to enable - this is logs in our case, which is the only option that Cohatoe currently provides.

Note that it is not enough to just check the option, you must also check the plugin! If you check the option and not the plugin, you won't see anything. This is how the tab should look like now:

When you run your launch configuration now and (in the second Eclipse instance that was fired up) trigger some action that calls a Haskell function, you will see some output in the console that obviously came from the Cohatoe server. Here is what I got:
[Cohatoe Server] No entry 'inexistent.obj' in bundle 'de.leiffrenzel.cohatoe.server.core.test'.
[Cohatoe Server] Initializing server.
[Cohatoe Server] Setting port to 36522
[Cohatoe Server] Setting executable to C:\workspaces\cohatoe\de.leiffrenzel.cohatoe.server.core.win32.win32.x86\server\haskellserver.exe
[Cohatoe Server] Evaluating de.leiffrenzel.cohatoe.example.core.functions.IHelloFunction
[Cohatoe Server] C:\workspaces\cohatoe\de.leiffrenzel.cohatoe.example\os\win32\x86\obj\Hello.o
[Cohatoe Server] > Server process ready
[Cohatoe Server] Response was: Hi, thanks for saying 'Hi Haskell!' to me.§
Now what does all this tell us?

The first line in this tracing output complains about a non-existing object file. This comes from the initialization phase of the Cohatoe server. The server, in that phase, reads all extensions to the haskellFunctions extension point. In these extensions, the object code for the Haskell functions is declared, and the server needs to locate the object files so that it knows (on the Haskell side of things) where to get them. The line in our trace states that one object file, named 'inexistent.obj', was not actually found, and that this file was declared by the bundle (for which read 'Eclipse plugin') with the id de.leiffrenzel.cohatoe.server.core.test. In this case, that's fine, because that plugin is the one which contains my unit tests for Cohatoe, and the object file declaration declared deliberately a non-existing object file (because I wanted to test the behaviour of the server in such a case). But if you see such a line in the trace, that indicates that you have either forgotten to provide an object file, or you have mistyped its declaration in the plugin.xml.

The next 3 lines tell us a bit about the proceedings inside Cohatoe. The server singleton starts its init routine, determines a free port on the local host, and then fires up a native executable in an external process which listens on that port. That executable knows how to load and execute Haskell functions from object code (using hs-plugins), and is the Haskell side of the Cohatoe server. I will add details about all this in some later posts.

After these inits, we see now the tracing output for all the calls to Haskell functions. You see the fully qualified name of a Java interface and the file name of an object file. Cohatoe uses Java interfaces as keys for Haskell functions, namely the interfaces that have been declared together with the object code in the plugin.xml. You can think of these Java interfaces as the placeholders or representatives of your Haskell functions in the Eclipse-plugin world. For every Haskell function that is called, you will see the interface name, object file name, and the response from the Haskell side in the trace.

You will have noticed that there was another line in between these function call tracing outputs: "> Server process ready". The '>' character in front of a tracing line indicates that this line comes from the standard output or standard error stream of the native server process. In this case, that's only a friendly message that the server is ready. (It is read asynchronously with low priority on the Java side, and therefore appears sometimes a little later in the trace.) But if, for some reason, your Haskell code causes the server process to write to the error stream, you'll see that output here.

Finally, when you shut down your debugging Eclipse, you will see some more output:
[Cohatoe Server] Disposing server.
[Cohatoe Server] Finished out and err of the server process.
which obviously informs us about the cleanups which go on at that time.

Thursday, December 28, 2006

Simple example walkthrough II

In the last post, we have prepared an Eclipse plugin project for a Hello-World Cohatoe plugin. Lets have a look at the Haskell side now.

The Haskell code that we are going to contribute will be loaded at runtime by the Cohatoe server core plugin, which in turn delegates the work to hs-plugins. hs-plugins is a library by Don Stewart written for exactly this purpose: dynamically locating compiled Haskell code at runtime, loading it and executing it, and all this in a typesafe manner. (hs-plugins can also load and on-the-fly-compile Haskell source code, but that's not a functionality I'm using.)

Haskell code that can be loaded this way has to fulfill a few conditions: it must provide, by convention, a symbol called 'resource', which is a record and has a field, pluginMain :: [String] -> [String], which is the entry point function that Cohatoe will call. (Think of it as a fancy sort of main function.) Therefore, all Haskell code that is contributed via Cohatoe will include something like this after the module declaration:

-- We must import Cohatoe.API and implement resource 
--
so that this code can be dynamically loaded
import Cohatoe.API

resource = plugin {
pluginMain = sayHello
}

-- our code starts here
-- ...
sayHello = ...
As you can see, this needs to import the Cohatoe API. Let's make sure that we have that API available.

Remember that you downloaded a second file, cohatoe-api_0.1.0.zip? Unzip that archive now and open a command prompt in the cohatoe-api/ folder:
> cd C:\cohatoetest\cohatoe-api
Now run the following three commands:
> runhaskell Setup.hs configure
> runhaskell Setup.hs build
> runhaskell Setup.hs install
This compiles the cohatoe-api package (which is very small and simple), installs it and registers it as a package for GHC.

Now that we have this in place, we can compile the Haskell code that we want to call from our Eclipse plugin. Create a file, e.g. Example.hs, with this content:
module Example where

import Cohatoe.API

resource = plugin {
pluginMain = sayHello
}

sayHello :: [String] -> [String]
sayHello _ = ["Hello from Haskell"]
and compile it like so:
> ghc -c -package cohatoe-api Example.hs
Put the resulting Example.o and Example.hi files into a folder called os/win32/x86/obj/ in your plugin:

(You might have guessed that there is a naming convention for OS platforms at work here in this particular name for the folder. You're right. I'll elaborate on this in a later post.)

The next thing to do now that we have our Haskell object code, is to register it with Cohatoe as an executable function. This is done via Eclipse's extension mechanism. Cohatoe provides an extension point, and we'll provide an Eclipse extension to that extension point now.

Open the Plugin Manifest Editor by double-clicking the file plugin.xml in the plugin project. First switch to the Dependencies tab. In the Required section, press Add and select de.leiffrenzel.cohatoe.server.core. By doing this, we make our new plugin dependent on the Cohatoe server core plugin, which we must do because we want to use its extension point. Now switch to the Extensions tab and press Add. Choose the haskellFunctions extension point (actually, it appears under its fully qualified name, which is de.leiffrenzel.cohatoe.server.core.haskellFunctions), then right-click it and select New > haskellFunction. Here you get a form where you can fill in the details for the Haskell Function extension that we are just creating:

As you can see, we have basically to specify three things: a Java interface, a Java class that implements that interface, and the location of the object code of our Haskell function. For the latter, just specify the .o-file, i.e. $os$/obj/Example.o. We only need to specify the location of the .o-file, the location of the corresponding .hi-file is automatically determined by Cohatoe from that location. (But the .hi-file must be there in the same folder as the .o-file, otherwise hs-plugins will not be able to load the code.)

And what about the interface and implementation class? These are registered together with the object code so that Java code from Eclipse plugins can call our Haskell function. That Java code will always only see the interface that you specify here. (We will see in a moment what exactly will be visible for any Java code that wants to call our Haskell function.) As a provider of a Haskell function, however, you also have to provide an implementation class for that interface. This is necessary for Cohatoe itself. It will instantiate that implementation class and pass it on to the clients that want to call the Haskell function (they will only see the specified interface as the type of the object that they get). The primary role of this implementation class is that it gives the provider of a Haskell function a hook to implement data marshalling. In our case, there is no data to be transported from the Java to the Haskell side, and only a minimum from the Haskell side to the Java side - thus the implementation class will not have to do much.

In the screenshot above you see that I have specified cohatoetest.IExampleFunction as the name of the interface and cohatoetest.ExampleFunction as the name of the implementation class. These two types do not yet exist. You can easily create them by clicking the link before the type name in the Manifest editor.

After you have created the source files, their content should look like this:
// cohatoetest.IExampleFunction:
package cohatoetest;

public interface IExampleFunction {
String sayHello();
}

// cohatoetest.ExampleFunction:
package cohatoetest;

import de.leiffrenzel.cohatoe.server.core.CohatoeServer;

public class ExampleFunction implements IExampleFunction {

public String sayHello() {
CohatoeServer server = CohatoeServer.getInstance();
String[] retVal = server.evaluate( IExampleFunction.class, new String[ 0 ] );

String result = "[Got no answer from Haskell.]";
if( retVal != null && retVal.length == 1 ) {
result = retVal[ 0 ];
}
return result;
}
}
The only interesting thing here is the implementation of sayHello() in our implementation class. You see that there is a singleton object of type CohatoeServer, to which the call to our function is delegated. The Cohatoe server singleton knows about all registered Haskell functions, and it knows how to forward calls to them. Since it is a singleton and shared between all plugins in the running Eclipse instance, we must pass a key to it so that it knows which Haskell function we actually want to evaluate. The Cohatoe server uses the class objects of the interfaces under which functions are registered as keys; you can see that we pass IExampleFunction.class as a key here. We also pass an empty string array as parameter list. The return value (if everything goes fine) is a one-element String array that holds the single string returned by our Haskell function.

(Note that this is a really, really trivial example. What you usually want to do in this implementation is to accept objects of all types as parameters and convert them into a string representation for marshalling, then convert back from strings what you receive as result from the call to the Haskell function. This is deliberately designed so that the interface for the Java code that calls the Haskell function is typesafe. So for example, instead of accepting a String for a file name, you would use a java.io.File object in the signature of your interface function, and only convert it into a string inside the implementation. Similarly, you should not return the bare strings that you get from the Haskell side, but instead convert them into objects of a type that really represent your data.)

This is all - we have now made our Haskell function available to any Java code in any Eclipse plugin that wants to call it. (Strictly speaking, we have made it available to any Eclipse plugin that depends on our own plugin.) To demonstrate how one would call it, we'll put such a call into the run() method of our 'Hello-World' action. Remember, this was in the class cohatoetest.actions.SampleAction. Modify the run() method so that it looks like this:
  public void run( final IAction action ) {
CohatoeServer server = CohatoeServer.getInstance();
Object fun = server.createFunction( IExampleFunction.class );
if( fun instanceof IExampleFunction ) {
IExampleFunction helloFunction = ( IExampleFunction )fun;
MessageDialog.openInformation(
window.getShell(),
"Cohatoetest Plug-in",
helloFunction.sayHello() );
}
}
You see again that we use the type of our declared interface as key, this time to obtain an object that (sort of) represents our Haskell function, so that we can call it via that object. The rule is that you can get an object complying to the interface that you pass in as key, or null (if no function is registered for that interface). Then you can cast and pass your function arguments in a typesafe manner.

Voila, if you run this, you'll see a friendly message from the Haskell side :-)

In my next post, I'll explain some troubleshooting aids that come with Cohatoe - just in case you encountered any problems on the way.

Simple example walkthrough I

In my last post I have pointed to the first preview of Cohatoe - so, how does one use it?

First of all a disclaimer: since this is a preview version, the usage of some of the APIs may change; this is so experimental that things may even change repeatedly. Of course I will make a notice of all changes to the APIs here, but before version 1.0 'API' just means 'candidate API'. The bright side of this is of course that it is also open for suggestions, so if you think the public interfaces of Cohatoe could be improved in a particular respect, please let me know.

Having said that, let me walk you through a very simple example usage of Cohatoe. In this first post, I will cover the basic installation and plugin development part of this. In a follow-up post we will then get to the Cohatoe-specific stuff (compiling the Haskell code, creating interfaces and classes for Eclipse).

As prerequisites, you need a Java 5 JRE, an Eclipse 3.2 (or higher) SDK installation and a GHC installation. I am myself currently using GHC 6.4.2 and the latest Eclipse 3.3 milestones. (You should be able to use other Haskell compilers, but I've not yet tried any.)

First of all, go to the download page and download the packaged update site archive and the cohatoe-api archive. Put them in some folder on your disk, say C:\cohatoetest\.

Fire up Eclipse and install the features in the update site archive, using Eclipse's update manager: select Help > Software Updates > Find and Install from the menu bar. Then choose Search for new features to install > New Archived Site, and browse to the downloaded file. Press OK and Finish. You get now a list of features to install:

Select all of them, press Finish and then confirm the licenses etc. etc., until Eclipse wants to restart. Confirm the restart too :-)

One of the features that you have just installed is the Cohatoe Examples feature. It contains some example code that demonstrates how to use Cohatoe, including the mandatory 'Hello World' menu item which you will find in the global menu bar right after the restart. You can use it to make sure that your installation of Cohatoe works. After clicking that menu item, you should see a dialog with a friendly message from Haskell:

In this example, we will just add another such 'Hello' menu item. It will evaluate a Haskell function and give the result of that computation to the user.

So let's first create a new plugin project. Select from the menu File > New > Project > Plugin Project, enter a name (like 'cohatoetest'), then press Next and again Next. Now select 'Hello world' and press Finish. We have now a Hello-World plugin that we can extend with some useful code. Before we look into the code, right-click the project and select Run As > Eclipse Application. A second Eclipse instance is now started that contains our newly created plugin project as runtime plugin, and you can see the 'Sample Menu' entry in the main menu bar. Make sure that it gives you a message box when clicking it:

The code that opens up this message box is in the SampleAction class:

and looks like this:


public void run( final IAction action ) {
MessageDialog.openInformation(
window.getShell(),
"Cohatoetest Plug-in",
"Hello, Eclipse world" );
}

We will shortly replace the text in this message box with the result of a computation implemented in Haskell. Stay tuned :-)

(... to be continued)

Wednesday, December 27, 2006

Cohatoe 0.1 preview

I have released a first preview version of Cohatoe. Cohatoe stands for 'Contributing Haskell to Eclipse'. It is a library in the form of a Haskell API and a set of Eclipse plugins, that can be used to access code written in Haskell from Eclipse plugins.

The Haskell code is contributed via an extension in the plugin.xml. Plugins that provide Haskell code in this manner must also contribute a (minimal) Java API for the Haskell code, and of course the object code of their Haskell functions. The Cohatoe runtime uses then hs-plugins to dynamically load and execute the object code.

You can find download links and the location of the Darcs repository here: http://leiffrenzel.de/eclipse/cohatoe/. I will add more information and announce further versions on this blog. Have fun :-)