Tuesday, November 20, 2007

Cohatoe 0.8 preview

This is an even smaller bugfix build. I mostly did it to have some fixes in for using Cohatoe in the new EclipseFP 2 development branch which is based on Cohatoe. (I'll post about that new development in EclipseFP shortly.)

Have fun :-)

Saturday, October 20, 2007

Cohatoe 0.7 preview

This version mostly completes what I started in the 0.6 version, with a particular focus on the tool support that comes with the Cohatoe SDK. It introduces the notion of a Cohatoe Plugin project, i.e. a project that contains both Java code and Haskell code for an Eclipse plugin. I have also added some more developer documentation to the Cohatoe entry in the Eclipse Help system.

Cohatoe Plugin projects

A Cohatoe Plugin project is a special type of project in the workspace. It inherits from Eclipse's (PDEs) own plugin project, but in addition it 'knows' about Cohatoe-based extensions and facilitates their development. For instance, it gives a warning if there is no dependency to the Cohatoe server core plugin.

More important, once a project is marked as a Cohatoe project, it is ensured whenever files in the project change that the Haskell source files in hs-src/ get compiled and the object code is placed at the object code folder for the current platform (ignoring contents of the object code folder which belong to other platforms). This should facilitate a lot of handling (and you can ignore the details of the naming conventions for platform-specific folder names).

If you use the Cohatoe extension wizard for creating new extensions, the project where you do this is automatically marked as a Cohatoe project. You can also set and unset this mark by right-clicking a project in the workspace and selecting Toggle Cohatoe Nature.

Autobuild of Cohatoe projects

When you change someting in a Cohatoe project, an automatic build is triggered in the background that re-compiles (if necessary) the sources in hs-src/. For this purpose, a simple Cabal configuration is generated (into the INTERNAL/cohatoe/ subfolder of the project) and executed. The outputs of the build are displayed on the Console View. If Haskell source files from the hs-src folder are mentioned in the build output (normally in error and warning messages), you can click them and an editor opens on the respective source location. (It helps if you have EclipseFP installed, of course, since in this case the editor that opens is EclipseFP's Haskell source editor). If you want to configure compiler options or package dependencies for the autobuild, you can do so in the Cabal configuration file.

I think this is an important step for EclipseFP (of which Cohatoe is a part). On the way to transform it into a self-hosted IDE for Haskell that is partly implemented in Haskell, it is necessary to have better support for the development of the Haskell bits in the workspace. EclipseFP has good editing support, but since it is a more general IDE for Haskell development, it would not be good to build knowledge about Cohatoe structures directly into it. The Cohatoe SDK plugins now add this on top of EclipseFP and JDT/PDE.

Have fun :-)

Sunday, October 14, 2007

Cohatoe 0.6 preview

Here comes another step towards the 1.0 release version of Cohatoe. This one has taken much longer than I had hoped, and I'm also somewhat running out of time for today, so I just give you the 'official changelog'. There'll be more soon.

Have fun :-)

Changes between 0.6 and 0.5

  • A binary version of the Cohatoe API against which the Haskell server executable was compiled is delivered with the Cohatoe server core plugin, and it is provided to the hs-plugins library when object code is loaded and when source code is compiled. This removes the requirement that the user must have the cohatoe-api package installed in addition to their GHC installation.
  • The Cohatoe API package (the GHC package against which Haskell code for Cohatoe-based Eclipse plugins must be compiled) is now finalized as version 1.0.0. The reason for this is mostly to avoid complications when versions are continuously updated: the .hi files that must be provided for hs-plugins to load the object code contain the versions of package dependencies, and thus fail to load if only a version of the API with a newer version number is available. On the other hand, the API doesn't change at all (apart from the version number itself), so it makes sense to keep the version number stable. There remains a minimal risk that the API must be changed again before the release, in which case some early adopters would possibly have to recompile. (But this is very unlikely.)
  • Fixed a bug in locating the Haskell server executable: when multiple platform-specific fragments of the Cohatoe server core plugin were present, an arbitrary executable was picked.
  • The function factory in the CohatoeServer singleton is now generic (in the Java sense, i.e. has a type parameter. This adds some compile-time typesafety and removes the need for some casts.
  • The Cohatoe development tools (Cohatoe SDK) contain now a Cohatoe project type that can be toggled via a context menu entry on project resources in the workspace. Adding the Cohatoe nature means that the Eclipse autobuild runs over Cohatoe projects and makes checks and re-compiles the object code for Cohatoe extensions. This feature is still experimental.
  • Started to write developer documentation; in particular, there is now a Plug-In Developer's Guide that describes the Cohatoe SDK tooling, and there are also some context help entries for the Cohatoe extension wizard.
  • Added an installation tester application to the SDK. This is a small, headless standalone application that runs a Cohatoe extension in order to check whether all pre-requisites in an installation are there for running Cohatoe-based extensions.

Sunday, September 30, 2007

Eclipse's IDE Metatooling project

There is a new initiative at Eclipse.org for supporting projects who want to build Eclipse-based IDEs for new programming languages: the IMP project ('Eclipse IDE Meta-tooling Platform'). It seems it is a continuation of an IBM-internal project called SAFARI (of which we have seen a presentation at the last EclipseCON). This is great news; I think it will make the 'small step' that I have described in an earlier post much more extensive.

The IMP approach

Basically the idea behind the IMP approach is that for any IDE you create, there is a lot of stuff that you have (or someone else has) done already for other IDEs. The language specifics (such as the concrete syntax and semantics) may differ, but when you want to color syntax in the editor, show an Outline view and so on, you find yourself doing basically the same things that are done for any other language IDE. Much of the specifics of a language can be captured in a condensed description such as a grammar - why not derive the common stuff from there?

With IMP, you start with identifying your language and some basic information (e.g. file extension), and then providing a syntactical description, from which the lexer and parser are generated. From this, IMP can already derive some development support (e.g. mark syntax errors in the code editor). Now the rest of your IDE is built up from 'IDE services', which you can add one by one.

These IDE services are an abstraction over the well-known Eclipse APIs that one would have to implement when creating an IDE, e.g. project natures, New Project wizards, Outline page, syntax colorer, Code Folding, Content Assist etc. Since the parser is already in place, such IDE services can be generated in order to make use of the AST delivered by it; the generated code can then be customized to particular needs.

More information in detail can be found in the IMP Users Guide and in the SAFARI presentation from EclipseCON 2007.


The small step and the big step

So much for the small step (that is, what you get for free). What the IMP approach doesn't help with, though, is making the big step, once the small step is done.

Well, perhaps for many languages an extensive small step is all that is needed. An IMP-based IDE will have a lot of nice features. Here's a list from their website:

  • syntax highlighting, outline view population, package explorer-like navigation, content assistance, project natures and builders, error markers
  • refactoring support (not only "Move" and "Rename", but type- and code-related refactorings requiring non-trivial analysis, e.g. "Extract Method" and "Infer Type Arguments")
  • static program analysis (pointer analysis, type analysis, etc.) in support of the above
  • execution and debugging support
For many languages, IDE support of this order will get them quite some way, in particular if it comes mostly for free, or is at any rate easier done than with the current Eclipse APIs :-) Especially experimental languages or languages with a focused field of application (DSLs) will benefit from having such IDE support, and they might have not a real demand for making the big step.

IMP and established languages

But what about established, general-purpose languages? They have been around for a while; there is a community, and there are programming tools. Such languages tend to be self-hosting: given one that is sufficiently general-purpose, as soon as there is some community around it, people will start to write development tools in that language, for that language. Furthermore, its community works in the target language. They're not so much interesting in building their IDE in Java. (Perhaps even less so in hacking around in generated Java files to 'customize' it?) For taking the big step, making the project attractive to a larger community is essential. To avoid duplication of functionality, it is important to integrate existing tools - but these are written in the target language and don't necessarily integrate quite well with Java and Eclipse concepts.

Quite possibly, having gained so much by the small step, it might even become more difficult to get started with the big step. If the big step is not done in Java, it will involve much re-work, and there is a certain discouragement for this in the fact that so much already works ...

As far I can see, the IMP approach doesn't take this into account, and will thus mostly be helpful in areas where languages are restricted to some area of application, or are expected to be experimental and short-lived (or are sufficiently close to Java).

However that may be, this is definitely an interesting project that addresses a real need, and I'll be interested to follow it and see how it develops. (As a side note: there may well be some potential to integrate Cohatoe with IMP, in order to further reduce the amount of code one has to write on the Eclipse/Java side when implementing plugins in Haskell. I'll have a closer look into this sometime in the future and keep you posted.)

Saturday, September 22, 2007

What Cohatoe is not

For understanding what a tool or framework is good for, it is sometimes helpful to eliminate a few potential misunderstandings, i.e. what one might think from a first glace at a description or screenshot, but what is in fact not the case. There is always room for such misunderstandings, since people come from different backgrounds and with different expectations, and since descriptions are never perfect.

So here is what Cohatoe is not:

  • a set of Haskell bindings to the Eclipse APIs

    Cohatoe is not for manipulating a running Eclipse instance (and its UI) from Haskell code, or for 'scripting Eclipse in Haskell'. What we want to do in Haskell is the internal logic, the UI and the general layout of the application are structured by Eclipse concepts.

    In this respect Cohatoe is different from the Haskell-JVM-Bridge (which enables to create and manipulate Java objects in a JVM instance from Haskell code), and from approaches such as EclipseMonkey (where JavaScript code is executed that binds to some Eclipse APIs).

  • support for Haskell-Java language interoperability

    The game is not in particular to make Haskell code talk to Java code, or vice versa. It is rather to integrate logic that is written in Haskell with Eclipse's plugin model in a way that preserves the modularity and extensibility that comes with this plugin model as it is implemented in Java. There will be some more support for transporting data structures between the Haskell side and the Java side, but the interoperability part of Cohatoe is just a generic vehicle (currently simply a list of strings), and the clients of Cohatoe on both sides (Haskell and Java) have currently to do their marshalling and unmarshalling themselves.

    Much more important than the language interoperability aspect is the modularity and extensibility that is so important in the architecture of Eclipse. We want to preserve this modularity and extensibility even when we write or integrate large parts of an application in Haskell.

  • an approach to UI programming in Haskell

    Basically the same as the first point above. The idea is to have the UIs built using Eclipse APIs. Only the logic is in Haskell.

  • a plugin system for Haskell

    This is what hs-plugins is, and Cohatoe uses hs-plugins in order to load (possibly compile) and execute Haskell code.

Thursday, September 20, 2007

Some reflections on Eclipse-based language IDEs

A few years ago, when Eclipse's popularity was quickly growing, an interesting phenomenon was going on: a number of small projects were popping up (for a time, at the rate of a dozen or more per week) that implemented some extension to Eclipse, often for supporting some programming language. Some of them died quickly away when the interest of the founders moved on to something else, some of them became obsolete when their main idea was implemented by the Eclipse project itself. But there were some language support plugins that remained active and continued development until today. (EclipseFP is one of them, others include PHPEclipse, RDT (Ruby), Erlide (Erlang), PyDev (Python), EPIC (Perl) and Eclipse.org's CDT project.)

The obvious paradigm for all Eclipse-based IDE's are are the Java Development Tools (JDT) from Eclipse.org. They were a most important factor in making Eclipse popular, and they pioneered many of the cool IDE features that we have come to expect from Java IDEs nowadays. However, keeping up with JDT proved hard for each of the language support projects. One obvious reason for this is of course that the effort that went into the JDT project exceeded by far the means available to the other projects, all of them (with the exception of CDT) being volunteer projects. But there are other reasons, too; I've got a few observations about one of them.


The small step and the big step

For building language support on the basis of the Eclipse Platform, there is a small step and a big step: the small step is to get basic support, the big step is deep support. In order to make the small step, you just have to do a bit Java coding, learn a few Eclipse APIs, and you need some modest understanding of the target language. To make the big step, you have to implement, or integrate someone else's implementation of, deeply language-aware functionality (i.e. functionality aware of the syntax and semantics of the target language).

The equivalent of the small step can be done even with many text editors: they often have a feature where you can enter some syntax rules and get syntax coloring for content formats such as HTML or Java source code. In Eclipse, the small step includes not only syntax coloring, but project types (called 'Project Natures' in Eclipse terminology), integration of external build tools (normally compilers), output parsers to populate the Problems View, import and export wizards, etc.

The big step, on the other hand, includes all the attractive features such as automated refactorings, search for references to language elements ('Find all calls to this function'), navigation to declarations, debugger integration and so on. (A while ago, I've co-authored an article in a German journal that discussed which features belong into the language-sensitive category; my co-author exemplified the main points from the RDT project.)


What the big step involves

What everybody wants, of course, is language support that has taken the big step. But this means that a backend must be built that can analyze the syntactical and semantical structure of source code written in the target language. (And not only source code in the language itself, but also supporting formats, for example - in the Haskell world - Cabal configuration files, GHC package configuration files, etc.) It must understand the concepts behind the structure of the language (e.g. it must know what a module is, and what it means to find the name of a module in the name of a source file, or in a declaration in a Cabal configuration, and so on).

JDT has achieved this. The Java IDE in Eclipse includes a fully standards- compliant Java compiler, and much of the interesting functionality relies on the source code parsing and incremental compilation facilities provided by it. In addition, JDT maintains a model of all language elements in the workspace; it has representations for all methods, types, (Java) packages and so on, both from source files and attached libraries. All this is naturally implemented in Java. (It is also certainly helpful that the JDT project can practice 'Eat your own dogfood', i.e. that Eclipse can be built using Eclipse itself.)

But what are you going to do if your target language isn't the same language as the one Eclipse is implemented in? First of all, there is not really a way around implementing Eclipse plugins at least partially in Java. That's unfavorable: even if you actually want to build all that functionality yourself, chances are that you want to do it in the target language, not in Java. If you don't want to re-implement everything (and that should be the common case), then it's squarely improbable that the existing stuff is written in Java. Haskell is a very good example: there is almost everything you'd like, in various degrees of maturity: source parsers (both for Haskell 98 code and many language extensions), type inference and typchecking engine (GHC API), refactoring (HaRe), API search (Hoogle), a debugger (in GHCi), and lots and lots more. But guess in what language all that is written ... ;-)


Interoperability

There are the usual options for language interoperability: you can run executables written in other languages and communicate via standard input and output, or use language interoperability interfaces (such as Haskell's FFI) in conjunction with Java's native interface, JNI. Some languages can be run in an interpreter; others can be compiled to Java Virtual Machine bytecode. Although this is feasible, it's sometimes ugly and complicated, and it often limits deeper integration (there's only so much information that you can send via console i/o, and in any case if your data is highly structured, you have to put additional effort in serializing and de-serializing it).

Apart from that, Eclipse has its own dynamic module system, which relies on loading code at runtime on demand - most of the plugins that are actually in an installation may not be loaded at all in a particular session. Furthermore, the entire extensibility model in Eclipse relies on the idea that plugins cannot just plug into pre-existing extension points, but may very well (and very easily) provide new extension points themselves, which are available for anyone else to extend. In order to honor these features of Eclipse, there is much more to interoperate with than just the programming language (Java) and its runtime (the JVM). It's the Eclipse platform concept that must be taken into account. And there is no obvious way to do that in any other language than Java. (There has been some experimentation recently with executing scripts in the place of Eclipse extensions, usually scripts written in languages that can be compiled to Java bytecode. If you are interested, look at this Google Summer of Code project and this post about Eclipse plugins in Scala).

Thus taking the big step is not just more difficult for others than JDT for lack of resources. It is inherently more complicated because it adds the burden of interoperating with the platform from a different language and/or runtime. It also doesn't help of course that naturally the group of potential contributors is smaller if they are required to be fluent in two different languages and sets of APIs. (I know from years of experience as a trainer for Eclipse plugin development that it takes some time to learn to find your way around in the set of Eclipse APIs, even for experienced Java programmers - simply because of their substantial size and the number of associated concepts.)


The way out

The lesson to be learned from this is, I think, clearly that a crucial piece in building an Eclipse-based IDE for a language must be to make code written in that language interoperate with the Eclipse Platform - that is, it is necessary to make it possible to write Eclipse plugins in that language. Once this is in place it becomes possible to

  • re-use existing tools written in the target language,

  • find and motivate contributors (i.e. who have an interest and a good knowledge about that language), and finally

  • make the big step :-)

Tuesday, September 18, 2007

Progress on Cohatoe

Here is another update about the latest additions to the Cohatoe repo :-)

We now manage the library files for the Coahatoe API (the GHC package against which Haskell code for Cohatoe-based Eclipse plugins must be compiled) next to the server executable in platform-specific fragments. They are automatically found, possibly extracted (when the plugin lives in a .jar file), and then provided to the hs-plugins library both when object code is loaded and when source code is compiled, in the form of an 'on the fly'-GHC package. This removes the requirement that Cohatoe users had to have the cohatoe-api package installed in addition to their GHCi installation.

So far, Cohatoe worked only on machines that had a Cohatoe API installed as GHC package. This is clearly not good, partly because it is not a reasonable pre-requisite (the API should only be needed for developing Cohatoe-based plugins, but not for using them), and partly because there might be version conflicts (e.g. if a plugin has been compiled against a later API version than is installed on the user's system).

The situation is now better: a user who has a GHC installation, but no cohatoe-api package installed, will be able to run Cohatoe-based plugins. The Cohatoe server locates the Cohatoe API library that is shipped together with it. The additional advantage is that the server executable will always get the version of the API that itself was compiled against. The shipped version has precedence. (Note that this does not guarantee that the plugins which are loaded are also compiled against that version of the API.) If you want to compile against the Cohatoe API, you still need to install the cohatoe-api package, of course.

I have also finalized the version of the cohatoe-api package to 1.0 (the final release version). There remains a minimal risk that the API must be changed again before the release, in which case some early adopters would possibly have to recompile.

On the other hand, I was having some trouble with object files that I had compiled against versions of the API (not actually different code, the only difference was in version number). So I think for this package I'm switching to a versioning policy that only updates the version number when the content actually changes. This means that even after the release the API will only change in version if there are actual changes to the interface. Version numbers will not be increased in step with the other Cohatoe plugins.

Later versions of Cohatoe will probably need some mechanism to enable running both code that was targeted at the 1.0 version and code that was written for later versions. This means that the old API will probably have to remain available, and the server will have to determine which API to use and to load against. (This will cause some work, but that is definitely not a topic for the 1.0 release, and I'm confident that it can be sorted out.)

Apart from that, I have also continued working on getting the Linux version ready, and I have cleaned up the Haskell code of the server executable a bit. I think I will be able to get the next preview version out soon.

Thursday, September 13, 2007

Cohatoe presentation video online

The video of my presentation at the Haskell in Leipzig 2 meeting is now online :-) You can watch it on Tobias' blog, along with the other talks from the event. (And don't ask me about the Stolperstein ;-)

As already said, the talk is in German; there is a summary that I've posted to the EclipseFP mailing list earlier, and here are the slides.

Thanks to Tobias and everybody who have made this fun possible :-)

Wednesday, September 12, 2007

Cohatoe development progress

Here's a brief status update about the latest few patches that just went into the Darcs repo.

I have fixed a bug that caused Cohatoe to select the wrong platform-specific fragment, and thus the wrong server executable, if there was more than one present. In other words, if you had Cohatoe running with both the Linux and the Windows fragment installed, it crashed under Windows because it tried to run the Linux executable - even though the Linux fragment wasn't activated. Cohatoe can now figure out which fragment is the correct one. (I suspect that has troubled nobody but me so far, but it was nagging me a bit. I wanted to catch up with the Linux version, but I had the fragment project disabled because of this bug. I think I will be able to get a well-working Linux version along with the Windows one for the next preview release. I've already re-activated my old Gentoo box for this :-)

I have also built in an automatism to pass on GHC runtime options to the Cohatoe runtime. If you have some Haskell code contributed via Cohatoe which needs to enable RTS options, you can now specify them on the command line for the Eclipse executable (or better, in the eclipse.ini file), and they are detected by Cohatoe and passed on to the Haskell server. You can specify the +RTS ... -RTS as you would pass them to any GHC-compiled executable. (In the eclipse.ini, take care that you have exactly one option per line, because that's how Eclipse expects them.)

At the moment, options that are not accepted by the GHC RTS break the Cohatoe server (i.e. the server executable doesn't even start at all). So if you are playing around with the options, you better have Cohatoe's tracing enabled to see what actually happens. Without that, all you see is an IllegalStateException that complains about a dead server. I will build in some more sophisticated recovery with more info within one of the next iterations.

Have fun :-)

Wednesday, September 05, 2007

Cohatoe 0.5 preview

Here comes another preview version of Cohatoe. I think it is now time to take a bold step and declare it 'beta' (no longer 'alpha' :-). This means in practice that the features which are in now will pretty much remain the same until version 1.0, but I still expect to have to do some more updates, mainly with fixes and documentation (and perhaps a little more tool support for PDE).

This version is only available from the new download site at EclipseFP! Please make sure that you update your bookmarks :-) Likewise, I have pushed the latest patches only to the new repo location. See this previous post for links and a little more info.

Version 0.5 has the extension wizardry that I have mentioned before, and it is well able to handle running extensions from Haskell source files now. (There were several fixes I had to do for this since my last post.) It is basically the version that I have presented at the Haskell in Leipzig 2 meeting.

Any feedback is appreciated - have fun :-)

Thursday, August 09, 2007

Next steps for Cohatoe

For those who are interested but haven't subscribed to the EclipseFP mailing list: I have posted a summary of what I presented at the Haskell in Leipzig 2 meeting, in my talk about the progress of Cohatoe. The post has also a link to the slides, but the slides are not the whole story of course, since most of the presentation was live demo.

(Oh, btw: perhaps you should consider to subscribe to the mailing list now, if you are interested in Cohatoe, because we have decided at the EclipseFP project that we would keep the discussion for both Cohatoe and the EclipseFP IDE project on the same mailing list. You can subscribe here :-)

We have now started to integrate Cohatoe back into the EclipseFP project. There won't be any significant changes in the Cohatoe code itself (at least for a while ;-) - but the location of the repo has changed, and the same will go for the download site. Both have been moved to the EclipseFP webspace, so please update your bookmarks :-)

The new Darcs repository is located here:

  • http://eclipsefp.sf.net/repos/cohatoe/trunk

There is a new section on the EclipseFP website for Cohatoe, which has all the links to the download files etc. that were also on the old Cohatoe website. You can either go there from
http://eclipsefp.sf.net/ or directly: http://eclipsefp.sf.net/cohatoe/.

Friday, July 06, 2007

Cohatoe talk at HaL 2

There will be another Haskell in Leipzig (HaL) meeting next week, and I'll be there too and present Cohatoe. The whole idea and many of its technical features owe a lot to the discussion at the first HaL meeting, so I'm thrilled that I was asked to report on the progress there :-)

So if you are are interested (and somewhere near Leipzig), why not drop by and have some fun?

Sunday, April 29, 2007

Cohatoe extension wizardry

I have started to add some helpful tools to the Plugin Development Environment (PDE) in Eclipse to facilitate working with Cohatoe. One of the things that still take some time (in my experience when implementing the examples) is creating the source files and manifest entries that are needed for a new Haskell contribution via Cohatoe. It's not much (and mostly boilerplate that is always almost the same stuff), and it can actually be generated from very little input by the user.


So I have first added a wizard for creating a new Cohatoe extension. Suppose that you already have a plugin project. Then you can now just go to the Extensions Tab on the Plugin Manifest editor, click Add > Extension Wizards > Cohatoe Development Tools > Haskell Contribution, and you get a wizard that generates all the files and manifest entries that you need, namely:

  • It adds the dependency to the Cohatoe Core plugin to the META-INF/Manifest.mf file (if it is not already there).
  • It adds an extension in the plugin.xml file and declares the Haskell source file along with the Java implementation and interface types.
  • It generates the Java interface and implementation types, together with the code necessary to call the server (i.e. what happens inside the implementation class) and with a usage example that can be copied to where you want to call the function from your Java code.
  • It generates the skeleton source file for the Haskell module that is called.
The only things you have to specify are some workspace locations, in particular the source folder (there is usually only one, so that's probably trivial), a Java package name, and a name for your contribution, from which the name of the Haskell module and the Java types are derived.


For the time being, this supports only contributing source files, if you want to change this into an object file contribution, you have to enter the path to the object files manually in the plugin.xml, in the codeFile attribute (where originally the source file is specified in what the wizard has generated).

I think this should considerably ease and speed up the process of creating the plumbing for a new Haskell contribution, and help you to focus on the actually more interesting bits, namely the Haskell code itself, and the places where it is eventually called in the Eclipse plugin :-)

(NB: this is not yet in the downloadable preview, but in the Darcs repo. But I think I'll do another preview build soon.)

Sunday, April 22, 2007

Contributing source files

When working on new examples for Cohatoe, I noticed that it is still somewhat tedious to go through the usual cycle: write some code, compile and run it, make some changes, compile and run again ... One thing that makes it tedious is the need to compile the Haskell code into object files all the time.

In order to make this nicer, I have extended Cohatoe to run Haskell code from contributed source files. (The actual work of compiling and loading the code is again done by hs-plugins.) You can now just put a Haskell source file into your plugin project, and declare it similarly to how you would have declared an object file.

There a few rather minor details connected to that change I'm a little unhappy about. One is that I had to change the format of a Cohatoe extension in the plugin.xml once more. Instead of a (mandatory) objectFile attribute, the haskellFunction has now a codeFile attribute (which says nothing anymore about the content of the file). Whether we treat the file as a source code file or an object file is determined from the file extension. Currently recognized are .o as extension for object files and .hs and .lhs for source files. The alternative would have been to keep the old objectFile attribute and introduce a new one called sourceFile. But that would have meant to make them both optional, and the only way to validate that exactly one of them was specified is then in the Java code that reads the extension. The current way has also the advantage that it can be easily extended to other types of contributed Haskell code (I'm in particular thinking of Haskell code that is compiled to JVM bytcode, e.g. via LambdaVM).

On the downside, of course, that's one more API change which is annoying to any early adopters out there :-( Still I think it's better to make such changes now, that is, early in the cycle.

Another less-than-optimal thing is that the .o and .hi files that hs-plugins generates before loading them end up in the same folder in which the original source file is. There is a number of situations where this could be a problem, including running from a jarred plugin, running from a write-protected folder, and so on. Cohatoe can handle a number of these situations transparently already, but very likely I have still overlooked one or the other. Therefore I regard this as a somewhat unfinished detail.

Of course, the additional compilation step has an effect on the time needed when first accessing the function; in production mode you would probably still prefer a pre-compiled object file to contributing a source file.

Looking at the bright side, this new possibility to run contributed Haskell code as source code will help to make life easier during development, and also make it easier to provide platform-independent functionality (always assuming that the code itself doesn't make any assumptions about the OS it runs on, in which case there is no way around keeping platform-specific code in one manner or the other).

Thursday, April 05, 2007

Unit testing in Haskell

Test-Driven-Development is a programming technique where you write small programs (test cases) in advance of coding the actual piece of software you want to write. If done well, this helps to clarify the various combinations of input/output/environment for your code, and makes it easier to implement it. As a nice extra, you speed up testing - all you have to do is to run all those test cases again, and they will point out whether everything still works as expected.

So far so good. (And if you're coming from Java development, you know that song by heart anyway ;-). But can we do this in Haskell? Sure. There are actually two different well-supported approaches in Haskell to do test-driven development. One is more oriented towards functional programming, focusing on declaring properties of functions - this one is supported by QuickCheck (and not covered in this post); the other is aligned with the 'classical' JUnit tool. There is an implementation of this approach in the form of HUnit. I have written a tutorial that gives an introduction to unit testing with HUnit.

Again, I would have have preferred to post the full text here, but that is too inconvenient with all the code listings. But you are welcome to comment on the tutorial here on this post.

Saturday, March 31, 2007

EclipseDarcs 0.4 available

A new version of EclipseDarcs, the Darcs integration in the Eclipse IDE, is now available. This is version 0.4 (still a preview, or milestone, version).

Thanks to Radek, we have now support for ssh repositories (note: this works only if you have passwordless login enabled, and on Windows you need the latest Darcs 1.0.9 release candidate build), and for pulling and pushing all patches from withing the workspace.

I think this is a great step forward. We're still not yet quite feature complete, and there are some bugs, so please be patient for a little while longer :-)

There are various options to get the latest version, please look at the installation instructions on the EclipseDarcs homepage. All feedback is very welcome. Enjoy.

Sunday, March 18, 2007

Command-line options in Haskell

I'm currently writing a small command-line application, and one of the tasks that needs to be done in that context is handling of command line options. So I've learned the basics of handling command line options in Haskell, and have logged my results. Nothing in this post is my own idea, I have just collected what I found when looking around :-)

My original intention was to post this as a regular blog entry here, but I did not manage to get the Haskell code listings properly formatted. I've tried systematically lots of combinations of the HTML 'code' and 'pre' tags, nbsp escapes and so on - nothing worked. Finally I gave up and put it all into a separate page.

Friday, March 16, 2007

Cohatoe 0.3 preview

The next preview of Cohatoe is now available from the download page, from the update site, and in the Darcs repository (all to be found here). The usual disclaimer (experimental state etc.) applies as before, but my feeling is that it is getting nearer some state where one can actually start some serious playing around with it. (I'm especially hopeful that there will be fewer API changes from now on.)

I have left out all the progress monitoring stuff I wrote about, but I have re-worked the Cohatoe API to execute Haskell functions within the IO monad. This means that the main entry point for Haskell code that is contributed in an Eclipse plugin (pluginMain, as declared in the Cohatoe API) is now of type [String] -> IO [String]. Please note that you have to re-compile any object code against that new API in order to run with the new version of the plugins.

Another important piece that is now in place is exception handling. Several sorts of exceptions that are raised during the execution of contributed Haskell code are caught and re-raised on the Eclipse side - instead of just making the server executable break and exit. This makes Cohatoe much more robust.

Any feedback is most welcome - have fun :-)

Wednesday, March 14, 2007

EclipseDarcs 0.3 available

A new version of EclipseDarcs, the Eclipse IDE support for the Darcs RCS, is now available. This is version 0.3. It is still a preview (milestone) version, and it contains mostly several fixes and minor improvements that have been done over the past months since the 0.2 release. (But watch out for more soon, there are things happening at the moment :-).

Tuesday, March 13, 2007

Progress on Progress?

In the past few days, I have been exploring a side alley of the main topics in Cohatoe: progress monitoring. I think it would be actually nice to have something like a progress monitor API on the Haskell side so that you can send back progress feedback to the omnipresent progress dialogs in Eclipse (and by the way, Eclipse users are very much used to a responsive UI that allows them to send everything into the background that takes more than a few hundred milliseconds time). So I poked around in that area for a while. But it looks as if I have hit a dead end.

On the Java side, taking progress messages and pass them on to an IProgressMonitor instance isn't a real problem (although it took a bit to implement, mostly for the message encoding and decoding). But I'm somewhat stuck on the Haskell side.

Once more, this is more a question of API design than anything else. I don't think there would be a problem at all if we would in general pass a progress monitor 'handle' to the main entry point function. An example would then look like so:


import Cohatoe.API

resource = plugin {
pluginMain = sayHello
}

sayHello :: ProgressMonitor -> [String] -> [String]
sayHello _ _ = ["Hello from Haskell"]
that is, adding another parameter to the signature of the main entry point function. The progress monitor type would be declared by the Cohatoe API, and the framework would actually create a channel behind that value that sends all progress feedback to the Eclipse side.

However, I'm uneasy with that. True, there will be a lot of occasions where it makes sense to send progress feedback, but there will be as may where it seems not necessary. Of course, in these cases one can always ignore the progress monitor argument, but it still seems not nice to me to make it a part of the function signature everytime. I would very much more like something similar to this
myFunction :: [String] -> IO [String]
myFunction args = do
pm <- getProgressMonitor
pmBegin pm "Starting" 100
-- actually do something
pmDone pm
But then where to get the progress monitor from? I have tried some approaches that seemed to lead nowhere. I suspect I could do it by designing my own monad, but I'm not deep enough into Haskell yet to do that :-) And I also can't see how that would keep the API neat. So I think for now I'll leave the progress monitoring out in the interest of simplicity and revert the code base.

(Btw, this was more of an experiment anyway; I don't think that it is a good idea in general to re-model the Eclipse APIs, such as the progress monitors, in Haskell. That's far too much work, and there is a limit anyway, since the Eclipse APIs really assume to be dealing with stateful objects all the time. I've written a bit more about that in my earlier contribution to the Eclipse Languages Symposium.)

Friday, March 09, 2007

Using Darcs (and EclipseDarcs)

Radek Grzanka has asked, on the EclipseDarcs mailing list, whether I could share some experiences using Darcs in my programming team (and I'm happy to do so :-). I have introduced Darcs to my team about one and a half year ago when we started to do some projects that involved developers sitting in locations other than our office in Karlsruhe, Germany (including external developers who were located in the US), and which also involved people who are, like I was myself at the time, frequently travelling.

For me this is one important plus for Darcs: that it allows fine control over what goes into the repository while being offline (on the train, on the plane, or in a Hotel room that hasn't internet access). With source code control systems that require the presence of a server you can check in less frequently, and therefore you tend to either check in too coarse-grained, or you spend much more time figuring out which source changes belonged to which activity when you are checking in. With Darcs, on the other hand, you can code, then record, then code, record and so on, and when you go online the next time, you can still push selectively.

The teams we are working in are relatively small (up to six people, with some fluctuation), so I couldn't say much on how it works with really big teams. But for us it worked out really well. Darcs is pretty easy to learn and also definitely fun to work with, and that was consistently the feedback I got from all developers. (It also has the occasional bug or sluggishness, especially on Windows, but that's far short of the big-time sucking that CVS can be, in my experience.)

Darcs is also helpful with branching, which we do a reasonable amount of. However, in this area one still needs some tools or techniques around it, mostly for figuring out diffs between branches. This is one of the areas where EclipseDarcs comes in very handy, namely the repository browsing perspective that allows to conveniently inspect patches. (I have also started work on a repo diff view for the Darcs repository browsing perspective, exactly from this background.) The most needed tool there is a viewer that supports viewing patch dependencies. For instance, imagine you have one release repository and one development repository. The latter holds a bunch of patches which are not yet in the release repository. Someone has the job to 'release' some patches from the development repo to the release repo. So it is important for that guy to see which patches depend on which others, in order to figure out what minimally must be pushed.

We typically have one central repository on an intranet server that we call the integration repo, or development repo. Developers work on local repos on their machines, and record changes there. All our work items, bug fixes and tasks that we do are structured in an issue tracker. So typically, when somebody has done some implementation task, that's associated with an issue tracker ID and recorded so (stating the issues tracker ID first thing in the patch comment). After that, it is usually either pushed to the integration repo, or sent to another developer who reviews it and then pushes it. On the integration repo, we have automated processes that build, collect compiler warnings, run CheckStyle, run JUnit, run PDE JUnit and so on. That way, we get early feedback about integration problems (that is, problems that go further than simple conflicts in the source text). This sort of setup is relatively common today (much more so than when we introduced it, some five years ago), and often called 'continuous integration'. It is one of the most valuable practices in software development in my view; however, it requires a certain insight and co-operation from the team. (As with many of these so-called 'agile' practices, the people in your team need to understand and value them - else they won't work much better than any other methodologies.) In our practice, we try to push and pull as often as possible (several times a day). This helps to have the entire team up-to-date with recent changes to the code base, and reduces the chance of integration problems.

In addition to the integration repo, we have branches that hold the released versions. In some cases, they are just subsets of the corresponding integration repo (the latter one holds more patches because some functionality is not yet mature enough to be released to the public, but ok to be integrated with the rest of the code base). In some cases, they are real functionality branches (branches of the main stem that contain some different functionality, e.g. for deployment of a customized version). Naturally, we try to keep the second variant rare - but in my view it is definitely more pleasant to do this with Darcs than with other systems I've worked with.

And of course we have also learned to love such techniques as those which are described on the Darcs wiki as spontaneous branches and preparation branches.

We use EclipseDarcs together with the Darcs command line (the latter one for more complicated recording that spans over multiple files, and for pulling and pushing). EclipseDarcs comes in helpful for inspecting repo contents (as I remarked above), and for adding files, quickly recording single-file changes, and it helps of course by showing which files in the repo have changed since the last recording. Well - and we all know that there is plenty of room for improvement :-) It would be nice to be able some day to have all Darcs functionality integrated in Eclipse, of course. We're only half-way there. But while that is a little inconvenient, I don't think it really should be a reason not to use Darcs. For one thing, the command line interface of Darcs is very convenient, and apart from that I'm convinced that a professional programmer should be able to use a command line tool.

The source trees we work on are almost always Eclipse Plug-In projects. This is a bit nasty when the contents of a repository must be imported into the workspace for the first time. (Especially because the contents of the pristine tree, under _darcs/pristine/, are detected by Eclipse and offered as projects to import; one must be very careful not to select these - the best technique for this is to temporarily move the entire _darcs folder out of the repository, import the projects all at once, and move the _darcs folder back.) On the other hand, this tends naturally to a well-structured and fine-grained organization of the source tree, which helps when recording, and probably also helps with the Darcs performance. For our purposes, the performance is ok (but again, that pretty much depends on the layout and size of your source trees).

Perhaps one or two more misc notes: one should checkpoint repos from time to time, it really helps to reduce the time to initially get repo contents. (It is also definitely faster to use darcs get instead of darcs init && darcs pull --all, but that's probably old news :-)

Sunday, March 04, 2007

A focused braindump

Over the last months, I have written an informal paper about some of the more arcane details of Eclipse technology, related to its update/install story. Currently, the approach in this area is based on so-called 'features' (roughly groups of plugins, which in turn are the functional units in Eclipse). The paper gives an overview over the different relationships that can obtain between features, and exposes some problems in the logical system that emerges from these relationships. It also highlights some issues that have arisen in the real-world use of the features approach.

My goal was to collect some thoughts that I have posted to mailing lists and points I made in technical discussions, mainly to have them all in one place. There is an ongoing discussion at the Eclipse project about this topic, and about replacing the features-approach in future versions of Eclipse, so hopefully this is somewhat useful for summarizing a few lessons learned. I’m confident that the problems described here will soon be a thing of the past.

Btw, if you are interested in the latest developments, here are a few pointers: the requirements pool for a new Update Manager, the Equinox provisioning project proposal, the Maya project proposal (somehwat related), and the Eclipse Packaging Project homepage (dito).

Sunday, February 25, 2007

About losing purity

Just so you don't think I've migrated to the Humanities Department - even though the last posts were about history, I'll finally get back to the technical stuff ;-) I'm currently working on a new example and a few API enhancements. Here's a few thoughts about a minor issue on the side:

When you are contributing Haskell code to Eclipse, via Cohatoe, the entry point so far had to look something like this (taken from the HelloWorld example):

import Cohatoe.API

resource = plugin {
pluginMain = sayHello
}

sayHello :: [String] -> [String]
-- and here goes your implementation of HelloWorld
I have been pondering for a while the question of purity in this. While I somewhat like having a pure function here ([String] -> [String]), I think for real world code what we want is rather something in the IO monad. This is not a technical problem (the code that loads pluginMain and executes it is in the IO monad anyway). It is just a question of API design. I think that in many cases the Haskell code will be used for doing things like reading source files, writing indices, and other such things that must be done in an IO action.

I have thought about enabling both (by passing, on the Java side, some flag that requests running in IO, or perhaps by making it something that you have to declare in the plugin.xml). However, this would load some more work on the user of the API, and I'd rather put a premium on simplicity here, seeing that we want to encourage programmers who don't know much about (or don't care much for) the Java and Eclipse side of things here. And Haskell programmers are used to having an entry point in the IO monad: the main function in every Haskell program is.

So, that settles it, from version 0.3 pluginMain will be of type [String] -> IO [String] :-)

Sunday, January 28, 2007

History of Haskell

Simon Peyton Jones has announced the final version of the History of Haskell paper that he has written together with Paul Hudak, John Hughes and Philip Wadler. It is a longish text, but a great read, especially if you have (like me) only started with Haskell in the recent few years.

(EclipseFP is also mentioned in the section about development tools :-)

Saturday, January 13, 2007

Some history

(This is not technical today but more a piece of personal history. I'm describing some of my background that led me to start the Cohatoe project.)

One of the key motivations for me to develop Cohatoe came from my experiences in the EclipseFP project. EclipseFP implements a set of plugins that support functional programming in the Eclipse IDE. Originally initiated by me, it has meanwhile got a momentum of its own, and there are a number of people who contribute to it. There is some support for OCaml (currently unmaintained and on hold), but the bulk of the functionality is for supporting Haskell development. When I started the EclipseFP project, I was about to learn Haskell, and coming from a Java background, I wanted to have Eclipse support similar to the one that existed for Java. There was no Eclipse plugin for Haskell, so I decided to write my own. (That was a bit of an unclever decision: I ended up with investing more time writing Eclipse code than actually using it for learning to write Haskell code ;-)

Usually, if one starts to develop language support in Eclipse, the first thing is to create a source code editor with some syntax coloring; perhaps some minimal Code Assist (automatic code completion) for keywords; a project type and some wizards that support the creation of source files; and a compiler integration that drives a build when files are saved and reports compiler errors and warnings as 'markers' (which show up in the Problems View and as line annotations in the code).

All these things are pretty easily done, and they can be implemented in Java without a lot of knowledge about the target language specifics going into that code. For instance, for syntax coloring, a simple set of coloring rules (e.g.: apply the comment color from the occurence of '--' until the end of the line) is enough to reasonably color Haskell source files without even actually knowing about the exact syntactical structure of Haskell source code. Eclipse's APIs provide good support for doing these things, and it is not too difficult or too much effort to get some Eclipse-based IDE up and running using them.

However, Eclipse's own Java Development Tools (JDT) have a lot of functionality that cannot be done without building some deep understanding of the language into the code. Here are some examples:

  • Mark Occurrences: given a source file and the cursor positioned on some identifier (e.g. in a call to a function), highlight all other occurrences of that identifier in the editor - and of course do not count just any occurrences of the same string of characters, but only those which actually are the same identifier; don't highlight them in comments, or occurrences of local variables with the same name in a different scope.
  • Source Outline and Code Folding: In the Outline View, there is a tree that represents the language elements (type declarations, field declarations, method declarations etc.) from the source code in the editor; and that tree can be used to navigate the code quickly and even change it (for example, deleting or moving field declarations). Code folding allows to collapse regions of the code (for instance a long comment at the top of the source file) into a single line to make the code more easy to read.
  • Refactoring: there are a lot of automated refactorings, like the simple Rename (of a method or type), which not only changes the type name in the source code, but also adjusts all code that refers to it, and also renames source files accordingly, or more complicated ones such as Extract Method, which allows to isolate a block of code and put it into a newly created method.
Such things usually require analysis of the source code as it is done by compilers and other tools.

(I have written a more extensive article about the topic of how much language-specific code is needed for which functionalities of an Eclipse-based IDE - it was written for a German journal, and is therefore in German, so you will unfortunately only be able to read it if you understand that language. My co-author, Markus Barchfeld, is a committer on the Ruby Development Tools, RDT, and he has made some similar experiences there like the ones I'm describing here.)

There is plainly no chance of re-implementing all this in Java for a language such as Haskell. (The JDT team was in the enviable position to be able to write their own Java compiler and in addition to use their own tool for building itself.) Therefore, the only sensible way of getting deeper integration would be to somehow re-use existing code that does the job. And happily, all that code exists, is free and could be used for an Eclipse-based IDE (think of the GHC API, HaRe, the Haskell Refactorer, and many other Haskell development tools). It is just necessary to find a way to use Haskell code in an Eclipse plugin without too much hassle and compromise.

This is what I hope to achieve with Cohatoe. On the way so far I have tried a number of different approaches, and I have discussed it with many people, notably the people on the EclipseFP mailing list. EclipseFP is now maintained by Thiago Arrais. Thiago has written a Haskell parser that does already a pretty cool job in supporting Code Assist and other helpers in the Haskell source editor. It has replaced an earlier approach of mine which used the Haskell parser from the haskell-src package in the standard libraries, and accessed it via FFI and JNI (Haskell's and Java's respective foreign and native interfaces). That approach had proved to be somewhat unstable (not the parser, but my code that drove it) and had the decisive disadvantage of working only on Windows (because it required the Haskell code to be run in a dll). I also wrote a contribution to the Eclipse Languages Symposium about a year ago. All this, however, left me a bit frustrated. Somehow I wasn't gettting the feeling that real progress could be made with all the approaches that I had tried.

When I was at the Haskell in Leipzig meeting last year, however, there was a very engaged discussion, and I decided to have another stab at the problem; I'd like to thank the participants in that meeting, and in particular Johannes Waldmann, for this very encouraging debate. This time I think I have made some progress, and it seems that is mostly thanks to a change in perspective. I'm focusing more on the programming model this time, instead of thinking in terms of choosing a technology that is able to make the connection between the Haskell code and the Java code that runs inside Eclipse. The goal is to design both the Haskell side and the Eclipse side of things in Cohatoe so that developers find themselves at home on either side, no distortion or unwieldy constructs, or extra efforts necessary. So far, this has worked out fine, so let's see what happens next :-)

Sunday, January 07, 2007

Cohatoe 0.2 preview

I have just released another preview version of Cohatoe. It contains a number of improvements and a cool new example.

As before, please keep in mind that this is still experimental and the APIs are bound to change a few times more. Actually, a rather severe change has been made already in 0.2 to the format of the plugin.xml when declaring a Haskell function (see below). (I promise I will stop doing such things of course once the APIs are more mature ;-).

There is now a changelog file in the SDK feature, where you can check what is new in which versions. The changelog is cumulative and contains all changes since the very beginning. In what follows, I'm giving a commented version of the changelog contents.

Declaring Haskell functions

As I mentioned, there have been changes to the schema of the haskellFunctions extension point (in the plugin de.leiffrenzel.cohatoe.server.core The attribute objectCode has been renamed to objectFile. This means that all plugin.xml declarations that use this extension point must be updated.

The old attribute is left in and tagged as deprecated, so that uses of the old format can be detected more easily. It will be removed in 0.3. That the attribute is still there doesn't mean that it still works, however. Eclipse will give a warning for each occurrence, so I have decided to leave it in. But the Cohatoe code doesn't read it anymore.

For situations where the Haskell code that you want to call is distributed over multiple modules, and you would consequently have more than one object file to declare, I have added some support. In the case where your modules are all in the same folder as the declared .o file, Cohatoe makes sure that they all are passed to hs-plugins, which will be able to load them. In the case where your object files are in a different folder (inside the plugin's directory structure, of course), you can specify that folder in the plugin.xml. Here is an example (taken from my unit test cases):

<haskellFunction
implementation="de.leiffrenzel.cohatoe.server.core.test.functions.MultiFolderModule"
interface="de.leiffrenzel.cohatoe.server.core.test.functions.IMultiFolderModule"
name="Multi-modules in different folders"
objectFile="$os$/obj/MultiFolderModule1.o">
<objectCodeFolder>$os$/obj2/</objectCodeFolder>
</haskellFunction>
There may be zero or more objectCodeFolder elements inside a haskellFunction element in your declarations now, each of which can declare a plugin-relative path, and all these paths are resolved and passed to the Haskell side where they are used for tracking module dependencies. (As usual, this works also when plugins are deployed and live in a .jar file.)

I'm planning to add further support for packages of object files (as created by the archiver tool). You will be able to declare these packages in a similar manner, but that support is not yet included as of version 0.2.

Next, I have added some helpful functions that can be used for marshalling data to the Haskell side and unmarshalling the results back. They are located in the class CohatoeUtils. Actually, they were already available in version 0.1, but there they were not well organized, well named and not even completely implemented. I've done a bit in that area, too.

Error handling

I have also started to improve Cohatoe in the area of error handling. This belongs to the large topic of robustness. Since Cohatoe will execute any Haskell code, it is important that its functioning is not impaired by code that crashes or perhaps is even malicious. Currently, a problem inside contributed code will cause the Haskell server executable to exit, and a restart of Eclipse is needed to get it working again. That is of course not acceptable, and so I have started to tackle the possibly problematic situations one by one.

The first case, the one that is now handled by Cohatoe, is when your contributed Haskell code calls the Prelude.error function. This will now no longer break the server, but is instead caught at the Haskell side and reported to the Eclipse side, where the exception is re-thrown as a CohatoeException.

The exception type (CohatoeException) is a subtype of Eclipse's CoreException, and it carries some useful information in addition to the usual Java exception information. It contains the error message that was used on the Haskell side, and it also knows the plugin identifier of the plugin that contributed the code which has caused the exception.

Since it is a CoreException, it also contains an Eclipse IStatus object that you can conveniently write to the workspace log, like so:
try {
// ...
} catch( final CohatoeException cohex ) {
// we don't handle this here, just
// write it to the workspace log
CohatoeExamplesPlugin plugin = CohatoeExamplesPlugin.getDefault();
plugin.getLog().log( cohex.getStatus() );
}
The exception knows also an identifier for the type of exception that was raised. I want to use that for distinguishing between calls to the error function and other types of errors, such as IO Errors, or problems that occured when hs-plugins tried to load code it couldn't. But that is not yet in - currently only Prelude.error is handled by Cohatoe.

New example - Sudoku solver

A new example has been introduced: a Sudoku solver View. The View can be found via the menu Window > Show View > Cohatoe Examples. The Haskell code that does the actual work is a Sudoku solver written by Graham Hutton which can be found at http://www.cs.nott.ac.uk/~gmh/sudoku.lhs (thanks to Graham for giving permission to include this code :-).

This example demonstrates how Haskell code is handled that spans more than just one object file (more background discussion can be found at in my earlier post about module dependencies in multiple modules), and it also gives an example for accessing multiple Haskell functions through a single interface on the Java side.

I will comment this example in more detail in one of my next posts, but I want to give you at least a screenshot here:

That's it from me for now - have fun! Any feedback is very welcome.

Saturday, January 06, 2007

Building a server executable

Part of the Cohatoe architecture is the Cohatoe server. On the Eclipse side it is represented by a singleton object (of class de.leiffrenzel.cohatoe.server.core.CohatoeServer), where clients can access it. The server runs in a separate process, and the singleton on the Java side communicates with it via a socket connection. It is implemented in Haskell, and it is able to locate object code that has been contributed via Cohatoe, load and execute it (using hs-plugins).

That the server is implemented in Haskell means also that it has to be compiled into a native executable, which is of course a different one for each supported OS/WS platform. Currently, that is only Windows (strictly speaking: only 32 bit Windows). But the mechanisms for supporting other platforms are there (mostly provided by Eclipse already), and the Haskell code for the server does not make use of platform specifics (as far as I can see). Cohatoe will automatically locate the correct executable for the platform it runs on (if one is there). Therefore, the only thing that is needed is to build the executable for all the other platforms. In what follows, I'll describe how to do this and explain a few background details on the way.

The Haskell source code for the Cohatoe server is located in the plugin de.leiffrenzel.cohatoe.server.core, in the directory INTERNAL/hs/src/. The native executables for the various platforms (i.e. Windows, Linux/GTK, MacOSX etc.) do not reside in the same plugin. Instead, they are contained in fragments. Fragments are a special concept in Eclipse's plugin architecture. (They have also recently become part of the OSGi specification, which is implemented by the Eclipse runtime.)

A fragment is, like a plugin, an OSGi bundle. In the workspace, it lives as a fragment project (instead of as a plugin project), and it contains a fragment.xml instead of a plugin.xml. But they have a similar directory structure, and they have the usual META-INF/MANIFEST.MF file where the meta information is located. When deployed, a fragment looks exactly like a plugin. It (usually) lives in a jar file in the plugins/ folder.

The specific thing about fragments is that they do not exist on their own - they are dependent on a plugin, which is called their host plugin. They have to declare which plugin their host is by the host's plugin id and version. As an example, here is how this looks like for the fragment with the Cohatoe server executable for Windows:


When Eclipse starts, it collects all fragments for a given host and adds their contents to the host plugin's. The contents are overlayed only virtually, they are not physically copied to the host plugin's folder. This means that all classes from .jar files in the fragment appear on the classpath of the host plugin, all files (such as icons, or internationalization properties files) appear on the host plugin's resource path, and so on.

Take an example: Suppose there is a plugin p, and a fragment f that declares a as its host plugin. Suppose further that f contains an icon file icons/eview16/myview.gif (p does not contain a file with that path and name). Now any code in a plugin that depends on p will get that icon file from p, if f is present, even though p does not contain it (and in fact doesn't even know about it).

This mechanism is used by the Eclipse project itself for providing internationalization texts. The plugins of the Eclipse IDE themselves contain only English UI texts. But there is a set of fragments (collectively called the 'language packs') that provide translations of these texts into a number of other languages. These fragments consist of just a lot of .properties files. When they are present, these files are used by Eclipse to display UI texts in their localized versions.

Fragments of the Cohatoe server plugin (de.leiffrenzel.cohatoe.server.core) which contain the server executables are platform-specific. That means that there is one fragment for Windows (which you can recognize from the win32.win32.x86 in the name), one for Linux (linux.gtk.x86) and so on. The three elements in the fragment name stand for Operating System, os (e.g. linux), Window System, ws (e.g. gtk) and OS Architecture, arch (x86).

The same naming convention is used in Eclipse for other things that are also related to platform-specific code. Another example (in addition to the platform fragments for the server executable which are my topic here) is the location of the Haskell object code which I have mentioned in an earlier post (the simple example walkthrough). Remember that the folder names there also had these 'os' and 'win32' bits? I'll explain the particulars of that in more details in an upcoming post.

For providing a server executable for a platform, the only thing that is needed is a fragment for that platform, and that fragment should contain a folder server, in which the executable is placed. The name of the executable must start with haskellserver. (On Windows, it is called haskellserver.exe, on other platforms it will be very likely just called haskellserver.)

I have organized the Windows fragment so that the build scripts for building the Windows executable are located with the fragment, not with the plugin (where the source code is). This is because I think that there will be platform-specific build scripts in all the other fragments too. The build scripts are in the folder INTERNAL/integration/ in the windows
fragment project.
As you can see, I'm using Cabal for building the executable. That makes it convenient to declare additional package dependencies. The packages that are currently needed are base, network and parsec, of course hs-plugins and the Cohatoe API package (cohatoe-api), and HaXml. Further dependencies may be added in the future, depending on how much more functionality will go into the server.

I have wrapped the several Cabal build steps (configure, build, clean) in a Windows batch file. There I'm also doing a pragmatic step to copy the created executable to the correct destination folder. I'm sure that there is a way to extend Cabal so that this can be done in a Cabal build hook - so if I've got a bit of time, I'll probably see to beautify that :-).

Finally, to execute that build script (i.e. the Windows batch file), I have declared an External Tool in Eclipse. This functionality in Eclipse is very useful, but seemingly not so well-known. You can create a launch configuration for an external tool ('external' means usually a separate process, but the file may be located inside the workspace, as it is in our case) and then have the launch configuration settings written into a workspace file. Mine is in the Haskell server exectuable - win32.launch file that you can see in the build folder in the screenshot above. You can configure it on the dialog that you get from the menu Run > External Tools > External Tools ....

You can see here that I have just declared the batch file as the executable and the folder which contains it as the working directory. In addition, it is very helpful to specify, on the Refresh tab, that the entire project should be refreshed automatically after the tool has run.


Finally, on the Common tab you can specify a location where the launch configuration settings should be stored.

Now that we have the launch configuration we can simply run it from the menu or from the global Eclipse toolbar (look out for the 'External Tools' button). What's even better, we can also register the External Tool launch configuration as a builder on the Eclipse project that contains the sources, so that it is automatically triggered when these sources change. (But that is left as an exercise to the reader ;-).

Wednesday, January 03, 2007

Indirect programming

Yesterday during a debugging session a colleague and I stumbled over a terrific example of what I have called 'indirect programming' elsewhere:

  private void start(String organizerId) {
IBreakpointOrganizer organizer = getOrganizer(organizerId);
IPropertyChangeListener listener = new IPropertyChangeListener() {
public void propertyChange(PropertyChangeEvent event) {
}
};
organizer.addPropertyChangeListener(listener);
organizer.removePropertyChangeListener(listener);
}
Why in the world would someone first add and then immediately remove a listener, and above all a listener with an empty implementation?? Here's why (taken from what happens to be one internal implementation class of IBreakpointOrganizer):
  public void addPropertyChangeListener(IPropertyChangeListener listener) {
getOrganizer().addPropertyChangeListener(listener);
}

[...]
protected IBreakpointOrganizerDelegate getOrganizer() {
if (fDelegate == null) {
try {
fDelegate = (IBreakpointOrganizerDelegate)
fElement.createExecutableExtension(ATTR_CLASS);
} catch (CoreException e) {
DebugUIPlugin.log(e);
}
}
return fDelegate;
}
Adding a listener has the (undocumented) side-effect of initializing the internals of the organizer. The implementation of the method above makes thus use of a) an implementation detail it shouldn't know; and b) a behaviour that is clearly a side-effect of an entirely different functionality. One can only hope that not, at some future time, a source code cleanup will just clean away that 'useless' add/remove a no-op piece of code ...

Tuesday, January 02, 2007

What's in which package?

There are three features in Cohatoe. (Features are a fancy term in Eclipse's terminology for 'installable package' - a feature consists usually of one or more plugins and can be installed or uninstalled via Eclipse's built-in Update Manager). Here is a short explanation what each of them is needed for:

  • Runtime: this contains only those plugins that are needed at runtime.
  • Examples: an example plugin that demonstrates various aspects of the use of Cohatoe.
  • SDK: contains the sources and developer documentation for all of Cohatoe.
(The Examples feature depends on the Runtime feature, and the SDK depends on both others. The Runtime feature itself has no dependencies apart from that to the very basic Eclipse runtime plugins. You can even use it in UI-less mode.)

When you develop an Eclipse plugin that has some functionality implemented in Haskell, you would therefore depend on the Runtime feature. This means that the plugins from the runtime feature must be present on the system of the user of your plugins. The best way to achieve this is to have the end user install the Cohatoe runtime feature and then your feature (which contains your plugins).

For development of such plugins, on the other hand, you will likely want to have the SDK feature around. It contains the API documentation, the extension point documentation, and the sources of Cohatoe itself (useful for debugging). Since the SDK feature depends on the Runtime feature, this means that you also must have the Runtime feature installed, which is obvious anyway ;-).

If you want to hack Cohatoe itself, you could theoretically also use the sources from the SDK feature. However, it is recommended that you get the latest sources from the Darcs repository instead. This means not only that you get the latest changes (which may not yet have been included in the released versions), but also that you will be able to send your changes as patches. Any contribution of patches is very welcome!

If you don't know Darcs yet - please consider to have a look at it. It is a really nice revision control system (i.e. an alternative to the more widely used like CVS or Subversion), very easy to learn and somewhat addictive once one has used it for a while. It differs from the more mainstream alternatives in that it is not only distributed, but also change-oriented (as opposed to version-oriented). You can get a Darcs client and find more information about Darcs at the main Darcs website at darcs.net.

If you want to access the Cohatoe repository, here's what you need:
> darcs get --partial http://leiffrenzel.eu/repos/cohatoe
Have fun!

Monday, January 01, 2007

Tracing deployed plugins

In a previous post I have explained how to use Cohatoe's debugging options (called Tracing in Eclipse) for getting more info about what goes on inside Cohatoe when Haskell functions are registered and called. This is useful when you are contributing an Eclipse extension and something goes wrong, so that your function call doesn't actually happen.

You can also use the same tracing options in an Eclipse installation where your plugins are deployed. This is in some respects a different situation, different from the debugging situation I have described. Your plugins may now reside in .jar archives, for instance, where they before had lived in project folders in the workspace, and that may have some unforeseen effects.

And there is another typical problem with deployed Eclipse installations. When you export a plugin, you have to declare, in the build.properties, which files should be exported and which not. You can find the build.properties in the Plugin Manifest editor, from the Build tab.


In the screenshot, you can see how this looks in my workspace for the Cohatoe examples. You see that the files needed only at development time (such as Eclipse's .project file of the .checkstyle configuration are not checked, but the files that we need to deploy, such as the plugin.xml or the object files, are checked. A typical case is where a developer first creates a project, later adds some files, such as icons or object files, forgets to check them in the build.properties, and then wonders why something doesn't work in the deployed version. To me at least, this happens all the time :-).

One way to detect such a problem, at least with object files contributed via Cohatoe, is to use tracing. In the post I mentioned above I have explained how the tracing output looks like. If an object file was missing, this would be reported in a line in the tracing output.

Now suppose you have some Eclipse installation, where Cohatoe is installed.


This is what a typical Eclipse installation looks like. The only thing that I have added is the .options file. Its content is quite simple:

# option for tracing calls to the Cohatoe server,
# including inits and function calls
de.leiffrenzel.cohatoe.server.core/logs = true
I have just copied it from the Cohatoe server core plugin. (You can add options for other plugins in here too, and get their tracing output too.) Note that I have switched the value to true; the original file has false in this place.

Now run Eclipse from the command line with the following options:
> cd C:\cohatoetest\eclipse
> eclipse -debug -consolelog
You get a console window where you will see the tracing output (and as a nice extra, you will also get the contents of the workspace log appended there, i.e. the output that ususally goes to yourWorkspaceLocation/.metadata/.log).