flowjo-plugin-testbed: Run and debug your FlowJo plugins without using FlowJo

Summary: Developing and debugging FlowJo plugins within FlowJo itself can be difficult and time consuming. By using flowjo-plugin-data-dump and flowjo-plugin-testbed you can simulate the FlowJo environment to run and debug your plugin in a standalone way, much faster and more conveniently than within FlowJo itself.

Rationale

The standard cycle for developing and debugging FlowJo plugins can be quite painful, and usually involves some variation of the following:

  • Build a fat .jar file including dependencies for your plugin
  • Copy it into your /Applications/plugins directory or windows equivalent
  • Start FlowJo from the command line
  • Open your workspace
  • Wait for FlowJo to finish scanning for .jar files
  • Run the plugin and check the console for errors

Sometimes this process can take upwards of 5 minutes for a single round trip, which is not feasible if you're developing a plugin or have a programming style that requires a lot of back and forth debugging. With these two libraries, the process is as simple as this:

  • Press "build and run" in your IDE and your plugin will be invoked with the correct arguments immediately.

Usage

  • Download the data-dump plugin .jar from the github releases page
  • Place it in your /Applications/plugins directory or windows equivalent
  • Start FlowJo
  • Select a sample or node then click Workspace -> Plugins -> FlowJoPluginDataDump
  • It will save files to the output folder and give you a command to copy paste that looks like this:
invokeAlgorithm(FlowJoPluginTestbed.getFcmlFromFile("/some/directory/FlowJo Plugin Data Dump/fcmlQueryElement.xml"), FlowJoPluginTestbed.createFileObject("/some/directory/FlowJo Plugin Data Dump/st_HM-1_CHECK192_001..ExtNode.csv"), FlowJoPluginTestbed.createFileObject("/some/directory/FlowJo Plugin Data Dump"));
    public static void main(String[] args) {
        YourPlugin plugin = new YourPlugin();
        try {
            plugin.invokeAlgorithm(FlowJoPluginTestbed.getFcmlFromFile("/some/directory/FlowJo Plugin Data Dump/fcmlQueryElement.xml"), FlowJoPluginTestbed.createFileObject("/some/directory/FlowJo Plugin Data Dump/st_HM-1_CHECK192_001..ExtNode.csv"), FlowJoPluginTestbed.createFileObject("/some/directory/FlowJo Plugin Data Dump"));
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
  • You'll now be able to use standard Java development and debugging practise to test if your plugin works.

Details

If you're interested in creating plugins for Flowjo, chances are you've read the FlowJo Plugin Development Guide. If you're writing a population plugin, you'll know that the bulk of the work is done by the invokeAlgorithm() method (documented here).

The invokeAlgorithm(SElement fcmlElem, File sampleFile, File outputFolder) function takes 3 arguments:

SElement fcmlElem

This is basically an XML document in "FCML" (flow cytometry markup language) format. It contains metadata describing the current node on which the plugin is being run. Note: this means that you need to dump the plugin data specifically for the sample or node you're intending to test with.

It looks something like this:

<FCML version="3" >
  <FcmlQuery queryId="FCML" >
    <DataSet uri="/Users/nicbarker/Documents/test gating/st_HM-1_CHECK192_001.fcs" >
      <transforms:spilloverMatrix prefix="Comp-"  name="Compensation-copy"  editable="1"  color="#00ccff"  version="FlowJo-10.4"  status="FINALIZED"  transforms:id="ce3c3c11-eef9-4505-b2c6-54495cce1ac4"  suffix="" >
        <data-type:parameters>
          <data-type:parameter data-type:name="B 530_B-A" />
          <data-type:parameter data-type:name="B 695_A-A" />
          <data-type:parameter data-type:name="R 780_A-A" />
          <data-type:parameter data-type:name="UV 379_C-A" />
          <data-type:parameter data-type:name="V 450_F-A" />
          <data-type:parameter data-type:name="V 525_E-A" />
          <data-type:parameter data-type:name="YG 582_D-A" />
          <data-type:parameter data-type:name="YG 780_A-A" />
        </data-type:parameters>

File sampleFile

When you call your population plugin, most of the time FlowJo will actually write out a new file to the disk containing the subpopulation at that node so you can work on it without damaging the original file. The structure and data in this file depends on which version of the plugin you use (FCS_FILE, TRANSFORMED_VALUES_CSV or RAW_VALUES_CSV) You then do your processing on that file and return your results. This is a basic java file object and you can read it from the disk as normal.

File outputFolder

This is just a File object that points to the directory where the previous two files will be stored.

How it works

flowjo-plugin-testbed will read the files off the disk generated by flowjo-plugin-data-dump and provide them to your plugins invokeAlgorithm function, emulating FlowJo. It actually uses one of FlowJo's internal libraries to parse the FCML file.

Accessing the FlowJo error log on Mac OS X (And potentially for any other Application)

Problem: Something is broken or isn't working correctly in FlowJo, but doesn't produce any error messages or debug friendly output.

I was recently trying to debug why a FlowJo plugin I was developing wasn't working. Unfortunately, FlowJo seems to swallow all of the errors that occur in the background, without reporting them to the user.

FlowJo 10.4 on Mac OSX doesn't appear to support the Open Log Window command shown here.

Here's a trick that will allow you to get much more detailed debugging information on FlowJo to help you work out what's going on.

Solution: Run the FlowJo app directly from the command line to capture outputted log messages

Open the terminal and run the following:

/Applications/FlowJo.app/Contents/MacOS/flowjoJavaApplicationStub

You'll see that by launching the FlowJo app directly from the command line, you'll get log output (including errors!) on the command line:

Expanding to other applications

You should theoretically be able to use the same approach (cd inside the .app folder, into whatever-app.app/Contents/MacOS/ and run the application in that folder directly.

Fixing the broken FlowJo 10.4 Installer for Mac OSX

I finally got around to downloading FlowJo 10.4 this morning. Double clicked on the installer as usual, expecting a disk image to mount or something to happen:

Screenshot 2017-11-10 11.40.02.png

 

 

 

After waiting around for a while, nothing seems to happen. It's possible that an embedded agreement is preventing the installer from opening.

There's an easy fix for this using hdutil tool on the OSX command line (terminal):

hdiutil attach ~/Downloads/FlowJo-OSX64-10.4.0.dmg

You'll be presented with a compatibility warning agreement on the command line that looks like the following:

Screenshot 2017-11-10 11.48.33.png

 

 

 

 

Simply press the q key to dismiss the agreement, then Y to agree:

Screenshot 2017-11-10 11.50.38.png

 

The .dmg should mount fine, and you'll see it in the finder.

Screenshot 2017-11-10 11.51.39.png