Natural Language Scripting: A first tutorial
Image Pre-processing

Why use a natural scripting interface

Graphical user interfaces can easily become complex and confusing as the number of user input parameters increases. This is particularly true if a workflow needs to be configured, where (i) each step has its own set of parameters, (ii) steps can occur in any order and (iii) steps can be repeated arbitrarily. Consider the configuration of an image pre-processing workflow, which consists of the following algorithms, each having its own set of parameters:

• Gaussian blurring (standard deviation)
• Median filtering (window radius)
• Background subtraction (window radius)
• Conversion to grayscale
• Intensity normalization

A traditional graphical user interface (GUI) could e.g. look like this:

where the user can activate the various algorithms and specify their parameters as necessary. This user interface however does not take into account that different algorithms could occur repeatedly, and it does not allow to change the order.

Using Natural Language Scripting, we want to implement a text-based interface which reads and executes text like:

Apply Gaussian blurring with a standard deviation of 3 pixel(s).
Subtract the background with a window readius of 30 pixel(s).
Apply Median filtering with a window radius of 1 pixel(s).
Normalize intensities.
Apply Gaussian blurring with a standard deviation of 1 pixel(s).

Create the backend

First of all, we'll implement a backend which does the actual processing. We therefore create a class that implements the actual algorithms. It uses Fiji as an underlying image processing library:

import ij.IJ;
import ij.ImagePlus;
import ij.process.ImageProcessor;

public class Preprocessing {

    private ImagePlus image;

    public Preprocessing(ImagePlus image) {
        this.image = image;
    }

    public void gaussianBlur(float stdDev) {
        IJ.run(image, "Gaussian Blur...", "sigma=" + stdDev);
    }

    public void medianFilter(int radius) {
        IJ.run(image, "Median...", "radius=" + radius);
    }

    public void subtractBackground(float radius) {
        IJ.run(image, "Subtract Background...", "rolling=50");
    }

    public void convertToGray() {
        if(image.getType() == ImagePlus.COLOR_RGB)
            IJ.run(image, "8-bit", "");
    }

    public void intensityNormalization() {
        convertToGray();
        ImageProcessor ip = image.getProcessor();
        double min = ip.getMin();
        double max = ip.getMax();
        ip = ip.convertToFloat();
        ip.subtract(min);
        ip.multiply(1 / (max - min));
        image.setProcessor(ip);
    }
}

Implement an interface that understands a first sentence

The Natural Language Scripting framework offers a convenient way to define the sentences your interface should understand, and provides an auto-completion enabled text editor for users to enter their instructions. The following code snippet shows how to create a parser, how to define a pattern for a sentence for it to parse, and how to display the editor:

Parser parser = new Parser();
parser.defineSentence(
    "Apply Gaussian blurring with a standard deviation of {stddev:float} pixel(s).",
    null);

new ACEditor(parser).setVisible(true);

parser = Parser();
parser.defineSentence(
    "Apply Gaussian blurring with a standard deviation of {stddev:float} pixel(s).",
    None);

ACEditor(parser).show();

parser = nlScript.Parser();
parser.defineSentence(
    "Apply Gaussian blurring with a standard deviation of {stddev:float} pixel(s).",
    undefined);

new nlScript.ACEditor(parser, document.getElementById("nls-container"));

Find the full code here: Java Python JavaScript.

In this example we state that we expect a literal "Apply Gaussian blurring with a standard deviation of ", followed by a floating point number, which we name "stddev" for later reference, followed by the literal "pixel(s).". There is a second parameter to defineSentence(), which we'll discover next.

The code snippet is sufficient to provide the means for user input, but nothing happens yet when the user clicks the Run button.

Find more information about How to specify variables.

Evaluating the parsed text

The second argument to parser.defineSentence(), which was omitted above, is of type Evaluator. Evaluator is an interface with a single function

interface Evaluator {
   Object evaluate(ParsedNode pn); 
}

class IEvaluator(ABC):
    @abstractmethod
    def evaluate(self, pn: ParsedNode) -> object:
        pass


class Evaluator(IEvaluator):
    def __init__(self, evaluate: Callable[[ParsedNode], object]):
        self._evaluate = evaluate

    def evaluate(self, pn: ParsedNode) -> object:
        return self._evaluate(pn)

export type Evaluator = (pn: ParsedNode) => any;

The task of the evaluator is to evaluate the expression (the sentence) we defined on the parser. In the example, it is responsible for the actual blurring. The argument to evaluate(), pn, is of type ParsedNode, which can be used to retrieve the parsed value for the standard deviation.

Variables in nlScript are defined hierarchically (see the paragraph "Custom types and type hierarchy" below). The result of parsing is also a hierarchical tree-like structure consisting of nodes of type ParsedNode. The ParsedNode has child ParsedNodes representing the variables the current type consists of. The sentence above consists of a string literal "Apply Gaussian blurring with a standard deviation of", a floating-point variable called "stddev", and a string literal "pixel(s).". Therefore, the ParsedNode given to the sentence's Evaluator has three child ParsedNodes, which can be accessed, e.g. by name, in the Evaluator and evaluated recursively.

ImagePlus image = IJ.openImage("http://imagej.net/images/clown.jpg");
Preprocessing preprocessing = new Preprocessing(image);
Parser parser = new Parser();
parser.defineSentence(
    "Apply Gaussian blurring with a standard deviation of {stddev:float} pixel(s).",
    pn -> {
        double stdDev = (double)pn.evaluate("stddev");
        preprocessing.gaussianBlur((float)stdDev);
        return null;
    });

new ACEditor(parser).setVisible(true);

preprocessing = Preprocessing(None)
preprocessing.open('http://imagej.net/images/clown.jpg')
preprocessing.show()

parser = Parser()

def evaluateSentence(pn):
    stddev = pn.evaluate("stddev")
    preprocessing.gaussianBlur(stddev)

parser.defineSentence(
    "Apply Gaussian blurring with a standard deviation of {stddev:float} pixel(s).",
    evaluateSentence)

editor = ACEditor(parser)
editor.show()

let preprocessing = new Preprocessing("output");

let parser = new nlScript.Parser();
parser.defineSentence(
  "Apply Gaussian blurring with a standard deviation of {stddev:float} pixel(s).",
  pn => {
    let stdDev = pn.evaluate("stddev");
    preprocessing.gaussianBlur(stdDev);
    preprocessing.show("output");
    return undefined;
  });

new nlScript.ACEditor(parser, document.getElementById("nls-container"));

Find the full code here: Java Python JavaScript.

This is the first fully working example:

Built-in types

Above, we specified the type of the standard deviation parameter stddev to be a floating point number, using {stddev:float}.

The following tables show other available built-in types:

Custom types and type hierarchy

Details about custom types

It is possible and also common to define custom types. We could e.g. define a type filter-size which consists of a floating point number and a unit (e.g. pixel(s)). In defineSentence we could then use filter-size as type for the standard deviation parameter:

parser.defineType(
    "filter-size",
    "{stddev:float} pixel(s)",
    pn -> pn.evaluate("stddev"));

parser.defineSentence(
    "Apply Gaussian blurring with a standard deviation of {stddev:filter-size}.",
    pn -> {
        double stdDev = (double)pn.evaluate("stddev");
        preprocessing.gaussianBlur((float)stdDev);
        return null;
    });

def evaluateFilterSize(pn):
    return pn.evaluate("stddev")

# Create a custom type 'filter-size'
parser.defineType(
    "filter-size",
    "{stddev:float} pixel(s)",
    evaluator=evaluateFilterSize)

def evaluateSentence(pn):
    stddev = pn.evaluate("stddev")
    preprocessing.gaussianBlur(stddev)

parser.defineSentence(
    "Apply Gaussian blurring with a standard deviation of {stddev:filter-size}.",
    evaluator=evaluateSentence)

parser.defineType(
  "filter-size",
  "{stddev:float} pixel(s)",
  pn => pn.evaluate("stddev"));

parser.defineSentence(
  "Apply Gaussian blurring with a standard deviation of {stddev:filter-size}.",
  pn => {
    let stdDev = pn.evaluate("stddev");
    preprocessing.gaussianBlur(stdDev);
    preprocessing.show("output");
    return undefined;
  });

Find the full code here: Java Python JavaScript.

Autocompletion and evaluation will work as before, but the type filter-size can be re-used, e.g. for other filters like the median filter.

Custom types can be defined using built-in types and other custom types. In this way, an entire hierarchy of types can be built up. Here are more Details about type hierarchy.

Fine-tuning autocompletion: Parameterized autocompletion

Defining custom types has an additional advantage: It allows to customize autocompletion. In the example above, with default autocompletion in place, autocompletion looks like

The user sees that a value for the stddev parameter is required, but doesn't know in which units the value needs to be entered. Usage is much clearer if we use inline, or parameterized autocompletion:

This is accomplished by specifying a 4th parameter to defineType, a boolean that specifies whether to use parameterized autocompletion or not (false is the default):

parser.defineType("filter-size",
                  "{stddev:float} pixel(s)",
                  pn -> pn.evaluate("stddev"),
                  true);

parser.defineType("filter-size",
                  "{stddev:float} pixel(s)",
                  lambda pn: pn.evaluate("stddev"),
                  True);

parser.defineType("filter-size",
                  "{stddev:float} pixel(s)",
                  pn => pn.evaluate("stddev"),
                  true);

Find the full code here: Java Python JavaScript.

Multiple-choice autocompletion: choose a unit for filter-size

Microscope images commonly have a calibrated pixel size, and it is beneficial to specify algorithm parameters in real-world units instead of pixels. This facilitates the re-use of processing workflows, even if images were acquired at different resolutions. To extend the script in this regard, we define another type units. To take different units into account, we can re-define units. Here we will define it as pixel(s) and calibrated units (i.e. the units in which the image is calibrated). If the user selects the first option, we let the units type evaluate to false, otherwise to true.

parser.defineType("units", "pixel(s)", pn -> false);
parser.defineType("units", "calibrated units", pn -> true);

parser.defineType("units", "pixel(s)", lambda pn:  false)
parser.defineType("units", "calibrated units", lambda pn: true)

parser.defineType("units", "pixel(s)", pn => false);
parser.defineType("units", "calibrated units", pn => true);

The definition of filter-size now becomes

parser.defineType("filter-size", "{stddev:float} {units:units}", pn -> {
    double stddev = (Double) pn.evaluate("stddev");
    boolean units = (Boolean) pn.evaluate("units");
    if(units)
        stddev /= image.getCalibration().pixelWidth;
    return stddev;
}, true);

def evaluateFilterSize(pn):
    stddev = pn.evaluate("stddev")
    units = pn.evaluate("units")
    if units:
        stddev /= preprocessing.getPixelWidth()
    return stddev


parser.defineType("filter-size",
                  "{stddev:float} {units:units}",
                  evaluateFilterSize,
                  True);

parser.defineType("filter-size", "{stddev:float} {units:units}", pn -> {
    let stddev = pn.evaluate("stddev");
    let units = pn.evaluate("units");
    if(units)
        stddev /= preprocessing.getPixelWidth();
    return stddev;
}, true);

Find the full code here: Java Python JavaScript.

In evaluate() we use pn to retrieve the values for stddev and units. If units evaluates to true (indicating that the user chose calibrated units), we calculate the filter-size in pixels, dividing by the pixel size.

Autocompletion will now look as follows:

Then, after typing 5, followed by tab:

Dynamic autocompletion at runtime using a custom Autocompleter

The completion option calibrated units obviously does not read very nice. Preferably we would replace it with the actual units the image was calibrated with, e.g. microns. This is runtime dependent, i.e. it depends on the image on which the script is executed. In the code example above, the image is opened before creating the language, but in a more realistic setting, we would like to run our script on any user-selected image.

To replace calibrated units with the real units, we use a custom Autocompleter, which needs to implement the Autocompleter interface:

public interface Autocompleter {
    Autocompletion[] getAutocompletion(ParsedNode pn, boolean justCheck);
}

class IAutocompleter(ABC):
    @abstractmethod
    def getAutocompletion(self, pn: DefaultParsedNode, justCheck: bool) -> List[Autocompletion] or None:
        pass


class Autocompleter(IAutocompleter):
    def __init__(self, getAutocompletion: Callable[[ParsedNode, bool], List[Autocompletion]]):
        self._getAutocompletion = getAutocompletion

    def getAutocompletion(self, pn: ParsedNode, justCheck: bool) -> List[Autocompletion] or None:
        return self._getAutocompletion(pn, justCheck)

interface Autocompleter {
    getAutocompletion(n: DefaultParsedNode, justCheck: boolean): Autocompletion[] | undefined;
}

getAutocompletion() returns an array of possible completions. As arguments, it takes a ParsedNode, which can be queried for what's already entered (pn.getParsedString()). A second parameter justCheck indicates that the actual completions are not needed and the function should just return whether it does autocomplete itself (return Autocompletion.doesAutocomplete()) or it leaves autocompletion to its children (return null). This is useful if the calculation of possible completion options is computationally expensive. Normally, getAutocompletion() returns an array of Autocompletions. There are different subclasses for Autocompletion: Autocompletion.Literal for string constants, Autocompletion.Parameterized for a (single) parametric completion, Autocompletion.EntireSequence for a sequence of literal and parameterized completions, Autocompletion.Veto, which is a special completion which, if present in the list of options, prohibits autocompletion totally. There are several convenience functions to create the different kinds of autocompletions in the Autocompletion class.

In the example at hand, instead of multiple definitions of the type units, we define units to be an arbitrary string that may contain characters a-z, A-Z, ( and ). Then, we use a custom Autocompleter as a 4th parameter to defineType, which returns as possible autocompletions pixel(s) and the actual string representing the units the open image is calibrated in (imageUnits).

parser.defineType(
    "units",                                                                // the type name
    "{unitstring:[a-zA-Z()]:+}",                                            // the pattern to parse
    pn -> !pn.getParsedString().equals("pixel(s)"),                         // the evaluator
    (pn, justCheck) -> Autocompletion.literal(pn, "pixel(s)", imageUnits)); // the autocompleter

def getAutocompletion(pn, justCheck):
    return Autocompletion.literal(pn, "pixel(s)", imageUnits)

parser.defineType(
    "units",                                                                // the type name
    "{unitstring:[a-zA-Z()]:+}",                                            // the pattern to parse
    lambda pn: pn.getParsedString() != "pixel(s)",                          // the evaluator
    getAutocompletion)                                                      // the autocompleter

parser.defineType(
    "units",                                                                // the type name
    "{unitstring:[a-zA-Z()]:+}",                                            // the pattern to parse
    pn => pn.getParsedString() !== "pixel(s)",                              // the evaluator
    (pn, justCheck) => Autocompletion.literal(pn, "pixel(s)", imageUnits)); // the autocompleter

Autocompletion.literal creates an array of literal completions from a ParsedNode and a variable number of String arguments.

So where does imageUnits come from? We need to catch it at runtime, so we register a ParseStartListener on the parser, and read the image calibration units at the time parsing is started:

StringBuilder imageUnits = new StringBuilder();
parser.addParseStartListener(() -> {
    imageUnits.setLength(0);
    imageUnits.append(image.getCalibration().getUnits());
});

imageUnits = ""

def parsingStarted():
    global imageUnits
    imageUnits = preprocessing.getUnits()

parser.addParseStartListener(listener=ParseStartListener(parsingStarted))

let imageUnits = "";
parser.addParseStartListener(() => {
  imageUnits = preprocessing.getUnits();
});

Find the full code here: Java Python JavaScript.

Once autocompletion hits the units phrase, it displays a dropdown menu with the 2 options pixel(s) and mm, as desired.

Prohibit further autocompletion: Autocompleter.VETO

However, when we start typing a value for units, e.g. we manually enter cm, it still shows pixel(s), mm and additionally .. That's because our Autocompleter always returns the two options. We could decide to just skip autocompletion once the user starts to type. We can do this in the Autocompleter, by checking if the user already entered something, and in case s/he did, we return Autocompletion.veto():

parser.defineType(
    "units",
    "{unitstring:[a-zA-Z()]:+}",
    pn -> !pn.getParsedString().equals("pixel(s)"),
    (pn, justCheck) -> pn.getParsedString().isEmpty()
                        ? Autocompletion.literal(pn, "pixel(s)", imageUnits)
                        : Autocompletion.veto(pn));

def getAutocompletion(pn, justCheck):
    if len(pn.getParsedString()) == 0:
        return Autocompletion.literal(pn, ["pixel(s)", imageUnits])
    return Autocompletion.veto(pn)

parser.defineType(
    "units",
    "{unistring:[a-zA-Z()]:+}",
    lambda pn: pn.getParsedString() != "pixel(s)",
    getAutocompletion)

parser.defineType(
    "units",
    "{unitstring:[a-zA-Z()]:+}",
    pn => pn.getParsedString() !== "pixel(s)",
    (pn, justCheck) => pn.getParsedString().length === 0
        ? nlScript.Autocompletion.literal(pn, ["pixel(s)", imageUnits])
        : nlScript.Autocompletion.veto(pn));

Find the full code here: Java Python JavaScript.

Dynamically re-defining types

The above solution works, but the meaning is a little different from what we wanted to achieve originally. Although it shows pixel(s) and mm as completion options, it accepts any string (we defined the pattern to parse to be a-zA-Z()). In fact, we'd ideally like to restrict possible types to pixel(s) and mm. And we can implement this by dynamically changing the parser, i.e. re-defining the type units. We do this again with a ParseStartListener:

parser.addParseStartListener(() -> {
    String unitsString = image.getCalibration().getUnits();

    parser.undefineType("units");

    // Re-define the 'units' type
    parser.defineType("units", "pixel(s)", pn -> false);
    parser.defineType("units", unitsString, pn -> true);
});

def parsingStarted():
    unitsString = preprocessing.getUnits()

    parser.undefineType("units")

    # Re-define the 'units' type
    parser.defineType("units", "pixel(s)", lambda pn: False)
    parser.defineType("units", unitsString, lambda pn: True)


parser.addParseStartListener(listener=ParseStartListener(parsingStarted))

parser.addParseStartListener(() => {
    imageUnits = preprocessing.getUnits();

    parser.undefineType("units");

    // Re-define the 'units' type
    parser.defineType("units", "pixel(s)", pn => false);
    parser.defineType("units", imageUnits, pn => true);
});

Find the full code here: Java Python JavaScript.

This does now exactly what we wanted, i.e. it acts like the units type was defined from the beginning on with the calibration unit of the image.

This is indeed a very powerful feature, as it allows for dynamic re-definition of the parsed language, based on current runtime circumstances.