Usе Expression Language

The NiFi Expression Language allows you to manipulate values of FlowFile attributes when setting properties of processors.

Expression Language can also be used:

Expression Language structure

The NiFi Expression Language always starts with a start delimiter ${ and ends with an end delimiter }. Between the start and end delimiters is the text of the expression itself. In its simplest form, an expression can only consist of an attribute name. For example, ${topic.name} will return the value of the topic name attribute.

To manipulate the value of an attribute, variable, or parameter, the Expression Language uses various functions.

You can replace part of an attribute’s value by calling the replace function — ${topic.name:replace('1', '2')}. In this case, we refer to the topic name attribute, and then change the value of this attribute from 1 to 2.

When a function call is made on an attribute, the attribute is the subject on which the function operates. It is possible to combine multiple function calls, where the return value of the first function becomes the subject of the second function, and its return value becomes the subject of the third function, and so on. There is no limit to the number of functions that can be linked together.

Function call structure

A function call consists of 5 elements:

  • Function call separator :.

  • The name of the function, in this case replace.

  • Opening parenthesis (.

  • Function arguments. The required arguments depend on which function is being called. The example above uses the arguments:

    • 1 — the text to be replaced;;

    • 2 — replacement text.

  • The closing parenthesis ) indicates the end of a function call.

Any FlowFile attribute can be referenced using an expression language. However, if the attribute name contains a "special character", the attribute name must be escaped by enclosing it in quotation marks. The following characters are considered "special characters" — $, |, {, }, (, ), [, ], ,, :, ;, /, *, ', space, \t, \r, \n.

In addition, a number is considered a "special character" if it is the first character of an attribute name. If any of these special characters are present in an attribute, it is enclosed in single or double quotes.

There are also some functions that assume the absence of a subject. These functions are called by a simple function call at the beginning of an expression, such as ${hostname()}. Attempting to evaluate a function with a subject will result in an error.

Expression Language hierarchy

When using an Expression Language to refer to a property by name, there is a specific hierarchy in which NiFi will look for the value.

The current hierarchy in NiFi looks like this:

  1. Finding a FlowFile for an attribute/key.

  2. Finding Process Group Variables for an attribute/key.

  3. Finding a file in the File Registry for an attribute/key.

  4. Finding NiFi JVM Properties for an attribute/key.

  5. Finding system environment variables for an attribute/key.

NiFi will search and return the first occurrence of the matching property. If no matching property is found, null is returned.

Expression Language editor

Setting the value of processor properties is more convenient when using the Expression Language editor. As soon as the expression starts with ${, the editor starts to highlight the parentheses and curly braces so that the user can easily tell which open brace or curly brace matches which close brace or curly brace.

Bracket selection
Bracket selection
Bracket selection
Bracket selection

The editor also provides context-sensitive help, showing a list of all functions that can be used at the current cursor position. To activate this feature, press Ctrl+Space.

Context sensitive help
Context sensitive help
Context sensitive help
Context sensitive help

The user can also type part of a function name and then press Ctrl+Space to see all available functions that start with the same prefix:

  • replace

  • replaceAll

  • replaceEmpty

  • replaceFirst

  • replaceNull

Quick function selection
Quick function selection
Quick function selection
Quick function selection

If you continue typing, you can narrow down the displayed functions. You can also select one of the functions from the list by double-clicking it or using the arrow keys to highlight the desired function and press Enter.

Data types for Expression Language

Each function subject value, argument value, and function return value has a specific data type. The type of the function subject (Subject Type), the type of arguments (Arguments), if any, or the return type of the function (Return Type), are specified in the description of each of functions.

The Expression Language supports four different data types:

  • String  — a sequence of characters that can be numbers, letters, spaces, and special characters.

  • Number — an integer consisting of one or more digits (from 0 to 9). When converted to numbers from the Date data types, they are represented as the number of milliseconds since midnight GMT on January 1, 1970.

  • Decimal — a numeric value that can support decimal and larger values ​​with minimal loss of precision. More precisely, it is 64-bit IEEE 754 double-precision floating point.

  • Date — an object that contains the date and time. Using the Date Manipulation and Type Coercion functions, this data type can be converted to/from strings and numbers.

  • Boolean — a boolean value that can be either true or false.

After evaluating the Expression Language functions, all attributes are stored as type String.

Use Expression Language when working with processors

Expression Language is widely used in the NiFi Server interface to set processor properties. Not all processor properties support Expression Language. Information about whether the Expression Language is supported for this processor property is indicated at the top of the window for entering a value under processor settings on the PROPERTIES tab.

Also, when you click on the info question mark icon next to the property name, you can see which scope for the Expression Language is supported:

  • NONE — Expression Language is not supported for this property.

  • VARIABLE_REGISTRY — Expression Language is supported for this property at the variables level. In this case, the search for a variable is performed hierarchically:

    • Variables are defined at the process group level, and then, recursively, up to the higher process group level up to the root process group.

    • Variables defined in custom properties files via the nifi.variable.registry.properties property in the nifi.properties file.

    • Environment variables defined at the JVM level and system properties.

  • FLOWFILE_ATTRIBUTES — attributes of each individual FlowFile, as well as variables can be used for Expression Language.

Variable reference example

For the PublishKafka processor, only variables can be used when setting the Kafka Brokers property.

Scope for Expression Language
Scope for Expression Language
Scope for Expression Language
Scope for Expression Language

By creating a kafka.broker variable with the kafka-1.ru-central1.internal:9092 value, this value can be manipulated by setting the ​​kafka-2.ru-central1.internal:9092, kafka-3.ru-central1.internal:9092 values and so on, as described by above. An expression to replace part of a variable’s value looks like this:

${kafka.broker:replace('1', '2')}
Specifying multiple values ​​with replacement of part of the text
Specifying multiple values ​​with replacement of part of the text
Specifying multiple values ​​with replacement of part of the text
Specifying multiple values ​​with replacement of part of the text
NOTE

Replacement of symbols can be used with a large number of Kafka brokers with hostnames that differ by a few symbols. When creating new processors, using one or more character substitutions, you can set the value of Kafka brokers without having to write the broker name manually, enter each Kafka broker in a parameter context or as a variable.

FlowFile attribute reference example

Further, setting the following Topic Name property for the PublishKafka processor, you can see that this property allows the use of FlowFile attributes.

Scope for Expression Language
Scope for Expression Language

Scope for Expression Language

Scope for Expression Language

For this property, you can set the value of the FlowFile attribute filename and append the current time value to it using the now function with the format function.

CAUTION

When using an Expression Language in the properties of the PublishKafka processor, consider the following:

  • To automatically create a topic, enable the auto.create.topics.enable parameter in the server.properties group when configuring the Kafka service.

  • The topic name can include alphanumeric characters (including ASCII), as well as the characters ., _, and -. Any other characters will generate an error and a bulletin.

The expression looks like this:

${filename}_${now():format("HH.mm.ss", "GMT+3")}
File name using the current time as the topic name
File name using the current time as the topic name
File name using the current time as the topic name
File name using the current time as the topic name

Then, if the value of the filename attribute in the given FlowFile is datafile, the PublishKafka processor publishes the data in the topic datafile_12.26.14, where 12.26.14 is the publication time of the topic in the format "HH.mm.ss", in the timezone "GMT+3".

NOTE

The file name with the current time as the topic name can be used when it is necessary to publish data from a FlowFile created from a frequently overwritten local file to Kafka topics.

Example of a parameter value in a parameter context

The Topic Name property of the PublishKafka processor can be configured via a parameter context. If you create the topic parameter with the value ${filename} and the time parameter with the value ${now():format("HH.mm.ss", "GMT+3")} in the parameter context, you can apply these parameters as a property.

The expression looks like this:

#{topic}_#{time}
File name using the current time as the topic name
File name using the current time as the topic name
File name using the current time as the topic name
File name using the current time as the topic name

Then, if the value of the filename attribute in the given FlowFile is datafile, the PublishKafka processor publishes the data to the topic datafile_12.26.14, as in the previous example.

Found a mistake? Seleсt text and press Ctrl+Enter to report it