Home
Arenadata Streaming
How to
NiFi
Basic operations
Work with attributes

Work with attributes

Olga Semme

Contents

FlowFile attributes
Processor properties
Parameters
Variables
Parameters using Expression Language

FlowFile attributes

All data in Apache NiFi is represented by an abstraction called FlowFile. A FlowFile has two main parts: content and attributes. Information about the FlowFile is available by clicking the info icon on the List queue page of the flow queue context menu.

FlowFile Data

FlowFile attributes

For a complete list of core attributes, see FlowFile in the NiFi Developer Guide.

FlowFile attributes can be retrieved and used as processor properties when configuring flows in the NiFi interface.

Processor properties

During processor сonfiguring, the PROPERTIES tab displays processor properties.

The Property column contains the name of the parameter. The Value column indicates the value of the parameter. Depending on the values allowed for the property, the Value column either provides the user with a drop-down list from which to select a value, or provides a text area for entering a value.

Processor properties

For parameters with a text area for entering values, the following forms of entering a value are available:

directly;
using parameters defined via the parameters context;
using variables set via Expression Language;
using parameters along with Expression Language (see Parameters and Expression Language).

The upper part of the value entry window indicates whether Expression Language (EL) and Parameters (PARAM) are supported for entering the value of a particular property of this processor.

Enter parameter value

Parameters have the following advantages over variables:

support for confidential values;
more detailed control over access policies;
properties that refer to parameters are checked against the replaced value, unlike most properties that refer to variables using an Expression language.

Parameters

Flow property values, including sensitive properties, can be parameterized using parameters. Parameters are created in the Parameter Contexts window of the NiFi Server interface global menu. Parameter contexts are globally defined for each NiFi instance. Access policies can be applied to parameter contexts to determine which users can create them.

Any property can be configured to refer to a parameter with the following conditions:

A private property can only refer to a private parameter.
A non-private property can only refer to a non-private parameter.
Properties that refer to controller services cannot use parameters.
Parameters cannot be referenced in reporting tasks or management controller services.

Create a parameter context

To create a parameter context, you need to:

Via NiFi Server interface global menu open Parameter Contexts window.
Click + to create the parameter context.
On the SETTINGS tab of the Add Parameter Context window that opens, enter the name and description of the parameter context.

Parameter context creation

Parameter context creation
Click APPLY.

As a result, the created parameter context is displayed in the Parameter Contexts window.

Created parameter context

Created parameter context

Create parameters in the parameter context

NOTE

You can both create parameters when creating a parameter context, and add parameters to an already created parameter context.

To create a new parameter in a parameter context, do the following:

Open the created parameter context for editing by clicking on the edit icon in the parameter context line.
On the PARAMETERS tab, click + to create a parameter.
In the Add Parameter window that opens, fill in the fields for the parameter:
- Name — the name that is used to designate the parameter. Only alphanumeric characters, hyphens, underscores, periods, and spaces are allowed.
- Value — the value that will be used when referring to the parameter. If the parameter uses an Expression language, it is important to note that the Expression language will be evaluated in the context of the component that references the parameter. See below for more information about using Expression language in parameters.
- Set empty string — check this box to explicitly set the value of the parameter to an empty string. Not checked by default. If the checkbox is checked but the value is set, the checkbox is ignored.
- Sensitive Value — set to Yes if the parameter value should be considered sensitive. If the setting is private, the setting’s value will not be displayed in the user interface after it is applied. The default value is No. Confidential parameters can only be referenced by confidential properties, and non-confidential parameters can only be referenced by non-confidential properties. Once a setting has been created, its sensitivity value cannot be changed.
- Description — parameter description. This field is optional.
  
  Create parameter
  
  Create parameter
Click APPLY.

As a result, an information window for updating or creating a parameter opens. To check all components that refer to added or changed parameters, the following operations are performed: stopping or restarting dependent processors, disabling or re-enabling dependent controller services, updating the parameter context.

Information window

Information window
Click CLOSE. As a result, the created parameter is displayed in the Update Parameter Context window.

Created parameter

Created parameter

Work with the parameter context in the NiFi server interface

In order to propagate the created parameter context to a process group, do the following:

On a free field, right-click to open the context menu and select the Configure command.
In the NiFi Flow Configuration window that opens, on the GENERAL tab, for the Process Group Parameter Context parameter, select the required parameter context in the pop-up list.
Click APPLY.

Enable parameter context for a process group

Enable parameter context for a process group

In order to apply the created parameter from the selected parameter context for a specific processor, do the following:

On the selected processor, right-click to open the context menu and select the Configure command.
On the PROPERTIES tab, select the required parameter and in the opened window enter the parameter value in the form #{name}, where name is the name entered when creating the parameter in the parameters context.

Apply the created parameter

image::nifi/basic/nifi_attrib_09_dark.png[width=724,alt="Apply the created parameter .Apply the created parameter
Click OK.

In case of successful application of the parameter, the icon arrow2 light appears in the last column of the assigned property. When clicked, the icon goes to the Update Parameter Context page, where this parameter is available to change.

Created parameter on the PROPERTIES tab

Update Parameter Context

When clicking on the icon arrow1 light , each parameter can also be written to the given parameter context.

Writing a parameter from the available processor properties

Check dependent components

On the PARAMETERS tab of the Parameter Contexts window for the selected parameter, the Referencing Components line displays the names of all processors that use this parameter.

Dependent components

Variables

Variables are created and configured in the NiFi user interface. They can be used in any realm that supports Expression Language. Variables cannot be used for sensitive properties. NiFi automatically picks up new or changed variables. Variables are defined at the process group level, so the access policies for viewing and modifying variables are derived from the process group access policies.

Create a variable

To create a variable, do the following:

On a free field, right-click to call the context menu and select the Variables command.
In the window that opens, click + to create a variable.
Enter variable name.
Click OK.

Creating a variable

Creating a variable
Enter the value of the variable.

Variable value entry

Variable value entry
Click OK.
In the Variables window, click APPLY. As a result, an information window for updating or creating a variable opens.

Information window

Information window

Use variables in processor properties via Expression Language

In order to apply the created variable for a specific processor, do the following:

On the selected processor, right-click to open the context menu and select the Configure command.
On the PROPERTIES tab, select the required parameter and in the opened window enter the parameter value in the form ${name}, where name is the name entered when creating the variable.

Apply the created parameter

Apply the created parameter
Expression Language functions

With the help of various Expression Language functions it is possible to manipulate the value of variables. Some functions provide string (text) manipulation, such as the toUpper function. Others, such as the equals and match functions, provide comparison functionality. There are also functions for manipulating dates and times and for performing mathematical operations.

Using the necessary expression structure it is possible, for example, to return an uppercase version of a filename by calling the toUpper function: ${filename:toUpper()}. In this case, the filename attribute is referenced and then manipulated with the value using the toUpper function.

A function call consists of 5 elements:
- Function call separator :.
- Function name, in this case toUpper.
- Opening bracket (.
- Function arguments. The required arguments depend on which function is being called. In this example, we are using the toUpper function, which has no arguments, so this element is omitted.
- The closing bracket ) indicates the end of a function call.
After entering the function for the variable, click OK in the property value input window and APPLY on the PROPERTIES tab in the Configure window of the processor.

NOTE

Variables in a child process group override values in the parent process group. In particular, if a variable is set in the root group and also gets a different value within the process group, then the components within the process group use the value defined directly in the process group itself.

Check dependent components

In the Variables window for the selected variable, the Referencing Components line displays the names of all processors that use this parameter.

Dependent сomponents

CAUTION

When creating custom properties, ensure that each custom property contains a distinct property value so that it is not overridden by existing environment properties, system properties, or FlowFile attributes.

Parameters using Expression Language

When adding a parameter that uses an Expression language, it is important to understand the context in which the Expression language will be evaluated. The Expression is always evaluated in the context of the processor or controller service that refers to the parameter. Take, for example, a scenario where a parameter named Time has a value of ${now()}. The result of the Expression language is a call to determine the system time when it is evaluated. When you add the Time parameter as a property, the system time is not evaluated when the parameter is added, but when the Expression is evaluated by the processor or controller service. That is, if the processor has a property set to #{Time}, it will behave exactly as if the property had been set to ${now()}. Each time the property is referenced, it creates a different timestamp.

In addition, some properties do not allow the use of an Expression language, while others allow the use of an Expression language but do not evaluate Expressions on the FlowFile attributes. To understand how this works, consider a parameter named File, whose value is ${filename}. Then consider three different properties, each with a different Expression language scope and a FlowFile with a file name of test.txt. If each of these properties is set to #{File}, the resulting value is shown in the following table.

Customized property value Expression language scope Property effective value Notes

#{File}

FlowFile Attributes

test.txt

The filename is resolved by looking at the filename attribute

Variable registry only

Empty string

FlowFile attributes are out of scope and we are assuming there is no variable named filename in the variable registry

${filename}

The literal text ${filename} will not be evaluated

Found a mistake? Seleсt text and press Ctrl+Enter to report it