Work with attributes

FlowFile attributes

All data in Apache NiFi is represented by an abstraction called FlowFile. A FlowFile has two main parts: content and attributes. Information about the FlowFile is available by clicking the info icon on the List queue page of the flow queue context menu.

nifi attrib 18 dark
FlowFile Data
nifi attrib 18 light
FlowFile Data
nifi attrib 19 dark
FlowFile attributes
nifi attrib 19 light
FlowFile attributes

For a complete list of core attributes, see FlowFile in the NiFi Developer Guide.

FlowFile attributes can be retrieved and used as processor properties when configuring flows in the NiFi interface.

Processor properties

During processor сonfiguring, the PROPERTIES tab displays processor properties.

The Property column contains the name of the parameter. The Value column indicates the value of the parameter. Depending on the values allowed for the property, the Value column either provides the user with a drop-down list from which to select a value, or provides a text area for entering a value.

nifi attrib 01 dark
Processor properties
nifi attrib 01 light
Processor properties

For parameters with a text area for entering values, the following forms of entering a value are available:

The upper part of the value entry window indicates whether Expression Language (EL) and Parameters (PARAM) are supported for entering the value of a particular property of this processor.

nifi attrib 02 dark
Enter parameter value
nifi attrib 02 light
Enter parameter value

Parameters have the following advantages over variables:

  • support for confidential values;

  • more detailed control over access policies;

  • properties that refer to parameters are checked against the replaced value, unlike most properties that refer to variables using an Expression language.

Parameters

Flow property values, including sensitive properties, can be parameterized using parameters. Parameters are created in the Parameter Contexts window of the NiFi Server interface global menu. Parameter contexts are globally defined for each NiFi instance. Access policies can be applied to parameter contexts to determine which users can create them.

Any property can be configured to refer to a parameter with the following conditions:

  • A private property can only refer to a private parameter.

  • A non-private property can only refer to a non-private parameter.

  • Properties that refer to controller services cannot use parameters.

  • Parameters cannot be referenced in reporting tasks or management controller services.

Create a parameter context

To create a parameter context, you need to:

  1. Via NiFi Server interface global menu open Parameter Contexts window.

  2. Click + to create the parameter context.

  3. On the SETTINGS tab of the Add Parameter Context window that opens, enter the name and description of the parameter context.

    nifi attrib 03 dark
    Parameter context creation
    nifi attrib 03 light
    Parameter context creation
  4. Click APPLY.

    As a result, the created parameter context is displayed in the Parameter Contexts window.

    nifi attrib 04 dark
    Created parameter context
    nifi attrib 04 light
    Created parameter context

Create parameters in the parameter context

NOTE

You can both create parameters when creating a parameter context, and add parameters to an already created parameter context.

To create a new parameter in a parameter context, do the following:

  1. Open the created parameter context for editing by clicking on the edit icon manage in the parameter context line.

  2. On the PARAMETERS tab, click + to create a parameter.

  3. In the Add Parameter window that opens, fill in the fields for the parameter:

    • Name — the name that is used to designate the parameter. Only alphanumeric characters, hyphens, underscores, periods, and spaces are allowed.

    • Value — the value that will be used when referring to the parameter. If the parameter uses an Expression language, it is important to note that the Expression language will be evaluated in the context of the component that references the parameter. See below for more information about using Expression language in parameters.

    • Set empty string — check this box to explicitly set the value of the parameter to an empty string. Not checked by default. If the checkbox is checked but the value is set, the checkbox is ignored.

    • Sensitive Value — set to Yes if the parameter value should be considered sensitive. If the setting is private, the setting’s value will not be displayed in the user interface after it is applied. The default value is No. Confidential parameters can only be referenced by confidential properties, and non-confidential parameters can only be referenced by non-confidential properties. Once a setting has been created, its sensitivity value cannot be changed.

    • Description — parameter description. This field is optional.

      nifi attrib 05 dark
      Create parameter
      nifi attrib 05 light
      Create parameter
  4. Click APPLY.

    As a result, an information window for updating or creating a parameter opens. To check all components that refer to added or changed parameters, the following operations are performed: stopping or restarting dependent processors, disabling or re-enabling dependent controller services, updating the parameter context.

    nifi attrib 06 dark
    Information window
    nifi attrib 06 light
    Information window
  5. Click CLOSE. As a result, the created parameter is displayed in the Update Parameter Context window.

    nifi attrib 07 dark
    Created parameter
    nifi attrib 07 light
    Created parameter

Work with the parameter context in the NiFi server interface

In order to propagate the created parameter context to a process group, do the following:

  1. On a free field, right-click to open the context menu and select the Configure command.

  2. In the NiFi Flow Configuration window that opens, on the GENERAL tab, for the Process Group Parameter Context parameter, select the required parameter context in the pop-up list.

  3. Click APPLY.

    nifi attrib 08 dark
    Enable parameter context for a process group
    nifi attrib 08 light
    Enable parameter context for a process group

In order to apply the created parameter from the selected parameter context for a specific processor, do the following:

  1. On the selected processor, right-click to open the context menu and select the Configure command.

  2. On the PROPERTIES tab, select the required parameter and in the opened window enter the parameter value in the form #{name}, where name is the name entered when creating the parameter in the parameters context.

    nifi attrib 09 dark
    Apply the created parameter
    nifi attrib 09 light
    Apply the created parameter
  3. Click OK.

In case of successful application of the parameter, the icon arrow2 light appears in the last column of the assigned property. When clicked, the icon goes to the Update Parameter Context page, where this parameter is available to change.

nifi attrib 10 dark
Created parameter on the PROPERTIES tab
nifi attrib 10 light
Created parameter on the PROPERTIES tab
nifi attrib 07 dark
Update Parameter Context
nifi attrib 07 light
Update Parameter Context

When clicking on the icon arrow1 light, each parameter can also be written to the given parameter context.

nifi attrib 11 dark
Writing a parameter from the available processor properties
nifi attrib 11 light
Writing a parameter from the available processor properties

Check dependent components

On the PARAMETERS tab of the Parameter Contexts window for the selected parameter, the Referencing Components line displays the names of all processors that use this parameter.

nifi attrib 12 dark
Dependent components
nifi attrib 12 light
Dependent components

Variables

Variables are created and configured in the NiFi user interface. They can be used in any realm that supports Expression Language. Variables cannot be used for sensitive properties. NiFi automatically picks up new or changed variables. Variables are defined at the process group level, so the access policies for viewing and modifying variables are derived from the process group access policies.

Create a variable

To create a variable, do the following:

  1. On a free field, right-click to call the context menu and select the Variables command.

  2. In the window that opens, click + to create a variable.

  3. Enter variable name.

  4. Click OK.

    nifi attrib 13 dark
    Creating a variable
    nifi attrib 13 light
    Creating a variable
  5. Enter the value of the variable.

    nifi attrib 14 dark
    Variable value entry
    nifi attrib 14 light
    Variable value entry
  6. Click OK.

  7. In the Variables window, click APPLY. As a result, an information window for updating or creating a variable opens.

    nifi attrib 15 dark
    Information window
    nifi attrib 15 light
    Information window

Use variables in processor properties via Expression Language

In order to apply the created variable for a specific processor, do the following:

  1. On the selected processor, right-click to open the context menu and select the Configure command.

  2. On the PROPERTIES tab, select the required parameter and in the opened window enter the parameter value in the form ${name}, where name is the name entered when creating the variable.

    nifi attrib 16 dark
    Apply the created parameter
    nifi attrib 16 light
    Apply the created parameter
    Expression Language functions

     

    With the help of various Expression Language functions it is possible to manipulate the value of variables. Some functions provide string (text) manipulation, such as the toUpper function. Others, such as the equals and match functions, provide comparison functionality. There are also functions for manipulating dates and times and for performing mathematical operations.

    Using the necessary expression structure it is possible, for example, to return an uppercase version of a filename by calling the toUpper function: ${filename:toUpper()}. In this case, the filename attribute is referenced and then manipulated with the value using the toUpper function.

    A function call consists of 5 elements:

    • Function call separator :.

    • Function name, in this case toUpper.

    • Opening bracket (.

    • Function arguments. The required arguments depend on which function is being called. In this example, we are using the toUpper function, which has no arguments, so this element is omitted.

    • The closing bracket ) indicates the end of a function call.

  3. After entering the function for the variable, click OK in the property value input window and APPLY on the PROPERTIES tab in the Configure window of the processor.

NOTE

Variables in a child process group override values in the parent process group. In particular, if a variable is set in the root group and also gets a different value within the process group, then the components within the process group use the value defined directly in the process group itself.

Check dependent components

In the Variables window for the selected variable, the Referencing Components line displays the names of all processors that use this parameter.

nifi attrib 17 dark
Dependent сomponents
nifi attrib 17 light
Dependent сomponents
CAUTION

When creating custom properties, ensure that each custom property contains a distinct property value so that it is not overridden by existing environment properties, system properties, or FlowFile attributes.

Parameters using Expression Language

When adding a parameter that uses an Expression language, it is important to understand the context in which the Expression language will be evaluated. The Expression is always evaluated in the context of the processor or controller service that refers to the parameter. Take, for example, a scenario where a parameter named Time has a value of ${now()}. The result of the Expression language is a call to determine the system time when it is evaluated. When you add the Time parameter as a property, the system time is not evaluated when the parameter is added, but when the Expression is evaluated by the processor or controller service. That is, if the processor has a property set to #{Time}, it will behave exactly as if the property had been set to ${now()}. Each time the property is referenced, it creates a different timestamp.

In addition, some properties do not allow the use of an Expression language, while others allow the use of an Expression language but do not evaluate Expressions on the FlowFile attributes. To understand how this works, consider a parameter named File, whose value is ${filename}. Then consider three different properties, each with a different Expression language scope and a FlowFile with a file name of test.txt. If each of these properties is set to #{File}, the resulting value is shown in the following table.

Customized property value Expression language scope Property effective value Notes

#{File}

FlowFile Attributes

test.txt

The filename is resolved by looking at the filename attribute

Variable registry only

Empty string

FlowFile attributes are out of scope and we are assuming there is no variable named filename in the variable registry

No

${filename}

The literal text ${filename} will not be evaluated

Found a mistake? Seleсt text and press Ctrl+Enter to report it