MiNiFi overview
MiNiFi features
MiNiFi — a subproject that extends the functionality of NiFi. MiNiFi provides the ability to collect data from the most remote peripheral devices where it is impossible to install a full-fledged NiFi Server, such as cell towers, network access devices, routers, etc. Collecting data directly at the source reduces the likelihood of data loss during transmission.
To ensure lightweight, MiNiFi lacks many of the features present in NiFi. For example, due to the lack of a user interface in MiNiFi, data flow construction can only be done in full-fledged NiFi to create templates that will be used in MiNiFi.
MiNiFi architecture
The figure below shows the architecture of MiNiFi.
MiNiFi works by building a custom NiFi data flow. Multiple agents (MiNiFi Agent) collect data from sensors or other devices (for example, via the MQTT protocol) and transmit them into the flow using specially configured processors (for example, ConsumeMQTT). Once the data flow is created and the required attributes for the FlowFile (sensor parameters, location, etc.) are added, the data is sent to the main NiFi instance via the Remote Process Group.
C2 Server
MiNiFi C2 Server — a central server that manages the configuration and operation of multiple agents via the C2 protocol. MiNiFi C2 Server is installed on the server that must be accessible from MiNiFi agents. Multiple C2 servers can be used for complex network structures.
The main functions of MiNiFi C2 Server are described below:
-
The server monitors heartbeats from agents and in response, if necessary, sends commands to update or write the flow configuration.
-
Each new agent is asked by the server for a manifest (all pre-installed configurations, processors, and controller services). The agent manifest is used in the C2 protocol to design a flow for an agent class — a group of agents with identical manifests.
-
The server manages MiNiFi flow configurations and publishes them via its own REST API. MiNiFi agents can connect to this API to update their own configuration at intervals defined by the
nifi.minifi.notifier.ingestors.pull.http.period.msparameter.The flow configuration is a YML file (config.yml) and can be created manually. Since this method is laborious, the most common flow is one created using the NiFI user interface and saved as template in XML file format. MiNiFi C2 Server accesses templates via the NiFi REST API. The template can be converted to YML format using the
minifi-toolkitutility. The template name usually corresponds to the agent class name.
MiNiFi Agent
MiNiFi Agent — an agent that runs data flows and is essentially a lightweight version of NiFi.
The main functions of MiNiFi Agent are described below:
-
In response to heartbeat messages, the agent receives a command from the C2 server to update the configuration.
-
The agent sends confirmation of all control commands received from the server via the C2 protocol.
-
The agent downloads the flow using the server’s REST API. The agent class is specified using the
nifi.minifi.notifier.ingestors.pull.http.queryparameter. -
The agent applies the new flow and sends the collected data via the Site-to-Site protocol to the remote NiFi Server instance using the Remote Process Group.
MiNiFi implementation
MiNiFi is available in two variants:
-
MiNiFi C++ — a binary, uses little system memory, but can run a limited subset of NiFi processors.
-
MiNiFi Java — contains most of the available NiFi processors, but is a larger binary distribution and uses more system resources.
MiNiFi in ADS
The MiNiFi service in the ADS cluster can only be installed simultaneously with the NiFi service or provided that NiFi is already installed.
After adding and installing the MiNiFi service in the ADS cluster, the following parameters are automatically set for each service component:
-
For the MiNiFi C2 Server component, the /etc/minifi-c2/conf/minifi-c2-context.xml file specifies the API address of the NiFi instance installed in the same ADS cluster, with which the flow templates will be loaded (
http://<hostname>:9090/nifi-api). -
For the MiNiFi Agent component, the /etc/minifi/conf/bootstrap.conf file automatically specifies the parameters for updating flow configurations and communicating with the MiNiFi C2 Server, and also loads the current flow configuration (if any) into the /etc/minifi/conf/config.yml file.
Configuring parameters and changing default parameters is performed on the configuration page of the MiNiFi service in the ADCM interface.
To connect to a NiFi instance via the Site-to-Site protocol, the NiFi service has parameters responsible for remote connection (with the nifi.remote prefix) set in the /etc/nifi/conf/nifi.properties file.
Descriptions of the parameters are given in the MiNiFi and NiFi sections of the ADS configuration parameters article.
After changing the parameters using the ADCM interface, restart the service. To do this, apply the Restart action by clicking 
in the Actions column.