Create ADS cluster

After you complete preliminary actions and upload ADB bundle, the ADCM cluster Manager contains the following list of objects (Pic.14.):

  • A monitoring cluster prototype and instantiated cluster (a real monitoring program running on a host, ADCM records its results and settings);
  • An ads cluster prototype for instantiation;
  • Pre-prepared hosts, the ADCM database stores records of them and their credentials - ssh keys or passwords.
../../../_images/hosts.png

Рис. 14. List of objects of ADCM

The following functionality is available:

Instantiate cluster

When cluster instance ads is created in web interface of ADCM it means only adding data about it to the database ADCM, this step doesn’t install ads on the hosts.

To instantiate cluster:

  1. Open the “CLUSTERS” tab in ADCM (Pic.15.).
../../../_images/cluster_mon.png

Рис. 15. “CLUSTERS” tab

  1. Press the “Add cluster” button and in the opened form create an instance of the cluster from the prototype ads obtained from the bundle (Pic.16.).
../../../_images/add_cluster.png

Рис. 16. Creating a cluster instance

  1. As a result of the actions performed, the creation of the cluster instance is displayed in the ADCM database on the “CLUSTERS” tab (Pic.17.).
../../../_images/clusters_list.png

Рис. 17. Cluster instance creation success

Configure cluster

To go to the settings of the cluster instance ads, press the button with the gear icon in the necessary row of the “CLUSTERS” tab (Pic.17.) and go to the “Configuration” menu section. This opens the configuration window for the selected instance (Pic.18.).

../../../_images/cluster_config.png

Рис. 18. Cluster configuration window

In the block “ADCM” it is possible to configure addresses:

  • Host (or fqdn and IP address);
  • Port.

In the “repos” settings block, you specify the packages required for ads installation from various yum repositories, and in each of the parameters you can change the default url to the required one:

  • ads;
  • monitoring;
  • epel.

Any cluster’s service component is able to send keep alive status information to the ADCM process in the docker container. Sometimes ADCM can be behind NAT and there is no evident way to automatically tell the address of ADCM to the service components located on cluster’s hosts. That is why it should be configured manually. During the ADB installation the ADCM address would be used to fill component’s keep alive status configuration.

Add services

Cluster ADS contains the following services:

  • Zookeeper – service for storing configurations, performing distributed process synchronization;
  • Kafka – distributed platform for streaming operations and data;
  • Nifi – distributed platform for building and automating data flows between different systems;
  • Monitoring Clients – agents that send host and ADS information to monitoring.

Not all services are required to be installed. For example if you do not plan to use Nifi then there is no need to add this service. If you have your own monitoring server (not Graphite based) there is no need for agents from the monitoring clients service. However, if you plan to use Kafka, the service of the same name and Zookeeper are required, the same can be said about the service Nifi. At the same time, the service may consist of mandatory and optional components. For example, the Kafka service consists of the mandatory component broker and the optional components: manager and schema-registry.

In this example, all services are added to the cluster:

Important

Currently it is not possible to remove already added service from the cluster

Setting up the Zookeeper service

To go to the settings of the service Zookeeper, press the button with the gear icon in the necessary row of the “SERVICES” tab in the ADCM and go to the “Configuration” menu section. This opens the configuration window for the selected service (Pic.19.).

../../../_images/zk_config.png

Рис. 19. Configuration window of Zookeeper service

The settings block “Main” contains the following parameters:

  • connect – the Znode connection string, in which Zookeeper stores the configuration of the current cluster, is used by the Kafka service. In the current implementation, this parameter is not editable and is automatically generated on the ADCM side;
  • data_dirs – directory for storing transaction logs Zookeeper. This parameter is specified as dataDir in the configuration file zoo.cfg;
  • client_port – the port on which Zookeeper listens to client connections.

The settings block “Advanced” contains the following parameters:

  • zoo_cfg_content – the contents of the file zoo.cfg, which is further templateable. This parameter can be used to make additional settings;
  • zookeeper_env_content – the contents of the file zookeeper-env.sh, which is further template. This parameter can be used to make environment variables.

Setting up the Kafka service

To go to the settings of the service Kafka, press the button with the gear icon in the necessary row of the “SERVICES” tab in the ADCM and go to the “Configuration” menu section. This opens the configuration window for the selected service (Pic.20.).

../../../_images/kafka_config.png

Рис. 20. Configuration window of Kafka service

The settings block “Main” contains the following parameters:

  • data_dirs – directory for storing data in Kafka. Indicated to the quality of the parameter log.dirs in the configuration file server.properties;
  • listeners – a list of URIs (protocol, host, and port on which the broker is run), separated by commas. If you are not using the PLAINTEXT protocol, then you must also specify listener.security.protocol.map. To bind to all interfaces, specify the host name as 0.0.0.0. Leave the host name blank for binding to the default interface. It is specified as the listeners parameter in the configuration file server.properties;
  • default_replication_factor – replication factor, with which topics are created and stored by default. It is specified as the parameter efault.replication.factor in the configuration file server.properties;
  • delete_topic_enable – this option allows you to delete topics. If the parameter is disabled, deleting the topic through the administration tools does not lead to the actual deletion. It is specified as the parameter default.replication.factor in the configuration file server.properties;
  • log_retention_hours – the number of hours that topics are stored in Kafka. It is specified as the log.retention.hours parameter in the configuration file server.properties;
  • log_roll_hours – the maximum time after which a new segment log appears, even if the old log is not full. It is specified as the parameter log.roll.hours in the configuration file server.properties;
  • broker_jmx_port – port on which Kafka broker gives jmx-metrics. It is specified as the parameter JMX_PORT in the file kafka-env.sh;
  • manager_port – port on which Kafka-Manager runs. Indicated in the file kafka-manager-env.sh;
  • schema_registry_heap_opts – heap size allocated to the schema-registry process. Specified as the SCHEMA_REGISTRY_HEAP_OPTS parameter in schema-registry-env.sh;
  • schema_registry_listener_port – port that is listening on schema-registry. Specified as the parameter listeners in the configuration file schema-registry.properties;

The settings block “Advanced” contains the following parameters:

  • server_properties_content – the contents of the server.properties file, which is further templateable. This parameter can be used to make additional settings;
  • kafka_env_content – the contents of the file kafka-env.sh, which is further template. This parameter can be used to make environment variables.

Setting up the Nifi service

To go to the settings of the service Nifi, press the button with the gear icon in the necessary row of the “SERVICES” tab in the ADCM and go to the “Configuration” menu section. This opens the configuration window for the selected service (Pic.21.).

../../../_images/nifi_config.png

Рис. 21. Configuration window of Nifi service

The settings block “Main” contains the following parameters:

  • nifi_ui_port – http-port on which the web interface of the service Nifi runs. It is specified as the parameter nifi.web.http.port in the configuration file nifi.properties;
  • nifi_node_jvm_memory – heap size allocated to the service process Nifi. It is specified in the configuration file bootstrap.conf.

The following optional parameters are set in the “Custom” settings block:

  • nifi_custom_nars – parameter should be used when adding custom nars; comma separated. Specified as nifi.nar.library.directory.lib… in the configuration file nifi.properties;

The settings block “Advanced” contains the following parameters:

  • nifi_properties_content – the contents of the file nifi.properties, which will be further templateable. This parameter can be used to make additional settings;
  • nifi_env_content – the contents of the file nifi-env.sh, which is further templateable. This parameter can be used to make environment variables;
  • bootstrap_content – the contents of the file bootstrap.conf, which is further templateable. This field can be used to make settings related to the start of the service;
  • logback_content – the contents of the file logback.xml, which is further templateable. This field can be used to make settings related to logging;
  • state_management_content – the contents of the file state_management.xml, which is further templateable. This field can be used to make settings related to storing the state of the service Nifi;
  • authorizers_content – the contents of the file authorizers.xml, which is further template. This field can be used to make authorization settings to the Nifi service if security policies are configured;
  • login_identity_providers_content – the contents of the file login_identity_providers.xml, which is further template. This field can be used to make authorization settings used by the state provider, if security policies are configured.

Setting up the Monitoring Clients service

To go to the settings of the service monitoring clients, press the button with the gear icon in the necessary row of the “SERVICES” tab in the ADCM and go to the “Configuration” menu section. This opens the configuration window for the selected service (Pic.22.).

../../../_images/mc_config.png

Рис. 22. Configuration window of Monitoring Clients service

The settings block “Advanced” contains the following parameters:

  • kafka_dashboard – a file in the format json, which is further template and sent to Grafana;
  • kafka_metrics – a file in the format yaml, which will be further template. Includes jmx broker metrics Kafka.

Add hosts

To add hosts to the cluster ads:

  1. Open the “Hosts” tab in the ads cluster menu (Pic.23.).
../../../_images/hosts_list.png

Рис. 23. “Hosts” tab of the ads cluster

  1. Press the “Add hosts” button and in the opened form select the required hosts (Pic.24.).
../../../_images/add_hosts.png

Рис. 24. Сhoose hosts

  1. As a result, the fact of adding hosts is displayed in the ads cluster in the list of the “Hosts” tab (Pic.25.).
../../../_images/hosts_list2.png

Рис. 25. Result of successfully adding hosts

Map service components to hosts

Each service consists of components that should be mapped to some hosts in the cluster. To do this, a component should be selected on the “Hosts - Components” tab by clicking on it in the “Components” column and mapped with a click to the required host in the “Hosts” column (Pic.26.).

../../../_images/components.png

Рис. 26. Map service components to hosts

Because the a*Zookeeper*, Kafka, Nifi and Monitoring Clients services added to the ADS cluster, but are not mapped yet, initially there are no components on any of the hosts:

  1. Components of the Zookeeper service are (Pic.27.):
  • Zookeeper.SERVER – should be added to one or an odd number of hosts (zk1); for large clusters no more than 5 are recommended.
../../../_images/zk_components.png

Рис. 27. Components of the Zookeeper service

  1. Components of the Kafka service are (Pic.28.):
  • kafka.BROKER – should be added to one or more broker hosts (kafka1, kafka2);
  • kafka.MANAGER – optionally could be added to any one host (kafka1);
  • kafka.SCHEMA_REGISTRY – optionally could be added to any one host (kafka2).
../../../_images/kafka_components.png

Рис. 28. Components of the Kafka service

  1. Components of the Nifi service are (Pic.29.):
  • nifi.SERVER – should be added to one or more hosts (df-mdw);
../../../_images/nifi_components.png

Рис. 29. Components of the Nifi service

  1. Components of the monitoring clients service are (Pic.30.):
  • monitoring_clients.diamond – must be added to all hosts (zk1, kafka1, kafka2, nifi2). Collects system metrics and sends them to ADCM;
  • monitoring_clients.jmxtrans – must be added to one host (zk1). Collects service-specific metrics and sends them to ADCM;
../../../_images/mc_components.png

Рис. 30. Components of the Monitoring Clients service

Ways to install services

There are two ways to install services:

  • Install all cluster services. The operation at the cluster level is a sequential call to the installation and configuration of all added services;
  • Custom installation service. The operation at the service level is the installation of a single service.

Important

Regardless of the type of installation selected, you initially need to import the settings from the Monitoring cluster in case the Monitoring Clients service is added

Install all cluster services

To install all added services in a cluster ads, select the corresponding cluster in ADCM and click the Install button in the upper right corner of the screen form (Pic.31.).

../../../_images/cluster_install.png

Рис. 31. Install all cluster services

As a result of the installation all the added services change the state from created to installed (Pic.32.).

../../../_images/cluster_actions.png

Рис. 32. Result of the installation

Install Zookeeper service

To install the Zookeeper service, you need to perform a number of actions on the cluster tab “Services”:

  • In the “Actions” field click on the icon in the Zookeeper service row and select the action Install (Pic.33.). At this point, the hosts are configured, the necessary packages are installed.
../../../_images/zk_install.png

Рис. 33. Install Zookeeper

  • As a result of the installation, the service Zookeeper changes state from created to installed.

Install Kafka service

To install the Kafka service, you need to perform a number of actions on the cluster tab “Services”:

  • In the “Actions” field click on the icon in the Kafka service row and select the action Install (Pic.34.).
../../../_images/kafka_install.png

Рис. 34. Install Kafka

  • As a result of the installation, the service Kafka changes state from created to installed.

Install Nifi service

To install the Nifi service, you need to perform a number of actions on the cluster tab “Services”:

  • In the “Actions” field click on the icon in the Nifi service row and select the action Install (Pic.35.).
../../../_images/nifi_install.png

Рис. 35. Install Nifi

  • As a result of the installation, the service Nifi changes state from created to installed.

Install Monitoring Clients service

The monitoring clients service is optional and requires importing the configuration parameters of the monitoring cluster (addresses, login/password) to the cluster ads:

  1. To import the monitoring configuration to the ads cluster, open the “CLUSTERS” tab in ADCM, select the Import option and mark the imported service settings using the checkboxes in the opened form (Pic.36.).
../../../_images/import_configs.png

Рис. 36. Import monitoring configuration

  1. To install monitoring clients in the cluster ads:
  • Return to the cluster tab “Services”. Press in the “Actions” field on the icon in the monitoring clients service row and select the action Install (Pic.37.).
../../../_images/mc_install.png

Рис. 37. Install monitoring clients

  • As a result of the installation, the service monitoring clients changes state from created to monitored.