# Config File¶

Config file is a way to describe how ADCM should run ansible playbooks and which parameters should be asked from End User.

Format of config file is YAML. For additional information about YAML syntax it is better to start digging from https://en.wikipedia.org/wiki/YAML.

That file has to be named as config.yaml or config.yml. Multiple config.yaml files are allowed in a bundle_.

## Prototype¶

Prototype is a description of one object.

Every prototype should be one of the following types:

• cluster;
• service;
• provider (host provider);
• host.

Every prototype has following mandatory properties:

• type;
• name;
• version.

So minimal working prototype should be like that:

---
- type: cluster
name: control
version: 1


Prototype definition can have same optional properties:

• Display Name and Description
---
- type: cluster
name: control
version: 1
display_name: "Controlling Software"
description: |
This software is intended for monitoring and controlling
you cluster and does it by ...

+ display_name is relatively short human readable name;

+ name is really short name intended for use in ansible scripts.

• Required

Every service prototype can have an optional required property. Default value is false. If cluster cannot work without this service, set it to true. When required service is not added to cluster, cluster will have an issue.

• Monitoring

Every service, component or host prototype can have an optional monitoring property. It defines, should their instances be monitored or not. Default value is active. If you want to exclude them from monitoring system, set it to passive.

Any cluster or host provider prototype can have an optional adcm_min_version property. This field define minimal version of ADCM required for this bundle_ to work correctly. If installed ADCM instance has version less than adcm_min_version defined in bundle, loading of this bundle cause a error.

• Allow maintenance mode

Every cluster prototype can have an optional allow_maintenance_mode property. This option enables support of the maintenance mode for a specific bundle. The default value is false.

---
- type: cluster
name: control
version: 1
allow_maintenance_mode: true


Every cluster or host provider prototype can have an optional license property (corresponding to the cluster or the host bundle_ respectively). The value of a license property is a path to the licence file relative to a bundle root.

If prototype has a license property, a user of ADCM will be asked to accept a license after the bundle is loaded. Before a user accept a license any action with the bundle will be blocked.

---
- type: cluster
version: 1


Note: You can use a special dot syntax to point out a path to a license file. In this case a license file will be searched in the same directory as config.yaml:

license: ./license.txt

• Edition

Every cluster or host provider prototype can have an optional edition property (corresponding the cluster or the host bundle_ respectively).

Default value of edition is community. Value of edition can be used in Upgrade definition.

---
- type: cluster
version: 1
edition: enterprise


## Actions¶

Every prototype can have actions. Actions should be described as follows:

---
- type: cluster
name: control
version: 1
description: "Monitoring and Control Software"

actions:
install:
display_name: "Install Monitor Server"
description: |
By click on this button you install monitoring and controlling server ...
type: job
script_type: ansible
script: ansible/site.yaml
allow_to_terminate: true
allow_in_maintenance_mode: true
params:
ansible_tags: install
jinja2_native: true
states:
available:
- created
on_success: installed
on_fail: created
config:
quorum:
type: integer
log_files:
- check


You can see one action called install. That action has a job type and its script property specify an ansible playbook that resides in directory <bundle_root>/ansible/site.yaml.

Note: You can use a special dot syntax to point out a path to a script file. In this case a script file will be searched in the same directory as config.yaml.

script: ./site.yaml


### Params¶

You can also pass any additional parameter to ansible call. They will be used in ansible-playbook calls.

One special case is parameter ansible_tags. That parameter turns to be --tags parameter of ansible-playbook call.

You can also pass the jinja2_native parameter. This parameter will be written to ansible.cfg file. This option preserves variable types during template operations.

### States¶

Action can have a states optional statement. It defines 3 clauses:

• available – list of states of object for which action will be available;
• on_success – state in which object will be set in case of action success;
• on_fail – state in which object will be set in case of action fail.

States on_success and on_fail are optional. If they are missed, state of an object will not be changed after action success or/and fail respectively.

Masking is a new DSL for state and multistate operation. This DSL is not compatible with States DSL provided above.

---
actions:
install:
display_name: "Install Monitor Server"
type: job
# Action will be shown if both condition under state and multi_state are met.
state:
available:
# If the state equal to any of this, then condition is true
- "state_value1"
- "state_value2"
unavailable:
# If you place unavailable, then no available should be there
- "state_value1"
- "state_value2"
multi_state:
available:
# If we have any of this multistate, then condition is true
- "multi_state_1"
- "multi_state_2"
unavailable:
# If you place unavailable, then no available should be there
- "multi_state_3"
- "multi_state_4"
on_fail:
state: "new_sate_value"
multi_state:
set:
- "multi_state3"
unset:
- "multi_state4"
on_success:
state: "new_sate_value"
multi_state:
set:
- "multi_state3"
unset:
- "multi_state4"


To simplify the rule description, you could use scalar “any” which is select all possible values “any” scalar selector.

---
state:
available: "any"
unavailable: "any"
multi_state:
available: "any"
unavailable: "any"


If masking state is omitted, than we think that action available on every state or multi_state.

So the following variants are equal:

---

---
state:

---
state:
available: "any"


The following variants are equal too:

---

---
multi_state:

---
multi_state:
available: "any"


### Config¶

Action can have a config optional statement. It can define a list of values as in Config Parameters. When action is run via UI, all this values will be asked from user before action start.

In action ansible script you can later refer this values:

{{ job.config.quorum }}


### Log Files¶

Action can have a log_files optional statement. It can define a list of log tags. At now, only one log tag check is supported. If you have check log tag, you can use adcm_check ansible module.

### Allow to terminate¶

Action can have a allow_to_terminate optional statement. Boolean value. If value not explicitly set, default value would be false. If the value allow_to_terminate is true, then it will be possible to cancel the running ansible playbook, otherwise it will be impossible to cancel the playbook.

### Allow in maintenance mode¶

Action can have a allow_in_maintenance_mode optional statement. Boolean value. If value not explicitly set, default value would be false. If the value allow_in_maintenance_mode is true, then it will be possible to run action even in case of just one object’s host being in maintenance mode.

### UI Options¶

That block allow you to tweak how UI show and perform an action.

Disclaimer: Ask user if he really want to run this action and show disclaimer text in a modal window.

actions:
install:
type: job
script_type: ansible
script: ansible/site.yaml
ui_options:
disclaimer: "Are you really want to click me?"


### Host action¶

Action can have a host_action optional statement. Boolean value. If value not explicitly set, default value would be false. If the value is true, then the action will be performed on the selected host, and a target group will be created, which will include one selected host.

actions:
install:
type: job
script_type: ansible
script: action.yaml
host_action: true


### Host Component Map ACL¶

Action can have a hc_acl optional statement. If hc_acl is present, when action is run via UI, new host component map will be asked from the user before action start.

actions:
expand:
type: job
script_type: ansible
script: ansible/site.yaml
states:
available: all
hc_acl:
-
component: datanode
-
component: server
action: remove


hc_acl define list of 3 clauses which specify permissions to modify component distribution by hosts:

• service – name of service to work with;
• component – name of component to work with;
• action – permission type. Can be add or remove.

### Multi Scripts¶

Action can have more then one ansible playbook. In this case type of action should be task. Ansible scripts will be executed one by one in the order of description. If any of script fail, execution will be stopped and state of the object will be set to the value of on_fail property.

actions:
install:
scripts:
-
name: prepare
display_name: Prepare to install
script_type: ansible
script: ansible/prepare.yaml
on_fail: created
-
name: install
display_name: Actual install
script_type: ansible
script: ansible/install.yaml
on_fail: prepare
states:
available:
- created
- prepare
on_success: installed


### Buttons¶

There is a way to bind an action to a button in UI. To do that check the following code:

actions:
install:
scripts:
-
name: prepare
display_name: Prepare to install
script_type: ansible
script: ansible/prepare.yaml
on_fail: created
button: "create_host"

Name Description
create_host “Create Host” button on Hosts page. If you bind multiple action to one button, than an action selection window appears

### venv¶

There is a way to specify ansible version for ansible script type actions using venv optional statement.

Valid values:

• default – launch action in ansible 2.8 environment;
• 2.9 – launch action in ansible 2.9 environment;

By default this value is default

Change default environment for all actions on object:

---
- type: cluster # service component hostprovider host
name: control
version: 1

venv: "2.9"


You also have a way to tweak this for any of actions:

actions:
install:
type: job
script_type: ansible
script: action.yaml
venv: "2.9"


## Components¶

Service prototype can have a components. Component is a mean of placing service components on hosts in cluster.

---
- type: service
name: server
version: 1

components:
master:
display_name: "Master Node"
description: "This node control all data nodes (see below)"
constraint: [1,2]
node:
display_name: "Data Node"
constraint: [+]
client:
monitoring: passive


### Constraint¶

Every component can has an optional constraint property. It describes how many instances of this component should be installed in one cluster:

• [1] – exactly once component should be installed;
• [0,1] – one or zero component should be installed;
• [1,2] – one or two component should be installed;
• [0,+] – zero or any more component should be installed (default value);
• [1,odd] – one or more component should be installed; total amount should be odd;
• [0,odd] – zero or more component should be installed; if more than zero, total amount should be odd;
• [odd] – the same as [1,odd];
• [1,+] – one or any more component should be installed;
• [+] – component should be installed on all hosts of cluster.

Every component can have an optional monitoring property. It defines, should component be monitored or not. Default value is active. If you want to exclude component from monitoring system, set it to passive.

### Requires¶

Components can have an optional requires property, which describe dependencies between components, i.e. which components need another components to be installed in cluster simultaneously.

Each component can require one or more another component in the same service or in another service in this cluster. If component required another component, you could not add it to host component map (hc) when required component absent in hc.

Requires can be circular. In this case all required components should be installed simultaneously.

So, dependencies are another type of constraints (or meta constraints) for a component.

---
- type: service
name: Hive
version: 1.0

components:
HiveServer:
requires:
- component: Metastore
Metastore:
constraint: [0,1]
TezUI:
requires:
- service: YARN
component: TimeLineServer
constraint: [0,1]

- type: service
name: YARN
version: 2.0

components:
TimeLineServer:
requires:
- service: Hive
component: TezUI


## Config Parameters¶

There are a number of user editable values that could be represented in UI and related to every object.

---
- type: cluster
name: control
version: 1
description: "Monitoring and Control Software"

config:
- name: repos
type: group
subs:
type: string
required: no


In the example you can see parameter ads in group repos. That parameter is not required. This means that it is possible for the user to leave it empty.

Group is not mandatory. So you can place your parameter on the highest level:

---
- type: cluster
name: control
version: 1
description: "Monitoring and Control Software"

config:
type: string
required: no


Group is just a visual representation in UI.

There is an old, deprecated way to define groups as map of map, not array of map:

---
- type: cluster
name: control
version: 1
description: "Monitoring and Control Software"

config:
repos:
type: string
required: no


### Activatable groups¶

Group can be activatable. This mean you can switch on or off this group via UI. When group is switched off (active = false) you don’t need to fill in its sub values in config UI and it value in ansible inventory file would be null.

---
- type: cluster
name: Kafka
version: 1

config:
- name: grafana
type: group
activatable: true
active: false
subs:
- name: endpoint
type: string
- name: port
type: integer


### Config customization¶

There is a functionality in ADCM which allow End User to customize some parameters of cluster (or service, or component, or provider) on some group of hosts. This functionality is disabled by default. To start working with it you have to option.

Change default policy for all config on object:

---
- type: cluster # service component hostprovider
name: control
version: 1

config_group_customization: true


By default this value is False

You also have a way to tweak this for any of parameters:

---
type: string
group_customization: true/false


### Common config parameters¶

Name Mandatory Values Default
type yes see below
display_name no
description no
default no user defined per type
required no yes/no yes
writable no see below yes
ui_options no see below no

If required set to yes, it is required to have a non empty and valid value for the variable.

You can make any config parameter read only (protected from change by user or API) for specified states of cluster/host/service.

Examples:

Parameter quorum will be read only (protected from change) when service state is installed or running:

config:
quorum:
type: integer


Parameter quorum will be read only (protected from change) for all service states except created:

config:
quorum:
type: integer
writable: [created]


Parameter quorum will be read only (protected from change) for all service states:

config:
quorum:
type: integer
default: 4


### UI Options¶

That block allow you to tweak how UI draw config.

We have the following options:

• Invisible

NOTE: That option doesn’t hide information from API, and doesn’t disable ReadOnly/Writable logic. It’s just an UI tweak.

int:
type: integer
default: 1
required: no
ui_options:
invisible: true


Sometimes you need to make a password field without confirmation (just one input). It especially useful for action’s parameters.

passwd_no_confirm:
required: yes
ui_options:
no_confirm: true


There is a new checkbox on config view that allow you to hide some of the parameter which is marked as advanced.

NOTE: Parameter marked as advanced is not searchable while they are not visible (when Advanced check box is unchecked).

float:
type: float
default: 1.0
required: no
ui_options:


### Types of config parameters¶

• String – arbitrary string. The size is not limited. If default is not explicitly set, default value would be an empty string "".
• Text – the same as string, but will be displayed as multi line text in user interface.
• Boolean – boolean value: True or False.
• Integer – integer number. If default is not explicitly set, default value would be 0. You can use optional min and max parameters to limit allowed values:
config:
memory_size:
type: integer
default: 16
min: 2
max: 64

• Float – float number. If default is not explicitly set, default value would be 0.0. You can use optional min and max parameters to limit allowed values:
config:
type: float
default: 1.0
min: 0.5
max: 3.5

• Password – the same as string type, but it displays as password field in user interface.
• Option – key-value dictionary type. Keys are showed in user interface, corresponding values are stored in DB:
config:
protocol:
type: option
option: {http: 80, https: 443}
default: 80

• List – dynamic array of arbitrary length. Array elements should be a string. Any element can be added, deleted or modified on every config update:
config:
mount_points:
type: list
default:
- /dev/rdisk0s1
- /dev/rdisk0s2
- /dev/rdisk0s3

• Map – dynamic key-value dictionary of arbitrary length. Keys and values should be a string. Any key or/and value can be added, deleted or modified on every config update:
config:
person:
type: map
default:
name: Joe
age: "24"
sex: m

• Variant – variant is a type where user can select one value from a list of options in UI. Sources of variants can be different. You can get a list of variants from another config parameter (type of this config parameter should be a list):
config:
- name: mount
type: variant
source:
type: config
name: mount_points

- name: mount_points
type: list
default:
- /dev/rdisk0s1
- /dev/rdisk0s2
- /dev/rdisk0s3
ui_options:
invisible: true


Also you can get a list of variants from builtin ADCM functions, such as cluster_hosts, which return list of all hosts in current cluster:

config:
- name: cluster_host
type: variant
source:
type: builtin
name: host_in_cluster


• host_in_cluster – return list of all hosts in current cluster;
• host_not_in_clusters – return list of all hosts not included in any cluster;
• service_in_cluster – return list of all services installed in current cluster;
• service_to_add – return list of all services not installed in current cluster;
• host – see below.
• Host – builtin function host can receive a lot of arguments in form of predicate and args: predicate is a name of internal host function; args are arguments for this function. All internal functions return list of hosts (of course, this list can be empty). For example:
config:
- name: vhost
type: variant
source:
type: builtin
name: host
args:
predicate: and
args:
- predicate: in_service
args:
service: UBER
- predicate: or
args:
- predicate: in_component
args:
service: UBER
component: UBER_SERVER
- predicate: in_cluster
args:


List of internal host functions:

• in_cluster – return list of all hosts in current cluster. Args: None.
• in_service – return list of all hosts in specified service. Args: service name.
• not_in_service – return list of all hosts not in specified service. Args: service name.
• in_component – return list of all hosts in specified component. Args: service and components names.
• not_in_component – return list of all hosts that are not in specified component. Args: service and components names.
• in_hc – return list of all hosts in host component map of this cluster. Args: None.
• not_in_hc – return list of all hosts that are not in host component map of this cluster. Args: None.
• and – return union of sets of its args.
• or – return intersection of sets of its args.

Or you can list variants right here in a bundle file:

config:
- name: city
type: variant
source:
type: inline
strict: False
value:
- Moscow
- Voronez
- Ufa
default: Samara


You can also specify optional argument strict for variants source. If strict is set to False, user can not only select value from predefined list, but also enter his own values in UI.

• JSON – this type can store arbitrary json-data. JSON structure is not preserved and can be modified on every config update:
config:
population:
type: json
default:
-
country: Russia
cities:
-
name: Moscow
population: 9
-
country: USA
cities:
-
name: LA
population: 5

• Structure – this type can store some regular data described by a yspec-specification. On every config update new data are checked against specification. If input data and specification do not match, ADCM will return a error.

Documentation on yspec is available at https://pypi.org/project/yspec/.

config:
country_codes:
type: structure
yspec: codes/schema.yaml
default:
-
country: Greece
code: 30
-
country: France
code: 33
-
country: Spain
code: 34

• schema.yaml file:
---
root:
match: list
item: country_code

country_code:
match: dict
items:
country: string
code: integer

string:
match: string

integer:
match: int


Note: You can use a special dot syntax to point out a path to a yspec-specification file. In this case a specification file will be searched in the same directory as config.yaml:

yspec: ./schema.yaml

• File – the content of this property is stored as a file on a file system. When you reference this variable in an ansible script it returns a full path to this file.

If default value is set, it should be the path to the file relative to the bundle root. When an object is created, content of this file will be read to the value of specified config parameter and stored on a file system (distinct file for each object).

User can edit content of this file via user interface. The file will be updated after each config saving.

It can be usable to store ssh private keys for an ansible script:

- type: host
name: ssh
version: 1

config:
ansible_private_key_file:
type: file
default: host/ssh.key


Note: You can use a special dot syntax to point out a path to a default file. In this case a default file will be searched in the same directory as config.yaml:

config:
ansible_private_key_file:
type: file
default: ./ssh.key


## Export¶

If you want to share some parts of config with other clusters, you can use export statement. In export you can specify any number of the first level config entries. You can export config in cluster or/and in service:

---
- type: service
version: 2.1

config:
core-site:
param1:
type: string
param2:
type: integer
quorum:
type: integer
default: 3

export:
- core-site
- quorum


## Import¶

To use exported parts of config from other cluster you should use import statement in your cluster. In import you should specify name. If you don’t need any optional parameters use empty dict.

---
-
type: cluster
name: Hive
version: 4

import:
versions:
min: 1.8
max_strict: 2.5

grafana: {}


When you bind your cluster and another cluster/service via ADCM user interface, you can refer exported config parameters in your ansible scripts like this:

{{  cluster.import.Hadoop.core-site.param2  }}


### Optional import parameters¶

#### Versions¶

You could specify the interval of versions, appropriate for import. Min and max use non strict comparison (i.e. boundaries include version of cluster/host). If you want strict comparison, you can use min_strict and/or max_strict instead of min/max. If you won’t specify borders appropriate version wouldn’t check.

#### Required¶

Import statement can have optional required parameter. If required is set to true, this cluster or service will raise issue and block any actions before required import is properly binded with export cluster/service. Default value for required is false.

#### Default¶

Import statement can have optional default parameter:

---
-
type: cluster
name: Hive
version: 4

config:
endpoints:
host:
type: string
port:
type: integer
default: 80

import:
Graphite:
versions:
min: 1.8
max: 2.5
default:
- endpoints


default parameter should be an array of names of config group of this cluster or service. If this cluster/service is not binded with any export cluster/service import section in ansible inventory file will be filled with values from specified config group.

To your own safety name of default groups should be the same as export group names.

#### Multibind¶

Import statement can have optional multibind parameter. If it set to true, you can bind many instances of cluster or service to the same cluster/service. In ansible inventory file this import values will be represented as an array.

Default value of multibind is false.

Cluster or host prototype can have an upgrade statement. This statement describes how to upgrade from one version of bundle_ to another. Prototype can have a few upgrade items to describe upgrade from different versions. There is two types of upgrades:

• The upgrade is just switching the prototypes of the current bundle to the prototypes of the new one.
• The upgrade is some kind of action, which may consist of several subactions including switching prototypes (see example: Upgrade 2)

Common options for each type of upgrade:

• versions – interval of versions, appropriate for upgrade;
• name;
• description (optional);
• states:
• available – list states of cluster, permitted to upgrade. Special value any can be used for upgrade, which available on any state;
• on_success – state of cluster after upgrade.

For upgrade based on action, also requred to specify scripts. See more Multi Scripts section. Also for upgrade implemented special type script_type and script, which is switching prototypes to newest:

• internal - script_type for internal adcm operaions
• bundle_switch - switching bundle prototype operation script

---
- type: cluster
name: control
version: 2.4
description: "Monitoring and Control Software"

versions:
min: 0.4
max: 2.0
states:
available: any
description: The cluster will be prepared for upgrade
from_edition: {{ from_edition[edition] | list }}
versions:
min: 2.1
max_strict: 3.0
scripts:
script: ansible/bundle_pre_check.yaml
script_type: ansible
params:
ansible_tags: os_check, version_check
script_type: internal
script: bundle_switch
script_type: ansible
params:
ansible_tags: hc_change
states:
available: [running]


About versions comparison: min and max use non strict comparison (i.e. boundaries include version of cluster/host). If you want strict comparison, you can use min_strict and/or max_strict instead of min/max.

### from_edition¶

Upgrade can has optional from_edition property. The value of from_edition property is list of editions from which upgrades are permitted.

Special value any can be used to specify permission to upgrade from bundles with any edition.

Default value of from_edition is [community].

upgrade:
-
versions:
min: 0.4
max: 2.0
states:
available: [created, installed]
from_edition:
- community
- enterprise


## Config order convention¶

To comply with the uniformity and transparency for configuration, it is desirable to follow the order of the prototypes. It is recommended to follow the order as shown in the example:

---
- name:
type: cluster
description:
version: &version

...

config:
...

import:
...

actions:

install:
...

- type: service
name:
version:
description:

config:
...

components:
...

actions:

install:
...