Product bundle

Adelya Zvyagintseva

This article describes an example of how to create a product bundle for installing and deploying a cluster.

To create a product bundle, perform the following steps:

  1. Create a working directory for the bundle. It should contain the following files:

    • a configuration file, config.yaml or config.yml;

    • Ansible playbooks used to describe and run actions;

    • additional files (libraries, auxiliary data, and so on).

An example of a bundle directory structure:

<product_bundle_name>/
├── ansible/        -- directory for Ansible that includes Ansible playbook and roles; it is recommended to follow best practices when organizing the structure of internal directories
├── config.yaml     -- configuration file
└── ...             -- additional supporting files and modules

  1. Create the config.yaml configuration file, which is a mandatory part of the product bundle and used by ADCM for creating and managing objects.

  2. In the configuration file, add a cluster description block. To do this, specify:

    • required properties:

      • name — internal cluster name used to refer to the cluster;

      • type — object type (for a cluster, use the cluster value);

      • version — cluster version;

      • venv — Ansible version that will be used when executing cluster actions;

      • contract_version — bundle contract version;

        NOTE
        The contract_version property is available starting from ADCM 2.10.0.

    • optional properties:

      • display_name — name of the cluster prototype displayed in the ADCM web interface;

      • description — description of the cluster prototype;

      • edition — cluster edition; any string value is allowed, for example, community or enterprise;

      • license — path to the license file that a user should accept when creating the cluster (the full license text is displayed on the bundle page);

      • adcm_min_version — minimum supported ADCM version required to work with the bundle;

      • flag_autogeneration — property used to control automatic generation of embedded flags when the configuration of clusters, services, or components changes.

        NOTE
        Other optional properties that can be defined in the cluster description block are listed in the Prototypes overview article.

        Example of a cluster description block:

        - name: ADPG
  type: cluster
  description: Arenadata PostgreSQL
  version: &version 16.3_arenadata5.4_b1-1
  edition: community
  license: adcm_cluster_eula/RUS_EULA_CE.txt
  adcm_min_version: 2.6.0
  flag_autogeneration:
    enable_outdated_config: False
  venv: &venv_cluster "2.9"

  3. To enable cluster configuration management, add a config block in the cluster description. The specified cluster configuration parameters can be accessed from the Ansible playbook (the parameters are added to the inventory file in {{ vars.cluster.config }}). For detailed information on the inventory file structure, refer to Ansible inventory structure.

    NOTE
    For more information on the properties used to define a cluster configuration, refer to the config article.

    Example of a cluster configuration parameters block:

      config:
    - name: __main_info
      type: file
      default:  adpg/info/cluster.html

    - name: repos
      display_name: "Centos 7 repositories"
      type: group
      subs:
        - name: use_adpg_repo
          display_name: Set up ADPG repo
          type: boolean
          default: true
        - name: adpg_core_repo
          display_name: ADPG CORE repo url
          type: string
          default: "https://downloads.arenadata.io/ADPG-CORE\
                    /16.3_arenadata5/centos/7/community/$basearch/"
          description: ADPG CORE repository
        - name: adpg_repo
          display_name: ADPG repo url
          description: "ADPG repository"
          type: string
          default: "https://downloads.arenadata.io/ADPG\
                    /16_arenadata4/centos/7/community/$basearch/"

    where __main_info — a specialized service configuration parameter that can use an HTML file. The file content is displayed in the Info section on the Overview tab of the cluster page.

  4. To execute various cluster actions, add an actions block to the cluster description. The specified actions will be available in the ADCM web interface on the Clusters page.

    NOTE
    For more information on the properties used to describe actions, refer to the actions article.

    Example of an actions description block for the Precheck and Install actions:

      actions:
    Precheck:
      type: task
      scripts_jinja: jinja/scripts_jinja/cluster/precheck.j2
      states:
        available:
          - created
          - ready to upgrade

    Install: &install_cluster
      allow_to_terminate: true
      type: task
      scripts:
        - name: Precheck ADPG Control agents
          script_type: ansible
          script: adpg/adpgc_agents.yaml
          params:
            actions:
              - precheck
            adcm_precheck_fail_when: true
      states:
        available:
          - created
      config: &auto_reboot
        auto_reboot:
          type: boolean
          default: true
          required: true
          display_name: Reboot cluster servers after installation
          description: |
            You will need to restart the cluster server manually
            after the installation is completed in order to apply
            some settings if you refuse automatic reloading.
    NOTE
    The specified Ansible playbook can use inventory groups that are generated according to the topology of the cluster created by the user. For detailed information about inventory file groups, see Ansible inventory groups.
    The adpgc_agents.yaml file
    ---
- name: Run config
  hosts: CLUSTER:!host_unreachable
  gather_facts: false
  roles:
    - role: config

- name: ADPGC_AGENTS
  hosts: adpgc_agents:!host_unreachable
  gather_facts: true
  tasks:
    - name: Run adpg control agents
      include_role:
        name: adpgc_agents
      tags: always

    - name: Run statuschecker adpgc-agent
      when: adcm.config.global.adcm_url | default('') | length > 0
      include_role:
        name: statuschecker
      vars:
        checker_type: services
        service_name: "adpgc-agent.service"
        component_id: "{{ services.adpgc_agents.adpgc_agent.component_id }}"
        host_id: '{{ adcm_hostid }}'
        token: '{{ env.status_api_token }}'
      tags: always

    - name: Run statuschecker adbm-agent
      when:
        - adcm.config.global.adcm_url | default('') | length > 0
        - groups['adpgc_agents.adbm_agent'] is defined
      include_role:
        name: statuschecker
      vars:
        checker_type: services
        service_name: "adbm-agent.service"
        component_id: "{{ services.adpgc_agents.adbm_agent.component_id }}"
        host_id: '{{ adcm_hostid }}'
        token: '{{ env.status_api_token }}'
      tags: always

  5. To describe a cluster upgrade from one bundle version to another, add an upgrade block to the cluster description. The specified upgrade will be available in the Actions column on the Clusters page of the ADCM web interface. To do this, specify the following properties in the upgrade block:

    • name — internal identifier of the upgrade operation.

    • description — textual description of the upgrade operation.

    • display_name — upgrade operation name displayed in the ADCM web interface.

    • versions — cluster versions allowed for upgrade operation.

    • scripts (optional) — sequence of Ansible playbooks executed during the upgrade and displayed on the Jobs page.

    • from_edition — list of cluster editions permitted for the upgrade operation.

    • states — cluster states:

      • available — list of cluster states in which the upgrade operation is allowed.

      • on_success — cluster state set after a successful upgrade operation.

      • on_fail — cluster state set after a failed upgrade operation.

        NOTE
        For more information on describing an upgrade operation, refer to the upgrade article.

        It is also possible to define multiple alternative cluster upgrade scenarios. An example for the ADB cluster is provided below:

          upgrade:
    - name: 6.29.1_arenadata1_b1 (community)
      description: |
        The cluster will be prepared for upgrade. During the upgrade process,
        the cluster will be stopped and started after the installation
        is completed. To start the upgrade, run the Update cluster action.
      from_edition: ['community']
      versions:
        min: 6.0.0
        max_strict: 6.29.1_arenadata1
      scripts:
        - name: Pre-upgrade check
          script: adb/bundle_pre_check.yaml
          script_type: ansible
          on_fail: running

    - name: 6.29.1_arenadata1_b1 (community)
      description: Bundle functional upgrade from current edition
      from_edition: ['community']
      versions:
        min: 6.29.1_arenadata1
        max_strict: *version
        The bundle_pre_check.yaml file
        ---
- name: Run upgrade precheck
  gather_facts: false
  hosts: CLUSTER:!host_unreachable
  roles:
    - role: config
      vars:
        config_run:
          - base

- name: Proceed old pg hba
  gather_facts: true
  hosts: adb.master:!host_unreachable
  tasks:
    - name: Process old pg_hba
      include_role:
        name: adb_edit_pg_hba
      vars:
        adb_edit_pg_hba_action: upgrade_pghba
      when: "'pg_hba_customs' not in services.adb.config"

  6. A cluster can include services. To implement a custom service, add the following:

    • block for the description of services included in the cluster similar to the cluster description;

    • blocks (directly in the description of each service similar to the description of cluster actions and configurations):

      • config — to manage service configuration;

      • actions — to perform actions on services.

        NOTE

        • The configuration of a specific service can be accessed via the {{ vars.services.<service name>.config }} variable. In this example, the ADPG service uses {{ vars.services.adpg.config }}.

        • Note that you should specify the service value for the type property.

        Example of a service description block:

        - name: adpg
  display_name: ADPG
  type: service
  required: true
  description: "Arenadata Postgres"
  version: *version
  venv: *venv_cluster

  actions:
    Install: &install_service_adpg
      display_name: Install
      allow_to_terminate: true
      type: task
      scripts_jinja: jinja/scripts_jinja/services/adpg/install.j2
      config_jinja: jinja/configs_jinja/common/auto_reboot.j2
      states:
        available:
          - created
    - name: datadir
      display_name: Data directory
      description: >-
        Directories used to store data on hosts
      type: string
      required: true
      writable: [created]
      default: "/pg_data1"
      ansible_options:
        unsafe: True

  config:
    - name: __main_info
      type: file
      default: adpg/info/adpg.html

    - name: datadir
      display_name: Data directory
      description: >-
        Directories used to store data on hosts
      type: string
      required: true
      writable: [created]
      default: "/pg_data1"
      ansible_options:
        unsafe: True

    - name: sysctl_params_map
      display_name: Sysctl parameters [key:value]
      description: >
        Linux kernel parameters for ADPG.
      type: map
      required: false
      default: {}

  7. The service without components is fictitious and is not distributed across hosts. To allow a user to allocate a component on hosts, add at least one component. To do this:

    • add blocks describing the components included in the service similar to the service description;

    • directly in the description of each component, define the following blocks:

      • config — to set configuration parameters and provide centralized management of the configuration similar to the service configuration;

      • actions — to execute actions on a component similar to the description of the service actions.

          components:
    adpg:
      constraint: [1]
      constraint: [1,+]
      display_name: ADPG

The final version of the config.yaml configuration file is shown below:

- name: ADPG
  type: cluster
  description: Arenadata PostgreSQL
  version: &version 16.3_arenadata5.4_b1-1
  edition: community
  license: adcm_cluster_eula/RUS_EULA_CE.txt
  adcm_min_version: 2.6.0
  flag_autogeneration:
    enable_outdated_config: False
  venv: &venv_cluster "2.9"

  upgrade:
    - name: 16.3_arenadata5.4_b1 (community) minor upgrade
      description: |
        The cluster will be prepared for upgrade. During the upgrade process,
        the cluster will be stopped and started after the installation
        is completed. To start the upgrade, run the Update cluster action.
      scripts:
        - name: Convert jinja configs adcm
          script_type: ansible
          script: adpg/convert_jinja_configs_adcm.yaml
        - name: bundle upgrade
          script_type: internal
          script: bundle_switch
        - name: Convert legacy configs
          script_type: ansible
          script: adpg/convert_legacy_configs.yaml
        - name: Save previous ADPG configuration
          script_type: ansible
          script: adpg/save_previous_adpg_config.yaml
      masking:
        state:
          available:
            - installed
            - ready to upgrade
      on_success:
        state: ready to upgrade
        multi_state:
            set:
              - ready for minor upgrade CE
      from_edition: [community]
      versions:
        min: '16.3_arenadata1'
        max_strict: 16.3_arenadata5.4

  actions:
    Precheck:
      type: task
      scripts_jinja: jinja/scripts_jinja/cluster/precheck.j2
      states:
        available:
          - created
          - ready to upgrade

    Install: &install_cluster
      allow_to_terminate: true
      type: task
      scripts:
        - name: Precheck ADPG Control agents
          script_type: ansible
          script: adpg/adpgc_agents.yaml
          params:
            actions:
              - precheck
            adcm_precheck_fail_when: true
      states:
        available:
          - created
      config: &auto_reboot
        auto_reboot:
          type: boolean
          default: true
          required: true
          display_name: Reboot cluster servers after installation
          description: |
            You will need to restart the cluster server manually
            after the installation is completed in order to apply
            some settings if you refuse automatic reloading.

  config:
    - name: __main_info
      type: file
      default:  adpg/info/cluster.html

    - name: repos
      display_name: "Centos 7 repositories"
      type: group
      subs:
        - name: use_adpg_repo
          display_name: Set up ADPG repo
          type: boolean
          default: true
        - name: adpg_core_repo
          display_name: ADPG CORE repo url
          type: string
          default: "https://downloads.arenadata.io/ADPG-CORE\
                    /16.3_arenadata5/centos/7/community/$basearch/"
          description: ADPG CORE repository
        - name: adpg_repo
          display_name: ADPG repo url
          description: "ADPG repository"
          type: string
          default: "https://downloads.arenadata.io/ADPG\
                    /16_arenadata4/centos/7/community/$basearch/"

- name: adpg
  display_name: ADPG
  type: service
  required: true
  description: "Arenadata Postgres"
  version: *version
  venv: *venv_cluster

  actions:
    Install: &install_service_adpg
      display_name: Install
      allow_to_terminate: true
      type: task
      scripts_jinja: jinja/scripts_jinja/services/adpg/install.j2
      config_jinja: jinja/configs_jinja/common/auto_reboot.j2
      states:
        available:
          - created
    - name: datadir
      display_name: Data directory
      description: >-
        Directories used to store data on hosts
      type: string
      required: true
      writable: [created]
      default: "/pg_data1"
      ansible_options:
        unsafe: True

  config:
    - name: __main_info
      type: file
      default: adpg/info/adpg.html

    - name: datadir
      display_name: Data directory
      description: >-
        Directories used to store data on hosts
      type: string
      required: true
      writable: [created]
      default: "/pg_data1"
      ansible_options:
        unsafe: True

    - name: sysctl_params_map
      display_name: Sysctl parameters [key:value]
      description: >
        Linux kernel parameters for ADPG.
      type: map
      required: false
      default: {}

  components:
    adpg:
      constraint: [1]
      constraint: [1,+]
      display_name: ADPG

- name: prometheus_monitoring
  display_name: Monitoring
  type: service
  version: &monitoring_version 1.3.1
  venv: "2.9"

  components:
    prometheus:
      constraint: [0,1]
      display_name: Prometheus Server
      config:
        - name: version
          type: string
          default: '2.39.2'
          read_only: any
    grafana:
      constraint: [0,1]
      display_name: Grafana
      requires:
        - component: prometheus
      config:
        - name: version
          type: string
          default: '9.5.8'
          read_only: any
Found a mistake? Seleсt text and press Ctrl+Enter to report it