Offline package usage

Sergei Tikhomirov

Offline packages are special shell scripts that contain necessary data for offline installation of the ADH services.

Prerequisites

Before using the offline pack, make sure that your target environment satisfies the following requirements:

  • Docker is installed and running.

  • Helm (v3.8.0 and above) is installed.

  • bash shell is available.

  • Sufficient disk space for images and charts is available.

  • Network access to OCI registry (to push images) is provided.

Setup

  1. Copy the <pack.sh> archive to your target environment, where <pack> is the name of the shell script provided to you by the Arenadata support team. You can use a <pack.sh>.xz archive as well, but unpack it before usage (e.g. with xz -d <pack.sh>.xz).

  2. Verify package integrity:

    $ bash <pack.sh>

    The expected output contains a greeting and a list of the available commands:

    -------------------------------------------------------------------
Hey there!
That is a pack for offline installation prepared by Arenadata team.

Use it wise and check out available options bellow.

And make sure there's no newline at the end of this script,
otherwise pack will be corrupted and you'll see "tar: This does not look like a tar archive"
-------------------------------------------------------------------

We have following options for that pack:

unpack_adcm - load ADCM image from pack to local docker image storage
create_adcm - create ADCM container
master_adcm - unpack ADCM image, create ADCM container and run it on port 8000
unpack_et - load Enterprise Tools images from pack to local docker image storage
unpack_repo - load repo data to HTTP repo
unpack_deps - unpack ET dependencies
unpack_images - unpack and load docker images
unpack_charts - unpack helm charts
list - list repos and images in this pack

  3. List bundled artifacts:

    $ bash <pack.sh> list

    The expected output is as follows:

    REPOS:
IMAGES:
hub.arenadata.io/adc-enterprise/trino-operator:1.3.1-0
hub.arenadata.io/adc-enterprise/impala-operator:1.3.1-0
hub.arenadata.io/adc-enterprise/kerberos-operator:1.3.1-0
hub.arenadata.io/adh-enterprise/trino-docker:476_arenadata2-adh-4.2.0-x86_64
hub.arenadata.io/adh-enterprise/impala-docker:4.5.0_arenadata1-adh-4.2.0-x86_64
hub.arenadata.io/adh-enterprise/spark3-docker:3.5.4_arenadata3-adh-4.2.0-x86_64
hub.arenadata.io/adh-enterprise/spark4-docker:4.0.1_arenadata1-adh-4.2.0-java17-x86_64
hub.arenadata.io/adh-enterprise/spark4-docker:4.0.1_arenadata1-adh-4.2.0-java21-x86_64
CHARTS:
oci://hub.arenadata.io/adc-enterprise/charts/trino-operator:1.3.1-0
oci://hub.arenadata.io/adc-enterprise/charts/trino-cluster:1.3.1-0
oci://hub.arenadata.io/adc-enterprise/charts/impala-operator:1.3.1-0
oci://hub.arenadata.io/adc-enterprise/charts/impala-cluster:1.3.1-0
oci://hub.arenadata.io/adc-enterprise/charts/kerberos-operator:1.3.1-0
oci://hub.arenadata.io/adc-enterprise/charts/kerberos-config:1.3.1-0
ET_IMAGES:
ET_DEPENDENCIES:
/ADC/cli/adc_v1.3.1-0_linux_amd64
/ADC/cli/adc_v1.3.1-0_darwin_arm64
/ADC/cli/adc_v1.3.1-0_darwin_amd64
/ADC/cli/adc_v1.3.1-0_linux_arm64

  4. If you plan to install the services via CLI, extract the ET_DEPENDENCIES artifacts using the unpack_deps option. Choose the file that corresponds to your OS and make it executable with chmod +x. Optionally, you can move it to /usr/local/bin and rename it to adc.

    Once you have unpacked the tool, verify its integrity:

    $ ./adc --help

    The expected output is:

    Usage:
  adc [command]

Available Commands:
  cluster     Manage product clusters
  completion  Generate the autocompletion script for the specified shell
  help        Help about any command
  operators   Manage operators

Flags:
  -h, --help      help for adc
  -v, --verbose   Enable verbose logging
      --version   version for adc

Use "adc [command] --help" for more information about a command.

  5. Extract and load Docker images:

    $ bash <pack.sh> unpack_images

    This will:

    • Extract images from the pack.

    • Load them into local Docker storage.

    • Make images available for the docker tag/docker push commands.

      The output should contain loading confirmations:

      Loaded image: hub.arenadata.io/adh-enterprise/spark4-docker:4.0.1_arenadata1-adh-4.2.0-java17-x86_64
Loaded image: hub.arenadata.io/adh-enterprise/spark4-docker:4.0.1_arenadata1-adh-4.2.0-java21-x86_64
Loaded image: hub.arenadata.io/adc-enterprise/trino-operator:1.3.1-0
Loaded image: hub.arenadata.io/adc-enterprise/impala-operator:1.3.1-0
Loaded image: hub.arenadata.io/adc-enterprise/kerberos-operator:1.3.1-0
Loaded image: hub.arenadata.io/adh-enterprise/trino-docker:476_arenadata2-adh-4.2.0-x86_64
Loaded image: hub.arenadata.io/adh-enterprise/impala-docker:4.5.0_arenadata1-adh-4.2.0-x86_64
Loaded image: hub.arenadata.io/adh-enterprise/spark3-docker:3.5.4_arenadata3-adh-4.2.0-x86_64

  6. Tag and push images into a private registry. The list of bundled images is available on the third step. For every image you see there, apply the tag and push commands below:

    $ docker tag hub.arenadata.io/adh-enterprise/spark3-docker:3.5.4_arenadata3-adh-4.2.0-x86_64 "$PRIVATE_REGISTRY"/adh-enterprise/spark3-docker:3.5.4_arenadata3-adh-4.2.0-x86_64
$ docker push "$PRIVATE_REGISTRY"/adh-enterprise/spark3-docker:3.5.4_arenadata3-adh-4.2.0-x86_64

    The output for each push should be similar to the following:

    .0-develop-x86_64
The push refers to repository [<PRIVATE_REGISTRY>/adh-enterprise/spark3-docker]
8bdcbd2226bf: Pushed
7c0e97019764: Pushed
ee6d25b218db: Pushed
b1c9872907bd: Pushed
bc9aad66e281: Pushed
fd7b85c1b84e: Pushed
6401a22b39d4: Pushed
6b0653d72989: Pushed
5f70bf18a086: Pushed
0761d1f99054: Pushed
3.5.4_arenadata3-adh-4.2.0-x86_64: digest: sha256:06976d3d99cb2d3d890d05fcb9f061c83ab2619e99f9c259bcf58070f6afea35 size: 1845

    The registry, repository, and tags are configurable on the installation step of operators, but please keep the mapping between images from pack and images in your registry documented.

  7. Extract the Helm charts to a local directory:

    $ bash <pack.sh> unpack_charts

    This creates the charts directory structure with all the Helm charts in the .tgz format.

    Run the command below to see an organized chart structure:

    $ tree charts/

    The output:

    charts
└── hub.arenadata.io
    └── adc-enterprise
        └── charts
            ├── impala-cluster-1.3.1-0.tgz
            ├── impala-operator-1.3.1-0.tgz
            ├── kerberos-config-1.3.1-0.tgz
            ├── kerberos-operator-1.3.1-0.tgz
            ├── trino-cluster-1.3.1-0.tgz
            └── trino-operator-1.3.1-0.tgz

3 directories, 6 files

  8. Upload the Helm charts into a private registry.

    Having all charts stored in the charts directory in the .tgz format, you can use the helm push command to upload a chart into a private registry:

    $ helm push ./charts/hub.arenadata.io/adc-enterprise/charts/impala-cluster-1.3.1-0.tgz oci://$PRIVATE_REGISTRY/adc-enterprise/charts/impala-cluster

    For each push, the output should be similar to:

    Pushed: <PRIVATE_REGISTRY>/adc-enterprise/charts/impala-cluster:1.3.1-0
Digest: sha256:8c93ec1fd886d492b0867b70aff0245efe133548f26aa67cd93cc0d72f311fe1

    Note that you will use your OCI registry’s charts during the installation and management later, but please keep the mapping between charts from pack and charts in your registry documented.

Troubleshooting

Doesn’t work on macOS

Please, use the operating systems declared in the Software requirements article.

Docker images aren’t found after unpacking

Verify the images are loaded:

$ docker images | grep -E "impala|spark|trino|operator"

If missing, rerun unpack_images:

$ bash pack.sh unpack_images
Insufficient disk space

Check the available space:

$ df -h
$ docker system df

Clean up if needed. Be careful as the command below removes unused images:

$ docker system prune -a
Found a mistake? Seleсt text and press Ctrl+Enter to report it