GitSync overview

Elena Kostyuchenko

GitSync is an ADO service designed for synchronizing content from remote Git repositories. Its primary use case is delivering Airflow DAGs, but GitSync can also synchronize other repository content, such as dbt projects or scripts, with configurable target directories.

GitSync main features:

  • Repository synchronization — cloning and updating files from one or multiple Git repositories.

  • Flexible synchronization scenarios — synchronization of multiple repositories, multiple branches, or multiple directories from the same repository.

  • Automated delivery — synchronization of repository content to configurable target directories.

  • Flexible filtering — selection of files using pattern-based filters.

  • Repository-specific dependency installation — optional installation of Python dependencies using a repository-specific Python environment.

  • Parallel processing — handling multiple synchronization tasks simultaneously using workers.

  • Cleanup support — optional removal of outdated files from target directories.

  • SSH key management — centralized handling of SSH credentials through service actions in ADCM.

  • Monitoring support — Prometheus metrics and Grafana dashboards for monitoring synchronization status and performance.

Workflow

GitSync operates as a standalone service and has only one component (gitsync).

The synchronization process consists of the following steps:

  1. Repository source code is stored in one or more Git repositories.

  2. GitSync clones or updates each configured repository entry independently.

  3. Files are filtered according to GitSync’s configuration.

  4. If enabled, Python dependencies are installed using the configured Python environment for that repository.

  5. Selected files are copied to the configured target directory.

  6. Optional cleanup removes outdated synchronized files.

  7. Logs and metrics are generated.

Airflow automatically discovers updated DAGs by scanning the configured DAG directory. It recursively scans all subdirectories inside the DAG folder (for example, /opt/airflow/dags).

Configuration

GitSync configuration consists of two levels:

Service-level configuration

Service-level parameters define the global behavior of the GitSync service. They are configured in the gitsync-env.sh option in ADCM.

Key parameters include:

  • number of parallel workers;

  • synchronization scheduling;

  • default synchronization timeout;

  • logging configuration;

  • default Python environment (TARGET_PYTHON) used for backward compatibility when a repository does not define its own Python interpreter.

Repository-level configuration

Repository settings are defined in the config.json option in ADCM.

Each repository configuration represents an independent synchronization task. Multiple entries may reference the same repository, branch, or directory.

Example repository configuration:

{
  "url": "git@ssh.gitlab.example.io:org/repo.git", (1)
  "branch": "main", (2)
  "directory": "./dags", (3)
  "files": "*.py", (4)
  "sync_interval": 60,
  "sync_timeout": 120,
  "ssh_key": "my-git-key", (5)
  "target_folder": "/opt/airflow/dags/project", (6)
  "sync_requirements": true,
  "requirements_path": "requirements.txt",
  "target_python": "/usr/lib/airflow/venv/bin/python", (7)
  "delete_old_files": true (8)
}
1 Git repository URL.
2 Repository branch.
3 Repository directory to synchronize.
4 File filtering rules.
5 SSH key name (for SSH repositories).
6 Target directory.
7 Python interpreter used to install dependencies for this repository. If "sync_requirements": true, the target_python value is used for installing dependencies. If omitted, the service-level TARGET_PYTHON value is used.
8 Optional cleanup behavior.

Each repository configuration is processed independently.

GitSync supports the following synchronization scenarios:

  • synchronization of multiple independent repositories;

  • synchronization of multiple branches from the same repository;

  • synchronization of multiple directories from the same repository;

  • synchronization of the same directory from different branches.

Each synchronization task should use a unique target_folder to avoid conflicts.

Example configuration
[
  {
    "url": "git@ssh.gitlab.example.io:org/project.git",
    "branch": "main",
    "directory": "./dags",
    "target_folder": "/opt/airflow/dags/main",
    "target_python": "/usr/lib/airflow/venv/bin/python",
    "files": "*.py"
  },
  {
    "url": "git@ssh.gitlab.example.io:org/project.git",
    "branch": "dev",
    "directory": "./dags",
    "target_folder": "/opt/airflow/dags/dev",
    "target_python": "/usr/lib/airflow/venv/bin/python",
    "files": "*.py"
  },
  {
    "url": "git@ssh.gitlab.example.io:org/project.git",
    "branch": "main",
    "directory": "./dbt",
    "target_folder": "/opt/dbt/project",
    "target_python": "/usr/lib/dbt/venv/bin/python",
    "files": "*"
  },
  {
    "url": "https://github.com/org/shared-dags.git",
    "branch": "main",
    "directory": "./",
    "target_folder": "/opt/airflow/dags/shared",
    "files": "*.py",
    "access_token": "******",
    "https_username": "oauth2"
  }
]

Repository synchronization

Each repository configuration is treated as an independent synchronization task.

GitSync maintains synchronization state separately for every unique combination of:

  • repository URL;

  • branch;

  • directory.

This allows different synchronization tasks to operate independently, even when they reference the same Git repository.

For example, the following configurations are supported simultaneously:

  • different branches of the same repository synchronized into different target directories;

  • different directories from the same repository synchronized independently;

  • combinations of both approaches.

SSH authentication

For SSH-based repositories, GitSync provides built-in key management:

  1. SSH keys are uploaded via the Upload private key action.

  2. Keys are stored and managed by GitSync according to the service configuration.

  3. Repository configuration references keys by name.

  4. Keys are injected at runtime.

The same SSH key can be reused across multiple repositories.

Usage

To start using GitSync:

  1. Add and install the GitSync service in ADO.

  2. Configure the service parameters.

  3. Upload SSH keys (if required) via the Upload private key GitSync action.

  4. Define repository configurations.

  5. Ensure that target directories are accessible by the corresponding applications.

After configuration is complete, GitSync automatically synchronizes each configured repository according to its synchronization interval.

Limitations

Consider the following limitations when configuring GitSync:

  • Runtime-generated files may be removed when delete_old_files=true if they are located inside the synchronized target directory.

  • dbt projects require special consideration because runtime-generated directories (for example, target/ and logs/) should not be synchronized. A typical configuration uses:

    • files = "*";

    • delete_old_files = false.

  • Requires network access to Git repositories.

  • SSH repositories require correctly configured SSH keys.

  • Multiple synchronization tasks must use different target_folder values.

  • Duplicate dag_id values across synchronized DAG repositories may lead to Airflow conflicts.

Found a mistake? Seleсt text and press Ctrl+Enter to report it