Configure Ozone metadata caching in S3 Gateway

Dmitry Bulgakov

Ozone S3 Gateway (S3G) is an Ozone component that provides an S3-compatible interface for accessing data stored in Ozone. It receives S3 requests from clients and translates them into operations against Ozone Manager (OM) and Ozone data storage.

S3G is designed to be stateless and does not persist data locally. Without optimization, each metadata request requires an RPC call to Ozone Manager to retrieve the required metadata. Operations such as HEAD Object, GET Object, and LIST Objects can therefore generate a large number of metadata requests to OM.

This behavior can significantly increase OM load in the following scenarios:

  • Object downloads. S3 clients commonly send a HEAD Object request before a GET Object request. This results in an additional metadata lookup.

  • Parallel downloads. Multi-threaded clients can split an object download into multiple ranged GET Object requests. Each request may require the same object metadata, resulting in multiple calls to OM.getKeyInfo.

As the number of concurrent S3 requests increases, the number of metadata requests also increases. This can make Ozone Manager a bottleneck for S3 workloads with a high request rate.

Metadata cache in S3 Gateway

ADH S3 Gateway can cache Ozone metadata using a metadata-caching OM client. The cache is implemented as a per-user local Caffeine cache within each S3G instance.

When S3G receives a metadata request, it first checks the local in-memory cache:

  1. If a valid cache entry exists and its age is within the configured TTL, S3G returns the cached metadata without sending an RPC request to OM.

  2. If the cache entry does not exist or has expired, S3G requests the metadata from OM.

  3. After receiving the metadata, S3G stores the value in the cache and returns the response to the client.

This reduces the number of repeated metadata requests to OM and can improve S3 request performance.

To reduce the possibility of serving stale metadata, S3G invalidates cache entries in the following cases:

  • The configured cache entry TTL expires.

  • A corresponding write operation is performed through S3G, such as object deletion or rename.

    NOTE
    Cache entry eviction is triggered only by write operations performed through S3G. If an object is modified by another client, such as an Ozone CLI client, the corresponding S3G cache entry is not immediately invalidated. The updated metadata becomes available to S3G after the cache entry expires.

To enable metadata caching in S3G, set the ozone.s3g.object.cache.enabled parameter to true.

The cache currently stores metadata for:

  • S3 volumes.

  • Buckets.

  • Object keys, including regular and extended key metadata.

Configuration parameters

The following parameters control the S3G metadata cache:

Ozone S3G component parameters
ozone-site.xml
Parameter Description Default value

ozone.s3g.object.cache.enabled

Enables S3 Gateway object cache

false

ozone.s3g.object.cache.metrics.enabled

Enables metrics for S3 Gateway object cache

true

ozone.s3g.object.cache.volume.size

Maximum number of entries in S3 Gateway cache for volumes

10

ozone.s3g.object.cache.volume.ttl.ms

TTL in milliseconds for entries in S3 Gateway cache for volumes

3000

ozone.s3g.object.cache.bucket.size

Maximum number of entries in S3 Gateway cache for buckets

100

ozone.s3g.object.cache.bucket.ttl.ms

TTL in milliseconds for entries in S3 Gateway cache for buckets

3000

ozone.s3g.object.cache.key.size

Maximum number of entries in S3 Gateway cache for keys

1000

ozone.s3g.object.cache.key.ttl.ms

TTL in milliseconds for entries in S3 Gateway cache for keys

3000

Limitations

Strict consistency is not guaranteed when caching is enabled, because an object may be modified externally during its TTL period, while S3 Gateway continues to return the stale cached value.

ADH monitoring source

Metadata caching introduces metrics that can be used to monitor cache behavior and evaluate its impact on S3G and OM.

The metrics can be visualized in the ADH monitoring system using a dedicated cache-related graph. The following metrics sources were added for all S3 object caches (s3VolumeCache, s3BucketCache, s3HeadKeyCache, s3KeyInfoCache).

Metrics name Description

hitCount

Cache hit count

hitRate

Cache hit rate

missCount

Cache miss count

loadSuccessCount

Successful cache loads

loadFailureCount

Failed cache loads

loadFailureRate

Failed cache loads rate

evictionCount

Cache eviction count

averageLoadPenalty

Average load penalty in nanoseconds
Found a mistake? Seleсt text and press Ctrl+Enter to report it