Composite cluster

The composite cluster type provides retry-aware cluster selection, allowing different retry attempts to automatically target different upstream clusters. Unlike the standard aggregate cluster which uses health-based selection, the composite cluster uses the retry attempt count to deterministically select which sub-cluster to route to.

Use cases

The composite cluster addresses several important scenarios:

Retry-based progression: Different clusters for retry attempts (primary → secondary → tertiary).
AI Gateway failover: Route initial requests to preferred providers and retries to fallbacks.
Cost optimization: Try expensive, high-performance services first, fall back to cheaper alternatives.

Configuration

The composite cluster is configured using the ClusterConfig.

Example configuration

The following example shows a composite cluster with three sub-clusters:

name: composite_cluster
connect_timeout: 0.25s
lb_policy: CLUSTER_PROVIDED
cluster_type:
  name: envoy.clusters.composite
  typed_config:
    "@type": type.googleapis.com/envoy.extensions.clusters.composite.v3.ClusterConfig
    clusters:
    - name: primary_cluster
    - name: secondary_cluster
    - name: fallback_cluster

In this configuration:

Initial requests (attempt 1) go to primary_cluster.
First retries (attempt 2) go to secondary_cluster.
Second retries (attempt 3) go to fallback_cluster.
Further retry attempts (attempt 4+) will fail with no host available.

Cluster selection

The composite cluster uses a sequential selection strategy based on retry attempt count:

Initial request (attempt 1): Uses the first cluster.
First retry (attempt 2): Uses the second cluster.
Second retry (attempt 3): Uses the third cluster.
Further retries: Fail with no host available.

When retry attempts exceed the number of configured clusters, requests fail with no host available. Configure the number of retries in your retry policy to match your cluster configuration.

Hosts availability

If the cluster selected for an attempt has no host available, for example because its endpoint list is empty after a DNS resolution failure or because all of its hosts have been ejected by outlier detection, the composite cluster fails over to the next cluster in the list within the same attempt. The clusters after it are tried in order until one of them provides a host. If none of the remaining clusters can provide a host, the request fails with no_healthy_upstream.

This behavior is guarded by the envoy.reloadable_features.composite_cluster_skip_clusters_without_hosts runtime feature which is enabled by default. When it is disabled, an attempt that maps to a cluster without an available host fails immediately instead of failing over.

Retry policy coordination

For the composite cluster to function correctly, configure an appropriate retry policy at the route level:

retry_policy:
  retry_on: "5xx,gateway-error,connect-failure,refused-stream"
  num_retries: 2  # Enables attempts 1, 2, and 3 (3 total attempts)

Important considerations

Sub-cluster independence: Each sub-cluster maintains its own health checking, load balancing, and outlier detection. If a selected sub-cluster has no healthy hosts available, the next sub-cluster in the list is used for the same attempt.
Deterministic routing: The same retry attempt will always target the same cluster given identical configuration and identical host availability. Cluster selection is based on the retry attempt count, and only fails over to the following clusters when the selected cluster has no host available.
Thread-local clustering: Cluster selection occurs at the thread-local level for optimal performance.
Sub-cluster health: Unlike the aggregate cluster, the composite cluster does not use sub-cluster health to spread load across sub-clusters. Health is only used to skip sub-clusters that cannot provide a host at all, so each attempt still targets the cluster that matches the attempt count whenever that cluster is usable.
Attempt progression after a failover: Because selection is driven by the attempt count, an attempt that failed over to a later cluster does not shift the mapping of the following attempts. For example, with [primary, secondary, fallback] and an empty primary, attempt 1 uses secondary, and attempt 2 also maps to secondary. Size the retry policy accordingly.

Comparison with aggregate cluster

Feature	Aggregate Cluster	Composite Cluster
Selection basis	Health status	Retry attempt count
Primary use case	Health-based failover	Retry progression
Overflow handling	Health-dependent	Fails request
Predictability	Health-dependent	Fully deterministic