Distributed Configuration Management — Consul, etcd, Spring Cloud Config, and Hot Reload

When you have three services, environment variables in a .env file work fine. When you have three hundred services across multiple regions, a single misconfigured timeout can cascade into a full outage. Distributed configuration management solves this by centralizing, versioning, and dynamically distributing configuration to every service in your fleet — without redeployments.

Why configuration management matters at scale#

Every non-trivial distributed system faces these configuration challenges:

• Consistency: All instances of Service A should use the same database connection pool size
• Environment parity: Dev, staging, and production need different values for the same keys
• Auditability: Who changed the rate limit from 1000 to 100 at 3 AM?
• Speed of change: Adjusting a circuit breaker timeout should not require a full CI/CD pipeline
• Safety: A bad configuration change should be reversible in seconds, not minutes

Config service architecture#

A well-designed configuration service has four layers:

1. Storage layer#

The source of truth for all configuration data. Must provide strong consistency guarantees — you never want two instances of the same service reading different values for the same key.

2. API layer#

A well-defined interface (REST, gRPC, or client library) that services use to read configuration. Supports namespacing, versioning, and access control.

3. Distribution layer#

Pushes configuration changes to running services in near real-time. This can be pull-based (polling), push-based (watches/subscriptions), or hybrid.

4. Client layer#

A lightweight library embedded in each service that handles fetching, caching, watching for changes, and applying updates without restarts.

+------------------+     +------------------+     +------------------+
|   Config Store   |---->|    Config API    |---->|   Service A      |
|  (Consul/etcd)   |     |  (REST / gRPC)   |     |  (client lib)    |
+------------------+     +------------------+     +------------------+
        |                         |                        |
        |                         |               +------------------+
        |                         +-------------->|   Service B      |
        |                                         |  (client lib)    |
        |                                         +------------------+
        |
+------------------+
|  Version History |
|  (audit log)     |
+------------------+

Consul KV#

HashiCorp Consul is a service mesh and configuration store built on the Raft consensus algorithm. Its key-value store is designed for distributed configuration.

Key features#

• Hierarchical keys: Organize configuration as service/payments/timeout or global/rate-limit
• Blocking queries: Long-poll on keys — the server holds the connection until the value changes
• Transactions: Atomically read-check-set multiple keys
• ACLs: Fine-grained access control per key prefix
• Multi-datacenter: Configuration replication across regions

Reading and writing configuration#

# Write a configuration value
consul kv put service/payments/timeout_ms 3000

# Read it back
consul kv get service/payments/timeout_ms

# List all keys under a prefix
consul kv get -recurse service/payments/

# Atomic check-and-set (CAS) — only update if current index matches
consul kv put -cas -modify-index=42 service/payments/timeout_ms 5000

Watching for changes#

import consul

c = consul.Consul()

index = None
while True:
    # Blocking query: waits until the value changes
    index, data = c.kv.get("service/payments/timeout_ms", index=index)
    if data:
        new_value = data["Value"].decode()
        apply_config_change("timeout_ms", new_value)

When to choose Consul#

• You already use Consul for service discovery
• You need multi-datacenter configuration replication
• Your team is comfortable with the HashiCorp ecosystem
• You want key-value configuration with service mesh integration

etcd#

etcd is a distributed key-value store that powers Kubernetes. It uses Raft for consensus and provides strong consistency guarantees.

Key features#

• Watch API: Server-push notifications when keys change (gRPC streaming)
• Leases: Time-to-live on keys — perfect for ephemeral configuration
• Transactions: Multi-key compare-and-swap operations
• Revisions: Every key modification increments a global revision, enabling point-in-time reads
• Range queries: Efficiently read all keys in a prefix range

Reading and writing#

# Write configuration
etcdctl put /config/service/payments/timeout_ms 3000

# Read it back
etcdctl get /config/service/payments/timeout_ms

# Read all keys under a prefix
etcdctl get /config/service/payments/ --prefix

# Read a specific historical revision
etcdctl get /config/service/payments/timeout_ms --rev=42

Watching for changes with gRPC#

package main

import (
    "context"
    "fmt"
    clientv3 "go.etcd.io/etcd/client/v3"
)

func watchConfig(client *clientv3.Client) {
    watchChan := client.Watch(
        context.Background(),
        "/config/service/payments/",
        clientv3.WithPrefix(),
    )

    for watchResp := range watchChan {
        for _, event := range watchResp.Events {
            fmt.Printf("Key: %s, Value: %s, Type: %s\n",
                event.Kv.Key, event.Kv.Value, event.Type)
            applyConfigChange(string(event.Kv.Key), string(event.Kv.Value))
        }
    }
}

When to choose etcd#

• You run Kubernetes and want to consolidate tooling
• You need strong consistency with revision history
• Your services are primarily written in Go
• You need watch-based (push) configuration updates

Spring Cloud Config#

For Java/Spring ecosystems, Spring Cloud Config provides a dedicated configuration server backed by Git, Vault, or a database.

Key features#

• Git-backed storage: Configuration files stored in a Git repository — version history for free
• Environment profiles: application-dev.yml, application-prod.yml with automatic profile resolution
• Encryption/decryption: Encrypt sensitive values at rest, decrypt on read
• Spring Boot integration: @RefreshScope beans automatically reload on configuration change
• Composite backends: Read from Git, then overlay with Vault secrets

Server configuration#

# application.yml for Config Server
spring:
  cloud:
    config:
      server:
        git:
          uri: https://github.com/myorg/config-repo
          default-label: main
          search-paths: "{application}"
          clone-on-start: true

Client usage#

# bootstrap.yml for a Spring Boot service
spring:
  application:
    name: payment-service
  cloud:
    config:
      uri: http://config-server:8888
      fail-fast: true
      retry:
        max-attempts: 5
        initial-interval: 1000

@RestController
@RefreshScope
public class PaymentController {

    @Value("${payment.timeout.ms:3000}")
    private int timeoutMs;

    @GetMapping("/config")
    public Map<String, Object> getConfig() {
        return Map.of("timeoutMs", timeoutMs);
    }
}

Triggering a refresh#

# Refresh a single service instance
curl -X POST http://payment-service:8080/actuator/refresh

# Or use Spring Cloud Bus to broadcast to all instances
curl -X POST http://config-server:8888/actuator/busrefresh

When to choose Spring Cloud Config#

• Your stack is Java/Spring Boot
• You want Git as the source of truth for configuration
• You need encryption of secrets at rest
• Your team prefers declarative YAML configuration

Hot reload patterns#

The ability to update configuration without restarting services is what separates configuration management from environment variables.

Pattern 1: Polling#

Services periodically fetch configuration from the server.

• Pros: Simple, works through firewalls, no persistent connections
• Cons: Update delay equals poll interval, wasted requests when config has not changed
• Best for: Non-critical configuration, environments where push is impractical

Pattern 2: Watch / Subscribe#

Services open a persistent connection and receive push notifications.

• Pros: Near-instant updates, no wasted traffic
• Cons: Connection management overhead, requires reconnection logic
• Best for: Critical configuration that must propagate quickly

Pattern 3: Sidecar / Agent#

A local agent (like Consul Agent or a custom sidecar) watches for changes and writes updated configuration to a local file or shared memory.

• Pros: Language-agnostic, service reads from local filesystem
• Cons: Extra process to manage, slightly higher resource usage
• Best for: Polyglot environments, legacy services that cannot use a client library

# Consul Template sidecar example
template {
  source      = "/etc/consul-template/payment.ctmpl"
  destination = "/etc/payment-service/config.json"
  command     = "pkill -HUP payment-service"
}

Configuration versioning#

Treat configuration changes like code changes. Every modification should be:

Versioned#

Store the full history of every configuration change. etcd does this natively with revisions. Consul supports it through audit logging. Spring Cloud Config gets it free from Git.

Diffable#

Before applying a change, show what is different. This prevents accidental overwrites and makes reviews meaningful.

# Compare current config with a proposed change
diff <(consul kv get -recurse service/payments/) proposed-config.json

Rollbackable#

Every configuration change should be reversible within seconds:

# etcd: restore a key to a previous revision
etcdctl get /config/service/payments/timeout_ms --rev=41

# Consul: use a transaction to atomically roll back multiple keys
consul kv put service/payments/timeout_ms 3000

# Spring Cloud Config: revert a Git commit
git revert HEAD && git push origin main

Validated#

Apply schema validation before writing configuration:

from jsonschema import validate

config_schema = {
    "type": "object",
    "properties": {
        "timeout_ms": {"type": "integer", "minimum": 100, "maximum": 30000},
        "max_retries": {"type": "integer", "minimum": 0, "maximum": 10},
    },
    "required": ["timeout_ms"]
}

def update_config(key, value):
    validate(instance=value, schema=config_schema)
    consul_client.kv.put(key, json.dumps(value))

Feature flags as configuration#

Feature flags are a specialized form of configuration. Rather than building a separate system, many teams manage them through their configuration service.

Simple boolean flags#

{
  "features": {
    "new_checkout_flow": true,
    "dark_mode": false,
    "beta_api_v2": true
  }
}

Percentage rollouts#

{
  "features": {
    "new_checkout_flow": {
      "enabled": true,
      "rollout_percentage": 25,
      "allowed_users": ["user-123", "user-456"]
    }
  }
}

Evaluating flags in code#

import hashlib

def is_feature_enabled(feature_name, user_id=None):
    flag = config.get(f"features.{feature_name}")

    if isinstance(flag, bool):
        return flag

    if not flag.get("enabled", False):
        return False

    # Check allowed users list
    if user_id and user_id in flag.get("allowed_users", []):
        return True

    # Percentage-based rollout using consistent hashing
    if "rollout_percentage" in flag and user_id:
        hash_val = int(hashlib.md5(
            f"{feature_name}:{user_id}".encode()
        ).hexdigest(), 16)
        return (hash_val % 100) < flag["rollout_percentage"]

    return flag.get("enabled", False)

Why configuration stores work for feature flags#

• Same versioning and rollback capabilities
• Same hot-reload infrastructure
• Same access control and audit logging
• No additional service dependency

For teams that need advanced targeting (user segments, A/B testing, analytics), dedicated feature flag services like LaunchDarkly or Flagsmith add value. But for straightforward boolean and percentage flags, your configuration store is enough.

Comparison table#

Conclusion#

Distributed configuration management is infrastructure that pays for itself the first time you need to change a timeout across 200 services at 2 AM without deploying anything. Consul KV fits HashiCorp-native stacks, etcd is the natural choice for Kubernetes environments, and Spring Cloud Config serves Java teams that want Git-backed configuration. Whichever you choose, invest in hot reload, version every change, validate before writing, and treat feature flags as first-class configuration citizens.

---

This is article #418 on Codelit.io — your deep-dive resource for system design, backend engineering, and infrastructure patterns. Explore more at codelit.io.