Service A occasionally fails to reach Service B with connection timeouts, even though both services report healthy, especially during traffic spikes.
Add retries with backoff, a circuit breaker, explicit timeouts, and use a service registry or Kubernetes DNS instead of hardcoded service addresses.
Step-by-Step Guide
Replace hardcoded URLs with a service registry (Consul, Eureka) or Kubernetes service DNS names
Add exponential backoff retries for transient failures, carefully for non-idempotent POSTs
Add a circuit breaker (e.g., opossum in Node.js) to fail fast when a downstream service struggles
Set explicit connect and read timeouts on HTTP clients instead of relying on defaults
Check for connection pool exhaustion on the calling service under concurrent load
Add distributed tracing (OpenTelemetry) to pinpoint exactly where latency is introduced
Verify Kubernetes readiness/liveness probes aren't marking healthy pods unhealthy under load
Scale the downstream service horizontally if timeouts correlate with traffic spikes
Found an issue with this solution?
