gRPC in Production: What the Documentation Doesn't Tell You
gRPC documentation is thorough on the protocol’s features and sparse on the operational realities of running it at production scale. The gap between the getting-started experience and the production experience is wide enough to have surprised most of the teams that have made the journey. The surprises are not fatal. They are the kind that would have been useful to know before the architecture decision was made.
The gRPC pitch is compelling: Protocol Buffers serialization that is faster and smaller than JSON, HTTP/2 multiplexing that reduces connection overhead, generated client and server stubs that eliminate serialization bugs, strong typing that catches integration errors at compile time rather than runtime. All of these benefits are real. All of them come with operational requirements that the pitch does not emphasize.
The Load Balancing Problem
HTTP/2’s multiplexed connection model — which allows multiple requests to share a single TCP connection — creates a load balancing problem that HTTP/1.1 does not have. With HTTP/1.1, load balancers distribute requests across backend instances at the connection level, because each request uses its own connection. With HTTP/2, a single long-lived connection carries many requests, and a load balancer that operates at the connection level sends all of those requests to the same backend instance.
The practical consequence is that an HTTP/2 gRPC service behind a standard L4 load balancer does not actually load balance in the way the operator expects. All traffic from a given client goes to the same backend for the lifetime of the connection. Backend instances that happen to establish more client connections receive more traffic. The distribution is uneven and not configurable through the load balancer’s routing rules.
The solutions require operating at the application layer: L7 load balancers that understand HTTP/2 and can distribute individual requests across backends (Envoy is the most commonly deployed option), client-side load balancing where the client distributes requests across a pool of backends it discovers through a service registry, or a service mesh (Istio, Linkerd) that handles load balancing transparently.
None of these solutions is complicated. All of them require additional infrastructure that the gRPC documentation’s getting-started examples do not prepare teams to operate.
Observability Gaps
The observability tooling that most teams have in place — HTTP access logs, curl-based health checks, browser-based debugging — does not work with gRPC. gRPC uses binary Protocol Buffers encoding over HTTP/2. Log lines that capture raw request and response bodies produce unreadable binary output. Tools that send HTTP/1.1 requests cannot call gRPC endpoints. Browser developer tools do not decode Protocol Buffers.
The tooling replacement is grpcurl — the gRPC equivalent of curl — for manual request testing, Protobuf-aware logging middleware that decodes binary messages into human-readable JSON for log analysis, and distributed tracing instrumentation that captures gRPC metadata in trace spans. All of these are available and functional. None of them are defaults that come with a gRPC service out of the box.
Schema Evolution
Protocol Buffers schema evolution is more constrained than JSON API evolution. Fields in Protobuf messages are identified by field numbers rather than names. Adding a new field requires assigning it a new, unused field number. Removing a field requires marking its number as reserved to prevent reuse by future fields — a future field with the same number would be misinterpreted by older clients as the deleted field.
These constraints mean that schema changes must be planned more carefully than JSON API changes, and that the schema file must be treated as a long-term historical record with reserved field numbers preserved indefinitely. Teams that delete reserved declarations to clean up the schema file create compatibility bugs between old clients and new servers.
When gRPC Is Worth It
The operational overhead of gRPC — the load balancing infrastructure, the observability tooling, the schema management discipline — is justified when the benefits it provides matter enough to absorb those costs. High-throughput internal microservice communication where Protocol Buffers’ serialization efficiency provides measurable infrastructure cost reduction. Systems where compile-time type safety across service boundaries catches a meaningful number of integration bugs. Streaming use cases where gRPC’s bidirectional streaming eliminates the polling overhead of REST.
For teams replacing REST with gRPC because gRPC seems more advanced or because benchmarks show it as faster: the operational overhead is not worth the marginal performance improvement in most application contexts. The teams that benefit most from gRPC are those where the specific technical properties — performance at scale, type safety, streaming — address specific production problems they have documented. The teams that regret gRPC adoptions are those that chose it on principle rather than on problem fit.