Demystifying remote procedure calls

Every distributed system eventually needs one component to ask another to do something. There are a few ways to structure that request, and RPC is the one that tries hardest to hide the fact that a network is involved.

I have spent a fair amount of time in this ecosystem — my Season of Docs work in 2020 was on gRPC-Gateway, and 2021 was on the gRPC and OpenAPI surface of Wechaty. Most of what follows is what I wish I had understood before starting.

The goal

A remote procedure call aims to make calling a function on another machine look like calling one locally.

Locally, this is unremarkable:

result := calculatePower(2, 3)

If calculatePower lives on another server, the honest version of that call is considerably longer. You need to find the server, open a connection, serialise the arguments into something it understands, send them, wait, receive a response, deserialise it, and decide what to do when any of that fails.

RPC frameworks aim to collapse all of it back to:

result := calculator.CalculatePower(ctx, 2, 3)

The property being sold here is location transparency — the call site does not encode where the code runs. That is genuinely valuable, and it is also the source of every complaint people have about RPC. Hold that thought.

How the trick works

Two pieces: an interface definition and generated stubs.

flowchart LR
    A[Client code] --> B[Client stub]
    B -->|marshal| N((network))
    N -->|unmarshal| C[Server stub]
    C --> D[Your implementation]

The interface definition describes the contract in a language-neutral format. With Protocol Buffers, that is a .proto file:

service NotificationService {
  rpc SendOtp (SendOtpRequest) returns (SendOtpResponse);
}
 
message SendOtpRequest {
  string email = 1;
}
 
message SendOtpResponse {
  bool success = 1;
  string message = 2;
}

This is the single source of truth. Both sides are generated from it, which is what makes them impossible to disagree about.

The stubs are generated from that file, in whatever languages you need. The client stub takes native arguments, marshals them into the wire format, sends them, waits, unmarshals the reply, and hands it back as a native object. The server stub does the reverse and calls the implementation you wrote.

Neither stub is code anyone maintains. That is the point — the tedious, error-prone half of network programming becomes a build step.

What you get

A contract that is checked. The IDL is strongly typed and both sides are generated from it, so an entire category of integration bug — the two services disagreeing about a field name — stops being possible.

Efficiency. Protobuf's binary encoding is substantially smaller than JSON, and gRPC runs over HTTP/2, so many concurrent calls share one connection instead of queueing behind each other.

Streaming as a first-class thing. Client-side, server-side, and bidirectional streams are part of the model rather than something bolted on with websockets.

Polyglot for free. Define once, generate clients for every language you use. For an organisation with services in four languages this is the practical argument that wins.

Client libraries you did not write. Consumers generate their own from your .proto. Nobody hand-maintains an SDK.

What it costs

The network is still there. This is the central tension. A local call takes nanoseconds and cannot fail on its own; a remote call takes milliseconds and fails routinely — timeouts, partial failures, the server being mid-deploy.

Making it look local is exactly what makes this easy to forget. Code that would obviously be wrong if the network were visible — a call in a tight loop, no timeout, no idea what happens on retry — looks perfectly normal when the call site is one line. The abstraction is doing its job, and its job includes hiding things you needed to know.

Coupling through the contract. Changing the IDL means regenerating and redeploying both sides. Protobuf's field-numbering rules make this manageable — add fields, never renumber, never reuse a number — but the discipline is real and version skew during a rollout is something you have to plan for.

Debugging is harder. A binary protocol is not something you can inspect with curl. You need grpcurl and reflection enabled, and a packet capture tells you much less than it would for JSON over HTTP.

Browsers cannot speak it directly. gRPC depends on HTTP/2 features browsers do not expose, so calling it from a web page needs gRPC-Web and a translating proxy — which is a meaningful chunk of what gRPC-Gateway exists to do.

The setup cost is front-loaded. Choosing a framework, writing the IDL, wiring generation into the build. Worth it at ten services; hard to justify at two.

When it fits

The consistent pattern: RPC wins for internal service-to-service traffic, where you control both ends, you care about latency and payload size, and the number of services is large enough that generated clients save real work.

REST tends to win at the public edge, where consumers are people you have never met, debuggability matters more than efficiency, and "it works with curl" is a feature rather than an afterthought.

Plenty of systems run both, and gRPC-Gateway exists precisely because that combination is common enough to automate — one .proto, gRPC internally, REST at the boundary.

The thing to hold on to

RPC is a good abstraction, and abstractions are judged by what they hide and what they let leak.

RPC hides serialisation, transport, and connection management, which are genuinely tedious and which you will not miss. It cannot hide latency or partial failure, because those are properties of the network rather than of the code, and no amount of generated stub will make a machine in another datacentre respond in a nanosecond.

Use it for what it hides. Stay aware of what it does not — the calls that look local are the ones most likely to surprise you at three in the morning.