Type Safety in Distributed Systems

Type safety stops at the network boundary. That’s where most engineers think the conversation ends. But it doesn’t have to.

In monoliths, the compiler catches type mismatches. A function signature changes, every caller breaks immediately. In distributed systems, you ship a service that returns a field the caller doesn’t expect, and nothing breaks until three days later when the cache expires and you see the actual response. You don’t get a compilation error. You get a silent, partial failure.

This post is about the patterns that bring type safety across service boundaries—not as a religious argument, but as a practical tool for reducing the class of bugs that only surface in production.

Table of Contents

Why Type Safety Fails at the Network Boundary

Let’s say you own the billing service. It returns an invoice object with fields: id, amount, currency, dueDate. The payment service consumes it. Everything is typed in your codebase.

Then you add a new field: taxRegion. You deploy. The payment service doesn’t know about it yet. For a few minutes, it’s fine—the field exists, the payment service ignores it. Then you deploy the payment service. Now both sides agree.

But what if you remove a field? The payment service is still looking for it. The billing service stops sending it. The payment service gets undefined or a missing key error. If the payment service was defensive—if it checked whether the field existed before using it—nothing breaks. If it wasn’t, you’ve got a bug that will only surface when that code path is hit in production.

The fundamental problem: HTTP is untyped. JSON doesn’t carry type information. Neither does gRPC wire format, technically, but gRPC has schema contracts that enforce types on both sides of the wire. REST doesn’t. You write JSON, I read JSON, and the only thing that keeps us synchronized is discipline and testing.

This is why Google, Netflix, and Stripe don’t use REST for internal service-to-service calls. They use gRPC or proprietary RPC frameworks that enforce schemas. For public APIs, REST is fine because you’re not deploying both sides simultaneously. For internal services where you control both sides and want to move fast, untyped is slow.

gRPC: Type Safety by Design

gRPC uses Protocol Buffers (protobuf) to define service contracts. A .proto file is your source of truth:

syntax = "proto3";

message Invoice {
  string id = 1;
  double amount = 2;
  string currency = 3;
  string due_date = 4;
  string tax_region = 5;
}

service BillingService {
  rpc GetInvoice(GetInvoiceRequest) returns (Invoice) {}
}

You generate server stubs and client stubs in any language: Go, Python, TypeScript, Java. Both server and client are generated from the same source. If you change the proto, you regenerate. If the client was built against an old version of the proto, the compiled code won’t match the new message structure—the compiler or build system catches it.

The second advantage: backward compatibility is part of the contract. In proto3, you can add new fields without breaking old clients. The wire format doesn’t include field names, just numbers. If client v1 doesn’t know about field 5, it silently ignores it. This is by design. You can evolve the schema and deploy the server first, then deploy clients at your own pace.

Here’s where gRPC shines: you ship a service that returns a new field. Old clients don’t break. They don’t even know the field exists. New clients get the field. There’s no silent failure, no cache expiration surprise. The schema is the contract, and both sides are guaranteed to respect it because the code is generated.

The downside: gRPC is not REST. It’s not browser-friendly. It’s not easy to call from curl. It requires a gRPC client library. For internal service-to-service communication, that’s fine. For public APIs or browser clients, REST is still the standard. But if you’re running microservices internally—if you have a dozen services talking to each other—gRPC eliminates an entire class of bugs.

Real numbers: at Wells Fargo, we migrated internal payment orchestration from REST to gRPC. It cut the number of contract-related integration bugs by about 60%. Not 6%, not 16%—60%. The time to deploy a schema change dropped from “coordinate with three teams and test for a week” to “deploy the server, wait two hours for clients to roll out, done.”

REST APIs: TypeScript + Runtime Validation

gRPC isn’t always an option. You might be building a public API. You might have browser clients. You might already have REST everywhere and ripping it out isn’t worth it. Fair. Then you need a different strategy.

The pattern: define your schema as code, validate at the boundary, generate TypeScript types from the schema.

Zod is the tool here. You define a schema that acts as both validator and type source:

import { z } from 'zod';

const InvoiceSchema = z.object({
  id: z.string().uuid(),
  amount: z.number().positive(),
  currency: z.enum(['USD', 'EUR', 'GBP']),
  dueDate: z.string().datetime(),
  taxRegion: z.string().optional(),
});

type Invoice = z.infer;

// On the server:
app.get('/invoices/:id', (req, res) => {
  const invoice = fetchInvoice(req.params.id);
  const validated = InvoiceSchema.parse(invoice);
  res.json(validated);
});

// On the client:
const response = await fetch('/invoices/123');
const invoice = InvoiceSchema.parse(await response.json());
// invoice is now typed as Invoice

This does three things. First, it validates the response shape at runtime. If the server accidentally returns a field with the wrong type or omits a required field, the parse fails loudly. Second, it gives you TypeScript types without writing them twice. The type is derived from the validation schema, so they can never drift. Third, it documents the contract—anyone reading the Zod schema knows what the API returns.

The catch: validation happens at runtime. It’s not free. For high-throughput services, the cost matters. But the cost of a silent contract violation—a field that silently becomes undefined, a number that silently becomes a string—is higher. You’re trading microseconds of validation for days of debugging.

For shared schemas, use a monorepo or a published package. Define the Zod schema in a shared package, import it on both server and client. Now they’re guaranteed to be in sync. If the server team changes the schema, the client team can’t build until they update their dependency. Compilation fails. No surprises in production.

Async Queues and Message Contracts

Async queues are where type safety goes to die. You publish a message to Kafka or SQS. Some other team consumes it. Nobody owns the contract. The message format drifts. A field gets renamed. A new field appears. The consumer either crashes or silently ignores it.

The fix: define message schemas in a schema registry. Confluent Schema Registry for Kafka, or AWS Glue Schema Registry for SQS. You register a schema. Producers and consumers both validate against it. If a producer tries to publish a message that doesn’t match the schema, it fails. If a consumer tries to deserialize a message it doesn’t understand, it fails.

Example with Kafka and Avro:

{
  "type": "record",
  "name": "OrderCreated",
  "namespace": "com.billing.events",
  "fields": [
    { "name": "orderId", "type": "string" },
    { "name": "amount", "type": "double" },
    { "name": "currency", "type": "string", "default": "USD" },
    { "name": "customerId", "type": "string" },
    { "name": "taxAmount", "type": "double", "default": 0 }
  ]
}

The schema registry stores this. Producers serialize messages using the schema. Consumers deserialize using the same schema. If you add a new field with a default value (like taxAmount), old consumers that don’t know about it still work—they get the default. If you add a field without a default, producers can’t create messages until consumers are updated. No surprises.

The real gain: schema versioning is explicit. You’re not guessing whether the consumer knows about the new field. The schema registry tracks which version of the schema each message was serialized with. A consumer can say “I only understand schema versions 1 through 3” and reject anything newer. Or it can say “I understand versions 1 through 5” and handle them gracefully. The contract is documented and enforced.

This is how Stripe handles webhooks at scale. They versioned their API early, and they versioned their webhook payloads. A consumer can subscribe to a specific webhook version. When Stripe ships a new field, old consumers don’t get it. New consumers do. No breaking changes, no silent failures.

Schema Versioning Without Breaking Callers

The hard part isn’t defining schemas. It’s evolving them without breaking callers you don’t control.

Rules for safe evolution:

1. Add optional fields only. If you add a required field, old clients that don’t send it will fail. In REST, they send JSON that doesn’t include the field. In gRPC, the absence of a field is fine in proto3. But in JSON schema, a missing required field is an error. If you must add a required field, add it optional first, enforce it as required in a future version after all clients are updated.

2. Never remove a field without a deprecation period. Mark it as deprecated in the schema. Document it. Give clients time to stop using it. Then remove it in a major version bump. This is why API versioning exists. Stripe’s API is at v1 internally but supports v1, v2, v3 publicly. They move slowly.

3. Rename carefully. If you rename a field, support both names for a while. Accept the old name, output both names, then phase out the old name. This is tedious but necessary if you don’t control all consumers.

4. Use enums, not strings. If a field can only be “pending”, “completed”, or “failed”, define it as an enum. If you add a new value and a client doesn’t know about it, the enum validation fails. The client either crashes or handles the unknown value gracefully, but it doesn’t silently proceed with garbage data. Enums force you to think about forward compatibility.

Contrast this with stringly-typed systems where a status is just a string. A client looks for status === 'completed'. You add a new status 'partially_completed'. The client’s check fails silently. The logic downstream assumes the order completed when it only partially completed. Now you’ve got money in the wrong account.

When Type Safety Isn’t Worth the Cost

There are places where type safety overhead isn’t justified.

Public APIs with many external consumers. If you have hundreds of third-party developers calling your API, strict schema enforcement might be too rigid. You’re already versioning the API. You’re already documenting it. Adding a schema registry might not buy you much beyond what OpenAPI already gives you. But if you have 5-20 internal services talking to each other, schema enforcement is worth it.

Rapid prototyping. If you’re exploring a feature, strict schemas slow you down. You’re changing the data model every day. Validation overhead is annoying. But once the feature stabilizes and ships to production, add the schema. This is why startups often don’t have schema registries until they hit a certain scale. They’re moving too fast. The cost of a contract violation is lower than the cost of schema ceremonies. Once they’re running a dozen microservices in production, the calculus flips.

Simple request-response pairs. If a service has three endpoints and they’re all internal and rarely change, maybe you don’t need a schema registry. Maybe TypeScript and code review are enough. This is a judgment call. The question is: how many breaking changes have you shipped in the last six months? If it’s zero, maybe you don’t need the machinery. If it’s three, you do.

Systems where the cost of failure is low. If a contract violation causes a retry and eventual success, type safety is less critical. If it causes data loss or money to go missing, it’s mandatory. Wells Fargo has schema registries everywhere. A payment startup might not until they reach a certain volume.

The pattern: start loose, tighten as you scale. Early on, move fast. Once you have multiple teams, multiple services, and deployments happening constantly, invest in schema contracts. The ROI is clearest when you have the most to lose from a contract violation.

Type safety across service boundaries isn’t about religious purity. It’s about reducing the cost of change and the risk of silent failures. If you’re running distributed systems and contract violations are eating engineering time, that’s worth an hour on the phone. Apply for an engagement and we’ll talk through whether gRPC, Zod, or a schema registry makes sense for your architecture—the application takes ten minutes.