Field Presence

A partial-update RPC writes every field in the message it receives. After a client release, support finds accounts whose discount percentage has become 0 and whose nickname has become "". No call failed, no validation rejected anything, and the audit log records a valid update. The fields are plain proto3 scalars.

A plain proto3 scalar has no presence information on the wire. The encoder omits a field whose value equals the type default, and the decoder cannot distinguish "omitted" from "explicitly set to the default". 0, "" and false are indistinguishable from absent, so the server reads defaults for fields the client never mentioned and treats them as instructions.

Why it matters

Presence decides whether "leave this field alone" is sayable. Without it, a partial-update endpoint is not implementable on that message type, however careful the handler is, and the failure is silent by construction: the only fields destroyed are the ones whose legitimate values collide with the type default. On a 40-field profile message that may be 2 of 40 fields, which is why the bug reads as selective and random rather than total.

Detection lags for the same reason. There is no error path and no rejected request. The sole signal is a distribution shift, a spike in the count of records sitting at exactly zero, which almost nobody alerts on. By the time finance notices, months of updates have applied.

Implementation patterns

• Mark the field optional. proto3 has supported explicit presence for scalar, enum, string and bytes fields since protobuf release 3.15 (February 2021), and the generated code then carries a has_ accessor. The field number and type do not move, so the wire change is backward compatible.
• Or carry a google.protobuf.FieldMask naming the paths to update. This makes intent part of the request rather than an inference from the payload, and it is the better answer for nested messages and for expressing "clear this field".
• Guard the degenerate case: reject an update whose mask is empty, or that sets no field with presence, rather than treating it as "set everything to default".
• Wrap scalars in message types (google.protobuf.Int32Value and friends) where you cannot change the proto edition; message fields have always tracked presence. The cost is an extra allocation and a clumsier API, which is why optional superseded the practice.
• Name full-replacement operations honestly. If omitted fields are cleared, call the method Replace and document it.
• Alert on the shape of the data, not just on errors: the count of records at the type default per field, compared with yesterday. A jump of a few hundred records a day on a field that normally moves by single digits is the only early signal this failure produces.

Industry example

The semantics are documented in the protobuf project's own field-presence application note, which states plainly that proto3 basic-type fields had no-presence semantics until release 3.15 added the optional label, and that repeated fields and maps still do not track presence. The same trap recurs outside protobuf: in JSON APIs it appears as the merge-patch null ambiguity, where null means "delete this field" and an omitted key means "leave it", so a client that serialises its whole model with nulls for unset values clears data it never meant to touch. The encoding changes; the question does not.

Failure scenarios

• Silent zeroing on partial update, as above, with no error and no rejected request.
• A boolean flag that cannot be turned off, because false is indistinguishable from absent, so the API can only ever set it true.
• Defaults overwriting good data on round-trip, where a client reads a record, re-serialises it through a generated type that drops unknown or default fields, and writes it back.
• Repeated fields and maps, where presence is still not tracked, so an empty list cannot be distinguished from an unspecified one and "remove all tags" is inexpressible.
• Cross-language skew, where one binding exposes a has-bit and another does not, so the bug exists only in some clients.

Trade-offs

When not to use it

If the operation genuinely replaces the whole object, do not add presence machinery — document that omitted fields are cleared and require the complete object. For high-volume telemetry where every byte counts and defaults are genuinely meaningful as "nothing happened", no-presence scalars are the right encoding and the smaller message is the point. The indefensible position is the middle one: partial-update semantics on an encoding that cannot say which fields were sent.

Interview question

Q: A team wants a PATCH-style RPC for a user profile with 40 fields, half of them optional, several boolean. They are using proto3 and want the smallest possible message. Walk me through the design, the compatibility implications of each option, and the one guard you would insist on before it ships.

What a strong answer covers: the presence problem stated in terms of the encoding, not the handler · optional versus FieldMask versus wrapper types, with the compatibility consequences of each · why booleans are the first casualty · rejecting empty-mask updates · the monitoring signal (records at the type default per field) · and the observation that conditional requests, not presence, solve the concurrent-update race that a partial update also has.

Quick check

Quiz: Why can a plain proto3 boolean field never be set back to false by a partial update? — Because false is the type default, so it is omitted on the wire and the server cannot distinguish it from an unset field.

Flashcard: Since which protobuf release can proto3 scalars carry explicit presence, and how? — Release 3.15, February 2021, by marking the field optional, which generates a has_ accessor without changing the field number or type.