Distributed Lock Service  ·  View 20 of 26  ·  6 · Operations

CI/CD — No Release Without a Checked History

Every change is proven under injected faults against a linearizability checker before it reaches a production arbiter or etcd member.

Editable source SVG draw.io All views
Source Pull request Gitea Proto check buf breaking Build Build + test Gitea Actions Sign image cosign → Harbor Prove Fault injection Chaos Mesh History checker Porcupine Zero double grants? Soak Staging clusters 72 h replayed load Release Argo CD sync Canary arbiter 1 of 9 etcd rolling one member at a time histories CI/CD — No Release Without a Checked History Security / platform Application we own Decision point Data store synchronous A failed history check blocks the release outright. It is not a flaky test to retry. v 1.0 · owner Platform Architecture · date 2026-09

Decisions

  • The release gate runs the real arbiter and a real etcd cluster under Chaos Mesh: partitions, member kills, clock steps, disk latency and process pauses. Every client operation is recorded with its start and end time.
  • Porcupine checks each recorded history against a sequential model of the lock. One history with two overlapping exclusive grants fails the pipeline, and the history is attached to the failure.
  • The same suite runs on every etcd version bump. The upgrade of the store is treated as a change to the lock semantics, because it is.

Rollout

  • Arbiter: one canary replica of nine for 24 hours, watched on grant latency and on any token regression detected by the reference resource.
  • etcd: one member at a time, followers first, leader last with a forced transfer, and a full health check between members.

Supply chain

  • Images are built in Gitea Actions, signed with cosign and pushed to Harbor. Admission control rejects unsigned images in the lock-system namespace.