A keyed Flink job running at parallelism 24 must reach 200 task slots before the next peak. It holds 4 TB of keyed state in RocksDB and restores from retained checkpoints. The first attempt to restart at parallelism 200 fails immediately with a message about max parallelism. Sequence the change so each step is reversible and say where the point of no return is.

Why the restart failed

Keyed state is not assigned to subtasks. It is assigned to key groups, and each subtask owns a contiguous range of them. The number of key groups is maxParallelism, fixed the first time the job snapshots its state and recorded inside every savepoint afterwards. Parallelism can be anything up to that number and nothing above it, because above it some subtask would own no key group and the state could not be split finely enough to redistribute.

Flink derives a default maxParallelism from the initial parallelism when none is set, with a floor of 128 and a ceiling of 32768. A job that started at parallelism 24 is almost certainly carrying 128, which caps it at 128 slots.

There is a second, quieter cost. Key groups are distributed evenly only when parallelism divides maxParallelism. At 128 key groups and 100 subtasks, 28 subtasks get two groups and 72 get one, so the busiest subtask carries twice the state and twice the work of the idlest, and the job's throughput is set by the busiest.

The sequence

1. Read the current maxParallelism from the latest savepoint's metadata. Read-only, reversible by definition, and it tells you whether the job is at 128 or something else.
2. Choose the new value as a power of two at roughly four to eight times the largest parallelism anyone can foresee - 1024 here. Key groups cost a small amount of metadata each; over-provisioning is far cheaper than discovering the ceiling again in a year. Plan the parallelisms you will actually run so they divide it.
3. Take a savepoint and rewrite it with the State Processor API into a new savepoint carrying the new maxParallelism. This is an offline batch job that reads the keyed state and writes it back redistributed. The original savepoint is untouched, so this step is reversible.
4. Start a second job from the rewritten savepoint at the new parallelism in shadow mode: same source topic, a different consumer group, sinks pointed at scratch tables. Compare its output against the live job over a full business cycle including a peak.
5. Stop the live job with a savepoint, start the shadow job's configuration against the live sinks from that stop savepoint's offsets, and retire the shadow instance.
6. Keep the old savepoint and the old job artifact for the agreed rollback window.

Where data can diverge and where the point of no return is

The shadow job writing to scratch tables is safe; the risk is a sink with shared external state such as a transactional producer identifier, which will fence one of the two jobs if both use it. Give the shadow job its own.

The point of no return is the first committed write by the new job past the old job's stop offset. Until then the old job restarts from its stop savepoint and loses nothing. After it, rolling back means reconciling whatever the new job has already published.

How long it really takes

Reading and rewriting 4 TB at an aggregate few hundred megabytes per second is several hours. The shadow comparison wants a full cycle. Plan two days end to end, not an evening.

When this is the wrong answer

If the job's state has a short time-to-live - a few hours of session state rather than a durable aggregate - throw the state away and rebuild it from the log at the new maxParallelism, which is faster than a 4 TB rewrite and has no savepoint surgery in it. And before any of this, check that the source has more than 24 partitions: a job reading 24 partitions gains nothing from the 25^th subtask no matter how the key groups are arranged, and the ceiling may not be the constraint anyone thought it was.