Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Root cutover

A DataflowRuntime configured by ReconfigurableDataflowRuntimeBuilder handles one control request at a time in the same owner loop that drives data ticks. It first constructs a complete resource-free candidate, then rebinds the persistent input and output sessions and applies the monitor change in a fixed order. Application has no rollback.

Planning

#Planning phaseResponsible entityProduces or validates
1Validate request structureDataflowRuntimeA structurally valid ReconfigurationRequest.
2Parse and compile replacementreconfiguration compilerA candidate DataflowProgram without live state.
3Resolve and plan inputInputPipelineCandidate ResolvedInput and InputPipelineReconfigurationPlan.
4Resolve and plan outputOutputPipelineCandidate ResolvedOutput and OutputPipelineReconfigurationPlan.
5Plan monitor replacementDataflowMonitorMonitorReconfigurationPlan::RetainExact, InstallCold, or Transfer.
6Assemble candidateDataflowRuntimeOne complete RuntimeReconfigurationPlan.

All six phases are resource-free. A failure leaves active owner structures unchanged, although DataflowRuntime treats the failed command as terminal.

The monitor plan is one of three forms:

  • MonitorReconfigurationPlan::RetainExact keeps the healthy active monitor when the definition is exact and policy permits retention.
  • MonitorReconfigurationPlan::InstallCold constructs initialized state when transfer is disabled or the active monitor cannot donate state.
  • MonitorReconfigurationPlan::Transfer constructs a target monitor and a validated semantic mapping from the active definition.

Application

#Application phaseResponsible entityApplied effect
1Rebind input sessionInputPipelineSessionEstablish the local boundary, evaluate already-admitted old batches through the callback, rebind retained sources, remove or open owners, resume ingress, and mark the input revision pending.
2Flush pending engine rowsDirectDataflowEngine and OutputWriterSubmit output produced by old-side batches before the candidate monitor and interfaces become active.
3Rebind output sessionOutputPipelineSessionFlush each affected destination writer, apply supported interfaces, and update router selection; mark the output revision pending.
4Apply monitor replacementDataflowMonitorRetain, cold-install, or transfer semantic state according to the plan.
5Commit session revisionsInputPipelineSession and OutputPipelineSessionAdvance both session revisions after input, output, and monitor application succeeds.
6Rebuild transient row layoutsDirectDataflowEngineRecreate reusable input/output rows and cached slot mappings around the active monitor.
7Send acknowledgementDataflowRuntimePublish revision and change flags after all local application phases complete.

Mutation begins at phase 1. Every later phase can fail after earlier effects have occurred, and acknowledgement is emitted only after phase 7 is reached.

Root cutover: old drain precedes candidate processing Read left to right on the single shared time baseline. Ordinary old data ticks d0 and d1 precede the delivered control request and barrier. The admitted old ticks d2 and d3 follow the barrier but are drained and evaluated under the old definition. Local input, output, monitor, revision, and row-layout changes then apply in sequence. The candidate state is active locally before acknowledgement is attempted, but the owner loop does not process candidate tick d4 unless acknowledgement succeeds. The barrier and processing-resumption positions are distinct. Dashed vertical guides mark those positions, while the solid horizontal arrow marks time. The sequence is ordered and non-transactional: earlier effects are not rolled back. The compact cells show temporal data and control facts only; phase internals are intentionally omitted. Root cutover: old drain precedes candidate processing OLD evaluation CANDIDATE evaluation delivered control barrier candidate processing resumes control request delivered acknowledgement succeeds processing may resume time data / evaluation d0 ordinary · OLD d1 ordinary · OLD d2 drain · OLD eval d3 drain · OLD eval d4 first · CANDIDATE ordered cutover; acknowledgement last non-transactional · no rollback

Reading rule. The first dashed line is the locally delivered control barrier, not the instant at which all old work disappears. Removed or changed owners can still hold admitted batches; DataflowRuntime evaluates those logical ticks with the old DataflowMonitor and flushes their output before the candidate monitor and output interfaces become active. Candidate state is applied before acknowledgement, but the owner loop resumes candidate processing only after acknowledgement succeeds. Neither line is a rollback boundary.

The old monitor evaluates every row drained from removed sources. Its resulting pending output rows cross the writer boundary before the candidate output interfaces and monitor become active.

Monitor replacement and revisions

Applying a monitor plan advances MonitorRevision for an accepted root activation, including exact retention. InterfaceRevision advances only when the effective input or output interface changed. The acknowledgement reports both revisions and both change flags, not the detailed transfer report.

After monitor application, the direct engine rebuilds reusable rows and cached slot layouts for the replacement monitor. This transient rebuild does not own language state.

Partial failure and cleanup

The cutover is serial but not atomic across subsystems. Examples of reachable partial state include:

  • removed input owners already detached when an addition fails;
  • input committed before output interface application fails;
  • an earlier destination interface updated before a later destination fails;
  • I/O changes applied before monitor transfer fails;
  • the entire local cutover applied before acknowledgement delivery fails.

Cleanup runs on failure, but prior effects are not reversed. The runtime terminates rather than continuing from a partially applied command.

Separate nested mechanism

dynamic and defer changes occur inside DataflowMonitor::evaluate. They resolve nested bodies and repair the active schedule before the tick's main range. They neither replace the root InputPipelineSession and OutputPipelineSession nor use this planning/application sequence.

Implementation mapping

The implementation mapping centers on plan_runtime_reconfiguration, apply_runtime_reconfiguration, and run_reconfigurable_dataflow in src/runtime/dataflow.rs, with plans supplied by input, output, and monitor modules.

Continue with replacement identity, context transfer, and failure containment.