When Messages Outlive the State That Created Them

Imagine an Embedded Linux device with several independent processes responsible for collecting different types of sensor data.

A temperature process talks to temperature sensors. A pressure process talks to pressure sensors. A motion control process collects encoder readings. A central SensorHub process keeps track of these processes and discovers what kind of data each one can provide.

The sensor processes are independent Linux processes, and they communicate with SensorHub through Remote Procedure Calls (RPCs).

When a sensor process starts, it initializes its hardware, registers itself with SensorHub, and becomes available to the rest of the system. When it uninitializes, it does the opposite: it unregisters from SensorHub, disconnects its sensors, releases its resources, and exits.

Under normal circumstances, the lifecycle looks straightforward:

sequenceDiagram
    participant H as SensorHub
    participant T as TempProcess
    participant S as Temperature Sensors
    autonumber
    T->>S: Initialize sensors
    activate S
    S-->>T: Sensors ready
    deactivate S
    T->>H: Register
    activate H
    H-->>T: Registration request queued
    deactivate H

    note over H: SensorHub picks up the request from queue
    H->>H: Add TempProcess to registry
    activate H

    H->>T: Discovery request
    T->>S: Query sensor information
    activate S
    S-->>T: Sensor information
    deactivate S
    T-->>H: Sensor capabilities
    deactivate H

    Note over T,S: Normal operation
    T->>H: Unregister
    activate H
    H-->>T: Unregistration request queued
    deactivate H
    H->>H: Remove TempProcess from registry

    T->>S: uninitialize sensors
    activate S
    S-->>T: Sensors powered-off
    deactivate S

    T->>T: Release resources
    T->>T: uninitialize

Nothing particularly unusual so far. Then we hit an interesting shutdown sequence.

TempProcess started its uninitialization sequence almost immediately after registering with SensorHub.

It initialized the sensors, sent register request, immediately sent an unregister request, and uninitialized the sensors.

However, the observation is: there is always an RPC timeout ! That was surprising !

The first clue: the process was already terminating

#

The issue first became visible through logging.

TempProcess was clearly going through its uninitialization sequence. Its sensors have been disconnected, and the process was shutting down.

Yet the logs showed incoming discovery requests from SensorHub, the kind of request that should have been generated as part of the registration process.

And since TempProcess has shutdown its sensors, it now makes sense why I am getting the RPC timeout. But I still need to know why SensorHub is sending discovery request in an inappropriate time ?

Looking at the RPC handler of SensorHub revealed that, when SensorHub receives a registration request, it needs to learn what the new provider can actually do. It therefore performs a discovery step, sending requests back to the registering process to query its capabilities.

It looked like this:

void rpc_handler_register(agent)
{
    add_agent_to_registry();
    // discover is called from the same context
    discover(agent)
}

The discovery wasn’t scheduled for later. It happened directly inside the RPC handler.

That detail turned out to be the key to the whole problem.

The detail that matters: That RPC request is asynchronous

#

At this point, there is an obvious question:

If SensorHub is still processing the registration, how can TempProcess already be sending an unregister request?

Because the discovery RPC request is asynchronous, which means that when TempProcess sends a request to SensorHub, it does not block until SensorHub finishes processing that request. TempProcess’s call to register only puts a request in the queue of SensorHub, and hence, TempProcess can add another request (like unregister).

From SensorHub’s perspective, it processes these requests one by one, so it will complete the whole logic of registration handler, then will start processing the logic of unregistration handler.

While SensorHub was busy with handling the logic of registration handler (including sending discovery requests), TempProcess was already uninitializing, and shutting down its sensors.

The failure sequence

#

Now consider what happens when TempProcess starts uninitializing immediately after registering.

It sends the registration request and continues executing.

Before SensorHub has finished processing that registration, TempProcess starts its uninitialization sequence and sends an unregister request.

From TempProcess’s perspective, everything looks perfectly reasonable: Register, then continue execution, then begin uninitialization, then unregister and disconnect sensors. It expects no more requests at this time.

But inside SensorHub, the requests are processed through its RPC queue.

At the critical moment, the SensorHub is processing the registration (that includes discovery request), while the unregisteration is queued.

So while the unregister request is waiting, SensorHub continues executing the registration logic and sends a discovery request to TempProcess.

The actual sequence therefore looks like this:

sequenceDiagram
    participant H as SensorHub
    participant T as TempProcess
    participant S as Temperature Sensors
    autonumber

    T->>S: Initialize sensors
    activate S
    S-->>T: Sensors ready
    deactivate S
    T->>H: Register
    activate H
    H-->>T: Registration request queued
    deactivate H
    T->>H: Unregister
    activate H
    H-->>T: Unregistration request queued
    deactivate H
    note over H: SensorHub will execute Register first, then Unregister
    T->>S: uninitialize sensors
    activate S
    S-->>T: Sensors powered-off
    deactivate S
    T->>T: Release resources
    T->>T: uninitialize
    note over H: SensorHub picks up first item from queue: Processing register request
    H->>H: Add TempProcess to registry
    activate H

    H->>T: Discovery request
    T->>S: Query sensor information
    activate S
    note over S: Sensors are off, request hangs forever, get RPC timeout
    deactivate H

This is the surprising part.

The unregister request was already sent.

But it was still sitting in SensorHub’s queue.

Therefore, from SensorHub’s point of view, TempProcess was still registered when the registration handler performed discovery.

Nothing was actually contradictory

#

The behavior looks contradictory if we think about registration and unregistration as instantaneous state transition: register means registered now, and unregister means unregistered now, but this is unfortunately not true. As call #5 in the sequence diagram above indicates, calling unregister doesn’t guarantee that TempProcess is immediately unregistered, but means that TempProcess announced its desire to unregister. SensorHub is yet to process this request at its own convenience.

TempProcess had decided to uninitialize.

SensorHub had not yet processed the request telling it that TempProcess was terminating.

Why the discovery request was particularly problematic

#

In some systems, receiving an unexpected request isn’t necessarily a big deal, but here it was.

The discovery request was asking TempProcess for information that ultimately depended on the sensors it managed.

But the uninitialization sequence had already disconnected those sensors.

So we had another lifetime mismatch: The process could therefore still receive an RPC while the resources required to process that RPC no longer existed.

So, the lifetime of a process, the lifetime of its registration, and the lifetime of the resources behind that registration don’t have to end at the same time.

In asynchronous systems, assuming that they do can be dangerous.

What could be done about it?

#

There are several ways to deal with this behavior. They don’t all solve the problem at the same layer, and they don’t all provide the same guarantees.

1. Delay the uninitialization

#

The simplest approach is to give the discovery traffic some time to complete before disconnecting the sensors.

This can be a practical workaround when changing the protocol isn’t an option.

But it relies on a timing assumption, “Wait long enough and all outstanding requests should have completed.”

But how long is long enough?

That depends on scheduling, system load, IPC latency, queue congestion, and potentially other work happening in SensorHub.

A delay can therefore reduce the probability of the problem without fundamentally changing the ordering guarantees.

It is a timing workaround, not a synchronization mechanism.

2. Ignore discovery requests during uninitialization

#

A cleaner approach is to make TempProcess explicitly aware of its lifecycle state.

Instead of assuming that unregistration means no more messages can arrive, it can enter a terminating state and reject discovery requests once uninitialization has started.

sequenceDiagram
    participant H as SensorHub
    participant T as TempProcess
    participant S as Temperature Sensors
    autonumber

    T->>S: Initialize sensors
    activate S
    S-->>T: Sensors ready
    deactivate S
    T->>H: Register
    activate H
    H-->>T: Registration request queued
    deactivate H
    T->>H: Unregister
    activate H
    H-->>T: Unregistration request queued
    deactivate H
    note over H: SensorHub will execute Register first, then Unregister
    T->>S: uninitialize sensors
    activate S
    S-->>T: Sensors powered-off
    deactivate S
    T->>T: Release resources
    T->>T: uninitialize
    note over H: SensorHub picks up first item from queue: Processing register request
    H->>H: Add TempProcess to registry
    activate H

    H->>T: Discovery request
    note over T: I have already terminated my sensors, let's ignore the request, and not pass it to the sleeping sensors
    T-->>H: negative response
    

This approach doesn’t depend on a particular amount of time passing.

It acknowledges the reality of asynchronous communication: A request can still arrive after a component has started shutting down.

That makes the receiving side responsible for handling that situation safely.

3. Defer discovery in SensorHub

#

There is another way to approach the problem: change when SensorHub performs discovery.

The original implementation effectively discovers and registers within the same handler, making it busy.

Instead, SensorHub could register the agent and defer discovery by posting an event to its main event loop or super loop.

The sequence then becomes:

sequenceDiagram
    participant H as SensorHub
    participant T as TempProcess
    participant S as Temperature Sensors
    autonumber
    T->>S: Initialize sensors
    activate S
    S-->>T: Sensors ready
    deactivate S
    T->>H: Register
    activate H
    H-->>T: Registration request queued
    deactivate H
    T->>S: uninitialize sensors
    activate S
    S-->>T: Sensors powered-off
    deactivate S
    T->>T: Release resources
    T->>T: uninitialize
    T->>H: Unregister
    activate H
    H-->>T: Unregistration request queued
    deactivate H
    note over H: SensorHub picks up the request from queue
    H->>H: Add TempProcess to registry
    H->>H: Adds Discovery request to the queue
    note over H: SensorHub picks up the next request from queue: Unregister
    H->>H: Remove TempProcess from registry
    note over H: SensorHub picks up the next request from queue: Discover
    note over H: But since TempProcess already unregistered, this is irrelevant: Do nothing
   

This changes the scheduling point, where registration handler finishes before discovery actually happens, giving other queued work, including the unregister request, a chance to be processed.

In this case, registration revalidatation is required before executing discovery.

However, simply deferring discovery does not guarantee that the unregister request will always be processed first. The event loop could still execute the discovery event before the unregister request.

Without the state check, we have merely moved the timing window:

The broader lesson: queued work has a lifetime

#

The bug wasn’t really about sensors, and wasn’t even specifically about RPC.

It was about asynchronous work outliving the state that originally made that work valid.

There may already be work in flight.

The same pattern appears in many other systems:

• asynchronous RPC
• Unix IPC
• event loops
• worker queues
• deferred callbacks
• timers
• message brokers
• network protocols
• background jobs

In all of these systems, cancelling or invalidating something doesn’t necessarily make previously generated work disappear.

The rule worth remembering

#

When designing asynchronous components, don’t assume that changing the state guarantees that nobody can still act on the old state.

However, it is relevant to ask: What work has already been queued or generated, and what happens if it executes after the state changes ?

Without this question, you might see bugs that will be classified as sporadic bug that are hard to reproduce, because their occurance is dependent on random factors, like scheduling, workloads on each involved process’s queue, and other factors.

If you consider this question, you fix the issue before it happens.

In this case, the unregister request was not wrong, and the discovery request was not necessarily wrong either.

The problem was the gap between when a request was sent, when it was processed, and when its consequences took effect.