Move the watchdog code to launch manager

[cameron-craig-etas]

What

Scope

Introduce watchdog initialization, servicing, and deinitialization into ProcessGroupManager::initialize(), run(), and deinitialize(). Only unconditional servicing (serviceWatchdog()) is in scope. Conditions for firing the watchdog (fireWatchdogReaction()) are out of scope and will be addressed separately.

Background

The watchdog is currently owned and operated by PhmDaemon (score/launch_manager/src/daemon/src/alive_monitor/details/daemon/PhmDaemon.hpp). The lifecycle is:

1. PhmDaemon::init(): loads config via MachineConfigFactory, calls watchdog->init(cycleTimeNs, configFactory) then watchdog->enable()

2. PhmDaemon::startCyclicExec(): calls watchdog->serviceWatchdog() each cycle (or fireWatchdogReaction() on critical failure)

3. On termination signal: calls watchdog->disable()

Since health_monitor and launch_manager now run in the same process, watchdog ownership should move to ProcessGroupManager. The watchdog will be initialized and enabled in ProcessGroupManager::initialize(), serviced every iteration of ProcessGroupManager::run(), and disabled in ProcessGroupManager::deinitialize().

Future watchdog trigger conditions (out of scope)

For context, the following conditions will eventually trigger fireWatchdogReaction() instead of serviceWatchdog():

• recovery_client_->hasOverflow() — recovery request queue overflow
• queuePosixProcess() failure — process state notification buffer full
• Other unrecoverable LaunchManager internal errors

Ownership and instantiation

• WatchdogImpl is instantiated in score/launch_manager/src/daemon/src/main.cpp (as it is today).
• ProcessGroupManager receives a std::unique_ptr<IWatchdogIf> via its constructor.
• The watchdog is only used from the main thread. No thread safety concerns.

Watchdog configuration

The watchdog requires two pieces of configuration for init():

• cycleTimeInNs (int64_t) — the expected servicing cycle time in nanoseconds
• An IDeviceConfigFactory — provides the list of watchdog device configurations (device path, timeout range, etc.)

A new configuration will be introduced within ProcessGroupManager that contains the relevant watchdog configuration data.

The semaphore wait is reduced from 100ms to 50ms to ensure the total cycle time (wait + processing) stays within the minimum supported watchdog timeout (100ms on QNX, 1000ms on Linux).

It needs to be clarified in the Score working group what minimal timeout should be supported in the future.

Acceptance Criteria (DoD)

• Watchdog moved out of PHM code and introduced as a new component in the core launch_manager
• Watchdog namespaces updated to remove the legacy "saf" namespace
• Add a createWatchdog() factory
• Give ProcessGroupManager watchdog ownership
• Add unit tests for watchdog

How

Implementation steps

1. Move watchdog source files from health_monitor_lib/src/score/lcm/saf/watchdog/ to src/launch_manager_daemon/src/watchdog/, separating public headers from private implementation:

   Public headers (src/launch_manager_daemon/src/watchdog/include/):

• IWatchdogIf.hpp (interface)

• IDeviceConfigFactory.hpp (configuration interface + DeviceConfig struct)

  Private implementation (src/launch_manager_daemon/src/watchdog/src/):
     - WatchdogImpl.hpp / WatchdogImpl.cpp
     - DeviceIf.hpp / DeviceIf.cpp
     - Watchdog.hpp (platform constants)

2. Create new BUILD target cc_library for watchdog in src/launch_manager_daemon/BUILD (or src/launch_manager_daemon/src/watchdog/BUILD).

3. Update ProcessGroupManager:
      - Add std::unique_ptr<IWatchdogIf> watchdog_ member.
      - Extend constructor to accept std::unique_ptr<IWatchdogIf>.
      - In initialize(): call watchdog_->init(cycleTimeInNs, configFactory) and watchdog_->enable(). Use new config if available, otherwise fall back to MachineConfigFactory.
      - In run(): call watchdog_->serviceWatchdog() at the end of each loop iteration.
      - In deinitialize(): call watchdog_->disable() before existing cleanup.

4. Update main.cpp:

   • Create watchdog via createWatchdog() and pass to ProcessGroupManager constructor.
   • Remove watchdog init/enable/disable from PhmDaemon path (watchdog is no longer passed to HealthMonitorImpl).

5. Update BUILD dependencies:

   • ProcessGroupManager BUILD target depends on the new watchdog cc_library.
   • If using fallback config: ProcessGroupManager BUILD target depends on health_monitor_lib:factory (temporary).

6. Add unit tests:

   • Add unit test for ProcessGroupManager verifying:
     • watchdog_->init() and watchdog_->enable() are called during initialize().
     • watchdog_->serviceWatchdog() is called each iteration of run().
     • watchdog_->disable() is called during deinitialize().
   • Add unit tests for watchdog

Pseudocode

Updated constructor signature:

ProcessGroupManager(std::unique_ptr<IAliveMonitorThread> alive_monitor_thread,

                    std::shared_ptr<IRecoveryClient> recovery_client,

                    std::unique_ptr<score::lcm::IProcessStateNotifier> process_state_notifier,

                    std::unique_ptr<IWatchdogIf> watchdog);

A free factory function is declared in the public header to create instances, keeping WatchdogImpl as a private implementation detail. This function does not exist yet and needs to be added:

// In IWatchdogIf.hpp (public header)

std::unique_ptr<IWatchdogIf> createWatchdog();

Updated instantiation in main.cpp:

auto watchdog = createWatchdog();

auto process_group_manager = std::make_unique<ProcessGroupManager>(
    std::move(aliveMonitorThread),
    recoveryClient,
    std::move(process_state_notifier),
    std::move(watchdog)
);

Updated initialize()

bool ProcessGroupManager::initialize()
{
    // ... existing initialization code ...
    // Watchdog initialization
    // Primary: use new ProcessGroupManager config for cycle time and device configs
    // Fallback: use MachineConfigFactory to load legacy hmcore.bin
    if (!watchdog_->init(cycleTimeInNs, deviceConfigFactory))
    {
        return false;
    }

    if (!watchdog_->enable())
    {
        return false;
    }
  
    return true;
}

Updated deinitialize()

void ProcessGroupManager::deinitialize()
{
    // ... existing deinitialization code ...
    watchdog_->disable();
}

Updated run()

bool ProcessGroupManager::run()
{
    OsHandler os_handler(*process_map_, process_interface_);
    bool result = startInitialTransition();

    if (result)
        while (!em_cancelled.load())
        {
            ControlClientChannel::nudgeControlClientHandler_->timedWait(std::chrono::milliseconds(50));
            for (auto pg : process_groups_)
            {
                controlClientHandler(*pg);
                processGroupHandler(*pg);
            }
            recoveryActionHandler();

            watchdog_->serviceWatchdog();
        }

    return result;
}