(SOAR) Automated response commands run again every 30 seconds forever: executions never leave PENDING #2942

Acknowledgements

• I have searched (https://github.com/utmstack/UTMStack/issues) for past instances of this issue
• I have verified that my UTMStack version is up-to-date

Describe the bug

When an incident response automation (SOAR) rule matches an alert, its command runs on the agent, but the execution record stays in PENDING. Every 30 seconds the scheduler picks up all PENDING executions of active rules and sends the same command again. The command keeps running on the agent indefinitely, as long as the rule is active. The UI shows the execution as pending the whole time.

For real response actions (block an IP, kill a process, disable a user, isolate a host) this means the action is repeated without limit.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

Each matching alert runs the rule's command once. The execution is marked EXECUTED, with its result, or FAILED with a cause, and is never sent again.

Current Behavior

Each matching alert runs the rule's command once. The execution is marked EXECUTED, with its result, or FAILED with a cause, and is never sent again.

Current Behavior

Test rule QA-SOAR shred: condition name IS "System Linux: File Deletion via Shred", platform linux. Its command appends one line to a file on the agent:

echo "SOAR-QA name=$(name) ds=$(dataSource) ..." >> /tmp/utm-soar-qa.log

One alert was generated at 22:12. A single execution record was created, and it is still PENDING:

 id |               alert_id               |       execution_date       | execution_status
  3 | 28c8236b-977e-438a-b1de-6f5c450d2190 | 2026-10-19 22:12:44.801523 | PENDING

On the agent, the command ran again and again. About 90 seconds after the alert the file had 3 lines; about an hour later it had 136, and it was still growing at 2 lines per minute. All the lines are for that same single alert:

$ wc -l /tmp/utm-soar-qa.log
136 /tmp/utm-soar-qa.log

The agent-manager command history (agentmanager.agent_commands, origin_type = INCIDENT_RESPONSE_AUTOMATION, status 3 = Executed) shows the real scale:

 rule (origin_id) | agent_id | status | total  | first                  | last
 37               | 10       | 3      | 39485  | 2026-10-06 16:02:28    | 2026-10-13 12:44:50
 5001             | 12       | 3      |   141  | 2026-10-19 22:13:01    | 2026-10-19 23:23:01
 37               | 10       | 4      |     4  | 2026-10-13 06:20:17    | 2026-10-13 12:45:20
 37               | 10       | 2      |     1  | 2026-10-11 09:21:35    | 2026-10-11 09:21:35

Interval between consecutive commands of rule 5001 in the agent-manager. Every one is executed (3), exactly every 30 s, which matches executeRuleCommands() @Scheduled(fixedDelay = 30s):

          created_at           |    interval     | command_status
 2026-10-19 22:13:01.335574+00 |                 | 3
 2026-10-19 22:13:31.339994+00 | 00:00:30.00442  | 3
 2026-10-19 22:14:01.344497+00 | 00:00:30.004503 | 3
 2026-10-19 22:14:31.348336+00 | 00:00:30.003839 | 3
 2026-10-19 22:15:01.352286+00 | 00:00:30.00395  | 3
 ... (141 in total until the rule was deactivated at ~23:23)

• Rule 37 (brute-force response, Windows) executed netsh advfirewall firewall add rule name="Block-Brute-Force-127.0.0.1" ... remoteip="127.0.0.1" 39,485 times over a week, about 4 per minute (2 executions, each re-sent every 30 s). It only stopped when the agent went offline. The backend then exhausted the offline retries and shows those executions as FAILED / AGENT_OFFLINE, which hides the fact that they ran tens of thousands of times. The target host most likely has tens of thousands of duplicate firewall rules.
• Rule 5001 (the test rule above) ran 141 times in 70 minutes and stopped when the rule was deactivated.

Reproduction Steps

1. Install an agent and make sure it is online.
2. Create a SOAR rule matching an alert that agent produces, with a harmless command such as echo test >> /tmp/soar-test.log. Activate it.
3. Trigger one matching alert.
4. Watch /tmp/soar-test.log on the agent: a new line every ~30 seconds. In the UI, the rule execution stays PENDING.
5. Deactivate the rule: the repetitions stop.

Possible Solution

The execution is marked EXECUTED only in the gRPC response observer's onCompleted(), and that callback is never invoked:

• IncidentResponseCommandService.sendCommand() opens a bidirectional stream (processCommand), sends the command, and never half-closes the stream. requestObserver.onCompleted() is commented out.
• The agent-manager (agent/agent_imp.go, ProcessCommand) runs the command, sends the result back with stream.Send(result), and loops back to stream.Recv(), waiting for more commands. The server only returns, with an error status, when the client closes the stream.
• So the backend receives the result in onNext() but never gets onCompleted(), and onError() is empty. The record stays PENDING.
• UtmAlertResponseRuleService.executeRuleCommands() (@Scheduled(fixedDelay = 30s)) re-sends every PENDING execution of an active rule.

Suggested fix, any of:

1. Mark the execution as dispatched before sending. Set the status to a non-PENDING value, for example EXECUTING, before calling sendCommand(), so the scheduler never picks it up twice. Update to EXECUTED or FAILED from the observer.
2. Complete on the first result. In onNext(), store the result and mark the execution EXECUTED, since the server sends exactly one CommandResult per command. Handle onError() by marking FAILED with the cause instead of ignoring it.
3. Agent-manager: return after sending the single result, so the stream completes and onCompleted() fires on the client.

Also, onError() should not be empty: an error should end the execution as FAILED, never leave it PENDING.

Additional Information/Context

No response

UTMStack Version

v11.3.0-community (also present since v11.2.6)

Operating System and version

Ubuntu 24.04.4 LTS (server); Linux agent on Ubuntu

Hypervisor and Version | Server Vendor and Model

Browser and version

Chrome 154.0.8037.98 (Official Build) (arm64), macOS