Tuning Action Runner Dispatcher Pool Size¶

When the action runner executes actions it uses two internal green thread pools - one for regular actions and one for workflows. Both of those pools have a hard limit on the maximum size. By default, it is 40 green threads for workflows and 60 green threads for non-workflow actions.

This hard limit is there to prevent resource exhaustion and to allow for a fair distribution across multiple action runner processes.

In some situations, such as if you have a lot of action-chain workflows with a delay policy, this limit can cause a deadlock to occur. (NB this deadlock can be reset by restarting action runner process which has dead locked).

To prevent such issues from occurring and to allow for a good server and action runner process resource utilization, you should adjust those pool sizes based on your configuration and workload requirements.

These settings are very hardware- and workload-specific (i.e. how many action runner processes are used, on how many servers, the type of actions being executed - long running vs short running, CPU intensive vs non-CPU intensive, etc.).

To effectively tune those limits you should monitor the following parameters:

Sizes of the RabbitMQ queues relating to the action executions
Action runner process resource usage (CPU and memory)

Once you have established baselines and have a good idea of your resource utilization and workloads, you can increase resource utilization and prevent deadlocks by tuning pool sizes and/or increasing/decreasing the number of action runner processes.

You can tune pool sizes by changing these settings in st2.conf:

[actionrunner]
# Internal pool size for dispatcher used by workflow actions.
workflows_pool_size = 40

# Internal pool size for dispatcher used by regular actions.
actions_pool_size = 60