Process queue
Introduction to Process queue
The Thinkwise Process queue is available for download from the Thinkstore. It offers an asynchronous queue that controls the execution of processes, so you no longer need to use database triggers. The architecture allows for seamless handling of subflows. This means you can invoke processes that integrate web connectors, application connectors, and other advanced functionalities. All of this is possible without affecting the integrity or performance of the database.
The Process queue is especially useful for asynchronous background processes, such as:
- Generating (management) data.
- Sending data to external systems.
- Processing uploaded files.
Architecture
The Process queue contains three core tabels in its model:
sub_flow_queue- acts as the primary table, where each record is managed on a first-in, first-out basis.sub_flow_queue_execution- records the execution details of each queue item, providing a clear log of all sub-flow actions performed.sub_flow_queue_setting- configures essential rules for each process queue-enabled sub-flow. Currently, this table mainly determines the maximum number of concurrent executions allowed per sub-flow, ensuring optimized and controlled processing throughout the system.
By default, a sub-flow can not be executed by the Process queue.
To enable a system flow for the Process queue, add the tag process_queue_enabled to the sub-flow in the menu Processes > Subflows > tab Process flow tags
You must run the Creation process to use the subflow in the Process queue.
After doing so, the subflow becomes visible in the sub_flow_queue_setting table.
Process queue tables
sub_flow_queue
| Column | Datatype | Description |
|---|---|---|
| sub_flow_queue_id | bigint identity, PK | Primary key of the queue item. |
| sub_flow_queue_status | tinyint | 1 – new 2 – processing 3 – processed 90 – error |
| scheduled_execution_date_time | datetime2 | When this is filled in, the queue item is only executed after this datetime. |
| start_date_time | datetime2 | Start date and time of the queue item. |
| end_date_time | datetime2 | End date and time of the queue item. |
| sub_flow_id | varchar(100) | The sub_flow_id used in the Software Factory for the sub flow to execute. |
| sub_flow_parameters | nvarchar(max) | Parameters to pass to the sub flow, which can be used in its actions. |
| sub_flow_status_code | int | Output status code of the executed subflow. |
sub_flow_queue_execution
| Column | Datatype | Description |
|---|---|---|
| sub_flow_queue_id | bigint | Corresponding sub_flow_queue_id (PK). |
| sub_flow_queue_execution_id | bigint | Identity, PK. |
| process_action_type | varchar(256) | Type of the process action (for example, decision, generate_report). |
| process_action_name | varchar(256) | Name of the action as defined in the Software Factory. |
| start_date_time | datetime2 | Start date and time of the process action. |
| end_date_time | datetime2 | End date and time of the process action. |
sub_flow_setting
| Column | Datatype | Description |
|---|---|---|
| sub_flow_id | varchar(100) | Sub flow ID from the Software Factory. |
| max_concurrent_queue_execution | Int | Mandatory. Default value is 1. |
Process flow
In this solution, we use a scheduled process flow to execute subflows in the queue.
This process flow consists of two decisions and a single process action: Execute subflow.
For each subflow that has process queue enabled in the Software Factory, a separate process action is added to the process flow.
For example, in the image below, the subflow is named test_sub_flow, resulting in the process action being named Execute subflow test_sub_flow.
The decision Read the sub_flow_queue table selects the following item from the sub_flow_queue table to execute.
The selection depends on the maximum concurrent executions and the scheduled start date and time.
The order is also determined by the sub_flow_queue_id.
Lastly, the decision Update the sub_flow_queue table updates the status of the sub_flow_queue item.
Process flow for executing subflows in the process queue
Retention
A scheduled system flow will run at 1:00 AM UTC to clean up the history of the process queue.
By default, items older than 90 days will be deleted.
You can change the default in the system flow sub_flow_queue_retention by changing the process flow parameter retention_in_days.