Instance Archiving and Purging

TODO: This page still needs work

Instance Archiving and Purging

This section covers retention and purging of completed or terminated process instances and their associated log records.

During process instance execution, the Flows for APEX workflow engine tracks process state in database tables in the Flows for APEX workspace. If logging is enabled, the engine also generates instance, step, and process variable event log records which are kept in the log tables inside the workspace, dependent on the level of logging that is enabled.

After the process instance completes its execution ( the process state will be completed or terminated), the instance archiving automation will create an instance archive document, a JSON document containing information about the process and its execution. This instance archive document is then stored in OCI object storage or in a database table that you have created (outside of the Flows for APEX tables) for long term archiving, with retention period being based on your requirements.

For performance reasons, you should purge process instances and associated log records once they are no longer required in the run-time system for business use. The archive provides a mechanism for longer term retrieval and as an audit trail for any future forensic investigation. Obviously, different organizations will have very different requirements here – but the implemented mechanisms should accommodate most requirements.

Log Archiving

Flows for APEX provides archiving of process instance execution details via the collection and packaging of all run-time details and event logs for the process instance into a single JSON document after the instance has completed execution.

The JSON instance archive document contains:

• details of the models and versions run
• how initiated the instance and when
• priority and due dates
• completion date
• whether any manual interventions, suspension, rewind, resumption or errors occurred
• process id and name
• any called diagrams
• all instance events
• all step events logged
• all step completions and timings
• all process variable changes logged

Archiving needs to occur before both the process instance and the log records for the instance are deleted.

The instance archive document is usually created by an APEX automation job that runs daily, creating archives for all process instances that recently completed. The automation will create archives for all un-archived process instances.

Additionally, if instance archiving is enabled and an instance is deleted before it has been archived, an archive will be created as part of the delete process instance action. This is to prevent a user from performing a process and deleting all record of it before it has been archived.

The instance archive can be stored in an Oracle database table or in OCI Object Storage (preferred).

• Using OCI Object Storage. You can define automatic retention policies (for example, retain for 7 years and then auto delete ), and implement retrieval quality-of-service rules to save cost (e.g., keep archive in Standard or Infrequent Access Tier to allow instant retrieval for 18 months, then move to lower cost Archive Tier (4-hr retrieval) until it is auto-deleted after 7 years).
• Using a Database Table. If yo choose to store the archive documents in the database, you must create a table to store the JSON documents.

Instance Purging / Deletion

Instance purging removes all of the run-time process state from Flows for APEX.

In technical terms, this deletes the process instance from flow_processes, deletes any remaining subflows from flow_subflows, deletes any process variables, and other information about the process execution. Once removed, the process instance cannot be viewed by the Flow Viewer.

Any log records created during the process execution are not affected by the instance being purged.

If you need an audit trail detailing all of the activities undertaken to execute the process instance, you should create an instance archive for the process before the instance is purged. In fact, if the logging_archive_instance_summaries configuration parameter is set to true, (i.e., log archiving is enabled), an instance archive will be created shortly after the instance execution completes. Any attempt to delete the instance will cause an instance archive to be created if it does not already, as part of the delete operation.

Process Instances can be deleted:

• manually from the Flow Monitor.
• by the Flows for APEX - Purge Completed Instances automation. This deletes any completed or terminated process instances a specified number of days after the instance completed / was terminated. The retention period is defined in Configuration - Instance Archiving and Purging page of the Flows for APEX application.
• programmatically via the PL/SQL API package, flow_api_pkg.flow_delete

Log Purging

Log purging removes all of the logging records created for a process instance.

In technical terms, this removes all records about the process instance from:

• the instance events log table, flow_instance-event_log
• the step event log table, flow_step_event_log
• the process variable event log table, flow_variable_event_log

If you need an audit trail of process instance execution, you should create an instance archive for the process before deleting log records for the instance.

Process logs can be purged:

• by the Flows for APEX - Purge Instance Logs automation. This deletes log records for all process instances that completed or terminated before a given date (specified as a retention period in days). The retention period is applied to all log records in a process instance, based on the retention period. (So a long running process instance which started on 1 January 2024 and completed on 30 June 2024 and with the log retention period set to 365 days, all of the log records will be retained until 30 June 2025, at which time they will all be deleted).
• programmatically via the Flows for APEX PL/SQL API package flow_admin_api.purge_instance_logs.