Event Logging and Auditing 🆕

Introduction

Flows for APEX has an optional event logging and auditing capability that was introduced in v21.1 and enhanced in v23.1.

Events are logged for three main purposes:

• to provide an audit trail for security or forensic purposes
• to provide monitoring information on currently running flow instances
• to provide data about many flow instances that can be used for process redesign and enhancement, sometimes called Process Performance Management (PPM).

Like all event logging and audit trails, capturing and storing event trails uses processing power and fills storage resources – so you should ‘log only what you need to log’ and ‘retain only what you need to retain for as long as you need it’! But if you are a ‘secure’ site and need to keep a forensic audit trail of everything that happened for 7-10 years, Flows for APEX can meet that requirement.

Logging Model Events

Model / diagram Events can be logged, to ensure that all changes to a production diagram is logged and audited. Logging at Model level ensures that

• Changes to Model meta-information, such as version information and status, are recorded.
• A copy of the BPMN diagram is logged when the model is released to become the Production version, and any time that a production diagram is modified.

Logging Instance Events

Instance Events provide information on the major events occurring for the execution of an instance. These include:

• Instance Creation
• Instance Start
• Instance Completion
• Instance Reset
• Instance Termination
• Instance Errors & Restarts

In the event of an error occurring on a step which causes the instance to be put into error status, the Instance Event Log contains a column error_info which will contain as much of the error stack from Flows for APEX, APEX, and Oracle as we are able to collect. This is particularly useful for debugging errors on steps that are performed outside of the user’s current step, such as errors from timers, scriptTasks, or process variable expression execution.

Logging Step Events

The Step Event Log contains one record for each completed process step, and includes information on:

• the preceding task
• time at which the step became the current task
• time at which work started (splitting waiting time from processing time) if the flow_start_work call is used.
• time at which the step was completed
• process status on completion
• subflow and subProcess information.

Logging Process Variable Changes

The Process Variable Event Log records the new value every time a process variable is set (created or updated).

Configuring Event Logging

Event logging requires configuration, in order to:

• enable or disable logging
• specify the amount of logging to do
• include or exclude personally identifying information is included
• specify whether to create instance summary archives for each completed process instance, and where to store them
• specify where to store diagram archives (log of each change to the process diagram)
• specify how long to retain the raw log data

Details on how to configure logging, log archiveing, and retention is shown here.

Managing Your Event Logs

Like any audit trail and event log, the Flows for APEX event log can rapidly collect a large volume of data. Some users will need to retain this for many years to meet their site security policy, while others will want to delete this periodically to prevent the logs from becoming too large and from either slowing system performance or from filling their storage device!

Flows for APEX will operate without event logging enabled, and after any event logging data has been deleted – so nothing is required for execution of BPMN instances. However, the Flow Monitor is unable to show detailed history, including error messages for tasks performed outside the users current step, unless the event logging data is captured and is still stored in the database. So you can delete anything you want from the event log tables, and Flows for APEX will continue to work.

Archiving your Log Information

Flows for APEX provides the following capabilities for managing your logging information.

1. Creation and Archiving of an Instance Summary. So that you can analyse exactly what happened when a process instance was run, the archiving process creates a single JSON document containing all log events for one instance. The archive document is created by an archive process, which should be configured to run as an APEX Automation, typically on a daily (nightly) basis. The archive document is created once the process instance has completed, or reached terminated status. The archive summaries create one JSON document per process instance. The archived JSON file can be stored in either an Oracle table or in OCI Object Storage.

2. Statistics. Separate from archiving detailed information about individual process instances, a separate process run daily by an APEX Automation creates daily statistical summary information about completed process instances and process steps. These are compiled into daily, monthly, quarterly summary statistics containing:

   • frequency (how many times a process or a task was completed)
   • minimum, median, 90-percentile and maximum step waiting time
   • minimum, median, 90-percentile and maximum step processing time

   These statistics are used to populate the enhanced dashboard pages in the Flows for APEX console application. For more information, see documetation on Statistics.

3. Log Record Purging. Another APEX Automation task, log purging can be configured to delete after a period of time since the process instance was completed or terminated.
4. Statistic Record Purging. Statistics purging can also be configured to remove daily and period summary statistics after they reach a specified age.

All of the logging and archiving features are configued using the Configuration pages of the Flows for APEX application. See the Configuration section for information on how to configure these options.