Resilient Design for Dynamic Job Generation: Creating Self-Healing, Reliable Workflow Structures

Enhance stability and ensure continuity for your dynamic masking workflows with a fault tolerant design that keeps operations reliable and secure.

The Challenge

Dynamic job generation workflows, often used for data masking and similar processes, give you high flexibility and automation by creating and executing job lists based on current database contents. Yet, even with automation, unexpected failures can occur. It is critical that when a job fails, the failure is detected immediately and subsequent jobs do not run incorrectly. For example, a downstream job should never execute if its upstream task has failed. By enforcing this level of control, you protect the integrity of your workflows and simplify operations management.

Solution

Create two jobs:
  • Job A: Executes the dynamically generated job list
  • Job B: Verifies the execution status of the jobs
Job B is automatically triggered using Job Stream, and incorporates Job Step Post Condition settings to enable automated retry and validation.

Reference Workflow Overview

Create the following two jobs:
  • 001_MASK_JOBLIST: Generates the list of jobs to be executed and triggers their execution
  • 002_CHECK_JOB: Monitors job statuses and completes only when all triggered jobs have successfully executed
Use Job Stream to link 001_MASK_JOBLIST to automatically trigger 002_CHECK_JOB.

Reference Steps

  • Create a new Trinity Job named “001_MASK_JOBLIST”, and add a new Trinity Job Step under it.
    • For the Step Type, select DM Step, and name the step “STEP01”.
  • Edit the DM Step with the following configuration:
    • Add a Reader to list the Jobs to be executed in sequence.
    • Add a Transformer to configure the Transformer rule, using the concat function to build the command.
    • Add a Writer to output a .sh file.
DM Step
Reader
Transformer 內容
Writer
Contents of the .sh file generated by the Writer
  • Create a new Trinity Job and name it 001_MASK_JOBLIST. Under this job, add a new Trinity Job Step. For the Step type, select External Command Step, and name the Step STEP02.
  • Edit the External Command Step and input the command to execute the .sh file generated by STEP01.
External Command Step
External Command Step Output
Shown: Executed jobs from the Job List displayed in the Task Console.
  • Create a new Trinity Job named ‘002_CHECK_JOB’, and check ‘Use Step Postcondition’ on the General page.
  • Under this Trinity Job, add a new Trinity Job Step. For the Step type, select DM Step.
  • Edit the DM Step, and set its content to SQL Executor.
SQL Executor
SQL Executor(continued from previous image)
SQL Executor(continued from previous image)
  • On the Postcondition page of the DM Step , set the execution rules.
    • If the QUITCODE value is 0, the execution of this STEP can be ended, and the STEP status is marked as Success.
    • If the QUITCODE value is 1, the execution of this STEP can be ended, and the STEP status is marked as Error.
    • If the value is neither of the above, wait for 10 minutes and then re-execute the content of this STEP (recheck the Job status).
Postcondition Page
Summary:
When you run the Job 001_MASK_JOBLIST, the required job list is automatically inventoried and triggered for execution. Using the Job Stream mechanism, the Job 002_CHECK_JOB then runs automatically to verify the execution status of each triggered job.

This process ensures that once the masking operations are dynamically generated for the current run, every operation proceeds in the correct sequence according to business rules. Each stage of the masking workflow is fully completed without omissions, protecting workflow integrity and simplifying operations management.

Scroll Top