Powerful i5/OS cleanup mechanisms allow application deadlock (cancel_handler and C++ automatic destructors)

i5/OS provides a set of powerful cleanup mechanisms. In i5/OS, an application has the ability to register a cancel handler. Your application can enable a cancel handler by using the #pragma cancel_handler preprocessor statement if it is written in C or C++ or by using the CEERTX() API.

A cancel handler is similar to a Pthread cancellation cleanup handler. However, a cancel handler runs whenever the stack frame or function for which it was registered ends in any way other than a normal return. Pthread cancellation cleanup handlers run only when the thread is terminated with pthread_exit() or pthread_cancel() or when the thread returns from the threads start routine.

The cancel handler is guaranteed to run for all conditions that cause the stack frame to end (other than return), such as thread termination, job termination, calls to exit(), abort(), exceptions that percolate up the stack, and cancel stack frames. Similarly, C++ destructors for automatic C++ objects are guaranteed to run when the stack frame (function) or scope in which it was registered ends.

These mechanisms ensure that your application can always clean up its resources. With the added power of these mechanisms, an application can easily cause a deadlock.

The following is an example of such a problem:

An application has a function foo() that registers a cancel handler called cleanup(). The function foo() is called by multiple threads in the application. The application is ended abnormally with a call to abort() or by system operator intervention (with the ENDJOB *IMMED CL command). When this job is ended, every thread is immediately terminated. When the system terminates a thread by terminating each call stack entry in the thread, it eventually reaches the function foo() in that thread. When function foo() is reached, the system recognizes that it must not remove that function from the call stack without running the function cleanup(), and so the system runs cleanup(). Because your application is multithreaded, all of the job ending and cleanup processing proceeds in parallel in each thread. Also, because abort() or ENDJOB *IMMED was used, the current state and location of each thread in your application is cannot be determined. When the cleanup() function runs, it is very difficult for the application to correctly assume that any specific cleanup can be done. Any resources that the cleanup() function attempts to acquire may be held by other threads in the process, other jobs in the system, or possibly by the same thread running the cleanup() function. The state of application variables or resources that your application manipulates may be in an inconsistent state because the call to abort() or ENDJOB *IMMED asynchronously interrupted every thread in the process at the same time. The application can easily reach a deadlock when running the cancel handlers or C++ destructors.

Do not attempt to acquire locks or resources in cancel handlers or C++ automatic object destructors without preparing for the possibility that the resources cannot be acquired.

Important

Neither a cancel handler nor a destructor for a C++ object can prevent the call stack entry from being terminated, but the termination of the call stack entry (and therefore the job or thread) is delayed until the cancel handler or destructor completes.

If the cancel handler or destructor does not complete, the system does not continue terminating the call stack entry (and possibly the job or thread). The only alternative at this point is to use the WRKJOB CL command (option 20) to end the thread, or the ENDJOB *IMMED CL command. If the ENDJOB *IMMED command causes a cancel handler to run in the first place, the only option left is the ENDJOBABN CL command because any remaining cancel handlers are still guaranteed to run.

The ENDJOBABN CL command is not recommended. The ENDJOBABN command causes the job to be terminated with no further cleanup allowed (application or operating system). If the application is suspended while trying to access certain operating system resources, those resources may be damaged. If operating system resources are damaged, you may need to take various reclaim, deletion, or recovery steps and, in extreme conditions, restart the system.

Recommendations

If you want to cleanup your job or application, you can use one of the following mechanisms:

• If you want to do process level or activation group cleanup for normal termination, use the C atexit() function to register your cleanup function. The atexit() function provides a mechanism to run cleanup after the activation group and possibly the threads, are terminated. This action significantly reduces the complexity.
  
  
• If you always want a chance to do process level or activation group cleanup in all cases (normal and abnormal), you could use the Register Activation Group Exit (CEE4RAGE()) system API. The CEE4RAGE() function provides a mechanism to run cleanup after the activation group (and possibly the threads) are terminated. This action significantly reduces the complexity.
  
  
• You can safely use cancel handlers. Simplify your cancel handlers so that they only unlock or release resources and do not attempt to acquire any new resources or locks.
  
  
• You can remove your cancel handlers and create a CL command, program, or tool that terminates your application in a more controlled fashion:
  
  
  • One possibility is a tool that uses a signal to terminate the application. When the signal comes in, your application can get control in a single location (preferably by using the sigwait() API to safely and synchronously get the signal), and then perform some level of cleanup. Then it can use exit() or abort() to end the application from within. Often this action is sufficient to remove the complexity.
    
    
  • A second possibility is to use the ENDJOB *CNTRLD CL command and have your application dedicate a thread to watching for the controlled end condition. The application thread can use the QUSRJOBI (Get Job Information) or the QWCRTVCA (Retrieve Current Attributes) APIs to look at the End Status information associated with your job. The End Status indicates that the job is ending in a controlled fashion, and your application can take safe and synchronous steps to clean up and exit.
    
    
  • A third possibility is to use the asynchronous signals support and set up a handler for the SIGTERM asynchronous signal. Support has been added to the system so that, if an ENDJOB *CNTRLD is done and the target job has a handler registered for the SIGTERM signal, that signal gets delivered to the target job. You should dedicate a thread for handling signals by using the sigwait() API in the dedicated thread. When the signal handling thread detects a SIGTERM signal using the sigwait() API, it can safely clean up and terminate the application. The system support for the delivery of the SIGTERM signal when ENDJOB *CNTRLD is issued was added in the base OS/400 in Version 4 Release 3 Modification 0 and is also available in Version 4 Release 2 Modification 0 through program temporary fixes (PTFs) 5769SS1-SF47161 and 5769SS1-SF47175. For more information about ending your job, see the Work Management topic.
    
    
  • A fourth possibility includes other interprocess communications (IPC) mechanisms that can also be used to indicate that your application should terminate in a safe and controlled fashion.
    
    
• If you want to do thread-level cleanup, use the pthread APIs, such as pthread_cleanup_push(), pthread_cleanup_pop(), and pthread_key_create() to create cancellation cleanup functions that run when the thread terminates under normal conditions. Often your cleanup functions do not need to run when the job ends. The most common use for these functions is to free heap storage or unlock resources. Unlocking resources is safe in a cancel handler, and you do not need to use free() on heap storage when the entire job is ending anyway.