Saturday, July 14, 2012

Advanced Concurrent Managers

A concurrent manager will run requests based on both its assigned:

  • Work shift/s - when the manager can run and the number of manager processes the manager should run and
  • Specialization Rules - The programs the custom manager can run.

Note: In general, all requests run through the Standard manager unless you create a custom manager with a work shift and assign a program via a specialization rule and then exclude the program from the standard manager.


Custom Concurrent Managers
A word of warning: - Creating a custom manager is relatively easy to process, getting rid of one is not. In fact you are not meant to delete a custom manager, only disable them, so plan carefully before you go creating custom managers.

Work shifts
A work shift defines when one or more manager/s will run. Out of the box – all Concurrent managers are assigned the “standard” work shift. This work shift runs between 00:00 am and 23:59 seven (7) days a week.

Before you go creating additional work shifts you really want to be very sure of why you require a custom work shift, what will run in that work shift and what will happen to any program assigned to the manager / work shift when the work shift defines the manager as inactive.
One of the most complex custom work shifts is one to only run selected concurrent programs overnight. There are three (3) basic steps to achieving this setup:
  • In order for a manager to only run requests overnight (spans midnight) you must create two (2) work shifts and assign both those work shifts to the “Overnight” Manager:
                        Overnight_am to run between 18:00 and 23:59
                        Overnight_pm to run between 00:00 and 07:00

  • You then create a custom concurrent manager usually with a descriptive name such as “Overnight_only” and assign it both the overnight_am and overnight_pm work shifts.
  • The final step is to assign the programs you want to be run by the “overnight_only” manager using specialisation rules.

Note: When a request is submitted to run in the “Overnight_only” manager before or after the “Overnight_only” manager is running, the request will be marked as Pending Error, as there are no managers to run the request until the managers start.

Specialization rules
Specialization rules are used to assign specific concurrent manager programs to specific queues.
Specialization rules can be one of the more complex Oracle Applications features to set up and once set up, one of the more complex to find using the current E-Business Suite screens. This makes it hard for an administrator taking over a site to understand and troubleshoot. Remember, with specialization rules simplicity is the key.
To add to the complexity, there are a number of specialization rule combinations:
  • By Program
  • By Application user
  • By Oracle User
  • By Request Type
  • By Complex Rule, which is a combination of Program, application user and / or Oracle user

In the case of a program rule, it’s a simple process of “including” a program to run in the custom manager and “excluding” the program from the standard manager.

Specialization Rules to concurrent manager (Custom/Standard) is set via the applications  
Concurrent > Manager > Define

Request types
Request types are one of the most powerful features of concurrent processing; they allow you to assign new requests of a program from one concurrent manager to another without the need to bounce the concurrent managers.

Note: Once a program has been submitted the queue cannot be changed.

A request type is a logical folder if you will, which is assigned to a specific queue by specialization rules the same way as a program is assigned e.g. The request type “Slow” is “included” to the concurrent manager “Slow” queue and “excluded” from the standard manager.

The request type for a concurrent program is set via the applications Concurrent > Program > Define screen

Incompatibility Rules
Programs may also not run when you expect based on one or more incompatibility rules.
An incompatibility rule defines that a selected program cannot run whilst a program that has been defined as incompatible is running.

An example of this is the GL Posting program “GLPPOS”, this program has been defined as incompatible with itself, as such a second or subsequent posting program being submitted must wait until the initial posting program has completed.

Much like specialization rules, incompatibility rules are easy to create but not so easy to keep track of.

Incompatibility rules are managed vie the application Concurrent > Programs > Define Screen. Select the incompatibilities button.

Role of the Conflict Resolution Manager (FNDCRM)
The conflict resolution manager assesses each requested concurrent program against programs that are running, referencing the applications incompatibility rules.

When a program lists other programs as being incompatible with it, the Conflict Resolution Manager prevents the program from starting until any incompatible programs in the same domain have completed running.


Run Alone Programs
You can make a program incompatible with all other concurrent programs by defining the program to be run-alone via the run alone option in the concurrent program define screen. Care should be exercised here as once this program is running no other programs can run at the same time.

How many concurrent managers should I have?

It is a common misconception that the more managers you have the better throughput you will get. This situation is further compounded once the business find out you can add managers. They too generally believe there are spare resources you are not utilising, and therefore that you should add more managers.

When configuring managers it is best to start low, and work up.

A general rule of thumb is you should have no more managers than 2 (standard and custom) times the number of CPUs (optimal is 1.8 as this leaves head room for user connections).

Always resist the urge to add managers

·         Generally no more than 2 managers per CPU
·         Custom managers for FSGs ( Assume 1 FSG will consume one CPU )
·         Custom managers for selected slow jobs
·         Custom managers for selected fast jobs


Note: When you creating custom managers do not forget to reduce the number of standard managers.


Sleep and cache settings in Concurrent Managers:

When a queue wakes up (sleep time) it queries the FND_CONCURRENT_REQUESTS table for any pending requests. The value of the cache size will determine the number of rows that will be returned. If a request completes within the managers sleep time, additional requests held in cache can be run prior to the next manager wake up time.

Some Important Documents for Patching..

Following list contains some of the important documents that are related to  EBZs- patches
  • EBS Quick Reference For MFG Rollup Patches (Doc ID 726229.1)
  • Procurement Family Patch Release History (Doc ID 222339.1)
  • Recommended patches for iProcurement (Doc ID 1399131.1)
  • Recommended patches for iSupplier (Doc ID 1401834.1)
  • Recommended patches for Purchasing Approvals (Doc ID 1122052.1)
  • Recommended patches for Purchase Order and Requisition Processing (Doc ID 1358356.1)
  • Procurement Suite Patching Strategy For One-Off Requests and RUPs (Doc ID 602754.1)
  • EBS Quick Reference for MFG Rollup Patches (Doc ID 726229.1)
  • What Are The Punchout Patches After Applying Latest CU And Procurement RUP 11 Or RUP 12 Patches (Doc ID 1305076.1)
  • Oracle iProcurement Patches Guide - Master List (Doc ID 362023.1)
  • INV/WMS/RCV Family Patch Release History / Patchsets / RUPs (Doc ID 726226.1)
  • Critical E-Business Suite11i (11.5.10) Extended Support Information on Minimum Baseline Patch Requirements (Doc ID 1116887.1)
  • How To Determine the Product Information Management (PIM) Patchset and Rollup Patch Version (Doc ID 414470.1)
  • Oracle E-Business Suite Release 12 Release Update Pack (RUP) (Doc ID 423541.1)
  • Logistics Suite Patching Strategy For One-Off Requests and RUPs (Doc ID 605248.1)
  • EBS Quick Reference For MFG Rollup Patche (Doc ID 726229.1)
  • Oracle E-Business Suite Release 12 Release Update Pack (RUP) (Doc ID 423541.1)
  • Introduction to Oracle Recommended Patches (Doc ID 756388.1)
  • How to Find E-Business Suite Recommended Patches (Doc ID 1400757.1)
  • Patch Wizard Utility  (Doc ID 976188.1)
  • Patch Wizard FAQ (Doc ID 976688.1)
  • Patch Wizard Overview Videos (Doc ID 1210479.1)
  • Patch Wizard Training (Doc ID 1085668.1)
  • New Required Patches for Patch Wizard,...for Oracle E-Business Suite Releases 11i, 12.0, and 12.1 (Doc ID 1267768.1)
  • Oracle E-Business Suite Releases 11i and 12 Critical Patch Update Knowledge Document (April 2011)        [Document 1272097.1]   
  • Patch Wizard Utility        [Document 976188.1]   
  • How to Apply a Patch and How to Retrieve Patch List for OSB 11g Using the OPatch Utility        [Document 1266548.1]   
  • HOW TO INSTALL A PATCH ON CLUSTERWARE 11.2.0.1 INTO GRID INFRASTRUCTURE HOME        [Document 1054439.1]   
  • Oracle Applications Patching FAQ for Release 12   [Document 459156.1]
  • Release 11i Adpatch Basics  [Document 181665.1]
  • Oracle Applications Patching FAQ for Release 11i  [Document 457566.1]
  • How to Apply an 11i Patch When adpatch is Already Running  [Document 175485.1]
  • 11.0.x : Patch Installation Frequently Asked Questions   [Document 60766.1]
  • Patching Best Practices and Reducing Downtime��  [Document 225165.1]
  • What Log Files are Generated with Adpatch, and Where are They Located?   [Document 314540.1]
  • How To Apply a Patch in a Multi-Server Environment  [Document 136342.1]
  • HRMS PATCH INSTALL FAQ  [Document 374915.1]

Overview – Patching in E-Business Suite

Overview
adpatch is one of AD utilities which can be used for Applying patches to the Oracle Applications and Oracle database (also you can add new languages and products as a patch).
--AD Utilities (abbreviation from Applications DBA) are programs which perform a variety of tasks, including generating files, updating your system, merging and applying patches, installing off-cycle products.  These AD Utilities are executable files in $AD_TOP/bin

Applying a patch updates your existing system in various ways, from adding a new feature or product to improving system performance.
patches are applied for a number of reasons, including:
Fixing an existing issue Adding a new feature or functionality Updating to a higher maintenance level Applying the latest product enhancements Providing interoperability to new technology stacks Determining the source of an issue Applying online help
Depending on the type of patch, it may update the file system, or the database, or both.

Unified Driver File
The unified driver, named u.drv, contains the commands necessary to change files and database objects, and to generate new objects. It contains copy,database, and generate portions and performs the copy, database, and generate actions in the stated order. You typically run the unified driver on all APPL_TOPs. AutoPatch
runs only the actions that are required for the current APPL_TOP.
C,D and G Driver File
C – copy driver
D – database driver
G – forms and reports generation driver

Apps patches may take a name like p111111_R12_Unix.zip where p menas patch next few digit are patch number Unix is platform or OS on which this patch can be applied.
When you unzip a patch you will see some files types like:
  1. readme.txt - This file contain steps to spply patches, List of Prerequisite patch should be there on apps Instance (If not apply that patch )
  2. cXXXXXXX.drv c stand for copy driver file , this copies patch content to respective patch location , driver is like bus driver which provides instruction on work adpatch need to perform.
  3. dXXXXXXX.drv d stand for Database driver & contain content related to database like creating packages, tables, adding column….
  4. gXXXXXXX.drv This contain files related to forms , reports, graphis or messages
  5. uXXXXXXX.drv Sometime these three types of files are bundled together into single driver file called Unified driver file

Hence as a system admin or as a DBA you have to be aware of which type of driver file to apply and where ? (its common sense if this is d driver you just need to apply once on concurrent manager tier and c & g on all tiers as database is single & if you run it once its enough . If you applied it again by mistake its no harm & will not cause any issues .Similary c & g you need to apply from all tiers , because c will copy your content & g will generate message files).
If you are not sure where to apply what just apply it on all tiers & depending on your context file in APPL_TOP/admin/SID_hostname.xml it will apply where to install what by checking xml tag TIER_ADFORMS , TIER_ADWEB , TIER_ADADMIN open your xml file & see these parameters are there or not ( For xml file your system should be Autoconfig enabled ) all system after 11.5.8 by default are autoconfig enabled .


Common Patch Terminology:

Standalone/Oneoffpatches : This is used to define patch created to fix single/particular problem.
Mini Pack : This is group of oneoff patches for a particular product like INV, GL, AP and named like 11i.GL.E ( means this group of patches contain fix for 11i GL product (General Ledger till time E is released ) This is cummutative which means it will include 11i.GL.A, 11i.GL.B ….11iGL.D till 11i.GL.E earlier in 10.7 it used to called as patchset.
Family Pack : Group of mini packs in one family buldeled together is called as family pack. they are usually named as 11i_PF. Few example of falilies are SCM ( 11i.SCM_PF.G ), ATG ( 11i.ATG_PF.H ) _PF indicate Product Family Pack
Maintenance Pack : Group of family pack together is called as maintenance pack. So if you say your Verison is 11.5.10 then its maintenance pack 10 ( 3rd digit is maintenance pack )

Hence, few one off patch make mini pack , few mini pack related to same family bundeled together as family pack & all family pack fixed till that time are bundeled in Maintenance pack

NLS Patch
 When you have more than one language , like English & French or Arabic) then apart from normal patch you have to apply patch for specific language Installed in your system called as NLS patch


Benifits of Oracle Recommended patches:
Oracle Recommended Patches provide the following benefits:
  • They fix a set of critical issues commonly encountered in targeted environments and configurations.
  • They stabilize production environments because the patches address known critical issues.
  • They help save time and cost by eliminating rediscovery of known issues.
  • They are tested as a single combined unit, resulting in increased quality and eliminating the risk of combining patches that are only independently tested.
  • They make it easier to identify patches applicable for a targeted environment/configuration.

 

How to troubleshoot a Patch Failure - adpatch



To Note that Workers are controlled by adctrl. Based on the error in the log , fix the problem ; Use adctrl ; select option restart failed worked & then give worker number you want to restart to confirm if problem is fixed.


Please check the following file types / errors causing job failures :
a) odf files : (Object Definition Files) -odf files are there to create tables ,indexes and view etc.
b) pls scripts : -If the failed pls scripts are for creating stored procedure or packages
c) sql Scripts: -Failure on the SQL Script
d) WFLOAD and FNDLOAD failure- Look for the error message in the log file of the worker that fails.


In order to resolve the error, you may consider the following action plan

1) Try to execute the failing sql / pls manually and see the results
2) Check the Alert Log  and If any of the workers failed due to insufficient tablespace, then add the space , restart the failed worker.
3) Check the database session waits - There could be enquires.
Check the locks and locked objects to ensure that the session applying the patch is now waiting on a lock. If so need to diagnose, kill the session holding the lock.
(Scripts for database Diagnosis provoded in the database Scripts Section)
4) -For WFLOAD and FNDLOAD, you need to look at the log file generated by the above command. The worker’s logfile should point you to the location of the log file.
5) If adpatch fails to generate reports or forms. it is ok to skip these failure as you can always regenerate them after patching.
6) If it fails for a Pre-req patch and a lot of downtime has already lapsed :

If Adpatch Fails due to pre-requisite Patch or for some other reason and a lot of time has already been spent since the patch started running then , instead of applying this patch from begining you need to do the following steps in order to save the time you spend while applying the patch.
1. Using the adctrl utility, shutdown the workers.
a. adctrl
b. Select option 3 “Tell worker to shutdown/quit”

2. Backup the FND_INSTALL_PROCESSES table which is owned by the APPLSYS schema
a. sqlplusapplsys/<password>
b. create table fnd_Install_processes_back
as select * from fnd_Install_processes;
c. The 2 tables should have the same number of records.
select count(*) from fnd_Install_processes_back;
select count(*) from fnd_Install_processes;

3. Backup the AD_DEFERRED_JOBS table.
a. sqlplusapplsys/<password>
b. create table AD_DEFERRED_JOBS_back
as select * from AD_DEFERRED_JOBS;
c. The 2 tables should have the same number of records.
select count(*) from AD_DEFERRED_JOBS_back;
select count(*) from AD_DEFERRED_JOBS;

4. Backup the .rf9 files located in $APPL_TOP/admin/<SID>/restart directory.
At this point, the adpatch session should have ended and the cursor should
be back at the Unix prompt.
a. cd $APPL_TOP/admin/<SID>
b. mv restart restart_back
c. mkdir restart

5. Drop the FND_INSTALL_PROCESSES table and the AD_DEFERRED_JOBS table.
a. sqlplusapplsys/<password>
b. drop table FND_INSTALL_PROCESSES;
c. drop table AD_DEFERRED_JOBS;

6. Apply the new patch ( say pre-req patch).
Once this Pre-req Patch is applied successfully.
7. Restore the .rf9 files located in $APPL_TOP/admin/<SID>/restart_back directory.
a. cd $APPL_TOP/admin/<SID>
b. mv restart restart_<patchnumber>
c. mv restart_back restart

8. Restore the FND_INSTALL_PROCESSES table which is owned by the APPLSYS schema.
a. sqlplusapplsys/<password>
b. create table fnd_Install_processes
as select * from fnd_Install_processes_back;
c. The 2 tables should have the same number of records.
select count(*) from fnd_Install_processes;
select count(*) from fnd_Install_processes_back;

9. Restore the AD_DEFERRED_JOBS table.
a. sqlplusapplsys/<password>
b. create table AD_DEFERRED_JOBS
as select * from AD_DEFERRED_JOBS_back;
c. The 2 tables should have the same number of records.
select count(*) from AD_DEFERRED_JOBS_back;
select count(*) from AD_DEFERRED_JOBS;

10. Re-create synonyms
a. sqlplus apps/apps
b. create synonym AD_DEFERRED_JOBS for APPLSYS.AD_DEFERRED_JOBS;
c. create synonym FND_INSTALL_PROCESSES FOR APPLSYS.FND_INSTALL_PROCESSES;

11. Start adpatch, it will resume where it stopped previously.

If you want to skip a worker , this option is not visible & do it using adctrl option 8 . It is not a good idea to skip the failed worker without consulting from oracle.

adpatch Options


Type "adpatch help=y" on unix prompt to display all the options. You can execute adpatch by logging in as the applications OS user and sourcing the appropriate environment file.

#su – applmgr
$. ./APPSORA.env
$adpatch

By default adpatch does not take any clause, but there are some clauses that you could use with adpatch

Running a patch in test mode
You can use the apply clause with adpatch to specify weather to run the patch in TEST mode or not, when you run the patch in  test mode it does not do any changes but runs generates a log fiule with all the actions it would have performed.

$adpatch apply=n|y
The default is apply=y

Pre-install Mode
You can run a patch in pre install mode, this would be done normally during an upgrade or consolidated update. When a patch is applied in a preinstall mode the all the AD utilities are updated before the upgrade or update.

$adpatch preinstall=y
The default is preinstall=n

Other Options with adpatch
You can use the options clause to specify some of the other options available with adpatch.
Autoconfig
You can use the options=noautoconfig top specify autopatch that you do not wish to run autoconfig as a part of the patch  application. This can be useful when applying a large number of patches when they are not merged. By default autoconfig is run  as a part of adpatch.

$adpatch options=noautoconfigCheckfile
The chekfile option of adpatch tells adpathc to check for already executed exec, SQL, and exectiercommands.You can use  options=nocheckfile skips this check, however this can cause performance overheds so should be used only when specified.

$adpatch options=nocheckfile

Compile Database
By defaultyautopatch compiles the invalid objects after the patch application, in case you wish not to do so you can specify  options=nocompiledb along with autopatch.
$adpatch options=nocompiledb

Compile JSP
By defaultyautopatch compiles the java server pages (jsp) after the patch application, in case you wish not to do so you can specify options=nocompilejsp along with autopatch.
$adpatch options=nocompilejsp

Copy Portion
If you wish adpatch not to execute the commands present in the copy driver portion of the patch you can use the  options=nocopyportion.
$adpatch options=nocopyportion

Database Portion
If you wish adpatch not to execute the commands present in the database driver portion of the patch you can use the options=nodatabaseportion.
$adpatch options=nodatabaseportion

Generate Portion
If you wish adpatch not to execute the commands present in the generate driver portion of the patch you can use the options=nogenerateportion
$adpatch options=nogenerateportion

Maintenance Mode
If you wish to apply a patch regardless of the system being in maintenance mode you can use options=hotpatch.
$adpatch options=hotpatch

Integrity Check
If you wish to check the integrity of the patch you can use the options=integrity. Since metalink patches are pre checked for  their integrity it is generally not required to do an explicit check and the default value is nointegrity.
$adpatch options=integrity

Maintain MRC
You can use the maintainmrc option to specify weather you wish adpatch to execute the Maintain MRC schema as a part of the patch application or not. By default maintain MRC is done for standard patches and is disbaled for tarnslation and documentation patches.
$adpatch options=nomaintainmrc

Pre requisite Patch Check
If you wish adpatch not to check for pre requisite patches before application of the main patch you can use options=noprereq.By default pre requsite checking is enabled.
$adpatch options=noprereq

Validate Schemas
If you wish adpatch to explicitly validate all the registedschems by making a connection you can use options=validate. By  default this validation is not performed.
$adpatch options=validate

Java Classes
If you wish adpatch not to copy new java classes from the patch you can use options=nojcopy.By default java classes are copied.
$adpatch options=nojcopy

Force Copy
By default adpatch copies the files without check the version of the existing files already present on the system.If you do  not wish the newer version of the file to be replaced by the older version contained in the patch use options=noforcecopy.
$adpatch options=noforcecopy

Relinking
If you wish adpatch not do perform relinking you can use options=nolink.
$adpatch options=nolink

Generate Forms
If you wish adpatch not to generate the forms files you can specify options=nogenform.
$adpatch options=nogenform

Generate Reports
If you wish adpatch not to generate the report files you can specify options=nogenrep.
$adpatch options=nogenrep

You could specify multiple options at the command line using the , delimiter.

$adpatch options=hotpatch,nojcopy


  1. Check the patch log file : Log file for adpatch or apps patch will exist as specified while running adpatch The default location is $APPL_TOP/admin/SID/log with name adpatch.log.
  2. See on which worker the Patch has failed.
  3. Check the Worker log for the failed Worker at the same location ($APPL_TOP/admin/SID/log) by the name adwork<worker_number>.log.