Unlocking oci.dll when Upgrading or Patching Oracle

       

   If the patch fails with a message as

$ORACLE_HOME/bin/oci.dll
Recommended actions: OPatch needs to modify files which are being used by some processes.


Please follow the following procedure,


1) Try to check the OPatch version using the following command and upgrade the optach with the latest version using the patch

   opatch version


2) If still the problem persists, then try to use the following procedure:


 a) Run the following tasklist command to list the processes that are using the file mentioned in the error message. In our example, the file is: $ORACLE_HOME\bin\oci.dll. Use the file name that is referenced in your error message.

Example Command:

tasklist /m oci.dll

Note that using the full path (for example, $ORACLE_HOME\bin\oci.dll) in the tasklist command will not work.



Command Output:
Image Name PID Modules
========================= ======== ========
dllhost.exe 3544 oci.dll

Use the following command to identify the process associated with the Image Names from the tasklist command output.

tasklist /svc
 
Command Output:
Image Name PID Services
========================= ======== =======================
dllhost.exe 3544 ComSysApp


Check if the services related to the Image Names can be stopped until the patch is applied.

recheck if still the oci.dll is used by the process and if not continue the patching.

ADOP New Tool in R12.2

Online patching is the most important new feature in Oracle EBS 12.2. It is the ability to patch a running system without having to take the system down for a significant period of time while the patches are applied. Even you can apply Maintenance Pack 12.2.2 or 12.2.3 or 12.2.4 once you install or upgrade to R12.2.0 base.

•     It will enable R12.2 PATCH Environment.

•     Dependency on opatches. All Opatches must be applied before enabling ADOP.

•     Missing Opatches will be reported by EBS DB CodeLevel Checker script.

•     All AD_ZD packages should be valid before enabling them.

•     To implement Online Patching Run reports/scripts mentioned in Using the Online Patching Readiness Report in Oracle E-Business Suite Release 12.2 [1531121.1]

•     Once you enable ADOP i.e. Online Patching you cannot apply any patches through ADPATCH.

•     If you still want to apply any patch via adpatch, source ENABLE_ADPATCH=yes. (if oracle recommends)

•     All patches applied using adop utility will be applied to PATCH environment i.e. fs2.

•     When you want to make the patches live you can execute adop using cutover phase. It will switch filesystems.

•     To synchronize both filesystems i.e. fs1 and fs2 you can execute adop using fs_clone phase.

1. In Oracle Apps R 12.2 onwards all patching operations are online ( EBS system remains available to users during patching) unlike previous versions where system is unavailable during patching (Maintenance Mode)

2. Online Patching uses Edition Based Redefinition (EBR) at Database Layer and Multiple File System at Application Tier level

3. Edition Based Re-Definition provides two copies of Database object (schema) pre and post patch .

a) Run Edition of database objects : used by online users and is not changed by patching

b) Patch Edition of database objects : used by patching tool and do not affect the running application

Note: There is third edition (zero or more editions) called Old Edition – Patch Edition when prompted to Production i.e. new Run Edition, previous Run Edition is marked as Old Edition. Full Cleanup Operation (part of ADOP) removes Old Edition (More on Cleanup Operation later)
 
 3. In Oracle Applications 12.2 there are two file system

a) FS1 : Production File System used by users

b) FS2 : Copy of production File System used by patching tools (adop : More on adop later)

Note: FS1 and FS2 can switch role i.e. FS2 will become production File system and FS1 will become File System for Patching at Cutover Phase of online patching (ADOP). More on various phases in AD Online Patching (adop) later.

Note: Third files system fs_ne stores data on filesystem like log, out, data import/export files

4. Patches are applied to copy of production environment (copy of code and NOT data) where as users access the original production environment (Patching applied is to file system and Patch Ediiton of database objects and not applicable to transactions tables/data in database)

5. Actual downtime during patching (ADOP) is to switch users from pre-patched copy to patched copy.

6. Online Patching Tool ADOP (AD Online Patching) now replaces adpatch in 12.2

7. Phases on ADOP (replacement of ADPATCH) are Prepare -> Apply -> Finalize -> Cutover -> Cleanup (more on these phases later)

8. Patching can be aborted any time before Cutover Phase


PATCH TOP DIRECTORY IN R12.2:



In R12.2 there is a new directory location environment variable called $PATCH_TOP which points to $NE_BASE/EBSapps/patch
$NE_BASE points to <Non-Editioned-filesystem-directory>
Download the patch into the patch top directory and unzip it. This is the default location where the adop will look for patch files.
If you are planning to put patches in non-defualt location then you need to use adop parameter ‘patchtop=<patch_path>’ to explicitly define this location.



PATCH MERGING IN ADOP
In earlier EBS releases, the AD Merge Patch tool was used to merge multiple patches into a single patch, so that the common tasks only needed to be performed once.
In Oracle EBS 12.2, all the functionality of AD Merge Patch has been included in the adop patching tool itself.
By default, adop will automatically merge all patches specified with the ‘patches’ parameter.
You can still use earlier AD Merge Patch tool but you should disable adop’s merging of patches, by adding to the adop command line > ‘merge=no’




WHAT HAPPENS WHEN ADOP COMMAND IS RUN?

adop will perform all the tasks required to apply the patch:
> Reads patch metadata to determine patch dependencies and requirements
> Attempt to recover previously failed patching session (if any)
> Reads and validate the patch/product driver files
> Compares version numbers of existing files against the patch files and Backs up all existing files that will be changed by the patch
> Copies files
> Archive files in libraries
> Relinks executables, Generates forms, reports, messages, graphics, and Java archive (JAR) files
> Compiles JSP files and invalid database objects. Updates database objects
> Runs AutoConfig if required
> Saves patch information to the database
All tasks are similar to what adpatch utility used to do earlier.


FROM WHICH FILESYSTEM ADOP COMMAND SHOULD BE RUN?
We always run adop utility from the run edition file system. The adop process automatically sets its environment correctly, regardless of the edition it is run from.

 

Phases:

    adop phase=prepare -> copies the application code
    adop phase=apply -> apply patches to PATCH Environment
    adop phase=finalize -> makes ready the system for cutover
    adop phase=cutover -> bounce the system and does filesystem switchover. fs2 becomes RUN environment.
    adop phase=cleanup -> remove obsolete objects.
    adop phase=fs_clone -> synchronize filesystems

 

Important parameters and commands related to ADOP are documents below.
ADOP PATCHING PARAMETERS DETAILS:

PARAMETER	DEFAULT VALUE	POSSIBLE VALUES	DESCRIPTION
phase	N/A	PHASE=PREPARE PHASE=APPLY PHASE=CUTOVER PHASE=CLEANUP PHASE=FINALIZE PHASE=ACTUALIZE_ALL PHASE=FS_CLONE PHASE=ABORT	These are the eight phases in which adop can run. It is most important and mandatory parameter that is used with adop. You can also club multiple phases in single command like ‘PHASE=PREPARE,APPLY’ although abort and fs_clone need to be run alone and can’t be clubbed.Standard phases: prepare – Prepare the instance for patch application. apply – Apply patches (to the patch edition). finalize – Ready the instance for cutover. It is run automatically. cutover – Make the patch edition the new run edition. cleanup – Drop obsolete objects and data from old editions. It is run automatically.  There are also three special phases, used as per requirement. Special phases: abort – Abort the current patching cycle. The abort phase can be run after either the prepare or apply phases have been run, but not after the cutover phase. actualize_all – Create new copies of all code objects in the patch edition. fs_clone – Copy the run file system to the patch file system.
loglevel	LOGLEVEL=EVENT	LOGLEVEL=STATEMENT LOGLEVEL=PROCEDURE LOGLEVEL=EVENT LOGLEVEL=WARNING LOGLEVEL=ERROR LOGLEVEL=UNEXPECTED	STATEMENT > for debugging. PROCEDURE > for debugging high level procedures. EVENT > to capture informational messages in normal processing. (default) WARNING > to capture any internal error that is handled by the system and does not affect processing. ERROR > indicates action failed, need to be reviewed, but the system continue processing. UNEXPECTED > indicates an unrecoverable error, requires user intervention before processing can continue.
cleanup_mode	CLEANUP_MODE=STANDARD	CLEANUP_MODE=FULL CLEANUP_MODE=STANDARD CLEANUP_MODE=QUICK	Cleanup processing needs to happen after adop finishes the patching work.Quick mode > shortest execution time, skips non-essential actions Standard mode > All quick mode action + drops covered objects Full mode > All quick mode action +  remove all unused code, data, and old editions and takes much longer
finalize_mode	FINALIZE_MODE=QUICK	FINALIZE_MODE=QUICK FINALIZE_MODE=FULL	Quick mode > shortest execution, skips non-essential actions, no gather statistics. Full mode > Gather statistics, may improve performance after cutover, can take an hour extra to complete.
input_file	N/A	INPUT_FILE=<Absolute input_file path>	To specify the name of the input_file supplied to adop. (see details on input_file later in this post)
workers	N/A (depends on number of available CPU cores)	WORKERS=<User-specified-value>	Number of parallel workers used to execute tasks.In earlier released  adpatch used to prompt for number of workers. With adop in R12.2, if you want  to override the default formula that oracle uses now to calculation number of workers, use the WORKERS parameter. Take care that you don’t specify very high number of workers or else adop will fail.
maxworkers	N/A (depends on number of available CPU cores)	MAXWORKERS=<User-specified-value>	Maximum parallel workers that can be engaged. maxworkers should always be set to greater than the desired number of workers.
runcontextfile	$CONTEXT_FILE	RUNCONTEXTFILE=<Absolute context_file path>	To specify the non-default context file patch in RUN filesystem
patchcontextfile	Standard context file path in patch FS	PATCHCONTEXTFILE=<Absolute context_file path>	To specify the non-default context file patch in PATCH filesystem
patches	N/A	PATCHES=<User-specified-value>For Standard Patch:when>patch directory is  a 6- to 8-digit numberPATCHES=<patch_number> For Non-Standard Patch when > patch directory is not a 6- to 8-digit number example NLS patches <patch_number>_<language_code>. > patch driver files are not named *<patchnum>.drv example merged patchesPATCHES=<patch_number>:<driver_file>.drv	This parameter specifies the patches adop needs to apply.Remember the numbered-only patches (standard) and containing-a-colon categories of patch (non-standard) can be mixed.Like:PATCHES= <patch_number1>,  <patch_number2>:<driver_file2>.drv
defaultsfile	$APPL_TOP/admin/<SID>_patch/adalldefaults.txt	DEFAULTSFILE=<Absolute defaults_file path>	Default file locations on both the run APPL_TOP and patch APPL_TOP is: $APPL_TOP/admin/<SID>_patch/adalldefaults.txtIn case you have created your own defaults file and want to use that instead, then use this parameter.
patchtop	$NE_BASE/EBSapps/patch	PATCHTOP=<Absolute patch_location_file path>	Default patch_top location is below.$NE_BASE/EBSapps/patchIf you want to keep your patches in some other lcoation, then you need to use this patrameter to let ADOP know where to search for patches pointed by ‘patches’ parameter.If you have a multi-node environment, you must download and unzip the patches (under $APPL_TOP_NE/EBSapps/patch) on the respective nodes.
merge	MERGE=NO	MERGE=NO MERGE=YES	In R12.2, oracle has integrated patch merging action in the patching command itself. In earlier releases we used to first merge patches using admrgpch command.By using MERGE=YES option ADOP will merge all the unified driver files into a single driver file.
abandon	ABANDON=NO	ABANDON=YES ABANDON=NO	If the patch you are applying went into error, you have two option when you start the adop utility again.1) you corrected error and want to continue with previous adop session:ABANDON=NO2) you decided that you don’t want to correct issue for now and want to abandon the previous adop session:ABANDON=YES
restart	RESTART=YES	RESTART=NO RESTART=YES	If the patch you are applying went into error and you corrected the issue and want to restart the previous patching session.It is just the reverse of ABANDON parameter.Remember ABANDON and RESTART will always have opposite value.
flags	N/A	FLAGS=AUTOSKIP	Use “flags=autoskip” in conjunction with the “abandon=no” parameter at the command-line to skip a failing patching step to “Continue as if a patch were successful”. You need to review the “autoskip” log that gets generated during the patching cycle in order to make sure that their were no errors and to take required actions in case of any errors
allnodes	ALLNODES=NO	ALLNODES=NO ALLNODES=YES	This parameter comes into picture when you have multi node setup. If you want to run adop on all nodes then use ALLNODES=YES.
action	ACTION=DB	ACTION=DB ACTION=NODB	Use this parameter to specify whether to perform database actions or skip. For example if you are in a multi-node environment and adop has already updated the database so when running on other node just use ACTION=NODB to save time.Remember when you are using ‘allnodes=yes’ in a multi-node ‘action=db’ must be specified.
apply	APPLY=YES	APPLY=YES APPLY=NO	To run adop in test mode (without applying any patches),specify apply=no
autoskip	AUTOSKIP=YES	AUTOSKIP=YES AUTOSKIP=NO	This parameter control whether the user is prompted about skipping actions in non-interactive patching. This is specifically useful when you are applying patches in multi node setup.
mtrestart	MTRESTART=YES	MTRESTART=YES MTRESTART=NO	This parameter specify whether to restart application tier services after cutover phase or not.
cm_wait	CM_WAIT=INFINITE (will wait forever)	CM_WAIT=<user_specified_number	Specifies the number of minutes to wait until the ICM will be forced down.
allowcoredump	ALLOWCOREDUMP=NO	ALLOWCOREDUMP=NO ALLOWCOREDUMP=YES	To specify that a core dump will be generated if adop crashes.
analytics	ANALYTICS=NO	ANALYTICS=NO ANALYTICS=YES	To specify that a report will be generated that can help debug certian adop issue.
preinstall	N/A	PREINSTALL=Y	This mode is used only if the patch readme instructs. Generally this mode is used during the upgrade process to update AD utilities, apply pre-upgrade patches, or work around other patching issues.It will Compares version numbers, Copies files, Relinks FND and AD executables, Saves patch information. It also runs autoconfig if required.The dual file system in Release 12.2 means that there is no need to shut down application tier services before running AutoConfig.
-help	N/A	N/A	Shows the help screen.
-status-status -detail	Latest Session	-STATUS (for latest session) -STATUS <SESSION_ID> (for specific session)	Display status of the latest adop session.Use ‘adop -status -detail’ for detailed info
-examples	N/A	N/A	Displays some commonly used adop ample commnads

ADOP PATCHING EXAMPLE COMMANDS:
‘Complete’ adop patching cycle using parameters input through command line  (INTERACTIVE MODE)
So in R12.2 we have complete patching cycle to follow to apply a patch. These are set of commands which needs to executed in specific order.
You must set the environment by executing the run file system environment file.
$ . <run APPL_TOP path>/APPS<CONTEXT_NAME>.env
1) adop phase=prepare
2) adop phase=apply patches=<patch_number1>,<patch_number2> workers=<number_of_worker>
3) adop phase=finalize workers=<number_of_worker> (called automatically)
4) adop phase=cutover workers=<number_of_worker>
5) adop phase=cleanup (called automatically)
OR
Running all phases in single command:
adop phase=prepare,apply,finalize,cutover,cleanup patches=<patch_number1>,<patch_number2>
All the phases need to be completed and you can’t skip any of these. For example, if you try to skip prepare phase, you may get error message like “Apply phase can only be run while in a patching cycle, i.e. after prepare phase.”
adop patching cycle in NON-INTERACTIVE mode
During apply phase, Non-interactive patching is a way to save time by avoiding some of the prompts and automating the patching process.
You can apply patches in non-interactive way by using a defaults file that contains much of the information you would have supplied at the adop prompts and by creating another file known as input file. Then, when you run adop, you specify the name of the input file. The location of the defaults file will also need to be included in the input file.
$ adop phase=apply input_file=<input_file.txt>
Location of Default file on both the run APPL_TOP and patch APPL_TOP is:
$APPL_TOP/admin/<SID>_patch/adalldefaults.txt
Just in case this file gets corrupted or lost, you can run AutoConfig and it will automatically instantiate a new copy.
NOTE: In R12.2, you don’t need to create this defaults file. The file is already created by oracle process. However you need to create one ‘input file’ to use with adop (The defaults file is not specified on the adop command line. It is the input file.)
The input_file contents should include the following required parameters:
patches=<patch number>
workers=<number of workers>
patchtop=<directory where patches are staged>
defaultsfile=<defaults file on patch APPL TOP>
To skip specific patching portion or action

During apply phase:
adop phase=apply options=nodatabaseportion, nogenerateportion
In above command we are instructing adop to not run database and generate portion.
Other options can be
options=noactiondetails  — if you do not want the details to be printed.
options=noautoconfig  — if you are applying a number of patches in sequence and want to run
AutoConfig only once in the end.
option=nocheckfile –to turn off the checkfile feature. Using checkfile adop skip some actions which are   already done.
option=nocompiledb –when applying multiple NLS, documentations patches etc
option=nocompilejsp –when applying multiple NLS, documentations patches etc
option=nocopyportion  –for skipping copy portion of the patch
To apply patch in ‘HOTPATCH’ mode

During apply phase:
adop phase=apply options=hotpatch
You can see couple of other examples commands by typing on Unix command prompt  ‘adop -examples‘ .

Environment Variables in Release 12.2

Below are some of the important Operating system level environment variables  related to R12.2.

$ echo $FILE_EDITION
<shows which file edition you are using, run or patch>



$ echo $RUN_BASE
<shows absolute path to run file system>



$ echo $PATCH_BASE
<shows absolute path to patch file system>



$ echo $NE_BASE
<shows absolute path to non-edition file system>



$ echo $APPL_TOP_NE
<non-editioned appl_top path. Equivalent to $NE_BASE/EBSapps/appl>



$ echo $LOG_HOME
<Application Instance Specific Log Directory>



$ echo $ADOP_LOG_HOME
<Online patching Specific Log Directory. Equivalent to $NE_BASE/EBSapps/log/adop>



$ echo $IAS_ORACLE_HOME
<FMW Web Tier Home Directory>



$ echo $ORACLE_HOME
< 10.1.2 ORACLE_HOME>



$ echo $CONTEXT_FILE
<Source for information populating template files (autoconfig)>



$ echo $EBS_DOMAIN_HOME
<WLS Deployment of Oracle E-Business Suite 12.2 Domain (instance specific)>



$ echo $ADMIN_SCRIPTS_HOME
<Shell scripts to control processes associated to the Applications Instance>



$ echo $EBS_ORACLE_HOME
<Oracle E-Business Suite 12.2 FMW Deployment directory>



$ echo $RW
<10.1.2 reports directory>



$ echo $HOSTNAME
<hostname without domain name>



$ echo $APPS_VERSION
<to get the EBS version>



And the most important part is for setting up the right environment don’t directly hard code the RUN environemnt .env file in the .profile of OS user as online patching switches the filesystem from RUN to PATCH and vice versa and it can really create confusion.
Instead use EBSapps.env environment file (created under BASE directory) with ‘RUN’ as argument. It will automatically find which file system (fs1 or fs2) is currently RUN file system and lay out the correct environment.

For example, in our case base directory is ‘/u01/oracle/dba’

. /u01/oracle/dba/EBSapps.env RUN

It will set environemnt and display below information on screen.
E-Business Suite Environment Information
 —————————————-
 RUN File System : /u01/oracle/dba/fs2/EBSapps/appl
 PATCH File System : /u01/oracle/dba/fs1/EBSapps/appl
 Non-Editioned File System : /u01/oracle/dba/fs_ne
 DB Host: dbhost.hertz.com Service/SID: dba
 Sourcing the RUN File System …