Friday, August 31, 2012

RAC and ASM Troubleshooting Part -1

Normal Commands to Maintain the RAC environment are as follows:

     In most of the cases and the recommended configuration, crsctl should be executed as a root  and the command srvctl should be executed as DB user.(For ease of administration, it is better to have 2 separate users for Grid and DB.Grid user will be responsible for the ASM ,Voting Disk(VD), Oracle Cluster Registry(OCR) and other Cluster Ready Services.)

i) To check the status of CRS (Cluster Ready Services)

           # crsctl check crs 

ii) To check the Status of services on all nodes
           
         $ srvctl status nodeapps
 or
         # crsctl status resource -t 
or
       # crsctl stat res -t

      # crsctl stat res -t -init
 

     #crsctl stat res -p


iii) To check the Clusterware servers:

# crsctl status server -f


iv) To check the status of the RAC High Availability Services

#crsctl status has

v) To check the status of Specific instance across the cluster nodes:

$srvctl status instance -d <rac_database> -i <rac_database_instance>

v) To check the status of all instances across the cluster nodes:


$srvctl status instance -d <rac_database> -i <rac_database_instance1>, <rac_database_instance2>,<rac_database_instance3>

vi) To check the Database status across all the cluster nodes:

$srvctl status database -d <rac_database> 

Maintaining the RAC Database:

i) To Stop the RAC Database 

# crsctl stop cluster -all

ii)  To stop crs on specific node

#crsctl stop has
or
# crsctl stop crs

iii) To stop the crs by force on specific node

# crsctl stop crs -f

iv) To start the crs on specific node

# crsctl start has
or
#crsctl start crs

v) To stop the specific instance across the cluster nodes:

$srvctl stop instance -d <rac_database> -i <rac_instance1>

vi) To start the specific instance across the cluster nodes:


$srvctl start instance -d <rac_database> -i <rac_instance1>

vii) To stop all the instances across the cluster nodes:

$srvctl stop instance -d <rac_database> -i <rac_instance1>, <rac_instance2>,<rac_instance3>

viii) To start all the instances across the cluster nodes:


$srvctl start instance -d <rac_database> -i <rac_instance1>, <rac_instance2>,<rac_instance3>

ix) To stop cluster Database

$srvctl stop database -d <rac_database>

x) To Start the Cluster Database

$srvctl start database -d <rac_database>





 




 

Friday, August 24, 2012

adcfgclone.pl error while adrelink.sh(sprintf_chk@GLIBC_2.3.4) while cloning on a new Linux Server


Cloning activity for the Release 12.1 on RHEL 5 on a new server gives the following error while running the adcfgclone.pl : 

adlnktools.sh started 
Log file located at /d01/inst/apps/DEV_uat/logs/ora/10.1.2/install/make_08241118.log

Error while running adlnktools.sh.
  return code = .46
Please check logfile located at /d01/inst/apps/DEV_uat/logs/ora/10.1.2/install/make_08241118.log

Further investigating the error in the logfile: 

adrelink.sh fails with the following error:

usr/lib/libXtst.so.6: undefined reference to `__stack_chk_fail@GLIBC_2.4'
/usr/lib/libXtst.so.6: undefined reference to `__fprintf_chk@GLIBC_2.3.4'
/usr/lib/libXtst.so.6: undefined reference to `__sprintf_chk@GLIBC_2.3.4'
collect2: ld returned 1 exit status

This error can occur only on OEL5 or RHEL 5 and to resolve please use the following steps:

ls -l /usr/lib/libXtst.so.6
lrwxrwxrwx 1 root root   /usr/lib/libXtst.so.6 -> libXtst.so.6.1.0

unlink /usr/lib/libXtst.so.6
ln -s /usr/X11R6/lib/libXtst.so.6.1 /usr/lib/libXtst.so.6
ls -l /usr/lib/libXtst.so.6
lrwxrwxrwx 1 root root   /usr/lib/libXtst.so.6 -> /usr/X11R6/lib/libXtst.so.6.1
Rerun to the apps user login and execute the following script to verify the adrelink.sh is successfully running with the above fix or not.
 $ORACLE_HOME/appsutil/clone/adlnktools.sh 
If it is completed successfully, then rerun the adcfgclone.pl script to complete rest of the process.



Thursday, August 23, 2012

Error: ORA-01031: insufficient privileges : SQLPLUS Fails After Rapid Clone adcfgclone.pl dbTier/dbTechStack With 11g Database



     Normally the insufficient privileges, do occur when the OS level privileges are missing for Oracle User. Like assigning the DBA group for oracle user in unix versions and ORA_DBA group in Windows User Group. The more detailed error study can be found in the following metalink note-id: Checklist for Resolving CONNECT AS SYSDBA (INTERNAL) Issues [ID 233223.1]
 
UNIX: Checklist for Resolving Connect AS SYSDBA Issues [ID 69642.1]
  
For us this error occured when we are trying to Clone the Oracle Database using the RMAN database and after running the adcfgclone.pl dbTechstack . Thanks for the blog(newappsdba.blogspot.com), who has the same situation and the same error.

After running the adcfgclone.pl, whenever we tried to run the sqlplus as "sysdba", it will throw error as 'ORA-01031: insufficient privileges'.

 

As per the metalink id: SYSDBA and SYSOPER Privileges in Oracle [ID 50507.1]

The fix is :

In $ORACLE_HOME/rdbms/lib

mv config.o config.o.orig make -f ins_rdbms.mk ioracle

May be the adcfgclone.pl is making some changes to this file or its corrupted. By applying the above solution it resolved the error.But each and everytime when we make the clone, it is quite necessary to do the above workaround.

But Oracle has provided a Permanent solution for the above error through the metalink ID : 1276420.1.  To apply the patch 8999153 in Source system.

 

Friday, August 17, 2012

Enabling the Debug Level for Apache in 11.5.10.2

Before Setting the Debug level for apache it is required to clear the old logfiles in the apache log directory and later the debug level to be enabled.

Steps:

i) Backup and delete the following files.
$IAS_ORACLE_HOME/Apache/Apache/logs/*

ii) Set the following debug parameters.
In httpd.conf file set LogLevel to debug

iii) Bounce your Apache.

iv) Check the log files generated under the following location:
 $IAS_ORACLE_HOME/Apache/Apache/logs/*
  


Custom Network Parameters to Override the Autoconfig on Oracle Apps Environment

     In any auto-config enabled Oracle Apps Server will over ride the custom network entries in the tnsnames.ora and listener.ora in network/admin directories of either DB node or Apps node. But at times it is required to include some custom entries either at the DB Tier or Apps Tier and which should not be wiped out by the Oracle adautonfig script.This type of changes can be included in the ifile which can be seen at the end of those files entries. Like in the DB Tier the the custom tnsnames.ora entries can be added into <Context_name>_ifile.ora (Context Name = SID_HOSTNAME) and the custom listener.ora entries can be added on listener_ifile.ora. 

  Recently we were upgrading the RAC database in one our client and after successful upgradation of the database, it was our turn to integrate the database with Oracle Apps Server by enabling the adconfig for 11.5.10.2. We were successful at DB Tier, later we started the Application Services. We could able to see the login page and also able to login into Oracle, but when we tried to open the forms based application, they were not opening. It was taking very long time, but never opened the forms. Immediately we have ran the adadmin to regenerate the jar files with FORCE option and it was completed successfully. Later We bounced all the services and rechecked, but still we were stuck with the same issue.

  After a very deep survey we found something, which very good lesson.Actually the the db upgrade was concluded in 2 parts: i) Upgrading the Server Configuration i.e. moving the DB from the lower-end resources to the Higher-end resources. ii) Upgrading the Database from 10g to 11g Release 2. There were some custom tnsnames.ora entries in the DB node which we were successfully over-raided while configuring the autoconfig and at the Application server node we were just making the changes in the Context file to see the new server in place of old server.But the there was an entry in ifile at Application Server node which was pointing to the old server, which internally disabling the Apache to start the Java Sessions.

Conclustion:

     An ifile with custom entires are available in 3 places in Oracle Apps Server :
    i) DB Tier 
   ii) the <806_ORACLE_HOME>/network/admin/<CONTEXT_NAME> and 
  iii) <IAS_ORACLE_HOME>/network/admin/<CONTEXT_NAME> 

All these ifile entries are not edited by the autofig scripts.Need to check everytime before upgradation or migration of the Database or Application Server.

Thursday, August 09, 2012

Active Form Sessions in Oracle Apps

Active Oracle Apps Form Sessions at any point of time can be know with the following query:

SELECT   fnd.user_name, icx.responsibility_application_id,
         fat.application_name, fat.application_id, icx.responsibility_id,
         frt.responsibility_name, icx.session_id, icx.first_connect,
         icx.last_connect,
         DECODE ((icx.disabled_flag),
                 'N', 'ACTIVE',
                 'Y', 'INACTIVE'
                ) status
    FROM fnd_user fnd,
         icx_sessions icx,
         fnd_responsibility_tl frt,
         fnd_application_tl fat
   WHERE fnd.user_id = icx.user_id
     AND frt.application_id = fat.application_id
     AND icx.responsibility_id = frt.responsibility_id
     AND icx.disabled_flag <> 'Y'
     AND TRUNC (icx.last_connect) = TRUNC (SYSDATE)
     AND frt.responsibility_name NOT LIKE '%Intelligence%'
ORDER BY fnd.user_name