adapcctl.sh: exiting with status 150

On E-Business Suite Release 12 environment, while starting midtier services using adstrtal.sh, below error occurs:

Executing service control script:
/oracle/apps/VIS/inst/apps/<CONTEXT>/admin/scripts/adapcctl.sh start
Timeout specified in context file: 100 second(s)
Script returned:
****************************************************
You are running adapcctl.sh version 120.7.12010000.2
Starting OPMN managed Oracle HTTP Server (OHS) instance ...
adapcctl.sh: exiting with status 150
adapcctl.sh: check the logfile /oracle/apps/VIS/inst/apps/<CONTEXT>/logs/appl/admin/log/adapcctl.txt for more information ...
.end std out.
.end err out.
****************************************************
Same error for adoacorectl.sh, adoafmctl.sh, adformsctl.sh.

CAUSE:
There is a states file called .opmndat at /oracle/apps/VIS/inst/apps/<CONTEXT>/ora/10.1.3/opmn/logs/states which gets created when services are started.
Whenever services are started this file gets created or updated.
Unfortunately for some reason this file is not updated when services are started.

SOLUTION:
1. Shutdown all Middle tier services and ensure no defunct processes exist running the following from the operating system:

# ps -ef | grep <applmgr>

If one finds any, kill these processes.
2. Navigate to $INST_TOP/ora/10.1.3/opmn/logs/states directory. It contains hidden file .opmndat:

# ls -lrt .opmndat
3. Delete this file .opmndat after making a backup of it:

# rm .opmndat
4. Restart the services.

5. Re-test the issue.

Steps to perform switchover in physical standby

 Steps to switchover the Standby database to Primary and Primary database to Standby in Oracle10g

1.    Shutdown the primary database.

SQL> shutdown immediate

2.    Shutdown the standby database.

SQL> shutdown immediate

3.    Startup standby database

SQL> startup nomount

SQL> alter database mount standby database;

4.    Startup primary database

SQL> startup

5.    SQL> alter system archive log current;                     (On primary)

       6.  Start the managed recovery operation:          (On standby)

SQL> recover managed standby database disconnect from session;

7.    Check the MRP process                                            (On standby)

SQL> select process, status from v$managed_standby;

8.    If MRP process exist than stop it                                (On standby)

SQL>alter database recover managed standby database cancel;

9.    Stop and start the listener                               (On both)

$ lsnrctl stop

$ lsnrctl start

10.  Shutdown the primary database.

SQL> shutdown immediate

11.  Shutdown the standby database.

SQL> shutdown immediate

12.  Startup standby database

SQL> startup nomount

SQL>alter database mount standby database;

13.  Startup primary database

SQL> startup

14.  Check switchover status                                (On primary)

SQL>select switchover_status from v$database;

            It should be to_standby.

15.  Now run commands                                       (On primary)

SQL>alter database commit to switchover to standby with session    shutdown;

SQL> alter database commit to switchover to physical standby;

SQL> shutdown immediate

SQL>startup nomount

SQL>alter database mount standby database;

SQL> alter system set log_archive_dest_state_2=defer;

SQL> recover managed standby database disconnect from session;

Now primary have converted in standby.

16.  Check switchover status                                (On old standby)

SQL>select switchover_status from v$database;

It should be switchover_pending.

17.  Run the command                                          (On old standby)

SQL> alter database commit to switchover to primary;

18.  Shutdown the database                                              (On old standby)

SQL> shutdown immediate

SQL> startup

SQL> alter database force logging;

SQL> alter system set log_archive_dest_state_2=enable;

Now old standby have converted in Primary database.

 NOTE:-   If you want to Check  that data of primary is applying on standby or not than you can check by these steps:-

1. Make a test table

            SQL> create table test(id number(20));         (On new primary)

2. Switch the log to send the redo data to the standby database:

SQL> alter  system switch logfile;

3. Start managed recovery, if necessary, and applies the archived redo logs by entering the following SQL statement:

SQL> alter database recover managed standby database;   (On new standby)

4. Cancel managed recovery operations.

                        SQL> alter database recover managed standby database cancel;

            This command will stop the MRP process. Now open the database in read only mode.

                        SQL> alter database open read only

5. After that if you want standby database in archive recover mode than firstly you will have to shutdown & startup the standby database.

                        SQL> shutdown immediate

                        SQL> startup nomount

                        SQL> alter database mount standby database;

6. Start the managed recovery operation on standby database:

SQL> recover managed standby database disconnect from session;

What Happens During a Hot Backup

What happens when we put Oracle database in hot backup mode

The BEGIN BACKUP command checkpoints the datafiles and advances the datafile header checkpoints to the begin backup checkpoint SCN. Until the END BACKUP command is executed, checkpoints will cease to update the datafile headers. It starts the logging of block images.Each time the block is read into cache an image of the whole block before the first change is logged.
 

The END BACKUP command creates a redo record containing the begin backup  checkpoint SCN. It stops the logging of block images and causes the datafile checkpoints to be advanced to the database checkpoint.  

Datafile updates by DBWR go on as usual during the time the backup is being copied so the backup gets a "fuzzy" copy of the datafile.Some blocks may be ahead in time versus other blocks and some may contain updates by transactions that are later rolled back. The "fuzzy" backup copy is unusable without the "focusing" via the redo log that occurs when the backup is restored and undergoes media recovery. Media recovery applies redo (from all threads) from the checkpoint SCN that was captured at BEGIN BACKUP time through the end-point of the recovery operation. 
 

Once redo application is finished, uncommited transactions are rolled back to leave a transaction-consistent "focussed" version of the datafile.