Oracle Solutions We believe in delivering tangible results for our customers in a cost-effective manner
Oracle E-Business Suite R12.2 Application Services Failed to Start After Server Crash
Recently,Β faced an issue in an Oracle E-Business Suite R12.2 environment where the application tier services failed to start after an unexpected server crash.
After the server came back online, I attempted to start the application services using the standard adstrtal.sh script. However, the startup process failed because the WebLogic AdminServer was not coming up. As a result, all dependent managed servers such as oacore, forms, oafm, and forms-c4ws were skipped during startup.
This type of issue is commonly seen after an abnormal shutdown or server crash, where stale WebLogic lock files remain under the domain server directories. In this case, the original error showed AdminServer startup failure and managed servers being skipped because AdminServer was down.
Β
Error Encountered
While starting the application services, the following error was observed:
Executing service control script:
/u01/app/inst/fs1/inst/apps/test_server/admin/scripts/adadminsrvctl.sh start -nopromptmsg
Timeout specified in context file: -1 second(s)
script returned:
****************************************************
You are running adadminsrvctl.sh version 120.10.12020000.9
Starting WLS Admin Serverβ¦
Refer /u01/app/inst/fs1/inst/apps/test_server/logs/appl/admin/log/adadminsrvctl.txt for details
adadminsrvctl.sh: exiting with status 1
adadminsrvctl.sh: check the logfile /u01/app/inst/fs1/inst/apps/test_server/logs/appl/admin/log/adadminsrvctl.txt for more information β¦
.end std out.
.end err out.
****************************************************
ERROR: Skipping startup of forms_server1 since the AdminServer is down. AdminServer needs to be up before starting/stopping the managed servers.
ERROR: Skipping startup of oafm_server1 since the AdminServer is down. AdminServer needs to be up before starting/stopping the managed servers.
ERROR: Skipping startup of oaea_server1 since the AdminServer is down. AdminServer needs to be up before starting/stopping the managed servers.
ERROR: Skipping startup of forms-c4ws_server1 since the AdminServer is down. AdminServer needs to be up before starting/stopping the managed servers.
ERROR: Skipping startup of oacore_server1 since the AdminServer is down. AdminServer needs to be up before starting/stopping the managed servers.
ERROR: Skipping startup of oacore_server2 since the AdminServer is down. AdminServer needs to be up before starting/stopping the managed servers.
adstrtal.sh: Exiting with status 1
adstrtal.sh: check the logfile /u01/app/inst/fs1/inst/apps/test_server/logs/appl/admin/log/adstrtal.log for more information β¦
Root Cause
After reviewing the issue, I found that the problem was caused by stale WebLogic lock files and persistent store files left behind after the server crash.
When WebLogic services are not stopped gracefully, .lok files and related .DAT files may remain under the server directories. During the next startup, WebLogic may assume that the server process is already running or that the store is locked, which prevents AdminServer or managed servers from starting properly.
In this case, the solution was to rename the old lock and related persistent store files for AdminServer and managed servers. The original fix was based on moving these stale .lok and .DAT files to .OLD files instead of deleting them directly.
Β
Solution
Before proceeding, make sure all application tier services are stopped and no WebLogic Java processes are running.
You can verify using:
ps -ef | grep java
ps -ef | grep weblogic
Then source the EBS application environment file:Β
. EBSapps.env run
Now move the stale lock and persistent store files as shown below.
Β
Step 1: Rename Lock Files for forms-c4ws_server1
cd $EBS_DOMAIN_HOME/servers/forms-c4ws_server1
mv ./tmp/forms-c4ws_server1.lok ./tmp/forms-c4ws_server1.lok.OLD
mv ./data/store/diagnostics/WLS_DIAGNOSTICS000000.DAT ./data/store/diagnostics/WLS_DIAGNOSTICS000000.DAT.OLD
mv ./data/store/default/_WLS_FORMS-C4WS_SERVER1000000.DAT ./data/store/default/_WLS_FORMS-C4WS_SERVER1000000.DAT.OLD
mv ./data/ldap/ldapfiles/EmbeddedLDAP.lok ./data/ldap/ldapfiles/EmbeddedLDAP.lok.OLD
Β
Β
Step 2: Rename Lock Files for oafm_server1
cd $EBS_DOMAIN_HOME/servers/oafm_server1
mv ./tmp/oafm_server1.lok ./tmp/oafm_server1.lok.OLD
mv ./data/store/diagnostics/WLS_DIAGNOSTICS000000.DAT ./data/store/diagnostics/WLS_DIAGNOSTICS000000.DAT.OLD
mv ./data/store/default/_WLS_OAFM_SERVER1000000.DAT ./data/store/default/_WLS_OAFM_SERVER1000000.DAT.OLD
mv ./data/ldap/ldapfiles/EmbeddedLDAP.lok ./data/ldap/ldapfiles/EmbeddedLDAP.lok.OLD
Β
Step 3: Rename Lock Files for AdminServer
cd $EBS_DOMAIN_HOME/servers/AdminServer
mv ./tmp/AdminServer.lok ./tmp/AdminServer.lok.OLD
mv ./data/store/diagnostics/WLS_DIAGNOSTICS000000.DAT ./data/store/diagnostics/WLS_DIAGNOSTICS000000.DAT.OLD
mv ./data/store/default/_WLS_ADMINSERVER000000.DAT ./data/store/default/_WLS_ADMINSERVER000000.DAT.OLD
mv ./data/ldap/ldapfiles/EmbeddedLDAP.lok ./data/ldap/ldapfiles/EmbeddedLDAP.lok.OLD
Β
Β
Step 4: Rename Lock Files for oacore_server1
cd $EBS_DOMAIN_HOME/servers/oacore_server1
mv ./tmp/oacore_server1.lok ./tmp/oacore_server1.lok.OLD
mv ./data/store/diagnostics/WLS_DIAGNOSTICS000000.DAT ./data/store/diagnostics/WLS_DIAGNOSTICS000000.DAT.OLD
mv ./data/store/default/_WLS_OACORE_SERVER1000000.DAT ./data/store/default/_WLS_OACORE_SERVER1000000.DAT.OLD
mv ./data/ldap/ldapfiles/EmbeddedLDAP.lok ./data/ldap/ldapfiles/EmbeddedLDAP.lok.OLD
Β
Β
Step 5: Rename Lock Files for oacore_server2
cd $EBS_DOMAIN_HOME/servers/oacore_server2
mv ./tmp/oacore_server2.lok ./tmp/oacore_server2.lok.OLD
mv ./data/store/diagnostics/WLS_DIAGNOSTICS000000.DAT ./data/store/diagnostics/WLS_DIAGNOSTICS000000.DAT.OLD
mv ./data/store/default/_WLS_OACORE_SERVER2000000.DAT ./data/store/default/_WLS_OACORE_SERVER2000000.DAT.OLD
mv ./data/ldap/ldapfiles/EmbeddedLDAP.lok ./data/ldap/ldapfiles/EmbeddedLDAP.lok.OLD
Β
Step 6: Rename Lock Files for forms_server1
cd $EBS_DOMAIN_HOME/servers/forms_server1
mv ./tmp/forms_server1.lok ./tmp/forms_server1.lok.OLD
mv ./data/store/diagnostics/WLS_DIAGNOSTICS000000.DAT ./data/store/diagnostics/WLS_DIAGNOSTICS000000.DAT.OLD
mv ./data/store/default/_WLS_FORMS_SERVER1000000.DAT ./data/store/default/_WLS_FORMS_SERVER1000000.DAT.OLD
mv ./data/ldap/ldapfiles/EmbeddedLDAP.lok ./data/ldap/ldapfiles/EmbeddedLDAP.lok.OLD
Β
Β
Step 7: Start the Application Services Again
After renaming the stale lock files, start the application services again:
$ADMIN_SCRIPTS_HOME/adstrtal.sh apps/<apps_password>
Or start AdminServer first:
$ADMIN_SCRIPTS_HOME/adadminsrvctl.sh start
Once AdminServer is up, start the remaining managed servers/application services.
Β
Β
Validation
After startup, verify the status of the services:
$ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh status oacore_server1
$ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh status forms_server1
$ADMIN_SCRIPTS_HOME/admanagedsrvctl.sh status oafm_server1
Also check the logs under:
$EBS_DOMAIN_HOME/servers/AdminServer/logs
$EBS_DOMAIN_HOME/servers/oacore_server1/logs
$EBS_DOMAIN_HOME/servers/forms_server1/logs
$INST_TOP/logs/appl/admin/log
Β
Important Notes
Do not delete the files directly. Rename them with .OLD so they can be reviewed later if required.
Before moving the files, ensure that no WebLogic or application tier process is running.
This activity should ideally be performed during an approved maintenance window or after proper internal approval, especially in production environments.
Β
