Error in OBIS1 after restart and repository is empty

Error in OBIS1 after restart and repository is empty

Detect a problem

I needed to restart OBIEE server.

I restarted it by  commands - 

sh /oracle/product/fmw1221/user_projects/domains/bi/bitools/bin/stop.sh

sh /oracle/product/fmw1221/user_projects/domains/bi/bitools/bin/start.sh

After that one of OBIEE process didn't work.

Name            Type            Machine                   Status      
----            ----            -------                   ------      
AdminServer     Server          lnxru0240vg701            RUNNING      
bi_server1      Server          lnxru0240vg701            RUNNING      
obips1          OBIPS           lnxru0240vg701            RUNNING      
obijh1          OBIJH           lnxru0240vg701            RUNNING      
obiccs1         OBICCS          lnxru0240vg701            RUNNING      
obisch1         OBISCH          lnxru0240vg701            RUNNING      
obis1           OBIS            lnxru0240vg701            SHUTDOWN

luckily I created a backup with repository and catalog).

Check a log 

Look at a log and find error message - 

cd /oracle/product/fmw1221/user_projects/domains/bi/servers/obis1/logs/obis1-diagnostic.log

Received error message from Node Manager Server: [Server start command for OBIS server 'obis1' failed due to: [Server failed to start up but Node Manager was not aware of the reason]. Please check Node Manager log and/or server 'obis1' log for detailed information.]. Please check Node Manager log for details.

Or go to EM  
http://your_host:9500/em/faces/logmgmt/msgs/logMessages?type=weblogic_domain&target=%2FDomain_bi%2Fbi

Check a NodeManager log

cd /oracle/product/fmw1221/user_projects/domains/bi/nodemanager/

cat nodemanager.log | grep obis1

<Dec 23, 2015 10:25:26 PM MSK> <WARNING> <Server start command for OBIS server 'obis1' failed due to: [Server failed to start up but Node Manager was not aware of the reason]. Please check Node Manager log and/or server 'obis1' log for detailed information.>

Check OBIS log

After I checked repository log

cd /oracle/product/fmw1221/user_projects/domains/bi/servers/obis1/logs
cat obis1-diagnostic.log | grep ERROR

[nQSError: 92006] Could not resolve the object

Went to check repositories

ls -l /oracle/product/fmw1221/user_projects/domains/bi/bidata/service_instances/ssi/metadata/datamodel/customizations/

total 12
drwxr-x--- 2 oracle oinstall 4096 Dec 23 14:11 default
-rwx------ 1 oracle oinstall 1176 Dec 23 15:34 liverpd.rpd
-rwx------ 1 oracle oinstall 1176 Dec 23 19:14 liverpd.rpd_1

Repositories had the same size this is a reason why your server wasn't started up.

Fix a problem

It means that 'liverpd.rpd_1' (your repository) suddenly became empty, just like the 'liverpd.rpd' one, while 'default/default-diff.xml' (the main source, which is generated from .rpd; it may be thought as a patch for an empty repository) is still full of metadata which is wrong now.
To solve this problem, you need to remove 'default-diff.xml' and 'liverpd.rpd_1' files. Also you need to make a copy of 'liverpd.rpd' for further restoration.

rm default/default-diff.xml liverpd.rpd_1
cp liverpd.rpd liverpd.rpd.copy

total 12
drwxr-x--- 2 oracle oinstall 4096 Dec 23 14:11 default
-rwx------ 1 oracle oinstall 1176 Dec 23 15:34 liverpd.rpd
-rwx------ 1 oracle oinstall 1176 Dec 23 15:34 liverpd.rpd.copy

---
old way 

I updated repositories and changed default_diff.xml in the folder default  to default values.

A link with these files
https://drive.google.com/folderview?id=0B6Owvik70w-1U2V4YnRTclVOWVk&usp=sharing

chown -R oracle:oinstall /oracle/product/fmw1221

find /oracle/product/fmw1221/ -user root|wc -l
---
After doing so, startup OBIEE:  
sh /oracle/product/fmw1221/user_projects/domains/bi/bitools/bin/start.sh

and obis1 would be able to run. If not, you may need to execute start.sh once again.

Status of Domain: /oracle/product/fmw1221/user_projects/domains/bi
NodeManager (lnxru0240vg701:9506): RUNNING
Name            Type            Machine                   Status     
----            ----            -------                   ------     
AdminServer     Server          lnxru0240vg701            RUNNING     
bi_server1      Server          lnxru0240vg701            RUNNING     
obips1          OBIPS           lnxru0240vg701            RUNNING     
obijh1          OBIJH           lnxru0240vg701            RUNNING     
obiccs1         OBICCS          lnxru0240vg701            RUNNING     
obisch1         OBISCH          lnxru0240vg701            RUNNING     
obis1           OBIS            lnxru0240vg701            RUNNING

Upload you repository:

$DOMAIN_HOME/bitools/bin/data-model-cmd.sh uploadrpd -SI ssi -U weblogic -I ~/backup.rpd

Finally, move empty repostory back in place:
mv liverpd.rpd.copy liverpd.rpd

total 12
drwxr-x--- 2 oracle oinstall 4096 Dec 23 14:11 default
-rwx------ 1 oracle oinstall 1176 Dec 23 15:34 liverpd.rpd

-rwx------ 1 oracle oinstall 345652 Dec 23 15:34 liverpd.rpd_2

Updated repository, catalog and all my reports are working!

Enjoy

Quick tip:

If you have no backup file made with 
./data-model-cmd.sh downloadrpd
but only live.rpd.bak, you may need to know the default repository password, in case when you would like to restore it using uploadrpd.

To retrieve the password, open Enterprice Manager → WebLogic Domain → System MBean Browser, and navigate to Application Defined MBeans → com.oracle.jps → your domain → JpsCredentialStore.

Choose the getPortableCredential from Operations tab and enter next two parameters:
p1: oracle.bi.enterprise
p2: metadata.encryption

Click the Invoke button.