Yesterday I went though the first stage in setting up clustering on OBIEE 12c which you can read here.
Today I want to finish off by creating the OBIEE domain on the second server. Oracle have made this easy by creating a process for essentially ‘zipping’ up the configuration files to create a domain template, which can then be used to set up the second machine.
Creating the Configuration File
The first step is to log into the first server and shut down OBIEE using the script:
<DOMAIN_HOME>/user_projects/domains/bi/bitools/bin/stop.sh
Then run the following command to create the template file:
<DOMAIN_HOME>/bitools/bin/clone_bi_machine.sh <IP Address> <NFS_Share>/pack.jar
Here the IP Address should be the one for the second server. Also, I’ve used the NFS share to store the output file pack.jar. That just saves me time FTP’ing the file over in the next step.
If everything runs you should get a message similar to this:
Copy pack file to remote machine and run the following on the remote machine to create the DOMAIN_HOME:
<OBIEE_HOME>/oracle_common/common/bin/unpack.sh -domain=<DOMAIN_HOME> -template=<NFS_Share>/pack.jar
Configuring the second server
The next step is move across to the second server and run the command that you received in the previous step. i.e:
<DOMAIN_HOME/oracle_common/common/bin/unpack.sh -domain=<DOMAIN_HOME> -template=<NFS_Share>/pack.jar
Then, start up the node manager using:
<DOMAIN_HOME>/bin/startNodeManager.sh &
And that’s it for the second server!
Now, all that’s left is to restart OBIEE on the first server using:
<DOMAIN_HOME>/user_projects/domains/bi/bitools/bin/start.sh
38 comments
Join the conversationVinay - 15th February 2016
I am seeing the below message in my secondary server after the scaleout.
[2016-02-15T09:45:34.434-05:00] [OBIS] [NOTIFICATION:1] [] [] [ecid: ] [sik: ssi] [tid: 1338] [13079] Started reading RPD credentials for key metadata.encryption from CSF.
[2016-02-15T09:45:34.466-05:00] [OBIS] [ERROR:1] [] [] [ecid: ] [sik: ssi] [tid: 1338] [46137] CSF error encountered. Error code: 43131. [[
file: server\Utility\Generic\Src\SUGCSF.cpp; line: 133
]]
Please advise
Vinay - 15th February 2016
Running sync_midtier_db.cmd resolved the issue. Please add it to your blog if possible.
Pete Bunning - 13th September 2016
Hi Vinay
I’ve never seen this problem but it seems to be another step in the instructions. There’s a quick reference on the Oracle Support site: OBIEE 12c Quick Reference: How To Cluster | Scale-out OBIEE 12c Including Examples (Doc ID 2101898.1)
Pete
Ömer Naci - 14th October 2016
Hi Pete,
You wrote a successful article.
Two errors present in the final steps. The main problem does not start OBIPS and OBIS service in this second machine.
obis2 :
[2016-10-13T12:42:06.465+03:00] [OBIS] [ERROR:1] [] [] [ecid: ] [sik: ssi] [tid: bf376720] [nQSError: 47024] The Oracle BI Server participates in a cluster but errors were found in the cluster configuration file. [[
[nQSError: 47051] The Oracle BI Server is specified as participating in a cluster, but global cache file storage location, /public/global_cache, is invalid.
obips2 :
[2016-10-13T12:42:02+03:00] [OBIPS] [ERROR:1] [] [saw.catalog.local.initialize] [ecid: ] [tid: ] [SI-Name: ] [IDD-Name: ] [IDD-GUID: ] [userId: ] Error initializing/loading existing Catalog: /public/bidata/metadata/content/livewebcat.
Another instance of Oracle Business Intelligence is trying to upgrade/update/create the catalog located at /public/bidata/metadata/content/livewebcat/root_startup.lck. Retry after it finishes.
Pete Bunning - 14th October 2016
That looks to me like a problem with accessing the Share from the second host. Can you check that and then see if it will startup?
Ömer Naci - 14th October 2016
Hi Pete,
catalog and global_cache directory are in the first node(/public/bidata and /public/global_cache).
On second node ,I can access the /public folder using “cd /public” command. So , there is no access problem NFS folder from second node.
Pete Bunning - 14th October 2016
I still think it has to be something to do with the share. Can you write to that folder from the second server?
Unless it’s something to do with the user. Were both instances installed by the same O/S user?
If you prove that the share is OK then you could try rebuilding the domain. A good thing about 12 is that you can just delete the ./user_domains/bi folder and rerun the config with having to re-install the software.
It’s a bit of a pain but it’s good practice!
Ömer Naci - 17th October 2016
Hi Pete,
/etc/export file are updated in the following way: /public *(rw,sync) in first machine. The problem is solved. Thanks for suggestions.
Roberto Contreras - 13th January 2017
Hello Pete,
Thanks for your great tutorial.
However, I`ve a question since Im` new in this OBIEE World.
In the part 1 tutorial you said “The second server just needs the Weblogic and OBIEE software”. Therefore I would like to know if I need to install Fusion and OBIEE ( just to the part 1 that installs the basic folders) or if I have to move some specific folders from the server 1 installation. If that`s the case, would you please let me know which folder do I need to move in order to implement Server 2.
Regards,
Roberto
Pete Bunning - 16th January 2017
Hi Roberto
The second server only needs the infrastructure and OBIEE software – just the basic installs. So there’s no need to move folders manually.
The clone_bi_machine.sh that you use will copy the Weblogic configuration and everything else over for you.
Cheers
Pete
Roberto Contreras - 16th January 2017
Hello Pete,
Thanks foor your response.
I followed your instructions and installed just the basic Weblogic and OBIEE configuration ( Part 1 and 2 of your OBIEE tutorial) on the 2nd server.
Then I made a clone on machine 1, move it to machine 2.
However when i ran the unpack command I got the following error:
>> warning:write Domain to “/u01/app/obiee/user_projects/domains/bi”
>> 40317: Invalid “Servers, Clusters and Machines” configuration.
40317: The “Servers, Clusters and Machines” configuration in your domain is invalid.
40317: Correct the configuration.
>> warning:write Domain to “/u01/app/obiee/user_projects/domains/bi”
>> 40326: Invalid coherence server address.
40326: At least one server in a Coherence cluster should have non localhost address.
40326: Enter a non localhost listen address for at least one Coherence server.
Kindly, do you know what might be the issue?
Regards,
Roberto
Roberto Contreras - 16th January 2017
One thing I can add is that I ran the clone command two times, so maybe that could be the issue…I`m investigating on how to delete machines from the main configuration…
Regards,
Roberto
Pete Bunning - 17th January 2017
It’s hard to say without looking into it a bit more but check that the IP addresses for the two servers are valid. And make sure you’re creating the 2nd BI machine for the second IP address.
This setup should be fairly straightforward and when it does go wrong it’s usually something daft that I’ve done.
Roberto Contreras - 17th January 2017
Hello Pete,
You were right, the problem was related to the iptables configuration on the 2nd server.
In machine 2, I had to open ports with the following lines:
-A INPUT -p tcp -m state –state NEW -m tcp –dport 9500 -j ACCEPT
-A INPUT -p tcp -m state –state NEW -m tcp –dport 9502 -j ACCEPT
-A INPUT -p tcp -m state –state NEW -m tcp –dport 9503 -j ACCEPT
-A INPUT -p tcp -m state –state NEW -m tcp –dport 9506 -j ACCEPT
Once i did that I restart node manager, then restart OBIEE in first server and everything works fine.
Thanks Pete for this blog, it saves me a ton of time.
If you come to Chile, please let me know so I can invite you some beers!
Regards,
Roberto
Pete Bunning - 18th January 2017
You’re on!
Glad I could help Roberto.
Roberto Contreras - 18th January 2017
Pete,
Thanks for all your help.
I finished my setup with 3 machines in a OBIEE Cluster.
However, Im now facing the following issue. As requested by our system administrator. they want to have only one Biservers up and running.Therefore I stopped obis1 and obis2 and only BIS3 is up and running.
If y try to open the analytics URL everything looks great but when I try to login with weblogic credential I just cant login. Do you know if this is expected or do I need to follow and extra step ( I already ran the sync_midtier_db command)
Regards,
Roberto
Pete Bunning - 23rd January 2017
Hi Roberto
There shouldn’t be any reason why the Weblogic login would not work. Can you log in OK when all three OBIS servers are up?
Check the logs and see what’s going wrong. Then check on My Oracle Support -there are quite a few notes on login issues.
Good Luck!
Pete
Mohan - 16th February 2017
Hi Pete,
First of all, thanks to your detailed blog. I am facing an issue which I would want to discuss here.
After unpacking the pack.jar file on the secondary node, the obis2 service is down with the below errors;
[ERROR:1] [] [] [ecid: 000001VROykF^6LqyOEgMG00kW2w00000n,0] [sik: service1] [tid: 3334] [nQSError: 13042] Repository password is wrong.
[nQSError: 61004] File error: path=/obilsappbin/orabase/middleware/Oracle_Home/bi/bifoundation/server/customizations/liverpd.rpd, code=-1: The specified file was not found.
I have modified the bi-environment.xml file and then created the pack.jar to unpack in the secondary node.
Unable to figure out how still it is referring the local path. I am facing this error for the second time. When I faced it the first time, I literally uninstalled everything on both servers and reinstalled all the stuff.
Request your help to resolve this issue.
Pete Bunning - 16th February 2017
Hi Mohan
Is the first server up and running and definitely using the share? And can the second server see the shared folder?
Pete
Mohan - 21st February 2017
Hi Pete,
Thanks for your reply! The first server is up and running. Below is the status shown on primary node;
bash-4.2$ ./status.sh
Domain status; Using domainHome: /obimxappbin/orabase/middleware/Oracle_Home/user_projects/domains/bi …
Initializing WebLogic Scripting Tool (WLST) …
Welcome to WebLogic Server Administration Scripting Shell
Type help() for help on available commands
/Servers/AdminServer/ListenPort=9700
Accessing admin server using URL t3://LSPRODEROBI12C2.LMGGI.COM:9700
Status of Domain: /obimxappbin/orabase/middleware/Oracle_Home/user_projects/domains/bi
NodeManager (LSPRODEROBI12C2.LMGGI.COM:9706): RUNNING
Name Type Machine Status
—- —- ——- ——
AdminServer Server LSPRODEROBI12C2.LMGGI.COM RUNNING
bi_server1 Server LSPRODEROBI12C2.LMGGI.COM RUNNING
bi_server2 Server LSPRODEROBI12C1.LMGGI.COM RUNNING
obips1 OBIPS LSPRODEROBI12C2.LMGGI.COM RUNNING
obijh1 OBIJH LSPRODEROBI12C2.LMGGI.COM RUNNING
obiccs1 OBICCS LSPRODEROBI12C2.LMGGI.COM RUNNING
obisch1 OBISCH LSPRODEROBI12C2.LMGGI.COM RUNNING
obis1 OBIS LSPRODEROBI12C2.LMGGI.COM RUNNING
obips2 OBIPS LSPRODEROBI12C1.LMGGI.COM RUNNING
obijh2 OBIJH LSPRODEROBI12C1.LMGGI.COM RUNNING
obiccs2 OBICCS LSPRODEROBI12C1.LMGGI.COM RUNNING
obisch2 OBISCH LSPRODEROBI12C1.LMGGI.COM RUNNING
obis2 OBIS LSPRODEROBI12C1.LMGGI.COM SHUTDOWN
The primary node is using shared folder because I have edited bi-environment.xml file to refer shared folder.
Second server can access the shared folder. Only difference I that I observe is that we have setup GPFS cluster shared path but NFS has been suggested in your blog
However, in obis2_diagnostics log I observe that second server is pointing to local path though the bi_environment.xml file is set properly.
Below are the error messages present in obis2 diagnostics log;
[nQSError: 61004] File error: path=/obilsappbin/orabase/middleware/Oracle_Home/bi/bifoundation/server/customizations/liverpd.rpd, code=-1: The specified file was not found.
[2017-02-20T15:03:25.786+05:30] [OBIS] [ERROR:1] [] [] [ecid: ] [sik: service1] [tid: 1] Oracle BI Server startup failed.[nQSError: 61004] File error: path=/obilsappbin/orabase/middleware/Oracle_Home/bi/bifoundation/server/customizations/liverpd.rpd, code=-1: The specified file was not found. [[
file: server/NQSNative/Driver/Src/NQDDriverImpl.cpp; line: 488
I deleted the secondary node completely and setup clustering twice as per steps suggested in the blog. Still facing the issue. I have setup two clusters earlier in lower environments as per the blog and both were successful. Unable to figure out the present issue.
Please let me know if any other details are required.
Thanks!
Mohan BN - 24th February 2017
Hi Pete,
The issue was with GPFS shared mount point configuration. We set a new NFS shared mount point. I deleted the secondary machine and did the setup of horizontal cluster once again. All services were UP properly. Even metadata refresh was consistent.
Thanks!
Pete Bunning - 13th March 2017
Excellent news Mohan
Mohamed - 12th March 2017
Many thanks for your great effort , but suppose that I do not have a shared storage , or I do not need to use it to not be a single point of failure, how can I make 2 nodes cluster without shared using shared storage? is there a note for that ?
thanks
Pete Bunning - 13th March 2017
Hi Mohamed
The cluster will only work if both nodes can access the RPD, Web Catalog etc. so they would have to be shared somehow. Otherwise you might as well just run two separate instances of OBIEE and replicate the contents across somehow to keep them in sync.
Could you set up the share on the main server? We did that when we were testing clustering on the new version. Just create an NFS share on one server and share it with the other.
Pete
Mohamed - 13th March 2017
Thanks for quick replay , You are better than Oracle Support, we made a shared storage , and it succeeded for installation , but I have now 2 URLs for BI , is there a Note in Oracle explain who to use a Load Balancer and make one URL for the 2 servers?
Pete Bunning - 14th March 2017
Hi Mohamed
All (or most at least) of the load balancer setup is in the Enterprise Deployment guide – which is quite a read!
For hardware load balancer requirements: Requirements
For general config: Prepare Load Balancer
Pete
Mohamed - 15th March 2017
Many thanks Pete
Reddy - 16th March 2017
I have installed cluster(node 1 and node 2) as per steps provided by you. In node 2 few services(obis2,obips2,bi_server2) are not running and showing shutdown state with the following errors.
1.The server ‘obis2’ with process id 3370 is no longer alive; waiting for the process to die.Server failed during startup. It may be retried according to the auto restart configuration.Server failed but will not be restarted because the maximum number of restart attempts has been exceeded.
2. The server ‘obips2’ with process id 3235 is no longer alive; waiting for the process to die.
3.The server ‘bi_server2’ with process id 4781 is no longer alive; waiting for the process to die
A MultiException has 2 exceptions. They are:
1. java.lang.AssertionError: Could not obtain the localhost address. The most likely cause is an error in the network configuration of this machine.
2. java.lang.IllegalStateException: Unable to perform operation: post construct on weblogic.rjvm.RJVMService
Please help me how to resolve the issue.
Pete Bunning - 17th March 2017
Hi Reddy
That looks to me like a problem with the hosts file. Can you check that and make sure you have all the right entries?
Pete
Reddy - 20th March 2017
Thanks a lot Pete. After entering right entries in /etc/hosts file , all the services running fine except obis2 service failed with below error .
The server ‘obis2’ with process id 10036 is no longer alive; waiting for the process to die
Server failed during startup. It may be retried according to the auto restart configuration
Server failed but will not be restarted because the maximum number of restart attempts has been exceeded>
Reddy - 21st March 2017
Hi Pete,
Once I given full permission to NFS singleton data directory, the server obis2 is running. Thanks..
Reddy - 21st March 2017
Hi Pete,
I can able to call all the urls from Node 1(console,em,Analytics,BI publisher) and able to call Analytics url from Node 2. But when I call the Publisher from Node 2 , I can login but as getting error ”
Error
The server can not be used due to a configuration error, please contact the administrator. If you are the administrator, please consult BI Publisher user guide for proper configuration”. oracle.xdo.servlet.resources.ResourceNotFoundException: ${xdo.server.config.dir}/repository/Admin/Security/principals.xml
I can see the principals.xml file in Node 1(Primary Machine). do we need to perform any post installation steps to fix the issue.
Thanks,
Hanu
narendra - 21st May 2017
I am also facing the same issue. if you resolved this, please help me get rid of this error
Pete Bunning - 22nd May 2017
Hi Reddy
Sorry, did you resolve this? I’m not aware of any extra steps for BIP and it works fine on the installations I have running.
You said you can see principals.xml from Node 1 but can you see from the OS on Node 2? It should be on the share with the other shared resources (RPD, web cat etc.)
Pete
Ratnadeep - 26th May 2017
Hi Pete/ Reddy,
I’m currently working on similar setup of OBIEE 12c, facing the similar error of missing principals.xml file when logging into the BIP portal on the node2, Can you please let me know if you have got any workarounds/fixes to this issue.
Thanks
Venkat - 15th June 2017
Hi, I had this issue after I made some changes to xmlp-server-config file so solution was just replace this modified file with original config file it’s working fine
Venkat
David - 17th May 2017
Ever run into an issue like the below?
Problem invoking WLST – Traceback (innermost last):
File “/opt/ssr/oracle/obiee12c/bi/modules/oracle.bi.sysman/syscomp/cloneBIMachine.py”, line 16, in ?
File “/opt/ssr/oracle/obiee12c/bi/lib/bi-sysman-syscomps-impl.jar!/wlstScriptDir/biSystemComponents.py”, line 149, in cloneBIMachine
File “/opt/ssr/oracle/obiee12c/bi/lib/bi-sysman-syscomps-impl.jar!/wlstScriptDir/biSystemComponents.py”, line 325, in _cloneSystemComponent
AttributeError: __getitem__
Error: Could not create clone to new machine
Pete Bunning - 22nd May 2017
Hi David
Have you got any more details from the logs?
So far most problems I’ve seen were caused by fairly obvious things like connection issues or share permissions etc. Apart from that it works (surprisingly) well.
So my best guess is to just confirm that’s all working properly.
Pete