My Blog

OBIEE 12c – Horizontal Scaling Part 2

by Pete Bunning on 19th January 2016 53 comments

Yesterday I went though the first stage in setting up clustering on OBIEE 12c which you can read here.

Today I want to finish off by creating the OBIEE domain on the second server.  Oracle have made this easy by creating a process for essentially ‘zipping’ up the configuration files to create a domain template, which can then be used to set up the second machine.

Creating the Configuration File

The first step is to log into the first server and shut down OBIEE using the script:


Then run the following command to create the template file:

<DOMAIN_HOME>/bitools/bin/ <IP Address> <NFS_Share>/pack.jar

Here the IP Address should be the one for the second server.  Also, I’ve used the NFS share to store the output file pack.jar.  That just saves me time FTP’ing the file over in the next step.

If everything runs you should get a message similar to this:

Copy pack file to remote machine and run the following on the remote machine to create the DOMAIN_HOME:
<OBIEE_HOME>/oracle_common/common/bin/ -domain=<DOMAIN_HOME> -template=<NFS_Share>/pack.jar

Configuring the second server

The next step is move across to the second server and run the command that you received in the previous step.  i.e:

<DOMAIN_HOME/oracle_common/common/bin/ -domain=<DOMAIN_HOME> -template=<NFS_Share>/pack.jar

Then, start up the node manager using:

<DOMAIN_HOME>/bin/ &

And that’s it for the second server!

Now, all that’s left is to restart OBIEE on the first server using:



Pete Bunning

Pete BunningOBIEE 12c – Horizontal Scaling Part 2

Related Posts

Take a look at these posts


Join the conversation
  • Vinay - 15th February 2016 reply

    I am seeing the below message in my secondary server after the scaleout.
    [2016-02-15T09:45:34.434-05:00] [OBIS] [NOTIFICATION:1] [] [] [ecid: ] [sik: ssi] [tid: 1338] [13079] Started reading RPD credentials for key metadata.encryption from CSF.
    [2016-02-15T09:45:34.466-05:00] [OBIS] [ERROR:1] [] [] [ecid: ] [sik: ssi] [tid: 1338] [46137] CSF error encountered. Error code: 43131. [[
    file: server\Utility\Generic\Src\SUGCSF.cpp; line: 133

    Please advise

    Vinay - 15th February 2016 reply

    Running sync_midtier_db.cmd resolved the issue. Please add it to your blog if possible.

  • Pete Bunning - 13th September 2016 reply

    Hi Vinay

    I’ve never seen this problem but it seems to be another step in the instructions. There’s a quick reference on the Oracle Support site: OBIEE 12c Quick Reference: How To Cluster | Scale-out OBIEE 12c Including Examples (Doc ID 2101898.1)


  • Ömer Naci - 14th October 2016 reply

    Hi Pete,
    You wrote a successful article.
    Two errors present in the final steps. The main problem does not start OBIPS and OBIS service in this second machine.
    obis2 :
    [2016-10-13T12:42:06.465+03:00] [OBIS] [ERROR:1] [] [] [ecid: ] [sik: ssi] [tid: bf376720] [nQSError: 47024] The Oracle BI Server participates in a cluster but errors were found in the cluster configuration file. [[
    [nQSError: 47051] The Oracle BI Server is specified as participating in a cluster, but global cache file storage location, /public/global_cache, is invalid.
    obips2 :
    [2016-10-13T12:42:02+03:00] [OBIPS] [ERROR:1] [] [saw.catalog.local.initialize] [ecid: ] [tid: ] [SI-Name: ] [IDD-Name: ] [IDD-GUID: ] [userId: ] Error initializing/loading existing Catalog: /public/bidata/metadata/content/livewebcat.
    Another instance of Oracle Business Intelligence is trying to upgrade/update/create the catalog located at /public/bidata/metadata/content/livewebcat/root_startup.lck. Retry after it finishes.

    Pete Bunning - 14th October 2016 reply

    That looks to me like a problem with accessing the Share from the second host. Can you check that and then see if it will startup?

    Ömer Naci - 14th October 2016 reply

    Hi Pete,

    catalog and global_cache directory are in the first node(/public/bidata and /public/global_cache).
    On second node ,I can access the /public folder using “cd /public” command. So , there is no access problem NFS folder from second node.

    Pete Bunning - 14th October 2016 reply

    I still think it has to be something to do with the share. Can you write to that folder from the second server?

    Unless it’s something to do with the user. Were both instances installed by the same O/S user?

    If you prove that the share is OK then you could try rebuilding the domain. A good thing about 12 is that you can just delete the ./user_domains/bi folder and rerun the config with having to re-install the software.

    It’s a bit of a pain but it’s good practice!

  • Ömer Naci - 17th October 2016 reply

    Hi Pete,
    /etc/export file are updated in the following way: /public *(rw,sync) in first machine. The problem is solved. Thanks for suggestions.

  • Roberto Contreras - 13th January 2017 reply

    Hello Pete,

    Thanks for your great tutorial.
    However, I`ve a question since Im` new in this OBIEE World.
    In the part 1 tutorial you said “The second server just needs the Weblogic and OBIEE software”. Therefore I would like to know if I need to install Fusion and OBIEE ( just to the part 1 that installs the basic folders) or if I have to move some specific folders from the server 1 installation. If that`s the case, would you please let me know which folder do I need to move in order to implement Server 2.


    Pete Bunning - 16th January 2017 reply

    Hi Roberto

    The second server only needs the infrastructure and OBIEE software – just the basic installs. So there’s no need to move folders manually.

    The that you use will copy the Weblogic configuration and everything else over for you.



    Roberto Contreras - 16th January 2017 reply

    Hello Pete,

    Thanks foor your response.
    I followed your instructions and installed just the basic Weblogic and OBIEE configuration ( Part 1 and 2 of your OBIEE tutorial) on the 2nd server.

    Then I made a clone on machine 1, move it to machine 2.
    However when i ran the unpack command I got the following error:

    >> warning:write Domain to “/u01/app/obiee/user_projects/domains/bi”
    >> 40317: Invalid “Servers, Clusters and Machines” configuration.
    40317: The “Servers, Clusters and Machines” configuration in your domain is invalid.
    40317: Correct the configuration.
    >> warning:write Domain to “/u01/app/obiee/user_projects/domains/bi”
    >> 40326: Invalid coherence server address.
    40326: At least one server in a Coherence cluster should have non localhost address.
    40326: Enter a non localhost listen address for at least one Coherence server.

    Kindly, do you know what might be the issue?


    Roberto Contreras - 16th January 2017 reply

    One thing I can add is that I ran the clone command two times, so maybe that could be the issue…I`m investigating on how to delete machines from the main configuration…


    Pete Bunning - 17th January 2017 reply

    It’s hard to say without looking into it a bit more but check that the IP addresses for the two servers are valid. And make sure you’re creating the 2nd BI machine for the second IP address.

    This setup should be fairly straightforward and when it does go wrong it’s usually something daft that I’ve done.

    Roberto Contreras - 17th January 2017

    Hello Pete,

    You were right, the problem was related to the iptables configuration on the 2nd server.
    In machine 2, I had to open ports with the following lines:

    -A INPUT -p tcp -m state –state NEW -m tcp –dport 9500 -j ACCEPT
    -A INPUT -p tcp -m state –state NEW -m tcp –dport 9502 -j ACCEPT
    -A INPUT -p tcp -m state –state NEW -m tcp –dport 9503 -j ACCEPT
    -A INPUT -p tcp -m state –state NEW -m tcp –dport 9506 -j ACCEPT

    Once i did that I restart node manager, then restart OBIEE in first server and everything works fine.

    Thanks Pete for this blog, it saves me a ton of time.

    If you come to Chile, please let me know so I can invite you some beers!


    Pete Bunning - 18th January 2017

    You’re on!

    Glad I could help Roberto.

  • Roberto Contreras - 18th January 2017 reply


    Thanks for all your help.
    I finished my setup with 3 machines in a OBIEE Cluster.
    However, Im now facing the following issue. As requested by our system administrator. they want to have only one Biservers up and running.Therefore I stopped obis1 and obis2 and only BIS3 is up and running.
    If y try to open the analytics URL everything looks great but when I try to login with weblogic credential I just cant login. Do you know if this is expected or do I need to follow and extra step ( I already ran the sync_midtier_db command)


    Pete Bunning - 23rd January 2017 reply

    Hi Roberto

    There shouldn’t be any reason why the Weblogic login would not work. Can you log in OK when all three OBIS servers are up?

    Check the logs and see what’s going wrong. Then check on My Oracle Support -there are quite a few notes on login issues.

    Good Luck!


  • Mohan - 16th February 2017 reply

    Hi Pete,
    First of all, thanks to your detailed blog. I am facing an issue which I would want to discuss here.
    After unpacking the pack.jar file on the secondary node, the obis2 service is down with the below errors;
    [ERROR:1] [] [] [ecid: 000001VROykF^6LqyOEgMG00kW2w00000n,0] [sik: service1] [tid: 3334] [nQSError: 13042] Repository password is wrong.
    [nQSError: 61004] File error: path=/obilsappbin/orabase/middleware/Oracle_Home/bi/bifoundation/server/customizations/liverpd.rpd, code=-1: The specified file was not found.

    I have modified the bi-environment.xml file and then created the pack.jar to unpack in the secondary node.
    Unable to figure out how still it is referring the local path. I am facing this error for the second time. When I faced it the first time, I literally uninstalled everything on both servers and reinstalled all the stuff.

    Request your help to resolve this issue.

    Pete Bunning - 16th February 2017 reply

    Hi Mohan

    Is the first server up and running and definitely using the share? And can the second server see the shared folder?


    Mohan - 21st February 2017 reply

    Hi Pete,

    Thanks for your reply! The first server is up and running. Below is the status shown on primary node;

    bash-4.2$ ./
    Domain status; Using domainHome: /obimxappbin/orabase/middleware/Oracle_Home/user_projects/domains/bi …

    Initializing WebLogic Scripting Tool (WLST) …

    Welcome to WebLogic Server Administration Scripting Shell

    Type help() for help on available commands

    Accessing admin server using URL t3://LSPRODEROBI12C2.LMGGI.COM:9700

    Status of Domain: /obimxappbin/orabase/middleware/Oracle_Home/user_projects/domains/bi

    Name Type Machine Status
    —- —- ——- ——

    The primary node is using shared folder because I have edited bi-environment.xml file to refer shared folder.
    Second server can access the shared folder. Only difference I that I observe is that we have setup GPFS cluster shared path but NFS has been suggested in your blog

    However, in obis2_diagnostics log I observe that second server is pointing to local path though the bi_environment.xml file is set properly.

    Below are the error messages present in obis2 diagnostics log;
    [nQSError: 61004] File error: path=/obilsappbin/orabase/middleware/Oracle_Home/bi/bifoundation/server/customizations/liverpd.rpd, code=-1: The specified file was not found.
    [2017-02-20T15:03:25.786+05:30] [OBIS] [ERROR:1] [] [] [ecid: ] [sik: service1] [tid: 1] Oracle BI Server startup failed.[nQSError: 61004] File error: path=/obilsappbin/orabase/middleware/Oracle_Home/bi/bifoundation/server/customizations/liverpd.rpd, code=-1: The specified file was not found. [[
    file: server/NQSNative/Driver/Src/NQDDriverImpl.cpp; line: 488

    I deleted the secondary node completely and setup clustering twice as per steps suggested in the blog. Still facing the issue. I have setup two clusters earlier in lower environments as per the blog and both were successful. Unable to figure out the present issue.

    Please let me know if any other details are required.


    Mohan BN - 24th February 2017 reply

    Hi Pete,

    The issue was with GPFS shared mount point configuration. We set a new NFS shared mount point. I deleted the secondary machine and did the setup of horizontal cluster once again. All services were UP properly. Even metadata refresh was consistent.


    Pete Bunning - 13th March 2017

    Excellent news Mohan

  • Mohamed - 12th March 2017 reply

    Many thanks for your great effort , but suppose that I do not have a shared storage , or I do not need to use it to not be a single point of failure, how can I make 2 nodes cluster without shared using shared storage? is there a note for that ?

    Pete Bunning - 13th March 2017 reply

    Hi Mohamed

    The cluster will only work if both nodes can access the RPD, Web Catalog etc. so they would have to be shared somehow. Otherwise you might as well just run two separate instances of OBIEE and replicate the contents across somehow to keep them in sync.

    Could you set up the share on the main server? We did that when we were testing clustering on the new version. Just create an NFS share on one server and share it with the other.


  • Mohamed - 13th March 2017 reply

    Thanks for quick replay , You are better than Oracle Support, we made a shared storage , and it succeeded for installation , but I have now 2 URLs for BI , is there a Note in Oracle explain who to use a Load Balancer and make one URL for the 2 servers?

    Pete Bunning - 14th March 2017 reply

    Hi Mohamed

    All (or most at least) of the load balancer setup is in the Enterprise Deployment guide – which is quite a read!

    For hardware load balancer requirements: Requirements

    For general config: Prepare Load Balancer


    Mohamed - 15th March 2017 reply

    Many thanks Pete

  • Reddy - 16th March 2017 reply

    I have installed cluster(node 1 and node 2) as per steps provided by you. In node 2 few services(obis2,obips2,bi_server2) are not running and showing shutdown state with the following errors.

    1.The server ‘obis2’ with process id 3370 is no longer alive; waiting for the process to die.Server failed during startup. It may be retried according to the auto restart configuration.Server failed but will not be restarted because the maximum number of restart attempts has been exceeded.
    2. The server ‘obips2’ with process id 3235 is no longer alive; waiting for the process to die.
    3.The server ‘bi_server2’ with process id 4781 is no longer alive; waiting for the process to die
    A MultiException has 2 exceptions. They are:
    1. java.lang.AssertionError: Could not obtain the localhost address. The most likely cause is an error in the network configuration of this machine.
    2. java.lang.IllegalStateException: Unable to perform operation: post construct on weblogic.rjvm.RJVMService

    Please help me how to resolve the issue.

    Pete Bunning - 17th March 2017 reply

    Hi Reddy

    That looks to me like a problem with the hosts file. Can you check that and make sure you have all the right entries?


  • Reddy - 20th March 2017 reply

    Thanks a lot Pete. After entering right entries in /etc/hosts file , all the services running fine except obis2 service failed with below error .
    The server ‘obis2’ with process id 10036 is no longer alive; waiting for the process to die
    Server failed during startup. It may be retried according to the auto restart configuration
    Server failed but will not be restarted because the maximum number of restart attempts has been exceeded>

  • Reddy - 21st March 2017 reply

    Hi Pete,
    Once I given full permission to NFS singleton data directory, the server obis2 is running. Thanks..

  • Reddy - 21st March 2017 reply

    Hi Pete,
    I can able to call all the urls from Node 1(console,em,Analytics,BI publisher) and able to call Analytics url from Node 2. But when I call the Publisher from Node 2 , I can login but as getting error ”

    The server can not be used due to a configuration error, please contact the administrator. If you are the administrator, please consult BI Publisher user guide for proper configuration”. oracle.xdo.servlet.resources.ResourceNotFoundException: ${xdo.server.config.dir}/repository/Admin/Security/principals.xml

    I can see the principals.xml file in Node 1(Primary Machine). do we need to perform any post installation steps to fix the issue.

    narendra - 21st May 2017 reply

    I am also facing the same issue. if you resolved this, please help me get rid of this error

    Pete Bunning - 22nd May 2017 reply

    Hi Reddy

    Sorry, did you resolve this? I’m not aware of any extra steps for BIP and it works fine on the installations I have running.

    You said you can see principals.xml from Node 1 but can you see from the OS on Node 2? It should be on the share with the other shared resources (RPD, web cat etc.)


    Ratnadeep - 26th May 2017 reply

    Hi Pete/ Reddy,

    I’m currently working on similar setup of OBIEE 12c, facing the similar error of missing principals.xml file when logging into the BIP portal on the node2, Can you please let me know if you have got any workarounds/fixes to this issue.


    Venkat - 15th June 2017 reply

    Hi, I had this issue after I made some changes to xmlp-server-config file so solution was just replace this modified file with original config file it’s working fine

  • David - 17th May 2017 reply

    Ever run into an issue like the below?

    Problem invoking WLST – Traceback (innermost last):
    File “/opt/ssr/oracle/obiee12c/bi/modules/”, line 16, in ?
    File “/opt/ssr/oracle/obiee12c/bi/lib/bi-sysman-syscomps-impl.jar!/wlstScriptDir/”, line 149, in cloneBIMachine
    File “/opt/ssr/oracle/obiee12c/bi/lib/bi-sysman-syscomps-impl.jar!/wlstScriptDir/”, line 325, in _cloneSystemComponent
    AttributeError: __getitem__

    Error: Could not create clone to new machine

    Pete Bunning - 22nd May 2017 reply

    Hi David

    Have you got any more details from the logs?

    So far most problems I’ve seen were caused by fairly obvious things like connection issues or share permissions etc. Apart from that it works (surprisingly) well.

    So my best guess is to just confirm that’s all working properly.


  • Gopi krishna Anabham - 28th October 2017 reply

    Can we create two managed servers,like
    bi_server1,bi_server2, if ok can you please share how to do that.

    Pete Bunning - 30th October 2017 reply

    Hi, the process of creating the 2nd cluster node will set up bi_server2 for you. Along with the all the other components needed for OBIEE to run on the second server.

    Run through it and see.


    Gopi krishna - 12th December 2017 reply

    Thanks for suggestion,

    I was created a domain, initially it has server bi_server1, I changed it to dev_msbie1 using wlst,
    when I run the script it is giving below error,
    Problem invoking WLST – Traceback (innermost last):
    File “/apps/oramw/products/fmw12213/bi/modules/”, line 16, in ?
    File “/apps/oramw/products/fmw12213/bi/lib/bi-sysman-syscomps-impl.jar!/wlstScriptDir/”, line 96, in cloneBIMachine
    File “/apps/oramw/products/fmw12213/bi/lib/bi-sysman-syscomps-impl.jar!/wlstScriptDir/”, line 305, in _assertValidChild
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
    at java.lang.reflect.Constructor.newInstance(

    java.lang.Exception: java.lang.Exception: No such server: bi_server1

    Error: Could not create clone to new machine


    Gopi krishna - 12th December 2017 reply

    IT is looking to default server name,
    please give me suggestions, but the change in server name dev_msbie1 is required

    appreciate quick response

    Pete Bunning - 12th December 2017

    Hi. Have you tried running the Weblogic Domain config tool?(/oracle_common/common/bin/

    That’s the safest way to do things like change the server names. Just start that and run through the screens until you find the one for editing the servers.

    Gopi krishna - 13th December 2017

    this issue is resolved by providing parameters like below,



    Gopi krishna - 13th December 2017


    Jayshree - 18th December 2017

    could you please provide exact syntax for cloneBIMachine(‘/apps/oramw/config/exc/domains/dev_biedomain’,’localhost’,b

    Gopi krishna - 26th December 2017


    Gopi krishna - 26th December 2017

    I don’t know I mentioned the exact syntax, but it was showing only like above only, sorry

  • Cezar Lopes - 1st November 2017 reply

    Thanks for this simple and powerfull tutorial.
    How about install FMW Infrastructure and OBIEE12c in the Share Folder that is stored on a NAS?
    I did it and I’m trying to scale out my Domain right now. I wonder if those steps are valid for this installation type.
    May the unpack script try to rewrite all or part of the installation directory?

    Pete Bunning - 9th November 2017 reply

    Hi Cezar

    Yes, the steps will work but there’s a bit of fiddling to do. You have to edit the config.xml file and do some renaming in the Weblogic domain etc.

    It’s a bit much to go through in a blog but it is possible. If fact I have used this method instead of the official Enterprise installation method.


    Cezar Lopes - 1st December 2017 reply

    Hi Pete

    I’m facing those fiddlings right now. My /bits scripts don’t work anymore an now I need then working ASAP, rsrsrs,, – All of then failed to connect to the domain but all things work fine through /em, /console and, of course, the traditional way to manage WeblLogic domains, but I must have those additional scripts running for other purposes.

    Pete Bunning - 5th December 2017 reply

    That’s the main reason I did the normal install and then adapted it to work for clustering.

    Without going into too much detail the main steps are:

    • Install Java (if necessary)
    • Install the Weblogic infrastructure
    • Install the OBIEE software
    • Configure the first OBIEE node using the non-enterprise method
    • Edit the configuration to set up the cluster correctly
    • Clone the first node to set up the second node

    And the configuration steps would be something like:

    Reset the Nodemanager to listen on the Admin VIP
    Using /oracle_common/common/bin/ (rather than the one you used before) do the following:
    Setup separate machines for BI Server and Admin Server
    Change the Listen Address for bi_server1 to be the physical IP address of the first cluster server
    Change the name of the machine to ADMINHOST and change its listen address to the Virtual IP address (VIP)
    Add a new machine – BIHOST1, set the listen address to the IP address of the first cluster server and change the Node Manager Listen Port to 9506
    Assign the servers to the right machines
    Finish off the config to make the changes

    Edit the file …/user_projects/domains/bi/config/config.xml
    Search for the string obips1 and you’ll find the five system components of OBIEE: obips1, obijh1, obiccs1, obisch1 and obis1. These will currently still be assigned to the ADMINHOST machine
    Change all five to the BIHOST1 machine
    Save and exit

    Now you should be able to clone that machine to make the second – using the IP address for the second server

    As I said, it’s a bit fiddly but it works and the control scripts all work perfectly to start and stop and give the status on all nodes etc.

    Good luck


  • Gopi krishna - 26th December 2017 reply

    HI i did the same process as you mentioned, bi_server1,bi_server2 and components obiccs1,obips1,obis1,obijh1,obisch1 are running fine, but components obis2,obips2,obiccs2,obijh2,obisch2 are not getting started they went to shutdown state,
    error is like below,
    row key might not be reset correctly at the end of the request. Component ID: :emT:avl_tt1 ViewId: /ai/bi/instance

Join the conversation