Oracle Database Appliance // Virtualization // Shared Repositories

Today I’vh setup my ODA as virtualized platform which is possible since 2.5 Image. I’vh done my installation with the latest stable release (2.9) which is available right now. I want you show you how setup an shared repository with is possible since 2.8 image. First the theory:


I will show from setup the repository until the installation of the virtual machine:

1. Create a shared repository

Show actual configuration:

>oakcli show repo

          NAME                          TYPE            NODENUM         STATE
          odarepo1                      local           0               N/A       
          odarepo2                      local           1               N/A    

Create repository shared1 in Diskgroup +RECO with the size of 250GB:

>oakcli create repo shared1 -dg reco -size 250

Created Shared Repo : SharedRepoType

Show new configuration:

>oakcli show repo       
          NAME                          TYPE            NODENUM         STATE                                                                
          odarepo1                      local           0               N/A 
          odarepo2                      local           1               N/A 
          shared1                       shared          0               ONLINE                                                               
          shared1                       shared          1               ONLINE                   

Show details:

akcli show repo shared1 -node 0
Resource: shared1_0
        AutoStart       :       restore        
        DG              :       RECO           
        Device          :       /dev/asm/shared1-72
        ExpectedState   :       Online         
        MountPoint      :       /u01/app/sharedrepo/shared1
        Name            :       shared1_0      
        Node            :       all            
        RepoType        :       shared         
        Size            :       256000         
        State           :       Online        

2. Check the implementation on the oda_base nodes

>showmount -e
Export list for oda_base_node0:

>showmount -e
Export list for oda_base_node1:

ATTENTION: The exports are done dynamically by oak. If you restart nfs service on ODA_BASE the configuration is lost until oak will reapply. >cat /etc/exports will be empty!

3. Check the implementation in vm hosts

[root@odavm01 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md1               19G  1.1G   17G   6% /
tmpfs                 1.7G     0  1.7G   0% /dev/shm
/dev/md3              519G  209G  284G  43% /OVS
/dev/md0              487M   41M  421M   9% /boot
none                  1.7G  112K  1.7G   1% /var/lib/xenstored
                      250G  649M  250G   1% /OVS/Repositories/shared1

[root@odavm02 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/md1               19G  1.1G   17G   6% /
tmpfs                 1.7G     0  1.7G   0% /dev/shm
/dev/md3              519G  181G  311G  37% /OVS
/dev/md0              487M   41M  421M   9% /boot
none                  1.7G  112K  1.7G   1% /var/lib/xenstored
                      250G  649M  250G   1% /OVS/Repositories/shared1

ATTENTION: The NFS mount point will be mounted with oracle best practice options, so this mounts are HARD mounts which will be a problem on simple reboot! on /OVS/Repositories/shared1 type nfs (rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,nfsvers=3,timeo=600,addr= on /OVS/Repositories/shared1 type nfs (rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,nfsvers=3,timeo=600,addr=

So please stop all VMs, execpt oda_base on vm host before reboot. Otherwise the reboot will hang on unmount NFS mountpoint!

4. Create vm template

There is nothing special, on local node import. So straight forward:

>oakcli import vmtemplate w2k8_template -assembly /tmp/stage/w2k8_template.ova -repo odarepo1

Imported VM Template 

>oakcli show vmtemplate

        NAME                                            MEMORY          VCPU            REPOSITORY
        w2k8_template                                   4096               1            odarepo1            

5. Create virtual machine

>oakcli clone vm w2k8 -vmtemplate w2k8_template -repo shared1 -node 0

Cloned VM Template 

We have created a virtual machine which starts only on NODE 0 and is located on shared Repository “shared1”

5. Configure virtual machine

>oakcli show vm w2k8
Resource: w2k8
        AutoStart       :       restore        
        CPUPriority     :       100            
        Disks           :       |file:/OVS/Repositories/shared1/Vir
        Domain          :       XEN_PVM        
        ExpectedState   :       online         
        FailOver        :       true           
        IsSharedRepo    :       true           
        Keyboard        :       de             
        MaxMemory       :       8192           
        MaxVcpu         :       4              
        Memory          :       8192           
        Mouse           :       USB_MOUSE      
        Name            :       w2k8      
        Networks        :       |type=netfront,bridge=net1||type=ne
        NodeNum         :       1              
        NodeNumStart    :       1              
        OS              :       WIN_2008       
        PrefNodeNum     :       0              
        PrivateIP       :       None           
        ProcessorCap    :       100            
        RepoName        :       shared1        
        State           :       Online         
        TemplateName    :       otml_w2k8_template
        Vcpu            :       4              
        cpupool         :       reports_pool   
        vncport         :       0              

You may configure your vm now.

ATTENTION: During clone of vm you may run in a BUG which causes one of both Nodes a Oracle Cluster Crash of ASM, the Private Interconnect may be not responed anymore until you force umount the ACFS Respository on vm host!
You may get the following:

64 bytes from icmp_seq=675 ttl=64 time=0.347 ms
64 bytes from icmp_seq=676 ttl=64 time=0.290 ms
64 bytes from icmp_seq=677 ttl=64 time=0.298 ms
ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available
ping: sendmsg: No buffer space available


nfs: server not responding, still trying
nfs: server not responding, still trying
INFO: task cp:29088 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
cp              D 0000000000000000     0 29088  29087 0x00000000
 ffff880146cddb68 0000000000000282 ffff880146cdda98 0000000000012180
 ffff88015121c240 ffff880154ebe2c0 ffff880146cddb28 ffff8801011f3bc0
 ffff880146cddab8 ffffffff810410c1 ffff880146cddab8 00000000009824aa
Call Trace:
 [] ? pvclock_get_nsec_offset+0x11/0x50
 [] ? pvclock_clocksource_read+0x29/0x80
 [] ? xen_clocksource_read+0x20/0x30
 [] ? xen_clocksource_get_cycles+0x9/0x10
 [] ? ktime_get_ts+0xb1/0xf0
 [] schedule+0x45/0x60
 [] io_schedule+0x71/0xb0
 [] sleep_on_page+0xe/0x20
 [] __wait_on_bit+0x5e/0x90
 [] ? __lock_page+0x80/0x80
 [] wait_on_page_bit+0x75/0x80
 [] ? autoremove_wake_function+0x50/0x50
 [] ? pagevec_lookup_tag+0x27/0x40
 [] filemap_fdatawait_range+0xf7/0x180
 [] ? nfs_file_direct_read_iter+0x50/0x50 [nfs]
 [] filemap_fdatawait+0x2b/0x30
 [] writeback_single_inode+0x1ce/0x230
 [] sync_inode+0x4a/0x80
 [] nfs_wb_all+0x46/0x50 [nfs]
 [] nfs_setattr+0x12f/0x140 [nfs]
 [] notify_change+0x18b/0x2f0
 [] do_truncate+0x63/0x90
 [] do_sys_ftruncate+0x114/0x130
 [] sys_ftruncate+0x13/0x20
 [] system_call_fastpath+0x16/0x1b


This issue is caused by known kernel bug 17896838, which is fixed in kernel 2.6.39-214.1 and above. ODA July ( release will come with the new kernel that includes the fix. In the meantime, the workaround is to set or increase the dom0_mem in /boot/grub/grub.conf to 8192M on dom0 then reboot the dom0:

kernel /xen.gz dom0_mem=8192M crashkernel=256M@64M

In cause of my SR there have been raised a note you can refer to:
ODA DOM0 Lost Network Communications Between Nodes While running ‘okacli clone vm’ or Copy (cp) Commands of Large Files over NFS (Doc ID 1665055.1)

6. Done, now start up vm and have fun!

>oakcli show vm

          NAME                                  NODENUM         MEMORY          VCPU            STATE           REPOSITORY
          w2k8                                  1               8192               4            ONLINE          shared1                  

8 thoughts on “Oracle Database Appliance // Virtualization // Shared Repositories

  1. after creating a repo,if i check the oakcli show command for that repo,the free space is not 100% but it is less than 100.Why is that happening.I did not create any vdisk or anything.

    • Hi Meghana,
      the shared repo is created as Oracle CloudFS or ACFS. The filesystem, as every other filesystem, has an overhead of less then 2%. May you provide more information:

      As GI owner (normaly GRID):

      acfsutil info fs
      df -m
      asmcmd volinfo --all


      • Where actually are the shared repos getting their memory from? While configuring the oda_base we give some memory,what is that?Arent the shared repos using that memory?

      • Hi Meghana,
        The ACFS shared repo is using memory as all other filesystems (os cache, kernel module memory). ASM is using additional Memory configured by ASM Instance Parameters (e.g. sga_target,memory_target).

        The ODA_BASE is the inital VM for your Oracle Environment so the memory assigned to it is the memory you need for your databases or cluster and and and …

        Futher you can assign memory for the Dom0 (the Hypervisor). This should be between 6-8GB on X3-2/X4-2/X5-2. This depends on the memory assigned to the Machine.


  2. Thanks, daniel.Can we know how much memory is still left out for creating the shared repos.Is there any command?

    • As already written, is the memory still as for normal OS Filesystems. Normally you should size your system by a total usage of all applications by ~80-85%. The rest should be free for the os stack and cache.

      To check the os cache usage you can use “free -m” or an equivalent tool.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s