Skip to main content
Pure Technical Services

Best Practices for Oracle on FlashBlade

Currently viewing public documentation. Please login to access the full scope of documentation.

This article covers the best practices to be followed for Oracle databases hosted on FlashBlade to achieve better performance and simplify operations.

NFS Filesystems for Oracle

Networked filesystems provide the flexibility to mount the same filesystem multiple times on the same host or across multiple hosts, for the RMAN backup target, you can create a single NFS filesystem that can be mounted across the available subnets on the host.  Make sure the NFS protocol is selected when creating the filesystem on FlashBlade.

Use dNFS over Kernel NFS

To scale up the bandwidth on FlashBlade enable numerous connections from the client than a single connection.  Oracle’s Direct NFS creates a separate connection to the storage system for every server process as opposed to a single connection per mount point by Kernel NFS.  

Enable Parallelism

To increase the read and write bandwidth on FlashBlade, use client-level parallelization techniques like parallel queries, multiple RMAN channels based on the CPU availability on your host in conjunction with dNFS.  This increases the number of connections to FlashBlade, especially with dNFS.

Use Multiple Network Interfaces

To enhance the network bandwidth, make sure to have multiple network interfaces on the client.  These multiple interfaces can be configured on a single subnet or on multiple subnets.

Single subnet

IO performance on a dNFS environment with multiple interfaces and single subnet is limited to the speed of the first interface that is picked by the OS.  This is because the OS returns the first path when multiple paths are available on the subnet and hence the traffic is always routed through the first path.

For example, in the setup below, the FlashBlade has a single IP address on which the NFS filesystem will be mounted from the client and the client has two interfaces.

Client interface       NFS server
ens7f0 10.21.108.193   10.21.108.10
ens7f1 10.21.108.194   10.21.108.10

[oracle@rlxora-b01-11-06 ~]$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.21.108.1     0.0.0.0         UG    0      0        0 ens7f0
10.21.108.0     0.0.0.0         255.255.255.0   U     0      0        0 ens7f0 <-- first route
10.21.108.0     0.0.0.0         255.255.255.0   U     0      0        0 ens7f1 <-- route ignored 


As per the routing table, both traffic can go through the first interface (ens7f0) as the destination and the mask fits for both routes and OS will invariably choose the first route.

In this case, to enhance the bandwidth, it is recommended to use NIC bonding at the client level.  Oracle support document (Doc ID 833481.1) provides other means to enable multiple paths in a single subnet using static routing but it doesn’t address the issue with availability when a NIC fails or network cable is pulled as the routing table will not be updated.

Multiple subnets

“Direct NFS client best practices recommend always to use multipaths in separate subnets”.

Oracle recommends using separate subnet for each interface and it supports up to four subnets. With multiple subnets, there is no need to bond the network interfaces to aggregate the bandwidth across the available interfaces. The routing will be automatic in the case of multiple subnets.


Client interface       NFS server
ens7f0 10.21.108.193   10.21.108.10
ens7f1 10.21.107.194   10.21.107.10

[oracle@rlxora-b01-11-06 ~]$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.21.108.1     0.0.0.0         UG    0      0        0 ens7f0
10.21.108.0     *               255.255.255.0   U     0      0        0 ens7f0
10.21.107.0     *               255.255.255.0   U     0      0        0 ens7f1 


In this case, these are two dynamic routes and based on the traffic, the route is selected automatically.

As such, if you decide to use multiple subnets, it should be configured at both the client and the FlashBlade side as well. Multiple subnets can be configured in FlashBlade GUI under the Network Settings.

Make sure to update the oranfstab with the subnets and the mount point details. In RAC environment, all RAC nodes should have the appropriate oranfstab file configured.

NFS volumes and Mount point requirements

In the case of multiple subnets, it is not required to have as many NFS filesystems/volumes as the subnets for dNFS to be effective.  Also, it is not required to mount a single volume on to all subnets for dNFS to be effective either.  Oracle dNFS reads the oranfstab and based on the storage paths and mount details, it will create multiple paths when the database files are accessed.

For example, with two subnets and two mounts, dNFS would create four paths to the storage system for every server process. 

Linux Mount options

For mounting the NFS filesystem on Linux, use the following mount options.  Do not specify the rsize, wsize options as the system can get the default offered by FlashBlade which is 524288).

rw,bg,nointr,hard,tcp,vers=3,actimeo=0

Note: The actimeo=0 is required for Oracle RAC databases to turn off attribute caching forcing the NFS client to look for the attributes like atime, ctime, file permissions, owner etc., from the server every time.  While this may increase the number of NFS operations, it guarantees the integrity of the file attributes across the RAC nodes.   For non-RAC Oracle databases, you can leave the default actimeo and it is a not a requirement to set it to 0.

Even for dNFS, the filesystem(s) should be mounted using the above mount options.  To persist these changes across reboots, please include them in the /etc/fstab file as given below. The IP address specified below refers to the data VIP from the FlashBlade.

10.21.214.100:/oradata01 /u02 nfs rw,bg,nointr,hard,tcp,vers=3,actimeo=0

For more details on the Mount Options for Oracle files for RAC databases when used with NFS, please see the Oracle Support document Doc ID 359515.1.

Enabling Oracle Direct NFS (dNFS) client

To enable Oracle dNFS on a database host, perform the following steps.

1. Shut down the running instance of the Oracle database software.

2. Change the directory to $ORACLE_HOME/rdbms/lib

3. Enable dNFS by running the following command.

$ make -f ins_rdbms.mk dnfs_on
  • The server parameter refers to the name of the FlashBlade storage.  This name is for informational use only and not validated.
  • The local parameter refers to the paths on the database host, specified either by IP address or by name as displayed using the ifconfig command
  • The path parameter should reflect the data VIP address(es) of the FlashStorage.
    • Oracle supports up to 4 paths
  • The nfs_version should refer to nfsv3 as at this time FlashBlade supports only the V3 of the NFS protocol.
  • The export parameters should reflect the mount points similar to the entries created in /etc/fstab but do not include any mount options.
     
  • Keywords (server, local, path, nfs_version, export, mount) should be immediately followed with ":" without any spaces.
  • The values should have a space after the ":"
  • Always group local and path together on the same line.  Similarly group export and mount together on the same line.
server: fb-dnfs-test-01
local: 192.168.20.50 path: 192.168.20.100
nfs_version: nfsv3
export: /oradata01 mount: /u02
export: /oradata02 mount: /u03
export: /redo01    mount: /u04

server: fb-dnfs-test-02
local: 192.168.10.25 path: 192.168.10.200
nfs_version: nfsv3
export: /rman01 mount: /r01
export: /rman02 mount: /r02

Sample oranfstab for multiple data VIP at the FlashBlade level.  Limit to four paths.

server: fb-dnfs-test-01
local: 192.168.20.50 path: 192.168.20.100
local: 192.168.20.50 path: 192.168.20.101
local: 192.168.20.50 path: 192.168.20.102
local: 192.168.20.50 path: 192.168.20.103
nfs_version: nfsv3
export: /oradata01 mount: /u02
export: /oradata02 mount: /u03
export: /redo01    mount: /u04

5. Restart the Oracle database instance(s). 

Note: The mount points referred to in the oranfstab file should be mounted on the system(s) by invoking the mount command directly or more commonly through an entry in the /etc/fstab file.  

Disabling the Oracle Direct NFS (dNFS) client

1. Shut down the running instance of the Oracle database software.

2. Change the directory to $ORACLE_HOME/rdbms/lib.

3. Disable dNFS by running the following command.

make -f ins_rdbms.mk dnfs_off

4. Restart the Oracle database instance(s). 

 

Verifying the use of Oracle Direct NFS client

1) If dNFS is enabled, the alert.log will show the following entry when the database is started.

Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 4.0

2) Query the dNFS server information from v$dnfs_servers view inside the database.

SQL> select svrname, dirname, mntport, nfsport, wtmax, rtmax from v$dnfs_servers;

SVRNAME            DIRNAME                 MNTPORT    NFSPORT      WTMAX      RTMAX
------------------ -------------------- ---------- ---------- ---------- ----------
fb-dnfs-test-02    /rman01                    2049       2049     524288      524288

Note: Even though dNFS is enabled, Oracle only mounts the volume/filesystem and opens the files when they are accessed. If no data files are accessed, then the above view will return no rows.

Troubleshooting oranfstab issues

Please make sure the svrname from the v$dnfs_servers show the server name from the oranfstab file.  If not, generally there is some misconfiguration in the oranfstab file.  Look for any dangling entries, lack of space after the keywords and clean up.

For example, in the following output the SVRNAME is not correct as it should be the actual server name from the oranfstab for all the exports under the given server but we see the IP addresses of the paths.  This means the oranfstab is not configured correctly.

SQL> SELECT svrname, dirname, mntport, nfsport, wtmax, rtmax FROM v$dnfs_servers;

Server                    Export Name                                                         MNTPORT    NFSPORT      WTMAX      RTMAX  
------------------------- ---------------------------------------------------------------- ---------- ---------- ---------- ----------  
192.168.20.100            /oradata01                                                             2049       2049     524288     524288  
192.168.20.101            /oradata02                                                             2049       2049     524288     524288 
192.168.20.102            /redo01                                                                2049       2049     524288     524288  

Looking at the oranfstab file, we see the following. 

server: fb-dnfs-01
local: 192.168.20.50
path: 192.168.20.100
local: 192.168.20.50
path: 192.168.20.101
local: 192.168.20.50
path: 192.168.20.102
local: 192.168.20.50
nfs_version: nfsv3
export: /oradata01 mount: /u02
export: /oradata02 mount: /u03
export: /redo01    mount: /u04

After rearranging them and removing the dangling local entry from the oranfstab file and restarting the database shows the expected output.

server: fb-dnfs-01
local: 192.168.20.50 path: 192.168.20.100
local: 192.168.20.50 path: 192.168.20.101
local: 192.168.20.50 path: 192.168.20.102

nfs_version: nfsv3
export: /oradata01 mount: /u02
export: /oradata02 mount: /u03
export: /redo01    mount: /u04
SQL> SELECT svrname, dirname, mntport, nfsport, wtmax, rtmax FROM v$dnfs_servers;

Server                    Export Name                                                         MNTPORT    NFSPORT      WTMAX      RTMAX 
------------------------- ---------------------------------------------------------------- ---------- ---------- ---------- ---------- 
fb-dnfs-01                /oradata01                                                             2049       2049     524288     524288  
fb-dnfs-01                /oradata02                                                             2049       2049     524288     524288
fb-dnfs-01                /redo01                                                                2049       2049     524288     524288