Skip to main content
Pure1 Support Portal

Best Practices for Oracle on FlashBlade

This article covers the best practices to be followed for Oracle databases hosted on FlashBlade to achieve better performance and simplify operations.

 

NFS Filesystems for Oracle

Networked filesystems provide the flexibility to mount the same filesystem multiple times on the same host or across multiple hosts, for the RMAN backup target, you can create a single NFS filesystem that can be mounted across the available subnets on the host.  Make sure the NFS protocol is selected when creating the filesystem on FlashBlade.

Use dNFS over Kernel NFS

To scale up the bandwidth on FlashBlade enable numerous connections from the client than a single connection.  Oracle’s Direct NFS creates a separate connection to the storage system for every server process as opposed to a single connection per mount point by Kernel NFS.  

Enable Parallelism

To increase the read and write bandwidth on FlashBlade, use client-level parallelization techniques like parallel queries, multiple RMAN channels based on the CPU availability on your host in conjunction with dNFS.  This increases the number of connections to FlashBlade, especially with dNFS.

Use Multiple Network Interfaces

To enhance the network bandwidth, make sure to have multiple network interfaces on the client.  These multiple interfaces can be configured on a single subnet or on multiple subnets.

Single subnet

IO performance on a dNFS environment with multiple interfaces and single subnet is limited to the speed of the first interface that is picked by the OS.  This is because the OS returns the first path when multiple paths are available on the subnet and hence the traffic is always routed through the first path.

For example, in the setup below, the FlashBlade has a single IP address on which the NFS filesystem will be mounted from the client and the client has two interfaces.

Client interface       NFS server
ens7f0 10.21.108.193   10.21.108.10
ens7f1 10.21.108.194   10.21.108.10

[oracle@rlxora-b01-11-06 ~]$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.21.108.1     0.0.0.0         UG    0      0        0 ens7f0
10.21.108.0     0.0.0.0         255.255.255.0   U     0      0        0 ens7f0 <-- first route
10.21.108.0     0.0.0.0         255.255.255.0   U     0      0        0 ens7f1 <-- route ignored 


As per the routing table, both traffic can go through the first interface (ens7f0) as the destination and the mask fits for both routes and OS will invariably choose the first route.

In this case, to enhance the bandwidth, it is recommended to use NIC bonding at the client level.  Oracle support document (Doc ID 833481.1) provides other means to enable multiple paths in a single subnet using static routing but it doesn’t address the issue with availability when a NIC fails or network cable is pulled as the routing table will not be updated.

Multiple subnets

“Direct NFS client best practices recommend always to use multipaths in separate subnets”.

Oracle recommends using separate subnet for each interface and it supports up to four subnets. With multiple subnets, there is no need to bond the network interfaces to aggregate the bandwidth across the available interfaces. The routing will be automatic in the case of multiple subnets.


Client interface       NFS server
ens7f0 10.21.108.193   10.21.108.10
ens7f1 10.21.107.194   10.21.107.10

[oracle@rlxora-b01-11-06 ~]$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.21.108.1     0.0.0.0         UG    0      0        0 ens7f0
10.21.108.0     *               255.255.255.0   U     0      0        0 ens7f0
10.21.107.0     *               255.255.255.0   U     0      0        0 ens7f1 


In this case, these are two dynamic routes and based on the traffic, the route is selected automatically.

As such, if you decide to use multiple subnets, it should be configured at both the client and the FlashBlade side as well.  Multiple subnets can be configured in FlashBlade GUI under the Network Settings.

Make sure to update the oranfstab with the subnets and the mount point details.  In RAC environment, all RAC nodes should have the appropriate oranfstab file configured.

NFS volumes and Mount point requirements

In the case of multiple subnets, it is not required to have as many NFS filesystems/volumes as the subnets for dNFS to be effective.  Also, it is not required to mount a single volume on to all subnets for dNFS to be effective either.  Oracle dNFS reads the oranfstab and based on the storage paths and mount details, it will create multiple paths when the database files are accessed.

For example, with two subnets and two mounts, dNFS would create four paths to the storage system for every server process. 

Linux Mount options

For mounting the NFS filesystem on Linux, use the following mount options.  Do not specify the rsize, wsize options as the system can get the default offered by FlashBlade which is 524288).

rw,bg,nointr,hard,tcp,vers=3,actimeo=0

Note: The actimeo=0 is required for Oracle RAC databases to turn off attribute caching forcing the NFS client to look for the attributes like atime, ctime, file permissions, owner etc., from the server every time.  While this may increase the number of NFS operations, it guarantees the integrity of the file attributes across the RAC nodes.   For non-RAC Oracle databases, you can leave the default actimeo and it is a not a requirement to set it to 0.

For more details on the Mount Options for Oracle files for RAC databases when used with NFS, please see the Oracle Support document Doc ID 359515.1.

Enabling Oracle Direct NFS (dNFS) client

To enable Oracle dNFS on a database host, perform the following steps.

1. Shut down the running instance of the Oracle database software.

2. Change the directory to $ORACLE_HOME/rdbms/lib.

3. Enable dNFS by running the following command.

$ make -f ins_rdbms.mk dnfs_on
  • The server parameter refers to the name of the FlashBlade storage.  This name is for informational use only and not validated.
  • The path parameters should reflect the data VIP address(es) of the FlashStorage.
  • The nfs_version should refer to nfsv3 as at this time FlashBlade supports only the V3 of the NFS protocol.
  • The export parameters should reflect the mount points similar to the entries created in /etc/fstab.
server: fb-dnfs-test-01
path: 192.168.201.100
nfs_version: nfsv3
export: /oradata01 mount: /u02
export: /oradata02 mount: /u03
export: /redo01    mount: /u04
server: fb-dnfs-test-02
path: 192.165.101.200
nfs_version: nfsv3
export: /rman01 mount: /r01
export: /rman02 mount: /r02

Sample oranfstab for multiple data VIP at the FlashBlade level.

server: fb-dnfs-test-01
path: 192.168.201.100
path: 192.168.202.100
path: 192.168.203.100
path: 192.168.204.100
nfs_version: nfsv3
export: /oradata01 mount: /u02
export: /oradata02 mount: /u03
export: /redo01    mount: /u04

5. Restart the Oracle database instance(s). 

 

Disabling the Oracle Direct NFS (dNFS) client

1. Shut down the running instance of the Oracle database software.

2. Change the directory to $ORACLE_HOME/rdbms/lib.

3. Disable dNFS by running the following command.

make -f ins_rdbms.mk dnfs_off

4. Restart the Oracle database instance(s). 

 

Verifying the use of Oracle Direct NFS client

1) If dNFS is enabled, the alert.log will show the following entry when the database is started.

Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 4.0

2) Query the dNFS server information from v$dnfs_servers view inside the database.

SQL> select svrname, dirname, mntport, nfsport, wtmax, rtmax from v$dnfs_servers;

SVRNAME            DIRNAME                 MNTPORT    NFSPORT      WTMAX      RTMAX
------------------ -------------------- ---------- ---------- ---------- ----------
fb-dnfs-test-02    /rman01                    2049       2049     524288      524288

Note: Even though dNFS is enabled, Oracle only mounts the volume/filesystem and opens the files when they are accessed. If no data files are accessed, then the above view will return no rows.