Best Practices for Oracle on FlashBlade
This article covers the best practices to be followed for Oracle databases hosted on FlashBlade to achieve better performance and simplify operations.
NFS Filesystems for Oracle
Networked filesystems provide the flexibility to mount the same filesystem multiple times on the same host or across multiple hosts, for the RMAN backup target, you can create a single NFS filesystem that can be mounted across the available subnets on the host. Make sure the NFS protocol is selected when creating the filesystem on FlashBlade.
Use dNFS over Kernel NFS
To scale up the bandwidth on FlashBlade enable numerous connections from the client than a single connection. Oracle’s Direct NFS creates a separate connection to the storage system for every server process as opposed to a single connection per mount point by Kernel NFS.
Enable Parallelism
To increase the read and write bandwidth on FlashBlade, use client-level parallelization techniques like parallel queries, multiple RMAN channels based on the CPU availability on your host in conjunction with dNFS. This increases the number of connections to FlashBlade, especially with dNFS.
Use Multiple Network Interfaces
To enhance the network bandwidth, make sure to have multiple network interfaces on the client. These multiple interfaces can be configured on a single subnet or on multiple subnets.
Single subnet
IO performance on a dNFS environment with multiple interfaces and single subnet is limited to the speed of the first interface that is picked by the OS. This is because the OS returns the first path when multiple paths are available on the subnet and hence the traffic is always routed through the first path.
For example, in the setup below, the FlashBlade has a single IP address on which the NFS filesystem will be mounted from the client and the client has two interfaces.
Client interface NFS server
ens7f0 10.21.108.193 10.21.108.10
ens7f1 10.21.108.194 10.21.108.10
[oracle@rlxora-b01-11-06 ~]$ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.21.108.1 0.0.0.0 UG 0 0 0 ens7f0
10.21.108.0 0.0.0.0 255.255.255.0 U 0 0 0 ens7f0 <-- first route
10.21.108.0 0.0.0.0 255.255.255.0 U 0 0 0 ens7f1 <-- route ignored
As per the routing table, both traffic can go through the first interface (ens7f0) as the destination and the mask fits for both routes and OS will invariably choose the first route.
In this case, to enhance the bandwidth, it is recommended to use NIC bonding at the client level. Oracle support document (Doc ID 833481.1) provides other means to enable multiple paths in a single subnet using static routing but it doesn’t address the issue with availability when a NIC fails or network cable is pulled as the routing table will not be updated.
Multiple subnets
“Direct NFS client best practices recommend always to use multipaths in separate subnets”.
Oracle recommends using separate subnet for each interface and it supports up to four subnets. With multiple subnets, there is no need to bond the network interfaces to aggregate the bandwidth across the available interfaces. The routing will be automatic in the case of multiple subnets.
Client interface NFS server
ens7f0 10.21.108.193 10.21.108.10
ens7f1 10.21.107.194 10.21.107.10
[oracle@rlxora-b01-11-06 ~]$ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 10.21.108.1 0.0.0.0 UG 0 0 0 ens7f0
10.21.108.0 * 255.255.255.0 U 0 0 0 ens7f0
10.21.107.0 * 255.255.255.0 U 0 0 0 ens7f1
In this case, these are two dynamic routes and based on the traffic, the route is selected automatically.
As such, if you decide to use multiple subnets, it should be configured at both the client and the FlashBlade side as well. Multiple subnets can be configured in FlashBlade GUI under the Network Settings.
Make sure to update the oranfstab with the subnets and the mount point details. In RAC environment, all RAC nodes should have the appropriate oranfstab file configured.
NFS volumes and Mount point requirements
In the case of multiple subnets, it is not required to have as many NFS filesystems/volumes as the subnets for dNFS to be effective. Also, it is not required to mount a single volume on to all subnets for dNFS to be effective either. Oracle dNFS reads the oranfstab and based on the storage paths and mount details, it will create multiple paths when the database files are accessed.
For example, with two subnets and two mounts, dNFS would create four paths to the storage system for every server process.
Linux Mount options
For mounting the NFS filesystem on Linux, use the following mount options. Do not specify the rsize, wsize options as the system can get the default offered by FlashBlade which is 524288).
rw,bg,nointr,hard,tcp,vers=3,actimeo=0
Note: The actimeo=0 is required for Oracle RAC databases to turn off attribute caching forcing the NFS client to look for the attributes like atime, ctime, file permissions, owner etc., from the server every time. While this may increase the number of NFS operations, it guarantees the integrity of the file attributes across the RAC nodes. For non-RAC Oracle databases, you can leave the default actimeo and it is a not a requirement to set it to 0.
Even for dNFS, the filesystem(s) should be mounted using the above mount options. To persist these changes across reboots, please include them in the /etc/fstab file as given below. The IP address specified below refers to the data VIP from the FlashBlade.
10.21.214.100:/oradata01 /u02 nfs rw,bg,nointr,hard,tcp,vers=3,actimeo=0
For more details on the Mount Options for Oracle files for RAC databases when used with NFS, please see the Oracle Support document Doc ID 359515.1.