Oracle Database Recommended Settings for FlashArray

Operating System Recommendations

Pure Storage's operating system recommendations apply to all deployments: databases, VDI, etc. These recommendations apply whether you are using Oracle Automatic Storage Management (ASM), raw devices or a file system for your database storage. 

grep PURE /sys/block/sd*/device/vendor

echo noop > /sys/block/sdx/queue/scheduler

for disk in $(lsscsi | grep PURE | awk '{print $6}'); do
    echo noop > /sys/block/${disk##/dev/}/queue/scheduler
done

[robm@robm-rhel7 ~]$ cat /sys/block/sdb/queue/scheduler
[noop] deadline cfq

/etc/udev/rules.d/99-pure-storage.rules

/lib/udev/rules.d/99-pure-storage.rules

# Recommended settings for Pure Storage FlashArray.
# Use none scheduler for high-performance solid-state storage for SCSI devices
ACTION=="add|change", KERNEL=="sd*[!0-9]", SUBSYSTEM=="block", ENV{ID_VENDOR}=="PURE", ATTR{queue/scheduler}="none"
ACTION=="add|change", KERNEL=="dm-[0-9]*", SUBSYSTEM=="block", ENV{DM_NAME}=="3624a937*", ATTR{queue/scheduler}="none"

# Reduce CPU overhead due to entropy collection
ACTION=="add|change", KERNEL=="sd*[!0-9]", SUBSYSTEM=="block", ENV{ID_VENDOR}=="PURE", ATTR{queue/add_random}="0"
ACTION=="add|change", KERNEL=="dm-[0-9]*", SUBSYSTEM=="block", ENV{DM_NAME}=="3624a937*", ATTR{queue/add_random}="0"

# Spread CPU load by redirecting completions to originating CPU
ACTION=="add|change", KERNEL=="sd*[!0-9]", SUBSYSTEM=="block", ENV{ID_VENDOR}=="PURE", ATTR{queue/rq_affinity}="2"
ACTION=="add|change", KERNEL=="dm-[0-9]*", SUBSYSTEM=="block", ENV{DM_NAME}=="3624a937*", ATTR{queue/rq_affinity}="2"

# Set the HBA timeout to 60 seconds
ACTION=="add|change", KERNEL=="sd*[!0-9]", SUBSYSTEM=="block", ENV{ID_VENDOR}=="PURE", ATTR{device/timeout}="60"

# Recommended settings for Pure Storage FlashArray.

# Use none scheduler for high-performance solid-state storage
ACTION=="add|change", KERNEL=="sd*[!0-9]", SUBSYSTEM=="block", ENV{ID_VENDOR}=="PURE", ATTR{queue/scheduler}="noop"

# Reduce CPU overhead due to entropy collection
ACTION=="add|change", KERNEL=="sd*[!0-9]", SUBSYSTEM=="block", ENV{ID_VENDOR}=="PURE", ATTR{queue/add_random}="0"

# Spread CPU load by redirecting completions to originating CPU
ACTION=="add|change", KERNEL=="sd*[!0-9]", SUBSYSTEM=="block", ENV{ID_VENDOR}=="PURE", ATTR{queue/rq_affinity}="2"

# Set the HBA timeout to 60 seconds
ACTION=="add", SUBSYSTEMS=="scsi", ATTRS{model}=="FlashArray      ", RUN+="/bin/sh -c 'echo 60 > /sys/$DEVPATH/device/timeout'"

# Recommended settings for Pure Storage FlashArray.
 
# Use noop scheduler for high-performance solid-state storage
ACTION=="add|change", KERNEL=="sd*[!0-9]|", SYSFS{vendor}=="PURE*", RUN+="/bin/sh -c 'echo noop > /sys/$devpath/queue/scheduler'" 

set sd:sd_io_time = 0x3c
set ssd:ssd_io_time=0x3C

SQL > sqlplus / as sysasm
SQL > alter diskgroup DATA set attribute 'thin_provisioned'='TRUE';

SQL > alter diskgroup DATA rebalance with balance compact wait;

SQL > alter diskgroup DATA add disk '/dev/oracleasm/disks/DATA2' name DATA2
      drop disk DATA1
      rebalance power 16 wait;

/etc/init.d/oracleasm createdisk MYASMDISK /dev/dm-2

filesystemio_options = SETALL

log_buffer = {at least 15MB}

Use the CALIBRATE_IO Utility

Oracle provides a built in package dbms_resource_manager.calibrate_io which, like the ORION tool, generates workload on the I/O subsystem. However, unlike ORION, it works with your running Oracle database, and it generates statistics for the optimizer. Therefore, you should run calibrate_io and gather statistics for your application schema at least once before launching your application.

The calibrate_io script as provided in the Oracle documentation and presented here using our recommended values for <DISKS> and <MAX_LATENCY>.

SET SERVEROUTPUT ON
DECLARE
 lat INTEGER;
 iops INTEGER;
 mbps INTEGER;
BEGIN
-- DBMS_RESOURCE_MANAGER.CALIBRATE_IO (<DISKS>, <MAX_LATENCY>, iops, mbps, lat);
 DBMS_RESOURCE_MANAGER.CALIBRATE_IO (1000, 10, iops, mbps, lat);
 DBMS_OUTPUT.PUT_LINE ('max_iops = ' || iops);
 DBMS_OUTPUT.PUT_LINE ('latency = ' || lat);
 dbms_output.put_line('max_mbps = ' || mbps);
end;
/

Typically you will see output similar to the following: 

max_iops = 134079
latency  = 0 
max_mbps = 1516

Advise that timed_statistics must be set to TRUE before calibrate_io is run.

This can be checked with:

SQL> show parameter timed_statistics
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
timed_statistics boolean TRUE

Regarding Front End Buffer Exhaustion and Large I/O Size Requests

The FlashArray has a limited number of available front end buffers to allocate for incoming writes and outgoing read information to hosts. For simplicity, imagine there are only 4 total front end read buffers with a total size of 512KB and Reads are allocated in 128KB chunks. Host1 sends one Read I/O with a size request of 400KB and Host2 sends two I/Os with a size request of 200KB totaling 400KB. Both hosts need buffers allocated at 128KB, which breaks down as 400KB / 128KB = 3.125 buffers needed, which is a minimum of 4 buffers. As Host1 reached the FA first there are now no more available front end buffers. Host2 I/Os are now queued and have to wait until Host1 single I/O is completed before any further I/Os can be serviced. If both hosts are sending Read sizes of 128KB each then four I/Os can now be processed in parallel rather than just a single I/O at 400KB or just 2 I/Os at 200KB. The larger the I/O size the more front end buffers that need to be allocated in order to service an I/O.

This issue can be exacerbated especially in situations where multiple hosts are sending Large I/O size requests at the same time, as well as when workflow demands increase over time where Large I/O has always been used. Because larger I/Os have consumed all front end buffers, Purity will now have to start queuing I/O, which will cause increased latency. This is known as serializing I/O (sequential processing of an I/O) rather than processing I/Os in parallel. This is why it is important to follow SAN best practices as having less I/O paths available means we are artificially creating a bottleneck by not maximizing the number of available front end buffers. It is vitally important that relevant host, workflow, and environment KBs are followed to help prevent potential performance issues (relevant KBs can be found through the Solutions Landing Page). Host Large I/O Size Requests are often overlooked as a source of performance issues. Make sure that I/O size requests are reasonable in size, as well as host and environment recommendations are being followed, to maximize host performance to the FA.

VMware

Pure Storage recommends grouping your ESXi hosts into clusters within vCenter—as this provides a variety of benefits like High Availability and Dynamic Resource Scheduling. To provide simple provisioning, Pure Storage also recommends creating host groups that correspond to VMware clusters. Therefore, with every VMware cluster that will use FlashArray storage, a respective host group should be created. Every ESXi host that is in the cluster should have a corresponding host (as described above) that is added to a host group.

Conclusion

Many of the traditional architecture decisions and compromises you have had to make with traditional storage are not relevant on a Pure Storage FlashArray. You do not need to sacrifice performance to gain resiliency, nor do you need to change existing policies that you may already have in place. In other words, there is no wrong way to deploy Oracle on Pure Storage; you can expect performance benefits out of the box.

• That said, there are some configuration choices you can make to increase flexibility and maximize performance:
• Use the Pure Storage recommended multipath.conf settings.
• Set the scheduler, rq_affinity, and entropy for the Pure Storage devices.
• Separate different I/O work loads to dedicated LUNS for enhanced visibility.
• If you use a filesystem for data files, use ext4.
• Always run calibrate_io.