When 8Gb FC was released, the fill word setting needed to be changed. The fill word is what we call the 'idle' frames that are sent through an unused stretch of fiber. For 4Gb the fill words were IDLE-IDLE. 8Gb speeds require ARB-ARB. On Brocade, there are modes 1 and 2, but mode 3 actually tests fill words and picks the correct one. For more information, see Brocade's help page for portcfgshow or portcfgfillword. Since this impacts only the fill words for quiet connections, there isn't any impact to data frames.
Using the incorrect setting can cause port errors and poor SAN performance.
While there are equivalent settings on Brocade and Cisco, you will encounter this more often on Brocade due to 8G switches defaulting to the incorrect fill word configuration (see below).
Applies to 8G switches only.
Starting from version 6.x of Brocade firmware (FOS), Brocade defaults to the 4Gb fill word configuration for devices (IDLE-IDLE). This leads to a very high rate of encoding outside of frame errors and invalid ordered sets even with fixed port speeds. This could cause sync and signaling problems leading to Link Resets and ultimately poorer performance.
Attached is the Brocade white paper "Link Initialization and Fill-Word Primitive Configuration on 8G FC Platforms". I will quote:
Up through the FOS v6.1.1 release, Brocade Fibre Channel switches used IDLE primitives both during link initialization and for fill words (denoted as “IDLE/IDLE”). This ensured successful link initialization between Brocade switch ports and end devices operating at 1G/2G/4G speeds.
With the introduction of 8G Fibre Channel, FC-PI-4 and FC-FS-3 standards stated that ARB should be used as fill words for lowering emission. To comply with the published FC standards, Brocade introduced options for ARB/ARB and IDLE/ARB link initialization/fill word support.
However, some 8G devices are not capable of properly establishing links with Brocade 8G Fibre Channel switches when ARB/ARB or IDLE/ARB primitives are used. These 8G devices require the legacy IDLE/IDLE sequence to achieve successful link initialization. To address this issue, Brocade has provided the ability to configure any of the three possible combinations (IDLE/IDLE, ARB/ARB, or IDLE/ARB) for link initialization and fill words. Any of these modes can be configured on an individual port basis using the CLI portcfgfillword. This CLI also provides an option to automatically switch from ARB/ARB to IDLE/ARB if the former is not able to successfully establish a link. (Please see the CLI documentation in the Appendix or in the FOS Command Reference Manual for further details). This capability is available in FOS v6.3.1 and later versions of firmware.
In summary, FOS v6.3.1 and later versions support the following modes for 8G F_Ports as a per port configuration.
MODE MEANING Mode 0 Use IDLE in link init and IDLE as fill word Mode 1 Use ARB in link init and ARB as fill word Mode 2 Use IDLE in link init and ARB as fill word Mode 3 Try mode 1 first; if it fails then try mode 2
From the Brocade "SAN Administration Best Practices" (attached), I quote:
Traffic outside of frame traffic is made up of fill words: IDLEs or ARB (F0) or ARB (FF). Encoding errors on fill words are generally not considered impactful. This is why you may see very high counts of enc_out (encoding outside of the frame) and not have customer traffic affected. If many fill words are lost at once, the link may lose synchronization.
On standard E_Ports, primitives are set to ARB, regardless of the portcfgfillword setting when not in R_RDY mode.
The recommended best practices are:
- Ensure that the fill word is configured to Mode 3.
- When connecting to a HDS storage device, set to Mode 2.
- When upgrading firmware, recheck the settings, since the fill word primitive has evolved over several Brocade FOS releases.
Find the current fill word value
In the logs of a supportshow, you can see the fill word by doing a search for "portcfgshow"
portcfgshow : Ports of Slot 0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 -----------------+---+---+---+---+-----+---+---+---+-----+---+---+---+-----+---+---+--- Speed AN AN AN AN AN AN AN AN AN AN AN AN AN AN AN AN Fill Word(On Active) 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 Fill Word (Current) 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
To check your current switch settings, log in to your Brocade switches that connect to any 8Gb device and issue
portcfgshow followed by the number of the port in question. (If you leave off a number, you'll see all of your port settings which is an effective way to confirm correct fill word settings for all of your attached 8Gb devices):
pure-brcd:FID128:admin> portcfgshow 39 Area Number: 16 Speed Level: AUTO(HW) Fill Word: 3(A-A then SW I-A) AL_PA Offset 13: OFF Trunk Port ON Long Distance OFF VC Link Init OFF Locked L_Port OFF Locked G_Port OFF Disabled E_Port OFF ISL R_RDY Mode OFF RSCN Suppressed OFF ...
Note the boldfaced type for "Fill Word" and note that the mode is 3. This is the correct mode for 8Gb devices.
Changing the fill word value
To change the fill word, use the command
portcfgfillword, followed by the port number and then the mode:
pure-brcd:FID128:admin> portcfgfillword 39 3
Operands The following are optional: --slot slot Specifies the slot on which to operate. If this option is not specified, the default slot is assumed. The default slot is 0 and designed to operate on fixed-port-count products, if -use_bports sets with nonzero value. -uports itemlist Specifies the list of user ports for which statistics are to be cleared. -bports itemlist Specifies the list of blade ports for which statistics are to be cleared. -use_bports value Specify a nonzero value to clear the diagnostics statistics for the blade ports specified in -bports clears. A value of zero (0) clears the user ports specified in -uports. The default value is 0. *Example: To clear port and diagnostic statistics: switch:admin> statsclear -bports 1/10-1/62 -use_bports 1
portcfgfillword command is applicable only to 8G FC ports; it is not supported on Condor 3 based platforms. Reference from Wikipedia:
The 7th generation of Brocade ASICs are Condor 3. Condor 3 supports 48 ports of 16 Gbit/s per ASIC. These ASICs are used in the DCX 8510 Backbone Family and port blades, and the 6505/6510/6520 switches. The initial 16 Gbit/s product line (DCX 8510-8, DCX 8510-4, 6510 48-port switch, and 1860 Fabric Adapter) was originally launched in 2011. The 6505 24-port switch was launched in May 2012. The 6520 96-port switch was launched in March 2013.
I quote from "FC Cookbook for HP Virtual Connect", page 150 (attached):
When Fibre Channel uplink ports on VC-FC 8Gb 20-port module or VC FlexFabric module are configured to operate at 8Gb speed and are connecting to HP B-series Fibre Channel SAN switches, the minimum supported version of the FOS is v6.4.x. In addition, “FillWord” on those switch be configured with option Mode 3 to prevent connectivity issues at 8Gb speed. This setting is only required:
- with VC-FC 8Gb 20-port module when running VC 3.70 or earlier.
- with VC FlexFabric module when running VC 3.60 or earlier.
Note: With VC-FC 8Gb 24-port module, FillWord is not required.
On Cisco systems, the value is often referred to as "fill-pattern" instead of "fill word".
Cisco NX-OS Release 5.2(8a) and later releases support IDLE as a fill word for the switchport fill-pattern command. Releases earlier than Release 5.2(8a) supported only the ARBFF fill word.
Find the current fill word value
To view all internal info for an interface (including fill-pattern) run:
show port internal info interface fc*/*
Example output (relevant line bolded):
swirly# show port internal info interface fc1/1 fc1/1 - if_index: 0x01000000, phy_port_index: 0x0 local_index: 0x0 Admin Config - state(up), mode(auto), speed(auto), trunk(on) beacon(off), snmp trap(on), tem(false) rx bb_credit(default), rx bb_credit multiplier(default) rx bb_credit performance buffers(default) bb scn config(on) speed group(NA) transceiver frequency (Unknown(0)) ignore flags (ignore:none), service state(in service) rxbufsize(2112), encap(default), user_cfg_flag(0x1) max-npiv-limit (0) max-npiv-limit for trunk port (0) description() port owner() admin rate-mode(default) port act license(acquire) congestion drop timeout mode F (500), congestion drop timeout mode E (500) no-credit-force mode F enable (0), no-credit-force mode F timeout (500) no-credit-force mode E enable (0) no-credit-force mode E timeout (500) slowport-monitor mode F enable (0), slowport-monitor mode F timeout (50) slowport-monitor mode E enable (0), slowport-monitor mode E timeout (50) admin port fill-pattern 8g (ARBFF) admin port fec state (down) Link failure syslog level (5) Link failure syslog level = 5 Port guard info - link failure state (disabled, 4) nt 0, dur 0 tsv state (disabled, 4) nt 0, dur 0 bit error state (disabled, 2) nt 0, dur 0 sig loss state (disabled, 2) nt 0, dur 0 sync loss state (disabled, 2) nt 0, dur 0 link reset state (disabled, 2) nt 0, dur 0 credit loss state (disabled, 2) nt 0, dur 0 Operational Info - state(down), mode(ALL), speed(auto), trunk(off) 32-bit state reason(7,7 - Link failure or not-connected), state reason(7,7 - Link failure or not-connected) phy port enable (1), phy layer (FC), port_rate_mode(dedicated) participating(1), fcid(0x000000), bb_scn(0) cached rate mode(Unknown(0)) config port vsan (1), oper port vsan (1) rx bb_credit(64), rx bb_credit multiplier(0) rx bb_credit performance buffers(0), tx bb_credit(0) not first time port bringup is 0 Port WWN is 20:01:8c:60:4f:80:e3:40 current state [PI_FSM_ST_LINK_INIT] port_init_eval_flag(0x00000001), cfg wait for none FC_PORT_INIT_DATA_VALID (0) FC_PORT_INIT_PORT_LICENSE_ELIGIBLE (45) FC_PORT_INIT_PORT_LICENSE_ACQUIRED (46) Mts node id 0x102 Lock Info: resource [fc1/1] type p_gwrap[(nil)] FREE @ 875355 usecs after Fri Apr 14 15:30:09 2017 type p_gwrap[(nil)] FREE @ 875698 usecs after Fri Apr 14 15:30:09 2017 type p_gwrap[(nil)] FREE @ 987712 usecs after Wed Feb 15 12:05:24 2017 0x100000001000000 Peer ELP Revision: 0 cnt_link_failure(0), cnt_link_success(0), cnt_port_up(0) cnt_cfg_wait_timeout(0), cnt_port_cfg_failure(0), cnt_init_retry(0) oper port fec state (down) nos link failures (0), debounce window (300 ms), elp wait tout (750 ms) link fail reason (0x0) num_elps_before_up (0) internal_reinit_cnt (0) external_reinit_cnt (0) external_laser_reinit_cnt (0) Port Capabilities - Modes: Min Speed: 2000 Max Speed: 16000 Max Sourcable Pkt Size: 0 Max Tx Bytes: 0 Max Rx Bytes: 0 Max Tx Buffer Credit: 0 Rx Buffer Credit in FRM mode: Min(1), Max(253), Default(64) Rx Buffer Credit in OSM mode: Min(0), Max(0), Default(0) Rx Buffer Credit(ISL) in FRM mode: Min(2), Max(253), Default(64) Rx Buffer Credit(ISL) in OSM mode: Min(0), Max(0), Default(0) Rx Buffer Credit Multiplier in FRM mode: Min(0), Max(0), Default(0) Rx Buffer Credit Multiplier in OSM mode: Min(0), Max(0), Default(0) Rx Buffer Credit Multiplier(ISL) in FRM mode: Min(0), Max(0), Default(0) Rx Buffer Credit Multiplier(ISL) in OSM mode: Min(0), Max(0), Default(0) Rx Buffer Credit performance buffers in FRM mode: Min(1), Max(0), Default(0) Rx Buffer Credit performance buffers in OSM mode: Min(1), Max(0), Default(0) Rx Buffer Credit performance buffers(ISL) in FRM mode: Min(1), Max(0), Default(0) Rx Buffer Credit performance buffers(ISL) in OSM mode: Min(1), Max(0), Default(0) Rx Buffer Credit change not allowed in FRM mode, not allowed in OSM mode Rx Buffer Credit performance buffer change not allowed in FRM mode, not allowed in OSM mode Port rate mode change not allowed BB_SCN processing supported Max BB_SCN 14 Max Private Devices: 0 Hw Capabilities: 0x0 Connector Type: 0x8 On demand port activation license is supported Port type is external Licensing scheme is sequential Default port activation license is required Default port activation license state is eligible License file used is PORT_ACTIV_9148S_PKG VPD updation is not needed SFP validation is done per port default mode is auto VF tagging not capable first flogi from hba (00:00:00:00:00:00:00:00) epp fc2 service (EPP_SERVICE_SWILS) is delayed trap timer active (0) Slowdrain capable Port cap bitset: FC_PORT_CAP_HW_SLOWDRAIN_CAPABLE (5) Port guard info - link failure ed FALSE, oper cnt 0 ts (n/a 0) tsv state ed FALSE, oper cnt 0 ts (n/a 0) bit error state ed FALSE, oper cnt 0 ts (n/a 0) sig loss state ed FALSE, oper cnt 0 ts (n/a 0) sync loss state ed FALSE, oper cnt 0 ts (n/a 0) link reset state ed FALSE, oper cnt 0 ts (n/a 0) credit loss state ed FALSE, oper cnt 0 ts (n/a 0) speed group convertible: 0x00 ( ) SFP Initializing:0 SFP present is :1 Trunk Info - trunk vsans (allowed active) (1) Pacer Info - Pacer state: released credits : granted_credits 0 F Port Pacer state: initialized : granted_credits 0 RNID info not found. PQP: not in progress
Changing the fill word value
To set, you need to go to `configure`, then into the interface, and set it there:
MDSSwitch(config-if)# switchport fill-pattern ARBFF speed 8000
Feb 21, 2017
Cisco MDS 9000 Series Multilayer Switches
Known Affected Releases
When some devices are connected at 8G , the interface may encounter bit error
and error disabled
connected devices use IDLE for fillword instead of ARB(FF) at 8G
Jan 31, 2017
Cisco MDS 9000 Series Multilayer Switches
Known Affected Releases
6.2(15)S23 7.3(0)D1(1) 7.3(0)DY(0.14)
ISL or Port-channel will not come up if the interfaces ae 16G with fill word set to ARBFF
Link will come up with 8G SFP,
MDS 9700 with DS-X9448-768K9 port is configured for 16 Gig.
CRC and Decode errors on 8-Gbps Fibre Channel ports connected to Brocade switches
Excessive CRC and decode errors on 8-Gbps Fibre Channel ports connected to other switch vendors.
Excessive errors can prevent servers from connecting to storage devices, resulting in devices not being detected and the system failing to boot from SAN.
Decode errors indicate failure on QLogic devices. Failure on a Brocade switch may be indicated by a few “er_enc_out” errors and large number of “er_bad_os” errors.
swd77:root> portstatsshow 6
> stat_wtx 547107199 4-byte words transmitted
> stat_wrx 785641731 4-byte words received
> stat_ftx 1082261 Frames transmitted
> stat_frx 1528326 Frames received
> stat_c2_frx 0 Class 2 frames received
> stat_c3_frx 1528326 Class 3 frames received
> stat_lc_rx 0 Link control frames received
> stat_mc_rx 0 Multicast frames received
> stat_mc_to 0 Multicast timeouts
> stat_mc_tx 0 Multicast frames transmitted
> tim_rdy_pri 0 Time R_RDY high priority
> tim_txcrd_z 0 Time TX Credit Zero (2.5Us ticks)
> tim_txcrd_z_vc 0- 3: 0 0 0 0
> tim_txcrd_z_vc 4- 7: 0 0 0 0
> tim_txcrd_z_vc 8-11: 0 0 0 0
> tim_txcrd_z_vc 12-15: 0 0 0 0
> er_enc_in 0 Encoding errors inside of frames
> er_crc 0 Frames with CRC errors
> er_trunc 0 Frames shorter than minimum
> er_toolong 0 Frames longer than maximum
> er_bad_eof 0 Frames with bad end-of-frame
> er_enc_out 318 Encoding error outside of frames
> er_bad_os 2016423236 Invalid ordered set
> er_rx_c3_timeout 0 Class 3 receive frames discarded due to timeout
> er_tx_c3_timeout 0 Class 3 transmit frames discarded due to timeout
> er_c3_dest_unreach 0 Class 3 frames discarded due to destination unreachable
> er_other_discard 0 Other discards
> er_type1_miss 0 frames with FTB type 1 miss
> er_type2_miss 0 frames with FTB type 2 miss
> er_type6_miss 0 frames with FTB type 6 miss
> er_zone_miss 0 frames with hard zoning miss
> er_lun_zone_miss 0 frames with LUN zoning miss
> er_crc_good_eof 0 Crc error with good eof
> er_inv_arb 0 Invalid ARB
> open 0 loop_open
> transfer 0 loop_transfer
> opened 0 FL_Port opened
> starve_stop 0 tenancies stopped due to starvation
> fl_tenancy 0 number of times FL has the tenancy
> nl_tenancy 0 number of times NL has the tenancy
> zero_tenancy 0 zero tenancy
By default, Brocade switches running at 8-Gbps speed send “IDLE” as the fill word. This is non-standard behavior (see standard below) and causes our receiver to train poorly resulting in decode errors.
Applicable Fibre Channel Switch Standard:
Unlike TCP/IP communication, Fibre Channel requires that all devices in a Fabric to communicate at all times, even when the device in question is neither sending or receiving data.
The two most common frames (Primitives) being sent are R_RDY (Receive Ready) and IDLE. IDLE is a Fill Word used to take up space along the wire. These two frames alternate R_RDY and IDLE,
R_RDY and IDLE, until data receive or transmit commands occur. IDLE can also be found when a device is waiting for a response from another device.
Prior to the creation of the 8 Gig standard, the T11 sanctioning body set IDLE as the required Fill Word, with ARB(ff) as an option. Effectively, 95% of FC manufacturers chose IDLE as the Fill Word.
When the 8 Gig standard was ratified, ARB(ff) was selected, instead of IDLE, as a means to help lower EMR emissions. Using ARB(ff) as a Fill Word, instead of IDLE, was established in FC-PI-4 sec 5.8that a port operating at 8G FC shall use scrambling/ELP.
FC-PI-4 Rev 8.0 states:
5.8 Frame scrambling and emission lowering protocol 8GFC shall use the frame scrambling as stated in FC-FS-2 AM1 (reference ) and emission lowering protocol as stated in FC-FS-2 (reference ). 1GFC, 2GFC, and 4GFC do not use scrambling.
FC-FS-3 Rev 0.70 states:
22.214.171.124 ARB(ff) ARB(ff) is a Primitive Signal used to maintain link synchronization on links using Emission Lowering Protocol (ELP). ARB(ff) shall be treated the same as Idle. ARB(ff) shall be transmitted on links using Emission Lowering Protocol during periods of time when frames, other Primitive Signals, or Primitive Sequences are not required to be transmitted.
See 8.3 for the requirements for the insertion of Fill Words between frames.
8G FC implementation is required to use the ELP, and if ELP is used, it is required to transmit ARB(ff). At the 8 Gig transmission rate, QLogic switches always use ARB(ff) as the Fill Word in ISL (E_Port), Fabric (F_Port), Loop (FL_Port), Transparent Routing (TR_Port), and Link Initialization modes. The ARB_FF port option is for devices that are running at transmission rates below 8 Gig, and use ARB(ff), instead of IDLE, when they have connected to the QLogic switch only in Fabric Loop mode (FL_Port). Very few devices use ARB(ff) in this manner, but if there is a communication problem between the device and the switch, setting the ARB_FF option to 'True' may resolve the problem. Some 8 Gig devices may continue to use ARB(ff) at lower transmission rates, instead of reverting to IDLE as the Fill Word. This setting is also found in firmware for QLogic 2 Gig and 4 Gig switches. This setting only applies to Loop devices at transmission rates less than 8 Gig.
To ensure interoperability between Brocade and QLogic switches, use the “portCfgFillWord” command to set the fill word of the connecting port to option (ARBFF/ARBFF). >
Brocade5100:admin> portcfgfillword 0 1
Usage: portCfgFillWord PortNumber Mode
Mode: 0/-idle-idle - IDLE in Link Init, IDLE as fill word (default)
1/-arbff-arbff - ARBFF in Link Init, ARBFF as fill word
2/-idle-arbff - IDLE in Link Init, ARBFF as fill word (SW)
3/-aa-then-ia - If ARBFF/ARBFF failed, then do IDLE/ARBFF
Want to learn more?
The Brocade free web-based course FC 101-WBT Introduction to Fibre Channel Concepts is an excellent start.