All About VMID+

VMID+ is a game changer. VMID+, a Fabric Vision FOS v9.1x feature, is exclusive to Gen 7 platforms. VMID+ provides IO Insight capabilities, but now down the VM-level with no requirement for storage array support. This is a feature many have been waiting for for years.

All About Traffic Optimizer

Brocade’s Traffic Optimization (TO) feature is utilized in both open systems and Mainframe environments – TO has been supported since FOS v9x. Traffic Optimization is a tremendous feature which automatically groups together traffic flows of particular types into their own separate performance groups, fabric wide! TO was designed to prevent slower Application traffic flows from negatively impacting much faster ones.

CLI Scripting – Important Warning

If you make heavy use of CLI scripting, this section applies to you!

Note: This technical note/section applies to all Brocade products running all FOS versions!

Frequent monitoring of switches via CLI scripting can lead to excessive use of the compact flash (CF). Specifically, of concern are scripting programs that log into a switch and issue one or more commands to every port on a switch. When this occurs, each command will generate a separate write to the CF. Even if the monitoring commands are “show” commands, a CLI history log will be captured and stored in CF to track the activity. To state the above another way, frequent polling of multiple ports will create one CLI history log entry for every “show” command issued, at the frequency of the polling script.

If a switch’s CF fails:
• The switch may become unstable
• The switch may hang and fault
• At next reboot, the switch may fail to come online
• After a firmware upgrade, the switch may fail to come online

To prevent early CF failures, in an environment that makes heavy use of CLI scripting, please review and implement the following “best practice” recommendations as required:

• Utilize the “-all” option on “show” commands vs issuing a separate “show” command for every port on a switch
• Reduce the polling frequency when possible to reduce the overall number of writes to CF
• Utilize REST or SNMP to conduct regular polling
• “Show” / “Get” commands issued from REST and SNMP are not captured in the CLI history log, and will not contribute to extra CF writes
• There will be a single audit-log entry to record the successful log in from REST or SNMP, however there will only be a single write regardless of how many “Show” or “Get” operations are performed

Dell PowerEdge MX7000 Bladed Chassis - Load Balancing

The Dell PowerEdge MX7000 bladed chassis supports multiple embedded switches. Two of those switching options have Fibre Channel capabilities, namely the MX9116n and the MXG610. The MXG610 can be configured in either native mode or NPG (NPIV Proxy Gateway) mode – aka Access Gateway mode, however the MX9116n only supports NPG mode.

Native mode on the MXG610s would use standard E-Port to E-Port ISL connections, and NPG mode on the MXG610 or MX9116ns would use standard N-Port to F-Port connections. When the MXG610 is configured in native mode, application IO is distributed on an exchange-by-exchange basis across all available ISLs. When the MXG610 or MX9116n are configured in NPG/Access Gateway mode, the PowerEdge servers distribute their logins in a round-robin fashion amongst all available uplinks between the embedded switch and the Brocade core switch.

Between the NPG-mode configured switch and the Brocade core switches, if new uplinks are added or if existing uplinks go down and come back up, you will have a server login imbalance. That is, server logins in these configurations are not automatically rebalanced/distributed amongst available uplinks when there is an uplink change.

As an example, if you had eight servers sharing two uplinks, typically you would have four server logins per uplink. Were an uplink to fail, all eight servers would then start to use the remaining uplink. Importantly, if the failed uplink came back up, even though two uplinks were now available and hopefully healthy, all eight servers would continue to use a single uplink.

To get around this, log into the MX9116n and issue a show fc switch command to ensure it indicates a Switch Mode of NPG.

You can then issue a command to see what the current state is and what would happen if you opted to re-balance your logins. The command is re-balance npg sessions vfabric 1 dry-run which will display the current state of the system, sessions that would be cleared, and the system state after the load balancing completed without actually doing it!

To actually run the command and re-balance your logins, you could issue the command
re-balance npg sessions vfabric 1. Upon doing this, there may be a brief pause in your IO through that fabric as the logins and sessions are rebalanced.

Note that after a manual rebalance, PowerPath may mark paths with an I/O Path Mode of asb:iopf as shown from the rpowermnt host=<host> display dev=all output. asb:iopf indicates Auto Standby intermittent I/O failure algorithm which means the path won’t be used unless all other active paths are dead. To force the Mode back to active, you can issue the command rpowermt host=<host> set mode=active force.

EZserver Vulnerability – CVE-2022-33186

Internal Brocade testing recently exposed a potential security threat associated with a long-since retired tool called EZserver. The hooks into this tool, which have been around for the better part of forever, can potentially be exploited by bad actors. Note: To our knowledge, this has never happened.

Out of an abundance of caution, it’s best to close off this vulnerability on all platforms not running FOS v7.4.2j1, v8.2.3c1, v9.0.1e1, or v9.1.1a or higher – all posted FOS versions plug this vulnerability.

Please either upgrade your FOS version, or, as of FOS v8.1.0b of higher, you can perform the following CLI command non-disruptively:

brocadeswitch:admin> configurechassis
Configure...
cfgload attributes (yes, y, no, n): [no]
ssl attributes (yes, y, no, n): [no]
webtools attributes (yes, y, no, n): [no] y
...
Login Session Timeout (in secs): (60..432000) [7200]
EZserver Enabled (yes, y, no, n): [yes] no
...
brocadeswitch:admin >

For Brocade DCX 8510-4 and DCX 8510-8 Customers

Note: This technical note/section does not apply to the X6 or X7 newer generations of Brocade Directors.

The Brocade Gen 5 DCX 8510 Directors (both the 4 and 8-slot versions) are now in their 12th year of service. As a precautionary measure, Brocade is recommending the following manual maintenance action: (please read this entire section for appropriate cautions)

Example: sw0:FID128:admin> configureChassis

Configure...
cfgload attributes (yes, y, no, n): [no]
ssl attributes (yes, y, no, n): [no]
webtools attributes (yes, y, no, n): [no]
Custom attributes (yes, y, no, n): [no]
system attributes (yes, y, no, n): [no] y
system.blade.bladeFaultOnHwErrMsk: (0x0..0x7fffffff) [0x1]
system.cpuLoad: (10..121) [121]
system.i2cTurboCnfg: (0..2) [1]
system.Enable.bladeAutoRecovery (yes, y, no, n): [no]

The default value of the system.blade.bladeFaultOnHwErrMsk setting is 0x0. Setting this value is non-disruptive and does not require a switch reboot to activate.

Note: The above recommendation is only for customers that have well designed storage networks including redundant fabrics, multiple paths from host to storage, and host fail-over detection software.

Enabling the above blade fault action will result in problematic Director blades being powered off if any nonfatal hardware ASIC data parity errors exist. Again, this is only recommended in environments designed with multiple alternate paths, redundant alternate fabrics, and storage target access using multipath IO software/drivers.

Note: We have not had any customers hit this issue, nor have any of the peers we typically correspond with.

Before making the recommended setting change, we would suggest you review the Director for occurrences of the following error messages IDs: C3-1006 / C3-1010. If these messages are being encountered, please contact support before making the fault mask change.

To level set, internal ASIC memory failures are rare. In the unlikely event they do occur, just like any other problem, they should be addressed. If repeated memory failures are encountered, RASlog entries of C3-1006 / C3-1010 should be present. The concern is if repeated memory failures are encountered but not dealt with, application traffic flowing through that ASIC could be impacted, but, traffic flowing through other paths will continue to work without issue. This can make the error condition difficult for host MPIO software to detect in order to re-route all traffic to all known-good alternate paths. Depending on many variables, this can cause application slowdowns or worse.

The goal here is to prevent the scenario in which a failing ASIC can result in disruption of traffic to multiple hosts or targets. The above setting will result in a hard fault of a port blade experiencing this quite rare condition. The hard fault of the port blade will trigger the multipath IO software to easily detect the failing condition (because the blade is powered off) and properly ensure that all host-target traffic is directed to alternate paths and/or alternate fabrics.

For Customers using 64Gb/s SFPs

Note: 64Gb/s SFPs are only supported in Gen 7 products. Therefore, this section only applies to customers using 64Gb/s SFPs in Gen 7 products.

Please ensure all of your products are running FOS v9.0.1b or higher to achieve the optimal SFP performance margins while operating at 64Gb/s. This is the most important thing.

In addition, all SFPs have firmware versions associated with them. The earliest versions of 64Gb/s SFPs naturally shipped with earlier versions of firmware. FOS v9.0.1b and later has the ability to upgrade 64Gb/s SFP firmware. SFP upgrades are recommended and desired, but they are not required or deemed critical for systems running in production.

For customer using 64Gb/s SFPs, please check your 64G SFP firmware levels to determine if they are down-level. The latest SFP module firmware versions available at the time of this publication were:

• MCU: 0x17 (Microcontroller)
• DSP: 0xda0718 (Digital Signal Processing)

The MCU 0x17 update reduces RX “jitter” in noisy environments. You can utilize the sfpupgrade ––show command (introduced in FOS v9.0.1b) to view your current SFP firmware levels. The command output is elegant in that it shows both the current levels as well as the latest levels available. Note: The sfpupgrade command is fully documented in the FOS v9.0x CLI guide.

If you are down-level and your switching products are not yet in production, please upgrade to the latest SFP firmware version available at your earliest convenience – the same if you’re performing maintenance operations in which any 64G port will be taken offline.

If your SFP firmware versions are down-level in production environments, while the upgrade process is not disruptive to the switch itself, it is disruptive to the individual ports (each must be taken offline before the upgrade can occur). Note that SFP firmware upgrades can be performed on individual ports, a range of ports, or all ports – there are many options as documented in the CLI Guide.

The following CLI example shows SFP firmware information for Director port 2/16 :

switch:admin> sfpupgrade 2/16 --show
[02/16 | ***]:Current MCU version : 0x16 (latest: 0x17)
[02/16 | ***]:Current DSP version : 0xda0710 (latest: 0xda0718)
[02/16 | ***]:MCU version : Out-dated, needs upgrade
[02/16 | ***]:LUT marker : Up-to-date
[02/16 | ***]:DSP version : Out-dated, needs upgrade

Notice that both the MCU and DSP versions shown above are out of date.

Monitoring for Errant SFPs and Cables

A single dying but not dead SFP can result in poor application performance, and in severe cases, application outages. This is true in every type of networking – regardless of type of infrastructure put in place - be it Hyper-converged, or the cloud. The low-level monitoring and alerting related to errant SFPs and cables is fundamental to the management of any type of Data Center.

Brocade Education has released three new Quick Hit Solution classes.

Quick Hit Solutions are short 2-10 minute solution based classes. They cover a verity of topics including basic switch setup, new features and common SAN management tasks.

New classes:
- Basic Switch Setup (QHS-008)
- How to Add a Switch to a Brocade Fabric (QHS-009)
- Fibre Channel Router Migration Mode (QHS-010)

Brocade has over 80 free online training classes grouped into topical playlists. Just search for "Brocade Playlist" in the Broadcom education learning portal

POD and ICL Kit Transaction key Retrieval

Throughout 2023, Brocade will be transitioning away from paper versions of transaction keys for all shipped POD upgrade and ICL kits. Traditionally, shipped out POD kits included the SFPs and QSFPs along with their associated transaction keys. Moving forward, SFP and QSFP POD kits will ship out with instructions (both in the box and in the internal clamshell containing the SFPs) on how to obtain transaction keys when these kits are ordered.

Moving forward, an extra step will be required to generate POD licenses. The process is to log into the CSP Licensing Portal (see Access to Free Brocade Education for full details), select the new Retrieve Optics Kit Transaction Key tab, and enter the serial number of the shipped kit (printed on the shipping box and on the internal clamshell) along with the serial number of any optic (SFP or QSFP) shipped with the kit.

If the box and clamshell cannot be found, open up a standard TAC case to get assistance generating the transaction key. Once you have the transaction key, the process remains identical for transacting it into a loadable license.

Secure Optics (SFPs)

• All Gen 7 products require Brocade Secure Optics
• Across the globe, counterfeit SFPs have made their way into Data Centers. When utilized, counterfeit SFPs often result in poor link connectivity which can in turn cascade into far more serious and complex problems
• This counterfeiting problem exists in any Ethernet-based networking technology as well as in fibre-channel-based technologies
• All major manufacturers of SFPs can have their products and product labeling spoofed to fool any networking platform into not knowing the SFPs are fake. It is imperative to source SFPs from reputable vendors. Various countries are rife with counterfeit SFPs
• Brocade Gen 7 products take this issue on directly in continued efforts to be the most secure networking technology in existence
• Without secure SFPs and Brocade Gen 7 products, there is literally no way to know if your SFP is genuine or not without physically opening up the SFP
• Brocade Secure Optics cannot be counterfeited and will ensure knock-off SFPs are not errantly installed in production systems
• Secure Optics are fully backwards compatible with all previous generations of Brocade products
• Note: One exception is the Brocade 7810 and SX6 FCIP extension products which will require the following FOS versions to support Secure Optics
• FOS v8.2.1e or later
• FOS v8.2.2c or later
• The easiest way to differentiate secure and non-secure SFPs is the SFP label itself - secure SFPs will indicate –SEC in the SFP description – for example: 32G LW-SEC 10km
• Also, the manufacturer part number as shown in the sfpshow output will have a 4 in the third to last major digit as in XBR-000492 or 57-1000487-01

Accessing Brocade Documentation

A comprehensive document has been written detailing where to find all Brocade documentation https://docs.broadcom.com/doc/locating-brocade-content.

Note that product manuals for all Brocade hardware products and FOS version-specific manuals such as the Brocade Fabric OS Command Reference, 8.2x or the Brocade Fabric OS Administration Guide, 9.1x can be located and downloaded from the public Broadcom website. Simply navigate to https://www.broadcom.com, enter what you’re looking for into the Search box located in the upper right-hand corner of the screen (e.g. 8.2.3, 9.1x, SANnav, G720, or Brocade Fabric OS Administration) and all related documentation will appear. You can further select the Documents check box to further cut down your search criteria to just documents.

For your convenience, if you search for a specific FOS version such as 8.2.3c or 9.0.1e, a Digest Edition of the Release Notes for that particular FOS version will appear – an example document title would be FOS v9.0.1e Release Notes Version 5.0 (Digest Edition). These digest versions are actually complete versions of Release Notes with the exception that only brief summaries for defect resolutions are given.

If you interested in reading the full version of Release Notes, Brocade direct support customers can locate them on the CSP, and all other customers can locate them on their respective OEM assist sites.

If you search above for Transceiver Support Matrix, you’ll find the document which contains the listing of all Brocade supported SFPs for each Brocade product (this can come in incredibly handy).

Note: By default, Brocade direct-support customers are sent out notifications whenever new software or documentation is posted to the Customer Support Portal (CSP). Last year, the option was added to allow users to turn off those notifications. Proceed as follows to disable notifications:

• Log into CSP
• Go to your profile manager
• On the right hand side of the screen, you will see a tab called “Notifications”
• “Notifications” are enabled as a default, to disable them, simple slide the button over to green
• Make sure you select Save Preferences

A Primer on Brocade FOS/SANnav Releases & Definitions

Brocade maintains a publicly available document titled Brocade Software: Software Release Support and Posting Matrices

Brocade Security Advisories (BSAs) and CVEs

Security and security vulnerabilities are of utmost importance to Brocade. To that end, Brocade maintains a publicly accessible website which can be referenced whenever questions surrounding such vulnerabilities arise: https://www.broadcom.com/support/fibre-channel-networking/security-advisories. Please use this site for researching CVE exposure to the Brocade product line. The site clearly matches CVE IDs with the associated Brocade Security Advisory.

Please keep in mind that CVE advisories are a necessary evil to ensure enterprises are quickly made aware of potential security threats to their environments. However, in making these advisories public knowledge, the natural byproduct is also informing bad actors, who now aware of the weaknesses, can try to exploit them.

Note: To report a potential security vulnerability not already listed, please contact brocade.sirt@broadcom.com.