Mainframe Solutions

Buffer Credits and Frame Size calculation in FOS 7.1

by Dr.Steve.Guendert on ‎02-27-2013 06:26 AM (25,628 Views)

Buffer-to-buffer credit management affects performance over distances; therefore, allocating a sufficient number of buffer credits for long-distance traffic is essential to performance. With FOS 7.1 Brocade introduced an enhancement to our FOS Extended Fabrics Feature:   The portCfgLongDistance CLI command now includes the option to configure the number of buffers by using the -frameSize option command along with the -distance optionPrior to FOS 7.1, the only option was the –distance option.


Buffer Credit Background and Review


To prevent a target device (either host or storage) from being overwhelmed with frames, the Fibre Channel architecture provides flow control mechanisms based on a system of credits. Each of these credits represents the ability of the device to accept additional frames. If a recipient issues no credits to the sender, no frames can be sent. Pacing the transport of subsequent frames on the basis of this credit system helps prevent the loss of frames and reduces the frequency of entire Fibre Channel sequences needing to be retransmitted across the link.


Because the number of buffer credits available for use within each port group is limited, configuring buffer credits for extended links may affect the performance of the other ports in the group used for core-to-edge connections. You must balance the number of long-distance ISL connections and core-to-edge ISL connections within a switch.Buffer-to-buffer (BB) credit flow control is implemented to limit the amount of data that a port may send, and is based on the number and size of the frames sent from that port. Buffer credits represent finite physical-port memory. Within a fabric, each port may have a different number of buffer credits. Within a connection, each side may have a different number of buffer credits.


Buffer-to-buffer flow control is flow control between adjacent ports in the I/O path, for example, transmission control over individual network links. A separate, independent pool of credits is used to manage buffer-to-buffer flow control. A sending port uses its available credit supply and waits to have the credits replenished by the port on the opposite end of the link. These buffer credits are used by Class 2 and Class 3 services and rely on the Fibre Channel Receiver-Ready (R_RDY) primitive to be sent by the receiving link port to the sender. The rate of frame transmission is regulated by the receiving port, and is based on the availability of buffers to hold received frames.   (If Virtual Channel technology is in use, the VC_RDY or EXT_VC control word is used instead of the R_RDY control word to manage buffer credits. A future blog post will focus on Virtual Channels).


Upon arriving at a receiver, a frame goes through several steps. It is received, deserialized,  decoded, and is stored in a receive buffer where it is processed by the receiving port. If another frame arrives while the receiver is processing the first frame, a second receive buffer is needed to hold this new frame. Unless the receiver is capable of processing frames as fast as the transmitter is capable of sending them, it is possible for all of the receive buffers to fill up with received frames. At this point, if the transmitter should send another frame, the receiver will not have a receive buffer available and the frame is lost. Buffer-to-buffer flow control provides consistent and reliable frame delivery of information from sender to receiver.


Optimal buffer credit allocation

The optimal number of buffer credits is determined by the distance (frame delivery time), the processing time at the receiving port, the link signaling rate, and the size of the frames being transmitted. As the link speed increases, the frame delivery time is reduced and the number of buffer credits must be increased to obtain full link utilization, even in a short-distance environment.


For each frame that is transferred, the hardware at the other end must acknowledge that the frame has been received before a successful transmission occurs. This flow requires enough capacity in the hardware to allow continuous transmission of frames on the link, while waiting for the acknowledgment to be sent by the receiver at the other end.

As the distance between switches and the link speed increases, additional buffer credits are required for the ports used for long-distance connections. Distance levels define how buffer credits are allocated and managed for extended ISLs. Buffer credits are managed from a common pool available to a group of ports on a switch. The buffer credit can be changed for specific applications or operating environments, but it must be in agreement among all switches to allow formation of the fabric.


Smaller frame sizes need more buffer credits. In FOS 7.1, two CLI commands are available to help you determine whether you need to allocate more buffer credits to handle the average frame size. The portBufferShow command calculates the average frame size. The portBufferCalc command uses the average frame size with the speed and link distance to determine the number of buffer credits needed.


Configuring buffers using actual frame size


You can configure the number of buffers by using the -frameSize option of the portCfgLongDistance command along with the -distance option. Fabric OS calculates the number of buffers from the -frameSize option value according to the following formula:


buffers_required = (2048/framesize) * data_vc_credits


If you enter a frame size of 1024, Fabric OS will allocate almost twice as many buffers as for the maximum frame size of 2048. The -frameSize option value is persistent across reboots and HA failover.




switch:admin> portcfglongdistance 2/35 LS 1 –distance 100 –framesize 1024


Calculating the number of buffers required given the distance, speed, and frame size


If you know the distance, speed, and frame size for a given port, you can use the portBufferCalc command to calculate the number of buffers required. If you omit the distance, speed, or frame size, the command uses the currently configured values for the port. Given the buffer requirement and port speed, you can use the same distance and frame size values when using theportCfgLongDistance command.


To determine the number of buffers required, perform the following steps:

  1. Connect to the switch and log in using an account assigned to the admin role.
  2. Enter the portBufferCalc command and provide values for the distance, port speed, and frame size.




The following example calculates the number of buffers required for an 8-Gbps port on a 100-km link with an average frame size of 512 bytes.

switch:admin> portbuffercalc 9/4 -distance 100 -speed 8 -framesize 512

1606 buffers required for 100km at 8G and framesize of 512 bytes


That’s a quick look at the FOS 7.1 Extended Distance Fabrics enhancement incorporating frame size as a variable in configuring buffer credits.  Questions, concerns, comments are always welcome.


Dr. Steve


by mtc
on ‎05-02-2016 10:56 PM

Hi Steve,


thank you for your article.

I understand that having not enought BB credits allocated on the port is harmfule fro performance. 

But I was wander if assigning to many of them can also negatively affect overall performance of fabric/switch?

I have seen that common practice is to assigne more BB credits then required (for example more then we can get from portbuffercalc command). For example by using portcfglongdistance usin LS mode with distance parameter much higher then in reality (for example actual distance is 15km, but when configuring one specifies let's say 100km). Kind of BB oversubscribtion.


Another question which I have is: is it better to use LD mode or LS mode in FC networks on long distance (let's say 100 km and more) which are connected with DWDM device?


Thanks and have a good day.