Mainframe & FICON

Reply
New Contributor
Posts: 3
Registered: ‎11-04-2013

We are receiving severe latency bottleneck warnings on an ISL port. How can I troubleshoot this?

We are receiving severe latency bottleneck warnings on an ISL port.  How can  I troubleshoot this?  I have run SAN health against our fabrics and the only thing that I can see that is different on this port is that it is a locked G port (not sure if this has any bearing on the situation.)  Any help/direction would be appreciated.

Regular Contributor
Posts: 161
Registered: ‎12-30-2009

Re: We are receiving severe latency bottleneck warnings on an ISL port. How can I troubleshoot thi

Locked G port only means it cannot become an FL port, so not of real concern.

 

using san health to Troubleshooting bottlenecks is not ideal.

Do you have BNA?

If not start with bottleneckmon on the CLI and/or look for slow drain devices in you fabric(s).

 

New Contributor
Posts: 3
Registered: ‎11-04-2013

Re: We are receiving severe latency bottleneck warnings on an ISL port. How can I troubleshoot thi

I really wasn't using San Health to troubleshoot, per se.  Just happened to notice that it was the only port that was in that locked state, and it just happened to be the port with the bottleneck.  I wasn't sure what that status was.

 

We do use CMCNE and I was told that BNA was wrapped up in there, and that is how the bottleneck was noticed in the first place via CMCNE.  

 

I'm extremely green to this and was just wondering what to do next?  The problem port is identified, how do I go about tracking down slow drain devices? How do I figure out what the issue is?  Also, bottleneckmon is disabled on our switches.

Regular Contributor
Posts: 161
Registered: ‎12-30-2009

Re: We are receiving severe latency bottleneck warnings on an ISL port. How can I troubleshoot thi

CMCNE is EMC's version of BNA.

If you know what the problem port is you don't need to troubleshoot further.

My guess is you mean with problem port the ISL port.

Usually the port reporting it is not the one causing it.

So your ISL port is most likely the symptom or result of the issue.

 

A slow drain device is a device lwhich cannot handle trafic being sent to it fast enough.

For instance a 8GB storage port sents traffic faster than a 1GB host port can consume.

The 1GB host port acts as the slow drain because it is not giving back buffer credit back to the switch fast enough.

So the switch will hold the frames in buffer against the switchport the host is using.

In turn buffers at ISLport start to will up  and buffers are not returned to the other end.

etc etc.

 

This happens on all ports along the IO path, unfortunately you'll see it most on the ISL as they process, in general, more traffic as other ports.

 

try to enable bottleneckmon, it could be helpful

New Contributor
Posts: 3
Registered: ‎11-04-2013

Re: We are receiving severe latency bottleneck warnings on an ISL port. How can I troubleshoot thi

Thanks for the reply!  Makes a lot more sense now, and the situation you describe is quite similar to what is present in the environment: 8GB storage ports to 4GB host port.  I will try and see if we can turn bottleneckmon; thanks for the suggestion on that.

 

This may be a dumb question, but when there is a situation in the environment as I described, a mismatch of speeds, can anything be done about that (to prevent the bottleneck, if indeed that is what is causing it) short of getting the 8GB host connections?

Regular Contributor
Posts: 161
Registered: ‎12-30-2009

Re: We are receiving severe latency bottleneck warnings on an ISL port. How can I troubleshoot thi

[ Edited ]

Change your storage port to 4G Smiley Wink

 

You could tinker with QoS or Ingress Rate Limiting, reducing buffer credits on port (if possible on F ports)

Join the Community

Get quick and easy access to valuable resource designed to help you manage your Brocade Network.