Application Delivery (ADX)

Reply
Occasional Contributor
Posts: 7
Registered: ‎06-22-2016

Truncated HTTP POST body on ADX 1000 with SSL termination

Hi,

 

We have spent the last couple of weeks trying to resolve a problem where a HTTP POST to our server from another server was occasionally missing its body (causing an error in our application). The issue seems to be with our ADX 1000 load balancer - if we bypass the LB then the problem no longer occurs.

 

The LB is configured for SSL termination, firmware version 12.5.02eT403. We get many GB of traffic per day, this seems to be the only problem.

 

We have this WireShark trace from one of our real servers that shows the issue:

 

https://www.dropbox.com/s/22g0cispjaho7f6/Screenshot%202016-06-15%2017.43.41.png?dl=0

 

You can see that we get the HTTP headers but the final packet (with the body) is missing. It also seems a bit odd that the LB has split the data into so many tiny TCP segments (most of them have a payload of only 15 bytes).

 

We see this with the successful traffic as well - the headers are split into 13 small frames, with the body coming as a single frame at the end. Here is an example:

 

https://www.dropbox.com/s/i0d2d4rngbg4mbb/Screenshot%202016-06-15%2015.11.42.png?dl=0

 

Does anyone have any idea of what the issue might be? RackSpace manage the LB for us, so far they have not been able to diagnose the problem. I'm hoping someone in this community might have some insight!

 

Cheers,

 

Steve

External Moderator
Posts: 4,809
Registered: ‎02-23-2004

Re: Truncated HTTP POST body on ADX 1000 with SSL termination

[ Edited ]

Hi Steve,

 

here is a DOC about SSL termination. Did you already tried to follow this procedure ?

 

http://community.brocade.com/t5/Application-Delivery-ADX/SSL-offload-and-acceleration-concepts-and-examples/ta-p/3871

 

 

TechHelp24
Occasional Contributor
Posts: 7
Registered: ‎06-22-2016

Re: Truncated HTTP POST body on ADX 1000 with SSL termination

Hi,

 

Yes, we have followed this (or rather Rackspace have followed it for us).

 

Our ADX SSL termination config has been working for several years and deals with millions of hits per month. It only seems to be traffic from this client that causes this problem, only in the last few months, and then only intermittently. We have 9 real servers, 107 SSL profiles and 75 virtual servers.

 

Cheers,

 

Steve

Occasional Contributor
Posts: 7
Registered: ‎06-22-2016

Re: Truncated HTTP POST body on ADX 1000 with SSL termination

Rackspace have tried increasing the TCP buffer size with no success. Does anyone have any ideas?

Occasional Contributor
Posts: 16
Registered: ‎06-13-2011

Re: Truncated HTTP POST body on ADX 1000 with SSL termination

I'm very interested in this as well.

 

We're currently running 12.5.02bT403.  We're running 10 real servers & get about 36m requests per day.  About 29K/day are reported as "400" errors.  I believe some of these are due to no request body - but I've not done wireshark captures yet.

 

Please post any further updates.

 

Mike

Occasional Contributor
Posts: 7
Registered: ‎06-22-2016

Re: Truncated HTTP POST body on ADX 1000 with SSL termination

It looks like a software issue on the ADX. Brocade are investigating...

Occasional Contributor
Posts: 16
Registered: ‎06-13-2011

Re: Truncated HTTP POST body on ADX 1000 with SSL termination

Extremely interested now.

 

Please let me know what you/they find.

 

Mike

Occasional Contributor
Posts: 7
Registered: ‎06-22-2016

Re: Truncated HTTP POST body on ADX 1000 with SSL termination

We are two months in now, and Brocade still don't seem to have any idea what the problem is. I will stay away from Brocade products in the future.

External Moderator
Posts: 4,809
Registered: ‎02-23-2004

Re: Truncated HTTP POST body on ADX 1000 with SSL termination

@Steve Baxter

 

-->>We are two months in now, and Brocade still don't seem to have any idea what the problem is. 

 

you wrote in preview post:

 

->It looks like a software issue on the ADX. Brocade are investigating...

 

did you opened a TAC with brocade ?

 

looking in Brocade KB, and  found several Article related to ADX and SSL termination, however i'm not sure what exact you problem.

 

 

TechHelp24
Occasional Contributor
Posts: 7
Registered: ‎06-22-2016

Re: Truncated HTTP POST body on ADX 1000 with SSL termination

[ Edited ]

Hi Antonio,

 

Rackspace have the relationship with Brocade, they will have opened the TAC (I don't have a Brocade reference unfortunately). 

 

It's Rackspace ticket 160607-03565, they say they opened the TAC with Brocade on Jun 28, 2016. The Rackspace employee who opened the case is Miroslaw Pabian. Maybe you can find the TAC from that?

 

Cheers,

 

Steve

Join the Community

Get quick and easy access to valuable resource designed to help you manage your Brocade Network.