HTTPS connection hangs during SSL handshake

Discussion:

Supratik Goswami

2012-09-04 13:51:46 UTC

I am using OpenSSL version : openssl-1.0.0j in our production.

I am facing a strange problem where the SSL connection simply hangs
during initial handshake when requested from our office IP address.
When I run the same command from another IP address it works fine.
[***@gateway ]# openssl s_client -connect test.mydomain.com:443
CONNECTED(00000003)
***@ip-10-0-0-10 (Development):~$ openssl s_client -connect
test.mydomain.com:443
CONNECTED(00000003)
depth=3 /L=ValiCert Validation Network/O=ValiCert, Inc./OU=ValiCert
Class 2 Policy Validation
Authority/CN=http://www.valicert.com//emailAddress=info-No2Jvm4j4PRWk0Htik3J/***@public.gmane.org
verify error:num=19:self signed certificate in certificate chain
verify return:0
---
Certificate chain
0 s:/O=*.mydomain.com/OU=Domain Control Validated/CN=*.mydomain.com
i:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com,
Inc./OU=http://certificates.godaddy.com/repository/CN=Go Daddy Secure
Certification Authority/serialNumber=07969287
1 s:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com,
Inc./OU=http://certificates.godaddy.com/repository/CN=Go Daddy Secure
Certification Authority/serialNumber=07969287
i:/C=US/O=The Go Daddy Group, Inc./OU=Go Daddy Class 2
Certification Authority
2 s:/C=US/O=The Go Daddy Group, Inc./OU=Go Daddy Class 2
Certification Authority
i:/L=ValiCert Validation Network/O=ValiCert, Inc./OU=ValiCert Class
2 Policy Validation
Authority/CN=http://www.valicert.com//emailAddress=info-No2Jvm4j4PRWk0Htik3J/***@public.gmane.org
3 s:/L=ValiCert Validation Network/O=ValiCert, Inc./OU=ValiCert Class
2 Policy Validation
Authority/CN=http://www.valicert.com//emailAddress=info-No2Jvm4j4PRWk0Htik3J/***@public.gmane.org
i:/L=ValiCert Validation Network/O=ValiCert, Inc./OU=ValiCert Class
2 Policy Validation
Authority/CN=http://www.valicert.com//emailAddress=info-No2Jvm4j4PRWk0Htik3J/***@public.gmane.org
---
Server certificate
-----BEGIN CERTIFICATE-----

REMOVED FOR SECURITY REASON

-----END CERTIFICATE-----
subject=/O=*.mydomain.com/OU=Domain Control Validated/CN=*.mydomain.com
issuer=/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com,
Inc./OU=http://certificates.godaddy.com/repository/CN=Go Daddy Secure
Certification Authority/serialNumber=07969287
---
No client certificate CA names sent
---
SSL handshake has read 4827 bytes and written 435 bytes
---
New, TLSv1/SSLv3, Cipher is RC4-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
Protocol : TLSv1
Cipher : RC4-SHA
Session-ID: 276ADBFB75336E7E870C5E109B4C5F6AFB8328C8775029EF135C5DA6F8608533
Session-ID-ctx:
Master-Key:
22B470A67XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXB50ED6237BE9
Key-Arg : None
Start Time: 1346765613
Timeout : 300 (sec)
Verify return code: 19 (self signed certificate in certificate chain

Any ideas ?

Supratik Goswami

2012-09-11 13:14:41 UTC

Permalink

Is there no one in the community who can help me to find the cause of
the problem ?

On Tue, Sep 4, 2012 at 7:21 PM, Supratik Goswami

Post by Supratik Goswami
I am using OpenSSL version : openssl-1.0.0j in our production.
I am facing a strange problem where the SSL connection simply hangs
during initial handshake when requested from our office IP address.
When I run the same command from another IP address it works fine.
CONNECTED(00000003)
test.mydomain.com:443
CONNECTED(00000003)
depth=3 /L=ValiCert Validation Network/O=ValiCert, Inc./OU=ValiCert
Class 2 Policy Validation
verify error:num=19:self signed certificate in certificate chain
verify return:0
---
Certificate chain
0 s:/O=*.mydomain.com/OU=Domain Control Validated/CN=*.mydomain.com
i:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com,
Inc./OU=http://certificates.godaddy.com/repository/CN=Go Daddy Secure
Certification Authority/serialNumber=07969287
1 s:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com,
Inc./OU=http://certificates.godaddy.com/repository/CN=Go Daddy Secure
Certification Authority/serialNumber=07969287
i:/C=US/O=The Go Daddy Group, Inc./OU=Go Daddy Class 2
Certification Authority
2 s:/C=US/O=The Go Daddy Group, Inc./OU=Go Daddy Class 2
Certification Authority
i:/L=ValiCert Validation Network/O=ValiCert, Inc./OU=ValiCert Class
2 Policy Validation
3 s:/L=ValiCert Validation Network/O=ValiCert, Inc./OU=ValiCert Class
2 Policy Validation
i:/L=ValiCert Validation Network/O=ValiCert, Inc./OU=ValiCert Class
2 Policy Validation
---
Server certificate
-----BEGIN CERTIFICATE-----
REMOVED FOR SECURITY REASON
-----END CERTIFICATE-----
subject=/O=*.mydomain.com/OU=Domain Control Validated/CN=*.mydomain.com
issuer=/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com,
Inc./OU=http://certificates.godaddy.com/repository/CN=Go Daddy Secure
Certification Authority/serialNumber=07969287
---
No client certificate CA names sent
---
SSL handshake has read 4827 bytes and written 435 bytes
---
New, TLSv1/SSLv3, Cipher is RC4-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
Protocol : TLSv1
Cipher : RC4-SHA
Session-ID: 276ADBFB75336E7E870C5E109B4C5F6AFB8328C8775029EF135C5DA6F8608533
22B470A67XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXB50ED6237BE9
Key-Arg : None
Start Time: 1346765613
Timeout : 300 (sec)
Verify return code: 19 (self signed certificate in certificate chain
Any ideas ?
--
Warm Regards
Supratik

Leonardo Laface de Almeida

2012-09-11 14:08:14 UTC

Permalink

For any SSL connection, you have to assure that:

1- The cpu's can reach each other (the hostname "test.mydomain.com" must be also resolved). You may use ping, HTTP, FTP to check it out;
2- Certificates or CA chain from each endpoint must be inserted in the opposite side as trust cert;
3- The both sides must have at least one cipher in common;
4- No NAT or Firewall is filtering the messages.

I have never made a connection by openssl command line, so, I can't tell you how to check it out .

I advice you to use some sniffer in at least one side, then you can reach the error, eg. where handshake is failuring, get the error code, etc... Using this you might be able to solve your problemm.

As I saw your logs, perhaps one side doesn't trust in the opposite cert received. That may happen for many reasons. I've already got some cases that the hostname (in your case "test.mydomain.com") must match with certificate common name (CN).

I hope it helps.
Leonardo

-----Mensagem original-----
De: owner-openssl-users-MCmKBN63+***@public.gmane.org [mailto:owner-openssl-users-MCmKBN63+***@public.gmane.org] Em nome de Supratik Goswami
Enviada em: terça-feira, 11 de setembro de 2012 10:15
Para: openssl-users-MCmKBN63+***@public.gmane.org
Assunto: Re: HTTPS connection hangs during SSL handshake

Is there no one in the community who can help me to find the cause of
the problem ?

On Tue, Sep 4, 2012 at 7:21 PM, Supratik Goswami

--
Warm Regards

Supratik
______________________________________________________________________
OpenSSL Project http://www.openssl.org
User Support Mailing List openssl-users-MCmKBN63+***@public.gmane.org
Automated List Manager majordomo-MCmKBN63+***@public.gmane.org

______________________________________________________________________
OpenSSL Project http://www.openssl.org
User Support Mailing List openssl-users-MCmKBN63+***@public.gmane.org
Automated List Manager majordomo-MCmKBN63+***@public.gmane.org

Dave Thompson

2012-09-12 02:55:07 UTC

Permalink

Almeida

Sent: Tuesday, 11 September, 2012 10:08
1- The cpu's can reach each other (the hostname
"test.mydomain.com" must be also resolved). You may use ping,
HTTP, FTP to check it out;

More exactly, the TCP stacks must be able to connect.
That requires slightly more than IP reachability --
not much more, but enough to be a problem in rare cases.
But "CONNECTED(fd)" from s_client means they *did* TCP
connect, so that's not the problem here.

2- Certificates or CA chain from each endpoint must be
inserted in the opposite side as trust cert;

A problem here would cause a handshake error not a hang.

3- The both sides must have at least one cipher in common;

A problem here would cause a handshake error not a hang.

4- No NAT or Firewall is filtering the messages.

Yes, or possibly other middlebox, see below.

I have never made a connection by openssl command line, so, I
can't tell you how to check it out .
I advice you to use some sniffer in at least one side, then
you can reach the error, eg. where handshake is failuring,
get the error code, etc... Using this you might be able to
solve your problemm.

Maybe both sides, see below.

As I saw your logs, perhaps one side doesn't trust in the
opposite cert received. That may happen for many reasons.
I've already got some cases that the hostname (in your case
"test.mydomain.com") must match with certificate common name (CN).

According to the log posted, his host is www.mydomain.com and
the cert is for *.mydomain.com . That is a valid wildcard match,
and should be acceptable to any conforming client. But openssl
library and s_client doesn't do hostname matching at all.
(*Apps* using openssl normally should, and at least some do.)

I don't know if "mydomain" is supposedly real or munged for posting.
mydomain.com is a real company and test.mydomain.com doesn't
resolve publicly and the cert chain used for {www.,}mydomain.com
publicly is wholly different from the OP's log.

OP's s_client fails to verify the received chain because it
(apparently) doesn't have the ValiCert root in its truststore.
Official openssl does not distribute any default trusted roots,
although custom packages of it may, as may apps using it.
OP probably didn't install a default truststore (or possibly
is using a build that has the default truststore wrong).

But failure to verify should cause a real app to reject the
connection, and s_client as a test tool overrides the verify
error and continues. Neither of these is a hang.

In the other direction, s_client doesn't do client authentication
and send a client cert unless explicitly specified, which the OP
didn't. If the server wants client-auth and client doesn't provide
it or provides a cert (chain) which server doesn't trust, that will
give a handshake error, not a hang.

-----Mensagem original-----
Enviada em: terça-feira, 11 de setembro de 2012 10:15
Assunto: Re: HTTPS connection hangs during SSL handshake
Is there no one in the community who can help me to find the cause of
the problem ?
On Tue, Sep 4, 2012 at 7:21 PM, Supratik Goswami

Use s_client with at least -state and preferably -debug or -msg
(you don't need both) to see how far it's getting in the handshake.

If you receive some handshake messages but not all, it practically
must be the server; talk to the server operator(s). It would be
unusual, but not impossible, for the server to mishandle connections
from one IP while it works for another. If you receive no message
at all, it might be server (try them) or it might be network
weirdness as (Mr?) de Almeida suggests; try a sniffer on your client
machine or near it (same LAN), and if that looks okay also try one
on or near the server (you may need server operator(s) to do that).

For Windows or Mac, I recommend www.wireshark.org . Very capable,
easy to install and use, well maintained. I don't know an equally
good solution for Linux, but there may be one, or at minimum you can
capture with tcpdump and if it's anything more complicated than
no-response you can copy the capture and decode with wireshark.

One possibility -- some servers want to lookup in DNS the address
of the client who connects to them (called reverse DNS or rDNS).
If this is done synchronously before the SSL handshake, and rDNS
is set-up correctly for client#2 but not client#1, then this could
cause the handshake for client#1 to stall. But even in worst case,
(r)DNS should timeout within a minute or so, and then the server
should either continue or reject the connection (disconnect).

Post by Supratik Goswami
test.mydomain.com:443
CONNECTED(00000003)

<snip: normal for unverified cert chain>

______________________________________________________________________
OpenSSL Project http://www.openssl.org
User Support Mailing List openssl-users-MCmKBN63+***@public.gmane.org
Automated List Manager majordomo-MCmKBN63+***@public.gmane.org

Aleksandr Konstantinov

2012-09-11 13:32:30 UTC

Permalink

Post by Supratik Goswami
Is there no one in the community who can help me to find the cause of
the problem ?

Maybe You have firewall issues on "office IP" macine. Have You tried tcpdump or
similar utility to check if there is something being sent/received?

Regards,

A.K.

Post by Supratik Goswami
On Tue, Sep 4, 2012 at 7:21 PM, Supratik Goswami

______________________________________________________________________
OpenSSL Project http://www.openssl.org
User Support Mailing List openssl-users-MCmKBN63+***@public.gmane.org
Automated List Manager majordomo-MCmKBN63+***@public.gmane.org

Supratik Goswami

2012-09-12 10:16:02 UTC

Permalink

It is not a firewall issue, I checked this from outside firewall. The
strange part of the problem is
it does not happen always, it works intermittently.

[***@gateway bin]# openssl s_client -bugs -connect
test.mydomain.com:443 -msg -state
CONNECTED(00000003)
SSL_connect:before/connect initialization

Post by Aleksandr Konstantinov

Post by Supratik Goswami

SSL 2.0 [length 0067], CLIENT-HELLO

01 03 01 00 4e 00 00 00 10 00 00 39 00 00 38 00
00 35 00 00 16 00 00 13 00 00 0a 07 00 c0 00 00
33 00 00 32 00 00 2f 03 00 80 00 00 05 00 00 04
01 00 80 00 00 15 00 00 12 00 00 09 06 00 40 00
00 14 00 00 11 00 00 08 00 00 06 04 00 80 00 00
03 02 00 80 00 00 ff c6 89 a6 e3 3e 51 4c 4b d9
e2 c4 29 01 63 54 06
SSL_connect:SSLv2/v3 write client hello A

It simply hangs after this.

* Here "test.mydomain.com" is not real it is used for posting.

On Tue, Sep 11, 2012 at 7:02 PM, Aleksandr Konstantinov

Post by Aleksandr Konstantinov

Post by Supratik Goswami
Is there no one in the community who can help me to find the cause of
the problem ?

Maybe You have firewall issues on "office IP" macine. Have You tried tcpdump or
similar utility to check if there is something being sent/received?
Regards,
A.K.

Post by Supratik Goswami
On Tue, Sep 4, 2012 at 7:21 PM, Supratik Goswami

I am using OpenSSL version : openssl-1.0.0j in our production.
I am facing a strange problem where the SSL connection simply hangs
during initial handshake when requested from our office IP address.
When I run the same command from another IP address it works fine.
CONNECTED(00000003)
test.mydomain.com:443
CONNECTED(00000003)
depth=3 /L=ValiCert Validation Network/O=ValiCert, Inc./OU=ValiCert
Class 2 Policy Validation
verify error:num=19:self signed certificate in certificate chain
verify return:0
---
Certificate chain
0 s:/O=*.mydomain.com/OU=Domain Control Validated/CN=*.mydomain.com
i:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com,
Inc./OU=http://certificates.godaddy.com/repository/CN=Go Daddy Secure
Certification Authority/serialNumber=07969287
1 s:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com,
Inc./OU=http://certificates.godaddy.com/repository/CN=Go Daddy Secure
Certification Authority/serialNumber=07969287
i:/C=US/O=The Go Daddy Group, Inc./OU=Go Daddy Class 2
Certification Authority
2 s:/C=US/O=The Go Daddy Group, Inc./OU=Go Daddy Class 2
Certification Authority
i:/L=ValiCert Validation Network/O=ValiCert, Inc./OU=ValiCert Class
2 Policy Validation
3 s:/L=ValiCert Validation Network/O=ValiCert, Inc./OU=ValiCert Class
2 Policy Validation
i:/L=ValiCert Validation Network/O=ValiCert, Inc./OU=ValiCert Class
2 Policy Validation
---
Server certificate
-----BEGIN CERTIFICATE-----
REMOVED FOR SECURITY REASON
-----END CERTIFICATE-----
subject=/O=*.mydomain.com/OU=Domain Control Validated/CN=*.mydomain.com
issuer=/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com,
Inc./OU=http://certificates.godaddy.com/repository/CN=Go Daddy Secure
Certification Authority/serialNumber=07969287
---
No client certificate CA names sent
---
SSL handshake has read 4827 bytes and written 435 bytes
---
New, TLSv1/SSLv3, Cipher is RC4-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
Protocol : TLSv1
Cipher : RC4-SHA
Session-ID: 276ADBFB75336E7E870C5E109B4C5F6AFB8328C8775029EF135C5DA6F8608533
22B470A67XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXB50ED6237BE9
Key-Arg : None
Start Time: 1346765613
Timeout : 300 (sec)
Verify return code: 19 (self signed certificate in certificate chain
Any ideas ?
--
Warm Regards
Supratik

______________________________________________________________________
OpenSSL Project http://www.openssl.org