Discussion:
Performance related queries for SSL based client server model
Alok Sharma
2014-09-07 07:30:17 UTC
Permalink
Hi,
I am writing one sample ssl based client server model which uses
SSL_Read & SSL_Write API provided by openssl. But I found that my
application is very slow it takes around 40 mins to copy 700MB file. While
same file using scp finishes in 10 mins.
So my query is that is there an alternative way to use open ssl read or
write to improve performance. I searched in scp code and found it does not
use SSL_read/SSL_write. So if there is another set of APIs which I can use
or any idea how I can meet the same performance as scp.
Regards,
Alok
Dave Thompson
2014-09-07 11:23:51 UTC
Permalink
This is not a –dev question, and there’s no need to send three times.



scp uses the SSH protocol. OpenSSL does not implement SSH.

OpenSSH, which is a different product from a different source, implements

SSH, although in their design the scp program doesn’t do any comms at all,

it just pipes to the ssh program which does.



What kind of network(s) are you transiting, and what are your endpoints?

On my dev LAN, which is one uncongested reliable 100Mbps switch, I get

plain TCP at nearly the hardware limit 8sec per 100MB, and within 10% of

that for SCP/SSH or trivial-app/SSL. These do 700MB in barely a minute.



SSL and SSH differ significantly in connection setup/handshake, and slightly

in multiplexing the data, but once actually sending application data they use

mostly the same range of ciphers and MAC, with openssh actually calling

libcrypto, and use TCP pretty much the same way, so unless you’re doing or

(perhaps unintentionally) invoking something wrong, you should get roughly

the same speed for both.



Try netcat to measure only the network (and disk) with almost no CPU;

that gives you an upper bound on any protocol – except one that can and does

compress well: I believe openssh can and openssl definitely can depending

on how it’s built, but many people disable it post-CRIME, and it certainly

depends very much on your data. You might try gzip on your data and

if that makes much difference send the gzipped form.





From: owner-openssl-***@openssl.org [mailto:owner-openssl-***@openssl.org] On Behalf Of Alok Sharma
Sent: Sunday, September 07, 2014 03:30
To: openssl-***@openssl.org; openssl-***@openssl.org
Subject: Performance related queries for SSL based client server model



Hi,

I am writing one sample ssl based client server model which uses SSL_Read & SSL_Write API provided by openssl. But I found that my application is very slow it takes around 40 mins to copy 700MB file. While same file using scp finishes in 10 mins.

So my query is that is there an alternative way to use open ssl read or write to improve performance. I searched in scp code and found it does not use SSL_read/SSL_write. So if there is another set of APIs which I can use or any idea how I can meet the same performance as scp.

Regards,
Alok
Viktor Dukhovni
2014-09-07 14:41:42 UTC
Permalink
Post by Alok Sharma
I am writing one sample ssl based client server model which uses
SSL_Read & SSL_Write API provided by openssl.
If you transfering each block of data as an RPC, with a round-trip
acknowledgement before sending the next block, and the blocks are
small enough, you're going to severely limit throughput.

In bulk data transfer applications that stream data, TLS typically
outperforms SSH, but a lot depends on the details.
--
Viktor.
______________________________________________________________________
OpenSSL Project http://www.openssl.org
User Support Mailing List openssl-users-MCmKBN63+***@public.gmane.org
Automated List Manager majordomo-MCmKBN63+***@public.gmane.org
Alok Sharma
2014-09-08 17:46:46 UTC
Permalink
Hi,
I am trying with simple client server model where it is reading datat
from a file and calling SSL_write and similar thing is happening on other
end i.e. reading from SSL using SSL_read and writing to file. Itis taking
40 mins of time to copy file of size 600 MB and if I run a non-ssl client
server (i.e. pure TCP socket application ) it finishes in 5-6 mins and also
on same network scp is taking 10 mins. One thing I observerd by looking
into scp code that it does not use SSL provided APIs (i.e.SSL_Read or
SSL_Write) but they use it differenly i.e. might be directly calling
encryption APIs and writing data to sockets. But I don't have much
understanding what SSL_Write or SSL_read does internally . So wanted to
understand if there is any way to improve performance of SSL_Read or
SSL_write to achive high performance. Following are my client server
programmes. Here client writes file on server machine in hardcoded
location and name.

Client
----------------------------

//SSL-Client.c
#include <stdio.h>
#include <errno.h>
#include <unistd.h>
#include <malloc.h>
#include <string.h>
#include <sys/socket.h>
#include <resolv.h>
#include <netdb.h>
#include <openssl/ssl.h>
#include <openssl/err.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <netinet/tcp.h>
#include <netinet/ip.h>

#define FAIL -1

int OpenConnection(const char *hostname, int port)
{ int sd;
struct hostent *host;
struct sockaddr_in addr;

if ( (host = gethostbyname(hostname)) == NULL )
{
perror(hostname);
abort();
}
sd = socket(PF_INET, SOCK_STREAM, 0);
bzero(&addr, sizeof(addr));
addr.sin_family = AF_INET;
addr.sin_port = htons(port);
addr.sin_addr.s_addr = *(long*)(host->h_addr);
if ( connect(sd, (struct sockaddr*)&addr, sizeof(addr)) != 0 )
{
close(sd);
perror(hostname);
abort();
}
return sd;
}

SSL_CTX* InitCTX(void)
{ SSL_METHOD *method;
SSL_CTX *ctx;

OpenSSL_add_all_algorithms(); /* Load cryptos, et.al. */
SSL_load_error_strings(); /* Bring in and register error messages */
method = SSLv2_client_method(); /* Create new client-method instance */
ctx = SSL_CTX_new(method); /* Create new context */
if ( ctx == NULL )
{
ERR_print_errors_fp(stderr);
abort();
}
return ctx;
}

void ShowCerts(SSL* ssl)
{ X509 *cert;
char *line;

cert = SSL_get_peer_certificate(ssl); /* get the server's certificate */
if ( cert != NULL )
{
printf("Server certificates:\n");
line = X509_NAME_oneline(X509_get_subject_name(cert), 0, 0);
printf("Subject: %s\n", line);
free(line); /* free the malloc'ed string */
line = X509_NAME_oneline(X509_get_issuer_name(cert), 0, 0);
printf("Issuer: %s\n", line);
free(line); /* free the malloc'ed string */
X509_free(cert); /* free the malloc'ed certificate copy */
}
else
printf("No certificates.\n");
}

int main(int count, char *strings[])
{ SSL_CTX *ctx;
int server;
SSL *ssl;
char buf[1024];
int bytes;
char *hostname, *portnum;
char recvBuff[16*1024];
int fd=0;
int mode,i,n,len;
int total=0;
if ( count != 4 )
{
printf("usage: %s <hostname> <portnum><FileName>\n", strings[0]);
exit(0);
}
SSL_library_init();
hostname=strings[1];
portnum=strings[2];

ctx = InitCTX();
server = OpenConnection(hostname, atoi(portnum));
ssl = SSL_new(ctx); /* create new SSL connection state */
SSL_set_fd(ssl, server); /* attach the socket descriptor */
if ( SSL_connect(ssl) == FAIL ) /* perform the connection */
ERR_print_errors_fp(stderr);
else
{ char *msg = "Hello???";

printf("Connected with %s encryption\n", SSL_get_cipher(ssl));
ShowCerts(ssl); /* get any certs */

fd=open(strings[3],O_RDWR);

while ( (n = read(fd, recvBuff, sizeof(recvBuff)-1)) > 0)
{
recvBuff[n] = 0;
mode=n;
i=0;
while(mode>0)
{
len=SSL_write(ssl, recvBuff,mode ); /* encrypt & send
message */
total+=len;
mode=mode-len;
}
}


bytes = SSL_read(ssl, buf, sizeof(buf)); /* get reply & decrypt */
buf[bytes] = 0;
printf("Received: \"%s\"\n", buf);
SSL_free(ssl); /* release connection state */
}
close(server); /* close socket */
SSL_CTX_free(ctx); /* release context */
return 0;
}


----------------------------------------------------

Server Programme
-------------------------------------------------


//SSL-Server.c
#include <errno.h>
#include <unistd.h>
#include <malloc.h>
#include <string.h>
#include <arpa/inet.h>
#include <sys/socket.h>
#include <sys/types.h>
#include <netinet/in.h>
#include <resolv.h>
#include "openssl/ssl.h"
#include "openssl/err.h"
#include <sys/stat.h>
#include <fcntl.h>
#include <netinet/tcp.h>
#include <netinet/ip.h>

#define FAIL -1

int OpenListener(int port)
{ int sd;
struct sockaddr_in addr;

sd = socket(PF_INET, SOCK_STREAM, 0);
bzero(&addr, sizeof(addr));
addr.sin_family = AF_INET;
addr.sin_port = htons(port);
addr.sin_addr.s_addr = INADDR_ANY;
if ( bind(sd, (struct sockaddr*)&addr, sizeof(addr)) != 0 )
{
perror("can't bind port");
abort();
}
if ( listen(sd, 10) != 0 )
{
perror("Can't configure listening port");
abort();
}
return sd;
}

SSL_CTX* InitServerCTX(void)
{ SSL_METHOD *method;
SSL_CTX *ctx;

OpenSSL_add_all_algorithms(); /* load & register all cryptos, etc. */
SSL_load_error_strings(); /* load all error messages */
method = SSLv2_server_method(); /* create new server-method instance */
ctx = SSL_CTX_new(method); /* create new context from method */
if ( ctx == NULL )
{
ERR_print_errors_fp(stderr);
abort();
}
return ctx;
}

void LoadCertificates(SSL_CTX* ctx, char* CertFile, char* KeyFile)
{
/* set the local certificate from CertFile */
if ( SSL_CTX_use_certificate_file(ctx, CertFile, SSL_FILETYPE_PEM) <= 0
)
{
ERR_print_errors_fp(stderr);
abort();
}
/* set the private key from KeyFile (may be the same as CertFile) */
if ( SSL_CTX_use_PrivateKey_file(ctx, KeyFile, SSL_FILETYPE_PEM) <= 0 )
{
ERR_print_errors_fp(stderr);
abort();
}
/* verify private key */
if ( !SSL_CTX_check_private_key(ctx) )
{
fprintf(stderr, "Private key does not match the public
certificate\n");
abort();
}
}

void ShowCerts(SSL* ssl)
{ X509 *cert;
char *line;

cert = SSL_get_peer_certificate(ssl); /* Get certificates (if
available) */
if ( cert != NULL )
{
printf("Server certificates:\n");
line = X509_NAME_oneline(X509_get_subject_name(cert), 0, 0);
printf("Subject: %s\n", line);
free(line);
line = X509_NAME_oneline(X509_get_issuer_name(cert), 0, 0);
printf("Issuer: %s\n", line);
free(line);
X509_free(cert);
}
else
printf("No certificates.\n");
}

void Servlet(SSL* ssl) /* Serve the connection -- threadable */
{ char buf[1024*16];
char reply[1024];
int sd, bytes;
int fd=0;
int mode,len,len1;
const char* HTMLecho="<html><body><pre>%s</pre></body></html>\n\n";

if ( SSL_accept(ssl) == FAIL ) /* do SSL-protocol accept */
ERR_print_errors_fp(stderr);
else
{
ShowCerts(ssl); /* get any certificates */
fd=open("/tmp/test1",O_WRONLY | O_CREAT, 0644);
while(1)
{
bytes = SSL_read(ssl, buf, sizeof(buf)); /* get request */
if ( bytes > 0 )
{
// printf("Client msg: \"%s\"\n", buf);
//sprintf(reply, HTMLecho, buf); /* construct reply */
//SSL_write(ssl, reply, strlen(reply)); /* send reply */
len=bytes;
mode=len;
while(mode>0)
{
len1=write(fd,buf,len);
mode=mode-len1;
len-=len1;
}

}
else {
ERR_print_errors_fp(stderr);
break;
}

}
}
sd = SSL_get_fd(ssl); /* get socket connection */
SSL_free(ssl); /* release SSL state */
close(sd); /* close connection */
}

int main(int count, char *strings[])
{ SSL_CTX *ctx;
int server;
char *portnum;

if ( count != 2 )
{
printf("Usage: %s <portnum>\n", strings[0]);
exit(0);
}
SSL_library_init();

portnum = strings[1];
ctx = InitServerCTX(); /* initialize SSL */
LoadCertificates(ctx, "mycert.pem", "mycert.pem"); /* load certs */
server = OpenListener(atoi(portnum)); /* create server socket */
while (1)
{ struct sockaddr_in addr;
socklen_t len = sizeof(addr);
SSL *ssl;

int client = accept(server, (struct sockaddr*)&addr, &len); /*
accept connection as usual */
printf("Connection: %s:%d\n",inet_ntoa(addr.sin_addr),
ntohs(addr.sin_port));
ssl = SSL_new(ctx); /* get new SSL state with context
*/
SSL_set_fd(ssl, client); /* set connection socket to SSL state
*/
Servlet(ssl); /* service connection */
}
close(server); /* close server socket */
SSL_CTX_free(ctx); /* release context */
}
Post by Viktor Dukhovni
Post by Alok Sharma
I am writing one sample ssl based client server model which uses
SSL_Read & SSL_Write API provided by openssl.
If you transfering each block of data as an RPC, with a round-trip
acknowledgement before sending the next block, and the blocks are
small enough, you're going to severely limit throughput.
In bulk data transfer applications that stream data, TLS typically
outperforms SSH, but a lot depends on the details.
--
Viktor.
______________________________________________________________________
OpenSSL Project http://www.openssl.org
Viktor Dukhovni
2014-09-08 21:34:37 UTC
Permalink
Post by Alok Sharma
while ( (n = read(fd, recvBuff, sizeof(recvBuff)-1)) > 0)
{
recvBuff[n] = 0;
mode=n;
i=0;
while(mode>0)
{
len=SSL_write(ssl, recvBuff, mode);
total+=len;
mode=mode-len;
}
}
The inner loop is wrong unless all the data is always written on
each SSL_write() call.
Post by Alok Sharma
bytes = SSL_read(ssl, buf, sizeof(buf)); /* get reply & decrypt */
buf[bytes] = 0;
printf("Received: \"%s\"\n", buf);
SSL_free(ssl); /* release connection state */
Why do you expect data back from the server? It is surely still
waiting for more data from the client, since there is no explicit
message framing to indicate to the server that all the data is
sent, and it should reply. You're freeing the SSL state without
a gracefull SSL_shutdown() (often called twice, see the docs).
Post by Alok Sharma
while(1)
{
bytes = SSL_read(ssl, buf, sizeof(buf)); /* get request */
if ( bytes > 0 )
{
// printf("Client msg: \"%s\"\n", buf);
//sprintf(reply, HTMLecho, buf); /* construct reply */
//SSL_write(ssl, reply, strlen(reply)); /* send reply */
len=bytes;
mode=len;
while(mode>0)
{
len1=write(fd,buf,len);
mode=mode-len1;
len-=len1;
}
}
else {
ERR_print_errors_fp(stderr);
break;
}
}
THe inner loop is wrong unless all the data is written on each
write(2) call. The server is in an infinite read loop, deadlocked
with the client.
Post by Alok Sharma
}
sd = SSL_get_fd(ssl); /* get socket connection */
SSL_free(ssl); /* release SSL state */
close(sd); /* close connection */
And sends no reply. This code is broken, and should block forever
with SSL_read()/SSL_write() replaced with read()/write(). Since
you're reporting finite completion times, you're not posting the
code you're testing, which wastes everyone's time.
--
Viktor.
______________________________________________________________________
OpenSSL Project http://www.openssl.org
User Support Mailing List openssl-users-MCmKBN63+***@public.gmane.org
Automated List Manager majordomo-MCmKBN63+***@public.gmane.org
Iñaki Baz Castillo
2014-09-08 22:24:59 UTC
Permalink
One thing I observerd by looking into scp
code that it does not use SSL provided APIs (i.e.SSL_Read or SSL_Write) but
they use it differenly i.e. might be directly calling encryption APIs and
writing data to sockets. But I don't have much understanding what SSL_Write
or SSL_read does internally.
It has been already replied above. SSH is not SSL so don't look for
SSL_xxxx methods on openssh. Said that, AFAIK openssh uses the crypto
library from openssl, but that is not SSL/TLS at all.
So wanted to understand if there is any way to
improve performance of SSL_Read or SSL_write to achive high performance.
Following are my client server programmes. Here client writes file on
server machine in hardcoded location and name.
You have lot of errors in your program. I suggest that you first
properly learn openssl, then measure your code if you need.
--
Iñaki Baz Castillo
<ibc-***@public.gmane.org>
______________________________________________________________________
OpenSSL Project http://www.openssl.org
User Support Mailing List openssl-users-MCmKBN63+***@public.gmane.org
Automated List Manager majordomo-MCmKBN63+***@public.gmane.org
Alok Sharma
2014-09-09 08:58:03 UTC
Permalink
Hi,
   Thanks all for your update. But functionality wise it is working
fine. I can remove the inner loop but that will require packet size to
be of 1K. I tried with that also but did not find any improvement in
performance. In my setup there are 8 routers between source &
destination. Can anyone share me sample SSL based client server
programme which I could use to measure performance.
Regards,
Alok
Post by Iñaki Baz Castillo
One thing I observerd by looking into scp
code that it does not use SSL provided APIs (i.e.SSL_Read or SSL_Write) but
they use it differenly i.e. might be directly calling encryption APIs and
writing data to sockets. But I don't have much understanding what SSL_Write
or SSL_read does internally.
It has been already replied above. SSH is not SSL so don't look for
SSL_xxxx methods on openssh. Said that, AFAIK openssh uses the crypto
library from openssl, but that is not SSL/TLS at all.
So wanted to understand if there is any way to
improve performance of SSL_Read or SSL_write to achive high performance.
Following are my client server programmes. Here  client writes file on
server machine in hardcoded location and name.
You have lot of errors in your program. I suggest that you first
properly learn openssl, then measure your code if you need.
--
Iñaki Baz Castillo
______________________________________________________________________
OpenSSL Project                                 http://www.openssl.org
______________________________________________________________________
OpenSSL Project http://www.openssl.org
User Support Mailing List openssl-users-MCmKBN63+***@public.gmane.org
Automated List Manager majordomo-MCmKBN63+***@public.gmane.org
Loading...