From ebrun at haproxy.com Fri Jun 2 10:00:27 2017 From: ebrun at haproxy.com (Emeric Brun) Date: Fri, 2 Jun 2017 12:00:27 +0200 Subject: [openssl-dev] Questions about ASYNC API for engines and moving read buffer. Message-ID: <14138849-5765-f58a-8905-203784947bb7@haproxy.com> Hi, I'm working on an ASYNC engine designed to offload crypto on multiple thread for monoproc applications. No issue on asymetric crypto part but i'm facing a problem concerning ciphers: When an SSL_read operation perform a do_cipher decrypt: My engine defer the do_cipher to be processed by an other thread using the output buffer provided by to the do_cipher method. After that it performs an ASYNC_pause_job(). On the application side it results on a WANT_ASYNC error. The application consider that nothing was written on its output, and the buffer read buffer is reused for something else. SSL_read is called a second time after receiving the event on the async fd. But this time the app used a different buffer. On the engine side, i don't see any change on the do_cipher output buffer which would allow to manage the issue with a memcopy. So my question is: Is there a way to perform ciphering with an ASYNC engine when the app don't reuse the same buffer. R, Emeric Brun From matt at openssl.org Fri Jun 2 10:40:47 2017 From: matt at openssl.org (Matt Caswell) Date: Fri, 2 Jun 2017 11:40:47 +0100 Subject: [openssl-dev] Questions about ASYNC API for engines and moving read buffer. In-Reply-To: <14138849-5765-f58a-8905-203784947bb7@haproxy.com> References: <14138849-5765-f58a-8905-203784947bb7@haproxy.com> Message-ID: <4baddc68-ed5f-569c-7375-b9523566b967@openssl.org> On 02/06/17 11:00, Emeric Brun wrote: > Hi, > > I'm working on an ASYNC engine designed to offload crypto on multiple thread for monoproc applications. > > No issue on asymetric crypto part but i'm facing a problem concerning ciphers: > > When an SSL_read operation perform a do_cipher decrypt: > > My engine defer the do_cipher to be processed by an other thread using the output buffer provided by to the do_cipher method. > > After that it performs an ASYNC_pause_job(). On the application side it results on a WANT_ASYNC error. > > The application consider that nothing was written on its output, and the buffer read buffer is reused for something else. > > SSL_read is called a second time after receiving the event on the async fd. But this time the app used a different buffer. > > On the engine side, i don't see any change on the do_cipher output buffer which would allow to manage the issue with a memcopy. > > So my question is: > > Is there a way to perform ciphering with an ASYNC engine when the app don't reuse the same buffer. No, the async operation is running in it own fibre which is initialised on first call, so it is still using the arguments from that initial call. Subsequent calls won't "see" any changes in those args. We should probably be more explicit about that in the docs. Matt From ebrun at haproxy.com Fri Jun 2 11:26:24 2017 From: ebrun at haproxy.com (Emeric Brun) Date: Fri, 2 Jun 2017 13:26:24 +0200 Subject: [openssl-dev] Questions about ASYNC API for engines and moving read buffer. In-Reply-To: <4baddc68-ed5f-569c-7375-b9523566b967@openssl.org> References: <14138849-5765-f58a-8905-203784947bb7@haproxy.com> <4baddc68-ed5f-569c-7375-b9523566b967@openssl.org> Message-ID: Hi Matt, On 06/02/2017 12:40 PM, Matt Caswell wrote: > > > On 02/06/17 11:00, Emeric Brun wrote: >> Hi, >> >> I'm working on an ASYNC engine designed to offload crypto on multiple thread for monoproc applications. >> >> No issue on asymetric crypto part but i'm facing a problem concerning ciphers: >> >> When an SSL_read operation perform a do_cipher decrypt: >> >> My engine defer the do_cipher to be processed by an other thread using the output buffer provided by to the do_cipher method. >> >> After that it performs an ASYNC_pause_job(). On the application side it results on a WANT_ASYNC error. >> >> The application consider that nothing was written on its output, and the buffer read buffer is reused for something else. >> >> SSL_read is called a second time after receiving the event on the async fd. But this time the app used a different buffer. >> >> On the engine side, i don't see any change on the do_cipher output buffer which would allow to manage the issue with a memcopy. >> >> So my question is: >> >> Is there a way to perform ciphering with an ASYNC engine when the app don't reuse the same buffer. > > No, the async operation is running in it own fibre which is initialised > on first call, so it is still using the arguments from that initial > call. Subsequent calls won't "see" any changes in those args. We should > probably be more explicit about that in the docs. > > Matt > I see that you're using EVP_CIPH_FLAG_PIPELINE in the dasync engine and that you're calling do_cipher to write in pipe_ctx->outbufs[i]. I also know that intel's qat uses such interface. Where points these buffers? directly to the read buffer provided during the first SSL_read call or are they temporary allocated buffers copied into the read buffer passed during the second SSL_read call? R, Emeric From ebrun at haproxy.com Fri Jun 2 14:20:55 2017 From: ebrun at haproxy.com (Emeric Brun) Date: Fri, 2 Jun 2017 16:20:55 +0200 Subject: [openssl-dev] Questions about ASYNC API for engines and moving read buffer. In-Reply-To: References: <14138849-5765-f58a-8905-203784947bb7@haproxy.com> <4baddc68-ed5f-569c-7375-b9523566b967@openssl.org> Message-ID: <1ac874e0-22f0-ce3f-bcae-4843025b4b02@haproxy.com> Hi Matt, On 06/02/2017 01:26 PM, Emeric Brun wrote: > Hi Matt, > > On 06/02/2017 12:40 PM, Matt Caswell wrote: >> >> >> On 02/06/17 11:00, Emeric Brun wrote: >>> Hi, >>> >>> I'm working on an ASYNC engine designed to offload crypto on multiple thread for monoproc applications. >>> >>> No issue on asymetric crypto part but i'm facing a problem concerning ciphers: >>> >>> When an SSL_read operation perform a do_cipher decrypt: >>> >>> My engine defer the do_cipher to be processed by an other thread using the output buffer provided by to the do_cipher method. >>> >>> After that it performs an ASYNC_pause_job(). On the application side it results on a WANT_ASYNC error. >>> >>> The application consider that nothing was written on its output, and the buffer read buffer is reused for something else. >>> >>> SSL_read is called a second time after receiving the event on the async fd. But this time the app used a different buffer. >>> >>> On the engine side, i don't see any change on the do_cipher output buffer which would allow to manage the issue with a memcopy. >>> >>> So my question is: >>> >>> Is there a way to perform ciphering with an ASYNC engine when the app don't reuse the same buffer. >> >> No, the async operation is running in it own fibre which is initialised >> on first call, so it is still using the arguments from that initial >> call. Subsequent calls won't "see" any changes in those args. We should >> probably be more explicit about that in the docs. >> >> Matt >> > > I see that you're using EVP_CIPH_FLAG_PIPELINE in the dasync engine and that you're calling > do_cipher to write in pipe_ctx->outbufs[i]. I also know that intel's qat uses such interface. > > Where points these buffers? directly to the read buffer provided during the first SSL_read call > or are they temporary allocated buffers copied into the read buffer passed during the second > SSL_read call? > > R, > Emeric > I've just read the code and I see it is not possible. I'm disappointed because i think that a lot of applications which are using openssl in asynchronous mode also uses SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER and have ephemeral/reused/circular buffers for both read and write operations. As documented: SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER "make it possible to retry SSL_write() with changed buffer location" SSL_read implicitly also support it, and since the support of openssl in our application (0.9.8), we never faced such issue. In my case, this is a dead end: Enabling the async mode on any engine (mine or qat), it looks good during the hanshake but it causes buffer overflows switching in the data stage. Is there a way for the application to disable the async mode after the asymetric stuff (handshake)? At least we could use the new async api for that. R, Emeric From juan.guerrero at gmx.de Sun Jun 4 00:15:04 2017 From: juan.guerrero at gmx.de (Juan Manuel Guerrero) Date: Sun, 04 Jun 2017 02:15:04 +0200 Subject: [openssl-dev] [PATCH] Some DJGPP specific fixes and improvements for OpenSSL_1_0_2-stable. Message-ID: <59335108.40804@gmx.de> I do not know if patches for DJGPP/FreeDOS are still welcome at all for the OpenSSL 1.0.2 branch but if yes, then I would like to propose some fixes and improvements. No one of the proposed changes have impact on any other port. The patch is based on openssl-1.0.2-stable-SNAP-20170602. Concerning its functionality, the patch is identical to the one proposed some months ago for the openssl-1.1.0 branch and that has found its way into that branch. The patch will fix/improve the following issues: 1) In Configure: For some reason -DTERMIO is set but DJGPP has never offered TERMIO making the build fail. I have changed this to -DTERMIOS as is used to be. 2) In crypto/bio/bss_dgram.c: I have removed superfluous macro definitions of sock_write, sock_read and sock_puts enclosed by WATT32. 3) In crypto/bio/bss_sock.c: Here the existing macro definitions for sock_write, sock_read and sock_puts are necessary and must be kept but they must be undefined before they can be defined. This is because newer versions of Watt-32 also redefine them. 4) In crypto/conf/conf_def.c: If this port is used on MS-DOS or FreeDOS it becomes necessary to check if the underlying file system supports long file names (aka LFN) or not. If it does not then file names with a leading dot like ".rnd" or ".ca_certs" are ilicit. In function def_load_bio, the macros IS_RANDFILE and IS_CERT_DIR are used to check if the file system offers LFN support so that the file names with leading dots are licit. If the tests fail then the new function dosify_filename is called and will substitute invalid characters in the file name by valid ones before using them. This check and the call of dosify_filename is enclosed by OPENSSL_SYS_MSDOS. 5) In e_os.h: In the DJGPP section the macros IS_RANDFILE and IS_CERT_DIR are defined. Also some auxiliar macros like HAS_LFN_SUPPORT and FILE_EXISTS are defined. Because neither MS-DOS nor FreeDOS provide 'egd' sockets, the DEVRANDOM_EGD macro is undefined. This shall inhibit the compilation of code that does not work on MS-DOS/FreeDOS. 6) In util/mklink.pl: Neither MS-DOS nor FreeDOS provide symlink support so copy files instead. 7) In INSTALL.DJGPP: Update URL of WATT-32 library. I have checked the modified version of OpenSSL_1_0_2-stable on linux and Cygwin. They are no issues. This is no surprise because the changes are either guarded by the __DJGPP__ or OPENSSL_SYS_MSDOS macros. If more informaton is required please mail me. Just in case it is required, I have attached the patch as gzip'ed files too. Regards, Juan M. Guerrero diff -aprNU5 openssl-1.0.2-stable-SNAP-20170602.orig/Configure openssl-1.0.2-stable-SNAP-20170602/Configure --- openssl-1.0.2-stable-SNAP-20170602.orig/Configure 2017-06-01 20:51:32 +0000 +++ openssl-1.0.2-stable-SNAP-20170602/Configure 2017-06-03 19:41:20 +0000 @@ -632,11 +632,11 @@ my %table=( "netware-libc-bsdsock", "mwccnlm::::::BN_LLONG ${x86_gcc_opts}::", "netware-libc-gcc", "i586-netware-gcc:-nostdinc -I/ndk/libc/include -I/ndk/libc/include/winsock -DL_ENDIAN -DNETWARE_LIBC -DOPENSSL_SYSNAME_NETWARE -DTERMIO -O2 -Wall:::::BN_LLONG ${x86_gcc_opts}::", "netware-libc-bsdsock-gcc", "i586-netware-gcc:-nostdinc -I/ndk/libc/include -DNETWARE_BSDSOCK -DL_ENDIAN -DNETWARE_LIBC -DOPENSSL_SYSNAME_NETWARE -DTERMIO -O2 -Wall:::::BN_LLONG ${x86_gcc_opts}::", # DJGPP -"DJGPP", "gcc:-I/dev/env/WATT_ROOT/inc -DTERMIO -DL_ENDIAN -fomit-frame-pointer -O2 -Wall:::MSDOS:-L/dev/env/WATT_ROOT/lib -lwatt:BN_LLONG ${x86_gcc_des} ${x86_gcc_opts}:${x86_asm}:a.out:", +"DJGPP", "gcc:-I/dev/env/WATT_ROOT/inc -DTERMIOS -DL_ENDIAN -fomit-frame-pointer -O2 -Wall:::MSDOS:-L/dev/env/WATT_ROOT/lib -lwatt:BN_LLONG ${x86_gcc_des} ${x86_gcc_opts}:${x86_asm}:a.out:", # Ultrix from Bernhard Simon "ultrix-cc","cc:-std1 -O -Olimit 2500 -DL_ENDIAN::(unknown):::::::", "ultrix-gcc","gcc:-O3 -DL_ENDIAN::(unknown):::BN_LLONG::::", # K&R C is no longer supported; you need gcc on old Ultrix installations @@ -1532,10 +1532,27 @@ if ($sys_id ne "") if ($ranlib eq "") { $ranlib = $default_ranlib; } +# DJGPP specific CFLAG adjustments +if ($target =~ /^DJGPP/) + { + my $gccver=0; + if (open(FD,"$cc --version |")) + { + while() { $gccver=$1 if (/ (([1-3])\.|4\.([0-1])([.0-9]*))/); } + close(FD); + } + if ($gccver==0) + { + # For gcc 4.3.0 and above ensure that always old GNU extern inline semantics + # are used (aka -fgnu89-inline) even if ISO C99 semantics has been specified. + $cflags=~s/-fomit-frame-pointer/-fgnu89-inline -fomit-frame-pointer/; + } + } + #my ($bn1)=split(/\s+/,$bn_obj); #$bn1 = "" unless defined $bn1; #$bn1=$bn_asm unless ($bn1 =~ /\.o$/); #$bn_obj="$bn1"; diff -aprNU5 openssl-1.0.2-stable-SNAP-20170602.orig/crypto/bio/bss_dgram.c openssl-1.0.2-stable-SNAP-20170602/crypto/bio/bss_dgram.c --- openssl-1.0.2-stable-SNAP-20170602.orig/crypto/bio/bss_dgram.c 2017-06-01 20:51:32 +0000 +++ openssl-1.0.2-stable-SNAP-20170602/crypto/bio/bss_dgram.c 2017-06-03 19:41:20 +0000 @@ -92,16 +92,10 @@ (((a)->s6_addr32[0] == 0) && \ ((a)->s6_addr32[1] == 0) && \ ((a)->s6_addr32[2] == htonl(0x0000ffff))) # endif -# ifdef WATT32 -# define sock_write SockWrite /* Watt-32 uses same names */ -# define sock_read SockRead -# define sock_puts SockPuts -# endif - static int dgram_write(BIO *h, const char *buf, int num); static int dgram_read(BIO *h, char *buf, int size); static int dgram_puts(BIO *h, const char *str); static long dgram_ctrl(BIO *h, int cmd, long arg1, void *arg2); static int dgram_new(BIO *h); diff -aprNU5 openssl-1.0.2-stable-SNAP-20170602.orig/crypto/bio/bss_file.c openssl-1.0.2-stable-SNAP-20170602/crypto/bio/bss_file.c --- openssl-1.0.2-stable-SNAP-20170602.orig/crypto/bio/bss_file.c 2017-06-01 20:51:32 +0000 +++ openssl-1.0.2-stable-SNAP-20170602/crypto/bio/bss_file.c 2017-06-03 19:41:20 +0000 @@ -93,10 +93,14 @@ # include # endif # if !defined(OPENSSL_NO_STDIO) +#ifdef OPENSSL_SYS_MSDOS +# include +static void dosify_filename(const char *filename); +#endif static int MS_CALLBACK file_write(BIO *h, const char *buf, int num); static int MS_CALLBACK file_read(BIO *h, char *buf, int size); static int MS_CALLBACK file_puts(BIO *h, const char *str); static int MS_CALLBACK file_gets(BIO *h, char *str, int size); static long MS_CALLBACK file_ctrl(BIO *h, int cmd, long arg1, void *arg2); @@ -159,10 +163,13 @@ static FILE *file_fopen(const char *file } } else if (GetLastError() == ERROR_NO_UNICODE_TRANSLATION) { file = fopen(filename, mode); } # else +# ifdef OPENSSL_SYS_MSDOS + dosify_filename(filename); +# endif file = fopen(filename, mode); # endif return (file); } @@ -477,8 +484,25 @@ static int MS_CALLBACK file_puts(BIO *bp n = strlen(str); ret = file_write(bp, str, n); return (ret); } +# ifdef OPENSSL_SYS_MSDOS +static void dosify_filename(const char *filename) +{ + if (filename && *filename && !HAS_LFN_SUPPORT(filename)) { + char *nextchar = unconst(filename, char *); + + do { + if (nextchar[0] == '/' && nextchar[2] != '.' && nextchar[2] != '/') { + + /* Leading dot not allowed on plain DOS. */ + if (nextchar[1] == '.') + *++nextchar = '_'; + } + } while (*++nextchar); + } +} +# endif # endif /* OPENSSL_NO_STDIO */ #endif /* HEADER_BSS_FILE_C */ diff -aprNU5 openssl-1.0.2-stable-SNAP-20170602.orig/crypto/bio/bss_sock.c openssl-1.0.2-stable-SNAP-20170602/crypto/bio/bss_sock.c --- openssl-1.0.2-stable-SNAP-20170602.orig/crypto/bio/bss_sock.c 2017-06-01 20:51:32 +0000 +++ openssl-1.0.2-stable-SNAP-20170602/crypto/bio/bss_sock.c 2017-06-03 19:41:20 +0000 @@ -64,11 +64,15 @@ #ifndef OPENSSL_NO_SOCK # include # ifdef WATT32 -# define sock_write SockWrite /* Watt-32 uses same names */ +/* Watt-32 uses same names */ +# undef sock_write +# undef sock_read +# undef sock_puts +# define sock_write SockWrite # define sock_read SockRead # define sock_puts SockPuts # endif static int sock_write(BIO *h, const char *buf, int num); diff -aprNU5 openssl-1.0.2-stable-SNAP-20170602.orig/e_os.h openssl-1.0.2-stable-SNAP-20170602/e_os.h --- openssl-1.0.2-stable-SNAP-20170602.orig/e_os.h 2017-06-01 20:51:32 +0000 +++ openssl-1.0.2-stable-SNAP-20170602/e_os.h 2017-06-03 19:41:20 +0000 @@ -240,10 +240,12 @@ extern "C" { # include # include # define _setmode setmode # define _O_TEXT O_TEXT # define _O_BINARY O_BINARY +# define HAS_LFN_SUPPORT(name) (pathconf((name), _PC_NAME_MAX) > 12) +# undef DEVRANDOM_EGD /* Neither MS-DOS nor FreeDOS provide 'egd' sockets. */ # undef DEVRANDOM # define DEVRANDOM "/dev/urandom\x24" # endif /* __DJGPP__ */ # ifndef S_IFDIR diff -aprNU5 openssl-1.0.2-stable-SNAP-20170602.orig/INSTALL.DJGPP openssl-1.0.2-stable-SNAP-20170602/INSTALL.DJGPP --- openssl-1.0.2-stable-SNAP-20170602.orig/INSTALL.DJGPP 2017-06-01 20:51:32 +0000 +++ openssl-1.0.2-stable-SNAP-20170602/INSTALL.DJGPP 2017-06-03 19:42:44 +0000 @@ -16,11 +16,11 @@ All of these can be obtained from the usual DJGPP mirror sites or directly at "http://www.delorie.com/pub/djgpp". For help on which files to download, see the DJGPP "ZIP PICKER" page at "http://www.delorie.com/djgpp/zip-picker.html". You also need to have the WATT-32 networking package installed before you try to compile - OpenSSL. This can be obtained from "http://www.bgnett.no/~giva/". + OpenSSL. This can be obtained from "http://www.watt-32.net/". The Makefile assumes that the WATT-32 code is in the directory specified by the environment variable WATT_ROOT. If you have watt-32 in directory "watt32" under your main DJGPP directory, specify WATT_ROOT="/dev/env/DJDIR/watt32". diff -aprNU5 openssl-1.0.2-stable-SNAP-20170602.orig/util/mklink.pl openssl-1.0.2-stable-SNAP-20170602/util/mklink.pl --- openssl-1.0.2-stable-SNAP-20170602.orig/util/mklink.pl 2017-06-01 20:51:32 +0000 +++ openssl-1.0.2-stable-SNAP-20170602/util/mklink.pl 2017-06-03 19:41:20 +0000 @@ -49,11 +49,11 @@ foreach $dirname (@from_path) { my $to = join('/', @to_path); my $file; $symlink_exists=eval {symlink("",""); 1}; -if ($^O eq "msys") { $symlink_exists=0 }; +if ($^O eq "msys" || $^O eq 'dos') { $symlink_exists=0 }; foreach $file (@files) { my $err = ""; if ($symlink_exists) { if (!-l "$from/$file") { unlink "$from/$file"; From juan.guerrero at gmx.de Sun Jun 4 00:35:54 2017 From: juan.guerrero at gmx.de (Juan Manuel Guerrero) Date: Sun, 04 Jun 2017 02:35:54 +0200 Subject: [openssl-dev] Fixing some warnings when compiling on non-posix systems. Message-ID: <593355EA.2060604@gmx.de> I do not know if someone really cares about this, but I have compiled openssl-1.0.2-stable-SNAP-20170602 using the DJGPP port of gcc-7.1.0. The code has been configured using --strict-warnings option. At least the following issues have appeared. Using the DJGPP port of gcc means that the code is configured and compiled on a non-posix system thus macros like OPENSSL_SYS_LINUX and FIONBIO are undefined rising warnings like the fixed ones. The patch is only intended to show the issue and how it could be fixed. Regards, Juan M. Guerrero * crypto/bio/bss_dgram.c (long dgram_ctrl): warning: variable 'sockopt_val' set but not used [-Wunused-but-set-variable] * crypto/bio/b_sock.c (BIO_socket_nbio): warning: variable 'l' set but not used [-Wunused-but-set-variable] diff -aprNU5 openssl-1.0.2-stable-SNAP-20170602.orig/crypto/bio/bss_dgram.c openssl-1.0.2-stable-SNAP-20170602/crypto/bio/bss_dgram.c --- openssl-1.0.2-stable-SNAP-20170602.orig/crypto/bio/bss_dgram.c 2017-06-03 21:36:52 +0000 +++ openssl-1.0.2-stable-SNAP-20170602/crypto/bio/bss_dgram.c 2017-06-03 21:45:22 +0000 @@ -492,12 +492,12 @@ static long dgram_ctrl(BIO *b, int cmd, { long ret = 1; int *ip; struct sockaddr *to = NULL; bio_dgram_data *data = NULL; - int sockopt_val = 0; # if defined(OPENSSL_SYS_LINUX) && (defined(IP_MTU_DISCOVER) || defined(IP_MTU)) + int sockopt_val = 0; socklen_t sockopt_len; /* assume that system supporting IP_MTU is * modern enough to define socklen_t */ socklen_t addr_len; union { struct sockaddr sa; diff -aprNU5 openssl-1.0.2-stable-SNAP-20170602.orig/crypto/bio/b_sock.c openssl-1.0.2-stable-SNAP-20170602/crypto/bio/b_sock.c --- openssl-1.0.2-stable-SNAP-20170602.orig/crypto/bio/b_sock.c 2017-06-01 20:51:32 +0000 +++ openssl-1.0.2-stable-SNAP-20170602/crypto/bio/b_sock.c 2017-06-03 21:42:30 +0000 @@ -949,14 +949,14 @@ int BIO_set_tcp_ndelay(int s, int on) } int BIO_socket_nbio(int s, int mode) { int ret = -1; +# ifdef FIONBIO int l; l = mode; -# ifdef FIONBIO ret = BIO_socket_ioctl(s, FIONBIO, &l); # endif return (ret == 0); } #endif From matt at openssl.org Mon Jun 5 08:26:16 2017 From: matt at openssl.org (Matt Caswell) Date: Mon, 5 Jun 2017 09:26:16 +0100 Subject: [openssl-dev] Questions about ASYNC API for engines and moving read buffer. In-Reply-To: <1ac874e0-22f0-ce3f-bcae-4843025b4b02@haproxy.com> References: <14138849-5765-f58a-8905-203784947bb7@haproxy.com> <4baddc68-ed5f-569c-7375-b9523566b967@openssl.org> <1ac874e0-22f0-ce3f-bcae-4843025b4b02@haproxy.com> Message-ID: <0ac709e1-8c95-660e-b1f1-9d239bb31c59@openssl.org> On 02/06/17 15:20, Emeric Brun wrote: > > I've just read the code and I see it is not possible. > > I'm disappointed because i think that a lot of applications which are using openssl in asynchronous mode > also uses SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER and have ephemeral/reused/circular buffers for > both read and write operations. > > As documented: SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER "make it possible to retry SSL_write() with changed buffer location" > > SSL_read implicitly also support it, and since the support of openssl in our application (0.9.8), we never faced such issue. > > In my case, this is a dead end: Enabling the async mode on any engine (mine or qat), it looks good during the hanshake but it > causes buffer overflows switching in the data stage. > > Is there a way for the application to disable the async mode after the asymetric stuff (handshake)? At least we could use the > new async api for that. Well I suppose in theory you could just switch the mode off: SSL_clear_mode(s, SSL_MODE_ASYNC); But I've not tested it so I don't know if there would be any unexpected problems as a result. Matt From matt at openssl.org Mon Jun 5 08:38:26 2017 From: matt at openssl.org (Matt Caswell) Date: Mon, 5 Jun 2017 09:38:26 +0100 Subject: [openssl-dev] [PATCH] Some DJGPP specific fixes and improvements for OpenSSL_1_0_2-stable. In-Reply-To: <59335108.40804@gmx.de> References: <59335108.40804@gmx.de> Message-ID: Please submit these patches as a github pull request. Matt On 04/06/17 01:15, Juan Manuel Guerrero wrote: > I do not know if patches for DJGPP/FreeDOS are still welcome at all for the > OpenSSL 1.0.2 branch but if yes, then I would like to propose some fixes > and > improvements. No one of the proposed changes have impact on any other > port. > The patch is based on openssl-1.0.2-stable-SNAP-20170602. Concerning its > functionality, the patch is identical to the one proposed some months > ago for > the openssl-1.1.0 branch and that has found its way into that branch. > > The patch will fix/improve the following issues: > 1) In Configure: > For some reason -DTERMIO is set but DJGPP has never offered TERMIO > making > the build fail. I have changed this to -DTERMIOS as is used to be. > 2) In crypto/bio/bss_dgram.c: > I have removed superfluous macro definitions of sock_write, > sock_read and > sock_puts enclosed by WATT32. > 3) In crypto/bio/bss_sock.c: > Here the existing macro definitions for sock_write, sock_read and > sock_puts > are necessary and must be kept but they must be undefined before > they can > be defined. This is because newer versions of Watt-32 also > redefine them. > 4) In crypto/conf/conf_def.c: > If this port is used on MS-DOS or FreeDOS it becomes necessary to > check if > the underlying file system supports long file names (aka LFN) or > not. If > it does not then file names with a leading dot like ".rnd" or > ".ca_certs" > are ilicit. In function def_load_bio, the macros IS_RANDFILE and > IS_CERT_DIR > are used to check if the file system offers LFN support so that the > file > names with leading dots are licit. If the tests fail then the new > function > dosify_filename is called and will substitute invalid characters in > the file > name by valid ones before using them. This check and the call of > dosify_filename > is enclosed by OPENSSL_SYS_MSDOS. > 5) In e_os.h: > In the DJGPP section the macros IS_RANDFILE and IS_CERT_DIR are > defined. > Also some auxiliar macros like HAS_LFN_SUPPORT and FILE_EXISTS are > defined. > Because neither MS-DOS nor FreeDOS provide 'egd' sockets, the > DEVRANDOM_EGD > macro is undefined. This shall inhibit the compilation of code > that does > not work on MS-DOS/FreeDOS. > 6) In util/mklink.pl: > Neither MS-DOS nor FreeDOS provide symlink support so copy files > instead. > 7) In INSTALL.DJGPP: > Update URL of WATT-32 library. > > > I have checked the modified version of OpenSSL_1_0_2-stable on linux and > Cygwin. > They are no issues. This is no surprise because the changes are either > guarded > by the __DJGPP__ or OPENSSL_SYS_MSDOS macros. > If more informaton is required please mail me. Just in case it is > required, I > have attached the patch as gzip'ed files too. > > > Regards, > Juan M. Guerrero > > From matt at openssl.org Mon Jun 5 08:39:58 2017 From: matt at openssl.org (Matt Caswell) Date: Mon, 5 Jun 2017 09:39:58 +0100 Subject: [openssl-dev] Fixing some warnings when compiling on non-posix systems. In-Reply-To: <593355EA.2060604@gmx.de> References: <593355EA.2060604@gmx.de> Message-ID: Again with this one, please submit as a github patch. Matt On 04/06/17 01:35, Juan Manuel Guerrero wrote: > I do not know if someone really cares about this, but I have compiled > openssl-1.0.2-stable-SNAP-20170602 using the DJGPP port of gcc-7.1.0. > The code has been configured using --strict-warnings option. At least > the following issues have appeared. Using the DJGPP port of gcc means > that the code is configured and compiled on a non-posix system thus > macros like OPENSSL_SYS_LINUX and FIONBIO are undefined rising warnings > like the fixed ones. The patch is only intended to show the issue and > how it could be fixed. > > > Regards, > Juan M. Guerrero > > > > * crypto/bio/bss_dgram.c (long dgram_ctrl): warning: variable > 'sockopt_val' set but not used [-Wunused-but-set-variable] > > * crypto/bio/b_sock.c (BIO_socket_nbio): warning: variable 'l' set > but not used [-Wunused-but-set-variable] > > > > > > > > diff -aprNU5 > openssl-1.0.2-stable-SNAP-20170602.orig/crypto/bio/bss_dgram.c > openssl-1.0.2-stable-SNAP-20170602/crypto/bio/bss_dgram.c > --- openssl-1.0.2-stable-SNAP-20170602.orig/crypto/bio/bss_dgram.c > 2017-06-03 21:36:52 +0000 > +++ openssl-1.0.2-stable-SNAP-20170602/crypto/bio/bss_dgram.c > 2017-06-03 21:45:22 +0000 > @@ -492,12 +492,12 @@ static long dgram_ctrl(BIO *b, int cmd, > { > long ret = 1; > int *ip; > struct sockaddr *to = NULL; > bio_dgram_data *data = NULL; > - int sockopt_val = 0; > # if defined(OPENSSL_SYS_LINUX) && (defined(IP_MTU_DISCOVER) || > defined(IP_MTU)) > + int sockopt_val = 0; > socklen_t sockopt_len; /* assume that system supporting IP_MTU is > * modern enough to define socklen_t */ > socklen_t addr_len; > union { > struct sockaddr sa; > diff -aprNU5 openssl-1.0.2-stable-SNAP-20170602.orig/crypto/bio/b_sock.c > openssl-1.0.2-stable-SNAP-20170602/crypto/bio/b_sock.c > --- openssl-1.0.2-stable-SNAP-20170602.orig/crypto/bio/b_sock.c > 2017-06-01 20:51:32 +0000 > +++ openssl-1.0.2-stable-SNAP-20170602/crypto/bio/b_sock.c 2017-06-03 > 21:42:30 +0000 > @@ -949,14 +949,14 @@ int BIO_set_tcp_ndelay(int s, int on) > } > > int BIO_socket_nbio(int s, int mode) > { > int ret = -1; > +# ifdef FIONBIO > int l; > > l = mode; > -# ifdef FIONBIO > ret = BIO_socket_ioctl(s, FIONBIO, &l); > # endif > return (ret == 0); > } > #endif From steven at raycoll.com Tue Jun 6 14:29:53 2017 From: steven at raycoll.com (Steven Collison) Date: Tue, 06 Jun 2017 07:29:53 -0700 Subject: [openssl-dev] [openssl-users] Problem in connecting to Java (Tomcat) server with ECDHE ciphers In-Reply-To: References: Message-ID: As a sanity check, are you using an ECDSA certificate on your Tomcat server? ECDHE-ECDSA-AES256-GCM-SHA384 can?t be negotiated without one. Perhaps you can try `openssl s_client -connect a.b.c.d: -msg -debug -cipher ?ECDHE-RSA-AES256-GCM-SHA384?` if you?re using an RSA cert. -Steven On 3 Jun 2017, at 22:01, Pravesh Rai wrote: > Hi, > > Even though I've disabled SSLvX protocols on both - client > (openssl-1.0.2k) > & server (Java 1.8 with Tomcat), still getting following handshake > error, > while executing: > > "openssl s_client -connect a.b.c.d: -msg -debug -cipher > ECDHE-ECDSA-AES256-GCM-SHA384" > > > ... > read from 0x213f50 [0x21c410] (7 bytes => 7 (0x7)) > 0000 - 15 03 03 00 02 02 28 ......( > <<< TLS 1.2 [length 0005] > 15 03 03 00 02 > <<< TLS 1.2 Alert [length 0002], fatal handshake_failure > 02 28 > 14756:error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert > handshake failure:.\ssl\s23_clnt.c:769: > ... > > And, such error happens, only when ECDHE ciphers are selected during > the > connection. > > Any clue on this? > > Thanks, > PR > -- > openssl-users mailing list > To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-users -------------- next part -------------- An HTML attachment was scrubbed... URL: From borisp at mellanox.com Wed Jun 7 09:59:01 2017 From: borisp at mellanox.com (Boris Pismenny) Date: Wed, 7 Jun 2017 09:59:01 +0000 Subject: [openssl-dev] test Message-ID: Test123 From borisp at mellanox.com Wed Jun 7 09:06:18 2017 From: borisp at mellanox.com (Boris Pismenny) Date: Wed, 7 Jun 2017 09:06:18 +0000 Subject: [openssl-dev] test Message-ID: From borisp at mellanox.com Wed Jun 7 12:35:48 2017 From: borisp at mellanox.com (Boris Pismenny) Date: Wed, 7 Jun 2017 15:35:48 +0300 Subject: [openssl-dev] [RFC 3/4] evp/e_aes: Expose GCM IV In-Reply-To: References: Message-ID: This commit exposes the generated GCM IV to applications. Change-Id: If058f9d55c101e3efb6c988c26071145ebcbf0e8 Signed-off-by: Boris Pismenny --- crypto/evp/e_aes.c | 8 ++++++++ include/openssl/evp.h | 1 + 2 files changed, 9 insertions(+) diff --git a/crypto/evp/e_aes.c b/crypto/evp/e_aes.c index f504c68..3da8bf2 100644 --- a/crypto/evp/e_aes.c +++ b/crypto/evp/e_aes.c @@ -1331,6 +1331,14 @@ static int aes_gcm_ctrl(EVP_CIPHER_CTX *c, int type, int arg, void *ptr) memcpy(ptr, EVP_CIPHER_CTX_buf_noconst(c), arg); return 1; + case EVP_CTRL_GCM_GET_IV: + if (gctx->iv_gen != 1) + return 0; + if (gctx->ivlen != arg) + return 0; + memcpy(ptr, gctx->iv, arg); + return 1; + case EVP_CTRL_GCM_SET_IV_FIXED: /* Special case: -1 length restores whole IV */ if (arg == -1) { diff --git a/include/openssl/evp.h b/include/openssl/evp.h index b9c83b2..39b0dbd 100644 --- a/include/openssl/evp.h +++ b/include/openssl/evp.h @@ -289,6 +289,7 @@ int (*EVP_CIPHER_meth_get_ctrl(const EVP_CIPHER *cipher))(EVP_CIPHER_CTX *, # define EVP_CTRL_GCM_SET_TAG EVP_CTRL_AEAD_SET_TAG # define EVP_CTRL_GCM_SET_IV_FIXED EVP_CTRL_AEAD_SET_IV_FIXED # define EVP_CTRL_GCM_IV_GEN 0x13 +# define EVP_CTRL_GCM_GET_IV 0x14 # define EVP_CTRL_CCM_SET_IVLEN EVP_CTRL_AEAD_SET_IVLEN # define EVP_CTRL_CCM_GET_TAG EVP_CTRL_AEAD_GET_TAG # define EVP_CTRL_CCM_SET_TAG EVP_CTRL_AEAD_SET_TAG -- 1.8.3.1 From borisp at mellanox.com Wed Jun 7 12:35:46 2017 From: borisp at mellanox.com (Boris Pismenny) Date: Wed, 7 Jun 2017 15:35:46 +0300 Subject: [openssl-dev] [RFC 1/4] e_os: hack! TLS offload In-Reply-To: References: Message-ID: <446eda56460024a1a62dba61e7c09cef62af8f4a.1496776577.git.borisp@mellanox.com> This workaround adds netinet/tcp.h modifications for TLS offload to the include directory of TLS. In the future it wouldn't be needed since netinet/tcp.h would have these changes. Change-Id: I1698a46d9f7cd6f1e63e30969d595475775a5072 Signed-off-by: Boris Pismenny --- e_os.h | 2 +- include/netinet/tcp.h | 305 ++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 306 insertions(+), 1 deletion(-) create mode 100644 include/netinet/tcp.h diff --git a/e_os.h b/e_os.h index eafa862..af9a9d1 100644 --- a/e_os.h +++ b/e_os.h @@ -413,7 +413,7 @@ struct servent *PASCAL getservbyname(const char *, const char *); # endif # include # include -# include +# include "netinet/tcp.h" // Add TLS stuff here.. # endif # ifdef OPENSSL_SYS_AIX diff --git a/include/netinet/tcp.h b/include/netinet/tcp.h new file mode 100644 index 0000000..dfa4b97 --- /dev/null +++ b/include/netinet/tcp.h @@ -0,0 +1,305 @@ +/* + * Copyright (c) 1982, 1986, 1993 + * The Regents of the University of California. All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * 4. Neither the name of the University nor the names of its contributors + * may be used to endorse or promote products derived from this software + * without specific prior written permission. + * + * THIS SOFTWARE IS PROVIDED BY THE REGENTS AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE REGENTS OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * @(#)tcp.h 8.1 (Berkeley) 6/10/93 + */ + +#ifndef _NETINET_TCP_H +#define _NETINET_TCP_H 1 + +#include + +/* + * User-settable options (used with setsockopt). + */ +#define TCP_NODELAY 1 /* Don't delay send to coalesce packets */ +#define TCP_MAXSEG 2 /* Set maximum segment size */ +#define TCP_CORK 3 /* Control sending of partial frames */ +#define TCP_KEEPIDLE 4 /* Start keeplives after this period */ +#define TCP_KEEPINTVL 5 /* Interval between keepalives */ +#define TCP_KEEPCNT 6 /* Number of keepalives before death */ +#define TCP_SYNCNT 7 /* Number of SYN retransmits */ +#define TCP_LINGER2 8 /* Life time of orphaned FIN-WAIT-2 state */ +#define TCP_DEFER_ACCEPT 9 /* Wake up listener only when data arrive */ +#define TCP_WINDOW_CLAMP 10 /* Bound advertised window */ +#define TCP_INFO 11 /* Information about this connection. */ +#define TCP_QUICKACK 12 /* Bock/reenable quick ACKs. */ +#define TCP_CONGESTION 13 /* Congestion control algorithm. */ +#define TCP_MD5SIG 14 /* TCP MD5 Signature (RFC2385) */ +#define TCP_THIN_LINEAR_TIMEOUTS 16 /* Use linear timeouts for thin streams*/ +#define TCP_THIN_DUPACK 17 /* Fast retrans. after 1 dupack */ +#define TCP_USER_TIMEOUT 18 /* How long for loss retry before timeout */ +#define TCP_REPAIR 19 /* TCP sock is under repair right now */ +#define TCP_REPAIR_QUEUE 20 +#define TCP_QUEUE_SEQ 21 +#define TCP_REPAIR_OPTIONS 22 +#define TCP_FASTOPEN 23 /* Enable FastOpen on listeners */ +#define TCP_TIMESTAMP 24 +#define TCP_NOTSENT_LOWAT 25 /* limit number of unsent bytes in write queue */ +#define TCP_CC_INFO 26 /* Get Congestion Control (optional) info */ +#define TCP_SAVE_SYN 27 /* Record SYN headers for new connections */ +#define TCP_SAVED_SYN 28 /* Get SYN headers recorded for connection */ +#define TCP_REPAIR_WINDOW 29 /* Get/set window parameters */ + +#ifdef __USE_MISC +# include +# include + +# ifdef __FAVOR_BSD +typedef u_int32_t tcp_seq; +/* + * TCP header. + * Per RFC 793, September, 1981. + */ +struct tcphdr + { + u_int16_t th_sport; /* source port */ + u_int16_t th_dport; /* destination port */ + tcp_seq th_seq; /* sequence number */ + tcp_seq th_ack; /* acknowledgement number */ +# if __BYTE_ORDER == __LITTLE_ENDIAN + u_int8_t th_x2:4; /* (unused) */ + u_int8_t th_off:4; /* data offset */ +# endif +# if __BYTE_ORDER == __BIG_ENDIAN + u_int8_t th_off:4; /* data offset */ + u_int8_t th_x2:4; /* (unused) */ +# endif + u_int8_t th_flags; +# define TH_FIN 0x01 +# define TH_SYN 0x02 +# define TH_RST 0x04 +# define TH_PUSH 0x08 +# define TH_ACK 0x10 +# define TH_URG 0x20 + u_int16_t th_win; /* window */ + u_int16_t th_sum; /* checksum */ + u_int16_t th_urp; /* urgent pointer */ +}; + +# else /* !__FAVOR_BSD */ +struct tcphdr + { + u_int16_t source; + u_int16_t dest; + u_int32_t seq; + u_int32_t ack_seq; +# if __BYTE_ORDER == __LITTLE_ENDIAN + u_int16_t res1:4; + u_int16_t doff:4; + u_int16_t fin:1; + u_int16_t syn:1; + u_int16_t rst:1; + u_int16_t psh:1; + u_int16_t ack:1; + u_int16_t urg:1; + u_int16_t res2:2; +# elif __BYTE_ORDER == __BIG_ENDIAN + u_int16_t doff:4; + u_int16_t res1:4; + u_int16_t res2:2; + u_int16_t urg:1; + u_int16_t ack:1; + u_int16_t psh:1; + u_int16_t rst:1; + u_int16_t syn:1; + u_int16_t fin:1; +# else +# error "Adjust your defines" +# endif + u_int16_t window; + u_int16_t check; + u_int16_t urg_ptr; +}; +# endif /* __FAVOR_BSD */ + +enum +{ + TCP_ESTABLISHED = 1, + TCP_SYN_SENT, + TCP_SYN_RECV, + TCP_FIN_WAIT1, + TCP_FIN_WAIT2, + TCP_TIME_WAIT, + TCP_CLOSE, + TCP_CLOSE_WAIT, + TCP_LAST_ACK, + TCP_LISTEN, + TCP_CLOSING /* now a valid state */ +}; + +# define TCPOPT_EOL 0 +# define TCPOPT_NOP 1 +# define TCPOPT_MAXSEG 2 +# define TCPOLEN_MAXSEG 4 +# define TCPOPT_WINDOW 3 +# define TCPOLEN_WINDOW 3 +# define TCPOPT_SACK_PERMITTED 4 /* Experimental */ +# define TCPOLEN_SACK_PERMITTED 2 +# define TCPOPT_SACK 5 /* Experimental */ +# define TCPOPT_TIMESTAMP 8 +# define TCPOLEN_TIMESTAMP 10 +# define TCPOLEN_TSTAMP_APPA (TCPOLEN_TIMESTAMP+2) /* appendix A */ + +# define TCPOPT_TSTAMP_HDR \ + (TCPOPT_NOP<<24|TCPOPT_NOP<<16|TCPOPT_TIMESTAMP<<8|TCPOLEN_TIMESTAMP) + +/* + * Default maximum segment size for TCP. + * With an IP MSS of 576, this is 536, + * but 512 is probably more convenient. + * This should be defined as MIN(512, IP_MSS - sizeof (struct tcpiphdr)). + */ +# define TCP_MSS 512 + +# define TCP_MAXWIN 65535 /* largest value for (unscaled) window */ + +# define TCP_MAX_WINSHIFT 14 /* maximum window shift */ + +# define SOL_TCP 6 /* TCP level */ + + +# define TCPI_OPT_TIMESTAMPS 1 +# define TCPI_OPT_SACK 2 +# define TCPI_OPT_WSCALE 4 +# define TCPI_OPT_ECN 8 + +/* Values for tcpi_state. */ +enum tcp_ca_state +{ + TCP_CA_Open = 0, + TCP_CA_Disorder = 1, + TCP_CA_CWR = 2, + TCP_CA_Recovery = 3, + TCP_CA_Loss = 4 +}; + +struct tcp_info +{ + u_int8_t tcpi_state; + u_int8_t tcpi_ca_state; + u_int8_t tcpi_retransmits; + u_int8_t tcpi_probes; + u_int8_t tcpi_backoff; + u_int8_t tcpi_options; + u_int8_t tcpi_snd_wscale : 4, tcpi_rcv_wscale : 4; + + u_int32_t tcpi_rto; + u_int32_t tcpi_ato; + u_int32_t tcpi_snd_mss; + u_int32_t tcpi_rcv_mss; + + u_int32_t tcpi_unacked; + u_int32_t tcpi_sacked; + u_int32_t tcpi_lost; + u_int32_t tcpi_retrans; + u_int32_t tcpi_fackets; + + /* Times. */ + u_int32_t tcpi_last_data_sent; + u_int32_t tcpi_last_ack_sent; /* Not remembered, sorry. */ + u_int32_t tcpi_last_data_recv; + u_int32_t tcpi_last_ack_recv; + + /* Metrics. */ + u_int32_t tcpi_pmtu; + u_int32_t tcpi_rcv_ssthresh; + u_int32_t tcpi_rtt; + u_int32_t tcpi_rttvar; + u_int32_t tcpi_snd_ssthresh; + u_int32_t tcpi_snd_cwnd; + u_int32_t tcpi_advmss; + u_int32_t tcpi_reordering; + + u_int32_t tcpi_rcv_rtt; + u_int32_t tcpi_rcv_space; + + u_int32_t tcpi_total_retrans; +}; + + +/* For TCP_MD5SIG socket option. */ +#define TCP_MD5SIG_MAXKEYLEN 80 + +struct tcp_md5sig +{ + struct sockaddr_storage tcpm_addr; /* Address associated. */ + u_int16_t __tcpm_pad1; /* Zero. */ + u_int16_t tcpm_keylen; /* Key length. */ + u_int32_t __tcpm_pad2; /* Zero. */ + u_int8_t tcpm_key[TCP_MD5SIG_MAXKEYLEN]; /* Key (binary). */ +}; + + +/****************************************************/ +/* TLS NEW STUFF */ +/****************************************************/ +#define TCP_ULP 31 + +#define SOL_TLS 282 +#define TLS_TX 1 +//#define TLS_RX 2 + +/* Supported versions */ +#define TLS_VERSION_MINOR(ver) ((ver) & 0xFF) +#define TLS_VERSION_MAJOR(ver) (((ver) >> 8) & 0xFF) + +#define TLS_VERSION_NUMBER(id) ((((id##_VERSION_MAJOR) & 0xFF) << 8) | \ + ((id##_VERSION_MINOR) & 0xFF)) + +#define TLS_1_2_VERSION_MAJOR 0x3 +#define TLS_1_2_VERSION_MINOR 0x3 +#define TLS_1_2_VERSION TLS_VERSION_NUMBER(TLS_1_2) + +/* Supported ciphers */ +#define TLS_CIPHER_AES_GCM_128 51 +#define TLS_CIPHER_AES_GCM_128_IV_SIZE ((size_t)8) +#define TLS_CIPHER_AES_GCM_128_KEY_SIZE ((size_t)16) +#define TLS_CIPHER_AES_GCM_128_SALT_SIZE ((size_t)4) +#define TLS_CIPHER_AES_GCM_128_TAG_SIZE ((size_t)16) +#define TLS_CIPHER_AES_GCM_128_REC_SEQ_SIZE ((size_t)8) + +#define TLS_SET_RECORD_TYPE 1 + +struct tls_crypto_info { + unsigned short version; + unsigned short cipher_type; +}; + +struct tls12_crypto_info_aes_gcm_128 { + struct tls_crypto_info info; + unsigned char iv[TLS_CIPHER_AES_GCM_128_IV_SIZE]; + unsigned char key[TLS_CIPHER_AES_GCM_128_KEY_SIZE]; + unsigned char salt[TLS_CIPHER_AES_GCM_128_SALT_SIZE]; + unsigned char rec_seq[TLS_CIPHER_AES_GCM_128_REC_SEQ_SIZE]; +}; +/****************************************************/ + +#endif /* Misc. */ + +#endif /* netinet/tcp.h */ -- 1.8.3.1 From borisp at mellanox.com Wed Jun 7 12:35:47 2017 From: borisp at mellanox.com (Boris Pismenny) Date: Wed, 7 Jun 2017 15:35:47 +0300 Subject: [openssl-dev] [RFC 2/4] bio: Linux TLS Offload In-Reply-To: References: Message-ID: <1ae2aac26528ffe6e30ddca8611b31580fd2c6cb.1496776577.git.borisp@mellanox.com> Add support for Linux TLS offload in the BIO layer and specifically in bss_sock.c. Change-Id: I64e08da83c595a9067a3c7de80f73408010fcde6 Signed-off-by: Boris Pismenny --- crypto/bio/bss_sock.c | 93 ++++++++++++++++++++++++++++++++++++++++++++++++++- include/openssl/bio.h | 32 ++++++++++++++++++ 2 files changed, 124 insertions(+), 1 deletion(-) diff --git a/crypto/bio/bss_sock.c b/crypto/bio/bss_sock.c index 570e898..5f02d04 100644 --- a/crypto/bio/bss_sock.c +++ b/crypto/bio/bss_sock.c @@ -27,6 +27,11 @@ # define sock_puts SockPuts # endif +#if defined(OPENSSL_LINUX_TLS) + #include "netinet/tcp.h" +#endif + + static int sock_write(BIO *h, const char *buf, int num); static int sock_read(BIO *h, char *buf, int size); static int sock_puts(BIO *h, const char *str); @@ -56,11 +61,21 @@ const BIO_METHOD *BIO_s_socket(void) BIO *BIO_new_socket(int fd, int close_flag) { BIO *ret; + int rc; ret = BIO_new(BIO_s_socket()); if (ret == NULL) return (NULL); BIO_set_fd(ret, fd, close_flag); +#ifdef OPENSSL_LINUX_TLS + rc = setsockopt(fd, SOL_TCP, TCP_ULP, "tls", sizeof("tls")); +#ifdef SSL_DEBUG + if (rc) { + printf("setsockopt failed %d\n", errno); + } +#endif + +# endif return (ret); } @@ -103,12 +118,54 @@ static int sock_read(BIO *b, char *out, int outl) return (ret); } +static int send_ctrl_message(int fd, unsigned char record_type, + const void *data, size_t length) +{ + struct msghdr msg = {0}; + int cmsg_len = sizeof(record_type); + struct cmsghdr *cmsg; + char buf[CMSG_SPACE(cmsg_len)]; + struct iovec msg_iov; /* Vector of data to send/receive into */ + + msg.msg_control = buf; + msg.msg_controllen = sizeof(buf); + cmsg = CMSG_FIRSTHDR(&msg); + cmsg->cmsg_level = SOL_TLS; + cmsg->cmsg_type = TLS_SET_RECORD_TYPE; + cmsg->cmsg_len = CMSG_LEN(cmsg_len); + *((unsigned char *)CMSG_DATA(cmsg)) = record_type; + msg.msg_controllen = cmsg->cmsg_len; + + msg_iov.iov_base = (void *)data; + msg_iov.iov_len = length; + msg.msg_iov = &msg_iov; + msg.msg_iovlen = 1; + + return sendmsg(fd, &msg, 0); +} + static int sock_write(BIO *b, const char *in, int inl) { int ret; clear_socket_error(); - ret = writesocket(b->num, in, inl); + if (BIO_should_offload_tx_ctrl_msg_flag(b)) { + unsigned char record_type = (unsigned char)b->ptr; + +#ifdef SSL_DEBUG + printf("\nsending ctrl msg\n"); +#endif + BIO_clear_offload_tx_ctrl_msg_flag(b); + ret = send_ctrl_message(b->num, record_type, in, inl); + if (ret >= 0) { + ret = inl; + } + } else { +#ifdef SSL_DEBUG + printf("\nsending data msg %p %d\n", b, b->flags); +#endif + ret = writesocket(b->num, in, inl); + } BIO_clear_retry_flags(b); if (ret <= 0) { if (BIO_sock_should_retry(ret)) @@ -121,6 +178,9 @@ static long sock_ctrl(BIO *b, int cmd, long num, void *ptr) { long ret = 1; int *ip; +# ifdef OPENSSL_LINUX_TLS + struct tls12_crypto_info_aes_gcm_128 *crypto_info; +# endif switch (cmd) { case BIO_C_SET_FD: @@ -148,6 +208,37 @@ static long sock_ctrl(BIO *b, int cmd, long num, void *ptr) case BIO_CTRL_FLUSH: ret = 1; break; +# if defined(OPENSSL_LINUX_TLS) + case BIO_CTRL_SET_OFFLOAD_TX: + crypto_info = (struct tls12_crypto_info_aes_gcm_128 *)ptr; + ret = setsockopt(b->num, SOL_TLS, TLS_TX, + crypto_info, sizeof(*crypto_info)); +#ifdef SSL_DEBUG + printf("\nAttempt to offload..."); +#endif + if (!ret) { + BIO_set_offload_tx_flag(b); +#ifdef SSL_DEBUG + printf("Success %p %p\n", b, &(b->flags)); +#endif + } else { +#ifdef SSL_DEBUG + printf("Failed ret=%ld\n", ret); +#endif + } + break; + case BIO_CTRL_GET_OFFLOAD_TX: + return BIO_should_offload_tx_flag(b); + case BIO_CTRL_SET_OFFLOAD_TX_CTRL_MSG: + BIO_set_offload_tx_ctrl_msg_flag(b); + b->ptr = (void *)num; + ret = 0; + break; + case BIO_CTRL_CLEAR_OFFLOAD_TX_CTRL_MSG: + BIO_clear_offload_tx_ctrl_msg_flag(b); + ret = 0; + break; +# endif default: ret = 0; break; diff --git a/include/openssl/bio.h b/include/openssl/bio.h index 31d41b4..c718627 100644 --- a/include/openssl/bio.h +++ b/include/openssl/bio.h @@ -146,6 +146,11 @@ extern "C" { # define BIO_CTRL_DGRAM_SCTP_SAVE_SHUTDOWN 70 # endif +# define BIO_CTRL_SET_OFFLOAD_TX 71 +# define BIO_CTRL_GET_OFFLOAD_TX 72 +# define BIO_CTRL_SET_OFFLOAD_TX_CTRL_MSG 73 +# define BIO_CTRL_CLEAR_OFFLOAD_TX_CTRL_MSG 74 + /* modifiers */ # define BIO_FP_READ 0x02 # define BIO_FP_WRITE 0x04 @@ -175,6 +180,13 @@ extern "C" { # define BIO_FLAGS_MEM_RDONLY 0x200 # define BIO_FLAGS_NONCLEAR_RST 0x400 +/* + * This is used with socket BIOs: + * BIO_FLAGS_OFFLOAD_TX means we are using offload with this BIO for TX. + * BIO_FLAGS_OFFLOAD_TX_CTRL_MSG means we are about to send a ctrl message next. + */ +# define BIO_FLAGS_OFFLOAD_TX 0x2000 +# define BIO_FLAGS_OFFLOAD_TX_CTRL_MSG 0x4000 typedef union bio_addr_st BIO_ADDR; typedef struct bio_addrinfo_st BIO_ADDRINFO; @@ -191,6 +203,18 @@ void BIO_clear_flags(BIO *b, int flags); # define BIO_set_retry_write(b) \ BIO_set_flags(b, (BIO_FLAGS_WRITE|BIO_FLAGS_SHOULD_RETRY)) +/* Offload related controls and flags */ +# define BIO_set_offload_tx_flag(b) \ + BIO_set_flags(b, BIO_FLAGS_OFFLOAD_TX) +# define BIO_should_offload_tx_flag(b) \ + BIO_test_flags(b, BIO_FLAGS_OFFLOAD_TX) +# define BIO_set_offload_tx_ctrl_msg_flag(b) \ + BIO_set_flags(b, BIO_FLAGS_OFFLOAD_TX_CTRL_MSG) +# define BIO_should_offload_tx_ctrl_msg_flag(b) \ + BIO_test_flags(b, (BIO_FLAGS_OFFLOAD_TX_CTRL_MSG)) +# define BIO_clear_offload_tx_ctrl_msg_flag(b) \ + BIO_clear_flags(b, (BIO_FLAGS_OFFLOAD_TX_CTRL_MSG)) + /* These are normally used internally in BIOs */ # define BIO_clear_retry_flags(b) \ BIO_clear_flags(b, (BIO_FLAGS_RWS|BIO_FLAGS_SHOULD_RETRY)) @@ -370,6 +394,14 @@ struct bio_dgram_sctp_prinfo { # define BIO_get_conn_address(b) ((const BIO_ADDR *)BIO_ptr_ctrl(b,BIO_C_GET_CONNECT,2,NULL)) # define BIO_get_conn_ip_family(b) BIO_ctrl(b,BIO_C_GET_CONNECT,3,NULL) # define BIO_set_conn_mode(b,n) BIO_ctrl(b,BIO_C_SET_CONNECT_MODE,(n),NULL) +# define BIO_set_offload_tx(b, keyblob) \ + BIO_ctrl(b, BIO_CTRL_SET_OFFLOAD_TX, 0, keyblob) +# define BIO_get_offload_tx(b) \ + BIO_ctrl(b, BIO_CTRL_GET_OFFLOAD_TX, 0, NULL) +# define BIO_set_offload_tx_ctrl_msg(b, record_type) \ + BIO_ctrl(b, BIO_CTRL_SET_OFFLOAD_TX_CTRL_MSG, record_type, NULL) +# define BIO_clear_offload_tx_ctrl_msg(b) \ + BIO_ctrl(b, BIO_CTRL_CLEAR_OFFLOAD_TX_CTRL_MSG, 0, NULL) /* BIO_s_accept() */ # define BIO_set_accept_name(b,name) BIO_ctrl(b,BIO_C_SET_ACCEPT,0,(char *)name) -- 1.8.3.1 From borisp at mellanox.com Wed Jun 7 12:35:49 2017 From: borisp at mellanox.com (Boris Pismenny) Date: Wed, 7 Jun 2017 15:35:49 +0300 Subject: [openssl-dev] [RFC 4/4] ssl: Linux TLS Tx Offload In-Reply-To: References: Message-ID: This patch adds support for Linux TLS Tx offload. The data-path of the TLS socket is offloaded to the kernel after CCS is complete. Change-Id: Ia966192a6704d1a57b74b2640ac04d55fb74c1c7 Signed-off-by: Boris Pismenny --- ssl/record/rec_layer_s3.c | 95 ++++++++++++++++++++++++++++++++++------------- ssl/t1_enc.c | 69 ++++++++++++++++++++++++++++++++++ 2 files changed, 139 insertions(+), 25 deletions(-) diff --git a/ssl/record/rec_layer_s3.c b/ssl/record/rec_layer_s3.c index 46870c0..a36be02 100644 --- a/ssl/record/rec_layer_s3.c +++ b/ssl/record/rec_layer_s3.c @@ -750,24 +750,31 @@ int do_ssl3_write(SSL *s, int type, const unsigned char *buf, /* Clear our SSL3_RECORD structures */ memset(wr, 0, sizeof wr); for (j = 0; j < numpipes; j++) { - /* write the header */ - *(outbuf[j]++) = type & 0xff; - SSL3_RECORD_set_type(&wr[j], type); + if (BIO_get_offload_tx(s->wbio)) { + /* IV will be generated by the kernel if offload is used */ + eivlen = 0; - *(outbuf[j]++) = (s->version >> 8); - /* - * Some servers hang if initial client hello is larger than 256 bytes - * and record version number > TLS 1.0 - */ - if (SSL_get_state(s) == TLS_ST_CW_CLNT_HELLO - && !s->renegotiate && TLS1_get_version(s) > TLS1_VERSION) - *(outbuf[j]++) = 0x1; - else - *(outbuf[j]++) = s->version & 0xff; + SSL3_RECORD_set_type(&wr[j], type); + } else { + /* write the header */ + *(outbuf[j]++) = type & 0xff; + SSL3_RECORD_set_type(&wr[j], type); - /* field where we are to write out packet length */ - plen[j] = outbuf[j]; - outbuf[j] += 2; + *(outbuf[j]++) = (s->version >> 8); + /* + * Some servers hang if initial client hello is larger + * than 256 bytes and record version number > TLS 1.0 + */ + if (SSL_get_state(s) == TLS_ST_CW_CLNT_HELLO + && !s->renegotiate && TLS1_get_version(s) > TLS1_VERSION) + *(outbuf[j]++) = 0x1; + else + *(outbuf[j]++) = s->version & 0xff; + + /* field where we are to write out packet length */ + plen[j] = outbuf[j]; + outbuf[j] += 2; + } /* lets setup the record stuff. */ SSL3_RECORD_set_data(&wr[j], outbuf[j] + eivlen); @@ -815,8 +822,13 @@ int do_ssl3_write(SSL *s, int type, const unsigned char *buf, } } - if (s->method->ssl3_enc->enc(s, wr, numpipes, 1) < 1) - goto err; + /* record sequence number is incremented inside ssl3_enc->enc() */ + if (!BIO_get_offload_tx(s->wbio)) { + if (s->method->ssl3_enc->enc(s, wr, numpipes, 1) < 1) { + goto err; + } + } + for (j = 0; j < numpipes; j++) { if (SSL_USE_ETM(s) && mac_size != 0) { @@ -826,12 +838,30 @@ int do_ssl3_write(SSL *s, int type, const unsigned char *buf, SSL3_RECORD_add_length(&wr[j], mac_size); } - /* record length after mac and block padding */ - s2n(SSL3_RECORD_get_length(&wr[j]), plen[j]); - - if (s->msg_callback) - s->msg_callback(1, 0, SSL3_RT_HEADER, plen[j] - 5, 5, s, - s->msg_callback_arg); + if (!BIO_get_offload_tx(s->wbio)) { + /* record length after mac and block padding */ + s2n(SSL3_RECORD_get_length(&wr[j]), plen[j]); + if (s->msg_callback) + s->msg_callback(1, 0, SSL3_RT_HEADER, plen[j] - 5, 5, s, + s->msg_callback_arg); + } else { + /* Create a fake SSL3_RT_HEADER msg to maintain compatibility */ + if (s->msg_callback) { + unsigned char header[5] = {0}, *pheader = header + 3; + + header[0] = type & 0xff; + header[1] = (s->version >> 8); + if (SSL_get_state(s) == TLS_ST_CW_CLNT_HELLO + && !s->renegotiate + && TLS1_get_version(s) > TLS1_VERSION) + header[2] = 0x1; + else + header[2] = s->version & 0xff; + s2n(SSL3_RECORD_get_length(&wr[j]), pheader); + s->msg_callback(1, 0, SSL3_RT_HEADER, header, 5, s, + s->msg_callback_arg); + } + } /* * we should now have wr->data pointing to the encrypted data, which is @@ -839,7 +869,11 @@ int do_ssl3_write(SSL *s, int type, const unsigned char *buf, */ SSL3_RECORD_set_type(&wr[j], type); /* not needed but helps for * debugging */ - SSL3_RECORD_add_length(&wr[j], SSL3_RT_HEADER_LENGTH); + + /* No header when using offload */ + if (!BIO_get_offload_tx(s->wbio)) { + SSL3_RECORD_add_length(&wr[j], SSL3_RT_HEADER_LENGTH); + } if (create_empty_fragment) { /* @@ -901,10 +935,21 @@ int ssl3_write_pending(SSL *s, int type, const unsigned char *buf, clear_sys_error(); if (s->wbio != NULL) { s->rwstate = SSL_WRITING; + if (BIO_get_offload_tx(s->wbio) && type != + SSL3_RT_APPLICATION_DATA) { + BIO_set_offload_tx_ctrl_msg(s->wbio, type); + } i = BIO_write(s->wbio, (char *) &(SSL3_BUFFER_get_buf(&wb[currbuf]) [SSL3_BUFFER_get_offset(&wb[currbuf])]), (unsigned int)SSL3_BUFFER_get_left(&wb[currbuf])); + /* To prevent coalescing of control and data messages, + * such as in buffer_write, we flush the BIO + */ + if (BIO_get_offload_tx(s->wbio) && + type != SSL3_RT_APPLICATION_DATA) { + (void)BIO_flush(s->wbio); + } } else { SSLerr(SSL_F_SSL3_WRITE_PENDING, SSL_R_BIO_NOT_SET); i = -1; diff --git a/ssl/t1_enc.c b/ssl/t1_enc.c index 4aa5ddd..6b85c5c 100644 --- a/ssl/t1_enc.c +++ b/ssl/t1_enc.c @@ -41,6 +41,10 @@ #include #include +#ifdef OPENSSL_LINUX_TLS +# include "netinet/tcp.h" // Add TLS stuff here.. +#endif + /* seed1 through seed5 are concatenated */ static int tls1_PRF(SSL *s, const void *seed1, int seed1_len, @@ -121,6 +125,11 @@ int tls1_change_cipher_state(SSL *s, int which) EVP_PKEY *mac_key; int n, i, j, k, cl; int reuse_dd = 0; +#ifdef OPENSSL_LINUX_TLS + struct tls12_crypto_info_aes_gcm_128 crypto_info; + BIO *wbio; + unsigned char geniv[12]; +#endif c = s->s3->tmp.new_sym_enc; m = s->s3->tmp.new_hash; @@ -303,6 +312,66 @@ int tls1_change_cipher_state(SSL *s, int which) SSLerr(SSL_F_TLS1_CHANGE_CIPHER_STATE, ERR_R_INTERNAL_ERROR); goto err2; } + +#ifdef OPENSSL_LINUX_TLS + if (!(which & SSL3_CC_WRITE)) { +#ifdef SSL_DEBUG + printf("\nSkipping offload for non-write context\n"); +#endif // SSL_DEBUG + goto skip_offload; + } + memset(&crypto_info, 0, sizeof(crypto_info)); + + /* check that cipher is AES_GCM_128 */ + if (EVP_CIPHER_mode(c) != EVP_CIPH_GCM_MODE) { +#ifdef SSL_DEBUG + printf("\nGot mode %08lx != GCM, skipping offload\n", + EVP_CIPHER_mode(c)); +#endif // SSL_DEBUG + goto skip_offload; + } + crypto_info.info.cipher_type = TLS_CIPHER_AES_GCM_128; + + /* check version is 1.2 */ + if (s->version != TLS1_2_VERSION) { +#ifdef SSL_DEBUG + printf("\nVersion mismatch got %08x, skipping offload\n", s->version); +#endif // SSL_DEBUG + goto skip_offload; + } + + if (EVP_CIPHER_key_length(c) != TLS_CIPHER_AES_GCM_128_KEY_SIZE) { +#ifdef SSL_DEBUG + printf("\nUnexpected key length %d, skipping offload\n", + EVP_CIPHER_key_length(c)); +#endif // SSL_DEBUG + goto skip_offload; + } + + crypto_info.info.version = s->version; + + /* This is not the IV that is used by OpenSSL, but rather the sequence + * number. We should actually provide both iv and sequence. + */ + EVP_CIPHER_CTX_ctrl(dd, EVP_CTRL_GCM_GET_IV, 12, geniv); + memcpy(crypto_info.iv, geniv + 4, TLS_CIPHER_AES_GCM_128_IV_SIZE); + memcpy(crypto_info.salt, geniv, TLS_CIPHER_AES_GCM_128_SALT_SIZE); + memcpy(crypto_info.key, key, EVP_CIPHER_key_length(c)); + memcpy(crypto_info.rec_seq, &s->rlayer.write_sequence, + TLS_CIPHER_AES_GCM_128_REC_SEQ_SIZE); + + wbio = s->wbio; + if (!wbio) { + goto skip_offload; + } + + (void)BIO_flush(wbio); + + BIO_set_offload_tx(wbio, &crypto_info); + +skip_offload: +#endif // OPENSSL_LINUX_TLS + #ifdef OPENSSL_SSL_TRACE_CRYPTO if (s->msg_callback) { int wh = which & SSL3_CC_WRITE ? TLS1_RT_CRYPTO_WRITE : 0; -- 1.8.3.1 From borisp at mellanox.com Wed Jun 7 12:35:45 2017 From: borisp at mellanox.com (Boris Pismenny) Date: Wed, 7 Jun 2017 15:35:45 +0300 Subject: [openssl-dev] [RFC 0/4] Kernel TLS socket API Message-ID: Hello all, I would like to introduce you to the new kernel API for TLS transmit-side data-path, and open a discussion regarding its support in OpenSSL. This is currently a V2 patch series in Linux net-next, and it is stabilizing. Dave has been working on this for a while [1][2], and Aviad, Ilya and myself have also been working on TLS transmit-side crypto offload to a NIC [3][4]. Similar work is also being done in FreeBSD by Netflix [5][6]. Placing the data-path of TLS in the kernel has several advantages. First, an efficient TLS sendfile system call implementation is possible with such a socket. Second, data from userspace could be encrypted when passing the user-kernel boundary, reducing the overhead of existing implementations that require at least encrypting the data and then copying from user to kernel. Third, TLS data-path crypto offload to hardware requires a kernel infrastructure for TLS. To sum up, a new kernel socket option is introduced to add support for the data-path of TLS and it will soon be common in operating systems. For anyone who is interested, here is the kernel code: https://github.com/Mellanox/tls-offload - branch tx_rfc_v10 And our openssl development branch: https://github.com/Mellanox/tls-openssl - branch tls_tx_v2 The user API is as follows: (also see [7]) ========================================== Creating a TLS connection ------------------------- First create a new TCP socket and set the TLS ULP. sock = socket(AF_INET, SOCK_STREAM, 0); setsockopt(sock, SOL_TCP, TCP_ULP, "tls", sizeof("tls")); Setting the TLS ULP allows us to set/get TLS socket options. Currently only the symmetric encryption is handled in the kernel. Patch 2 sets the TLS ULP in BIO_new_socket. All the parameters required to move the data-path to the kernel are available after processing the change cipher state message during the handshake. Patch 4 adds code similar to the one below in tls1_change_cipher_state. There is a separate socket option for moving the transmit and the receive into the kernel. Current code, supports only the transmit side, TLS1.2 and AES-GCM128. In the future, this will be extended. Below is the code that sets the socket option which moves the transmit data-path into the kernel. Patch 1 adds a temporary netinet/tcp.h file which includes these structures. Do you know if netinet/tcp.h is the right place for these? /* From linux/tls.h */ struct tls_crypto_info { unsigned short version; unsigned short cipher_type; }; struct tls12_crypto_info_aes_gcm_128 { struct tls_crypto_info info; unsigned char iv[TLS_CIPHER_AES_GCM_128_IV_SIZE]; unsigned char key[TLS_CIPHER_AES_GCM_128_KEY_SIZE]; unsigned char salt[TLS_CIPHER_AES_GCM_128_SALT_SIZE]; unsigned char rec_seq[TLS_CIPHER_AES_GCM_128_REC_SEQ_SIZE]; }; struct tls12_crypto_info_aes_gcm_128 crypto_info; crypto_info.info.version = TLS_1_2_VERSION; crypto_info.info.cipher_type = TLS_CIPHER_AES_GCM_128; memcpy(crypto_info.iv, iv_write, TLS_CIPHER_AES_GCM_128_IV_SIZE); memcpy(crypto_info.rec_seq, seq_number_write, TLS_CIPHER_AES_GCM_128_REC_SEQ_SIZE); memcpy(crypto_info.key, cipher_key_write, TLS_CIPHER_AES_GCM_128_KEY_SIZE); memcpy(crypto_info.salt, implicit_iv_write, TLS_CIPHER_AES_GCM_128_SALT_SIZE); setsockopt(sock, SOL_TLS, TLS_TX, &crypto_info, sizeof(crypto_info)); Sending TLS application data ---------------------------- After setting the TLS_TX socket option all application data sent over this socket is encrypted using TLS and the parameters provided in the socket option. For example, an application can send an encrypted hello world record as follows: const char *msg = "hello world\n"; send(sock, msg, strlen(msg)); send() data is directly encrypted from the userspace buffer provided to the encrypted kernel send buffer if possible. The sendfile system call will send the file's data over TLS records of maximum length (2^14). file = open(filename, O_RDONLY); fstat(file, &stat); sendfile(sock, file, &offset, stat.st_size); TLS records are created and sent after each send() call, unless MSG_MORE is passed. MSG_MORE will delay creation of a record until MSG_MORE is not passed, or the maximum record size is reached or an alert record needs to be sent. The kernel will need to allocate a buffer for the encrypted data. This buffer is allocated at the time send() is called, such that either the entire send() call will return -ENOMEM (or block waiting for memory), or the encryption will always succeed. If send() returns -ENOMEM and some data was left on the socket buffer from a previous call using MSG_MORE, the MSG_MORE data is left on the socket buffer. Send TLS control messages ------------------------- Other than application data, TLS has control messages such as alert messages (record type 21) and handshake messages (record type 22), etc. These messages can be sent over the socket by providing the TLS record type via a CMSG. Patch 2 adds the following function (in bss_sock.c) that sends @data of @length bytes using a record of type @record_type. /* send TLS control message using record_type */ static int klts_send_ctrl_message(int sock, unsigned char record_type, void *data, size_t length) { struct msghdr msg = {0}; int cmsg_len = sizeof(record_type); struct cmsghdr *cmsg; char buf[CMSG_SPACE(cmsg_len)]; struct iovec msg_iov; /* Vector of data to send/receive into. */ msg.msg_control = buf; msg.msg_controllen = sizeof(buf); cmsg = CMSG_FIRSTHDR(&msg); cmsg->cmsg_level = SOL_TLS; cmsg->cmsg_type = TLS_SET_RECORD_TYPE; cmsg->cmsg_len = CMSG_LEN(cmsg_len); *CMSG_DATA(cmsg) = record_type; msg.msg_controllen = cmsg->cmsg_len; msg_iov.iov_base = data; msg_iov.iov_len = length; msg.msg_iov = &msg_iov; msg.msg_iovlen = 1; return sendmsg(sock, &msg, 0); } Control message data should be provided unencrypted, and will be encrypted by the kernel. At a high level, the kernel TLS ULP is a replacement for the record layer of a userspace TLS library. References: [1] - https://netdevconf.org/1.2/session.html?dave-watson [2] - https://www.spinics.net/lists/linux-crypto/msg25953.html [3] - https://netdevconf.org/1.2/session.html?boris-pismenny [4] - https://www.spinics.net/lists/netdev/msg427122.html [5] - https://people.freebsd.org/~rrs/asiabsd_2015_tls.pdf [6] - https://people.freebsd.org/~rrs/asiabsd_tls_improved.pdf [7] - https://www.spinics.net/lists/linux-crypto/msg25957.html Boris Pismenny (4): e_os: hack! TLS offload bio: Linux TLS Offload evp/e_aes: Expose GCM IV ssl: Linux TLS Tx Offload crypto/bio/bss_sock.c | 93 +++++++++++++- crypto/evp/e_aes.c | 8 ++ e_os.h | 2 +- include/netinet/tcp.h | 305 ++++++++++++++++++++++++++++++++++++++++++++++ include/openssl/bio.h | 32 +++++ include/openssl/evp.h | 1 + ssl/record/rec_layer_s3.c | 95 +++++++++++---- ssl/t1_enc.c | 69 +++++++++++ 8 files changed, 578 insertions(+), 27 deletions(-) create mode 100644 include/netinet/tcp.h -- 1.8.3.1 From rsalz at akamai.com Wed Jun 7 15:19:21 2017 From: rsalz at akamai.com (Salz, Rich) Date: Wed, 7 Jun 2017 15:19:21 +0000 Subject: [openssl-dev] [RFC 0/4] Kernel TLS socket API In-Reply-To: References: Message-ID: <054105665b2146bea74babc27e5dfac3@usma1ex-dag1mb1.msg.corp.akamai.com> A couple of comments. First, until this shows up in the kernel adopted by major distributions, it is a bit premature to include in OpenSSL. Including netinet/tcp.h is seriously wrong to be part of openssl :) And finally, as I said before, the best way to get things in OpenSSL is to do pull requests on GitHub. Having said all that, the concept is interesting and exciting, and I hope it becomes widespread soon. From bkaduk at akamai.com Wed Jun 7 19:11:23 2017 From: bkaduk at akamai.com (Benjamin Kaduk) Date: Wed, 7 Jun 2017 14:11:23 -0500 Subject: [openssl-dev] [RFC 0/4] Kernel TLS socket API In-Reply-To: <054105665b2146bea74babc27e5dfac3@usma1ex-dag1mb1.msg.corp.akamai.com> References: <054105665b2146bea74babc27e5dfac3@usma1ex-dag1mb1.msg.corp.akamai.com> Message-ID: <3a85e1cb-17e8-1dd6-7551-48c875aabc21@akamai.com> On 06/07/2017 10:19 AM, Salz, Rich via openssl-dev wrote: > A couple of comments. > > First, until this shows up in the kernel adopted by major distributions, it is a bit premature to include in OpenSSL. Including netinet/tcp.h is seriously wrong I don't know that we would need to wait until it's in distributions, but we definitely shouldn't commit to supporting an API until mainline linux has [and then Linus's "don't break userspace" adage applies]. -Ben -------------- next part -------------- An HTML attachment was scrubbed... URL: From kurt at roeckx.be Wed Jun 7 22:05:41 2017 From: kurt at roeckx.be (Kurt Roeckx) Date: Thu, 8 Jun 2017 00:05:41 +0200 Subject: [openssl-dev] [RFC 0/4] Kernel TLS socket API In-Reply-To: References: Message-ID: <20170607220540.qsi2caboyvydiwii@roeckx.be> On Wed, Jun 07, 2017 at 03:35:45PM +0300, Boris Pismenny wrote: > Hello all, > > I would like to introduce you to the new kernel API for TLS transmit-side > data-path, and open a discussion regarding its support in OpenSSL. So my understanding is that there are really 2 parts in the kernel that change: - The kernel is aware of TLS and can do the symmetric encryption - The kernel can offload the symmetric encryption to the NIC And I guess you're mostly interested in the combination of the two where you would end up with the unencrypted data going go the NIC and that you might get speeds close to what you can do unencrypted. The performance gains would come from avoiding making copies and not doing the encryption on the CPU. My understanding from the old data is that moving the encryption to the kernel had a negative performance impact. So this at least looks like something we do not always want to enable. It might be useful to have an API where we can check that the offload is supported, or that we have an option to enable moving it to the kernel. Kurt From borisp at mellanox.com Wed Jun 7 18:41:16 2017 From: borisp at mellanox.com (Boris Pismenny) Date: Wed, 7 Jun 2017 18:41:16 +0000 Subject: [openssl-dev] [RFC 0/4] Kernel TLS socket API In-Reply-To: <054105665b2146bea74babc27e5dfac3@usma1ex-dag1mb1.msg.corp.akamai.com> References: <054105665b2146bea74babc27e5dfac3@usma1ex-dag1mb1.msg.corp.akamai.com> Message-ID: > A couple of comments. > > First, until this shows up in the kernel adopted by major distributions, it is a > bit premature to include in OpenSSL. Including netinet/tcp.h is seriously > wrong to be part of openssl :) And finally, as I said before, the best way to > get things in OpenSSL is to do pull requests on GitHub. > > Having said all that, the concept is interesting and exciting, and I hope it > becomes widespread soon. Thank you for your comments. I've posted this series as a pull request: https://github.com/openssl/openssl/pull/3631 These patch set isn't intended to be merged. It is posted to start the discussion about it and to gather some feedback from this community about how to integrate it into OpenSSL once it is available upstream. From hannes at stressinduktion.org Thu Jun 8 08:43:15 2017 From: hannes at stressinduktion.org (Hannes Frederic Sowa) Date: Thu, 08 Jun 2017 10:43:15 +0200 Subject: [openssl-dev] [RFC 0/4] Kernel TLS socket API In-Reply-To: <20170607220540.qsi2caboyvydiwii@roeckx.be> References: <20170607220540.qsi2caboyvydiwii@roeckx.be> Message-ID: <1496911395.1082191.1002676352.6EB05913@webmail.messagingengine.com> Hello, On Thu, Jun 8, 2017, at 00:05, Kurt Roeckx wrote: > On Wed, Jun 07, 2017 at 03:35:45PM +0300, Boris Pismenny wrote: > > Hello all, > > > > I would like to introduce you to the new kernel API for TLS transmit-side > > data-path, and open a discussion regarding its support in OpenSSL. > > So my understanding is that there are really 2 parts in the kernel > that change: > - The kernel is aware of TLS and can do the symmetric encryption > - The kernel can offload the symmetric encryption to the NIC > > And I guess you're mostly interested in the combination of the two > where you would end up with the unencrypted data going go the NIC > and that you might get speeds close to what you can do > unencrypted. The performance gains would come from avoiding making > copies and not doing the encryption on the CPU. > > My understanding from the old data is that moving the encryption > to the kernel had a negative performance impact. So this at least > looks like something we do not always want to enable. It might be > useful to have an API where we can check that the offload is > supported, or that we have an option to enable moving it to the > kernel. we have discussed this in the past on netdev at vger.kernel.org but I just want to point out here again, that renewing the symmetric crypto keys is not supported in the kernel part (for the time being). So in case the application depends on renegotiation (TLS1.2, which is the only version supported right now by the kernel AFAIK) as well key updates in TLS1.3 won't work. Because this feature is not transparent yet, I think it definitely needs a switch for applications to control it. Bye, Hannes From kurt at roeckx.be Thu Jun 8 16:26:35 2017 From: kurt at roeckx.be (Kurt Roeckx) Date: Thu, 8 Jun 2017 18:26:35 +0200 Subject: [openssl-dev] [RFC 0/4] Kernel TLS socket API In-Reply-To: <1496911395.1082191.1002676352.6EB05913@webmail.messagingengine.com> References: <20170607220540.qsi2caboyvydiwii@roeckx.be> <1496911395.1082191.1002676352.6EB05913@webmail.messagingengine.com> Message-ID: <20170608162635.sbneycw3uot3psth@roeckx.be> On Thu, Jun 08, 2017 at 10:43:15AM +0200, Hannes Frederic Sowa wrote: > > we have discussed this in the past on netdev at vger.kernel.org but I just > want to point out here again, that renewing the symmetric crypto keys is > not supported in the kernel part (for the time being). > > So in case the application depends on renegotiation (TLS1.2, which is > the only version supported right now by the kernel AFAIK) as well key > updates in TLS1.3 won't work. It might be useful to be able to transfer the state in both directions, so that those things are possible. > Because this feature is not transparent yet, I think it definitely needs > a switch for applications to control it. We will probably also at least need to have way to find out if a cipher is supported by the kernel we're running on or not. Kurt From kurt at roeckx.be Thu Jun 8 19:20:08 2017 From: kurt at roeckx.be (Kurt Roeckx) Date: Thu, 8 Jun 2017 21:20:08 +0200 Subject: [openssl-dev] [RFC 0/4] Kernel TLS socket API In-Reply-To: References: <20170607220540.qsi2caboyvydiwii@roeckx.be> Message-ID: <20170608192007.ly2l4pgt7cq7chg6@roeckx.be> On Thu, Jun 08, 2017 at 06:26:28PM +0000, Ilya Lesokhin wrote: > Hi Kurt, > I think this it's better to have this discussion in the kernel mailing list. > But basically, we were debating this issue ourselves. > Previously we had another field in the attach API which could be {SW only, HW only and auto}. > However, we thought that most application wouldn't know what to do with it. > So, we thought we could simplify the API, and make the HW/SW decision a global administrative configuration. > > The downside is that the ability to do HW offload depends on the routing, so the software > Can't be configured to use KTLS only if HW offload is available. > > Do you think we should restore the old {SW only, HW only and auto} API? I currently have no idea what the best way is. But I would like to avoid a regression in performance by default. So I guess part of the question is if you think this kind of hardware would become common, or that it's only going to be used in specialised settings? Kurt From borisp at mellanox.com Thu Jun 8 18:32:06 2017 From: borisp at mellanox.com (Boris Pismenny) Date: Thu, 8 Jun 2017 18:32:06 +0000 Subject: [openssl-dev] [RFC 0/4] Kernel TLS socket API In-Reply-To: <20170608162635.sbneycw3uot3psth@roeckx.be> References: <20170607220540.qsi2caboyvydiwii@roeckx.be> <1496911395.1082191.1002676352.6EB05913@webmail.messagingengine.com> <20170608162635.sbneycw3uot3psth@roeckx.be> Message-ID: > > we have discussed this in the past on netdev at vger.kernel.org but I > > just want to point out here again, that renewing the symmetric crypto > > keys is not supported in the kernel part (for the time being). > > > > So in case the application depends on renegotiation (TLS1.2, which is > > the only version supported right now by the kernel AFAIK) as well key > > updates in TLS1.3 won't work. > > It might be useful to be able to transfer the state in both directions, so that > those things are possible. Symmetrically to the setsockopt TLS_TX that provides keys, iv and record sequence number to the kernel there is getsockopt TLS_TX that retrieves the same parameters. Do you think that supporting renegotiation is a must to merge this kernel feature into OpenSSL? or is it possible to let users decide if they want to make the tradeoff? > > > Because this feature is not transparent yet, I think it definitely > > needs a switch for applications to control it. > > We will probably also at least need to have way to find out if a cipher is > supported by the kernel we're running on or not. Currently it is possible to call the setsockopt TLS_TX with some cipher. If it is not supported the operation will fail. From rsalz at akamai.com Fri Jun 9 17:51:24 2017 From: rsalz at akamai.com (Salz, Rich) Date: Fri, 9 Jun 2017 17:51:24 +0000 Subject: [openssl-dev] Code Health Tuesday -- Fix the FAQ Message-ID: <08d59baca17144d784af4e43b95ba9ac@usma1ex-dag1mb1.msg.corp.akamai.com> It's been awhile since we did a code health Tuesday and we're overdue for one next week. Our online FAQ is really old; it's outdated and incorrect. We haven't fully figured out how much of the older versions and older platforms we should document. So, let's fix it. Move anything older than 1.0.2 to the new "old" section. Move anything about really old platforms that aren't fully supported, or have strange wonky compilers, etc., to that same section. And along the way, let's fix up any other entries that come to mind. The repo is here: https://github.com/openssl/web The FAQ can be found here: https://github.com/openssl/web/tree/master/docs Feel free to clone the fork the repo and make pull requests. If that's too much work, open an issue with the suggested revisions (but please if it's about moving entries, do a PR). The FAQ is mostly plain text, with some markdown-like additions. It should be easy to figure out. Thanks! We'll post a reminder about this after the weekend. -- Senior Architect, Akamai Technologies Member, OpenSSL Dev Team IM: richsalz at jabber.at Twitter: RichSalz -------------- next part -------------- An HTML attachment was scrubbed... URL: From rees.bettell at identifiglobal.com Tue Jun 13 08:53:47 2017 From: rees.bettell at identifiglobal.com (Rees Bettell) Date: Tue, 13 Jun 2017 08:53:47 +0000 Subject: [openssl-dev] Quantum Random Number Generator - Consultancy Opportunity - Rees Bettell Message-ID: Hi All. The reason I am reaching out to you today is because I have a very interesting opportunity that I would like to bring to your attention. My client is currently in the process of building a quantum encrypted chip, and they are looking to connect with someone that has Quantum Random Number Generator experience/knowledge. Due to the sensitivity of the work I can't provide much more detail at this point. This role is very unique and interesting, and my client is happy for the successful person to work from home with occasional travel to the UK, so relocation wouldn't be needed. The work will take place on a consultancy basis, so the type of rate would be daily; and my client would be happy to accommodate your expected charge rate as they have very strong backing from sponsors. If you fit this requirement it would be great to speak with you. Alternatively if you know someone that isn't in this mailshot but fits the requirement, please pass the email on. Best regards, Rees Bettell. Rees Bettell | Researcher T: +44 (0)1908 886 048 | M: +44 (0)7730096013 | DDI: +44 (0)1908 886 031 E: rees.bettell at identifiglobal.com | W: www.identifiglobal.com Innovation Centre, A&E Block, Bletchley Park, Milton Keynes, Buckinghamshire, MK3 6EB [http://tiny.cc/23bq4x] [http://tiny.cc/o5bq4x] [http://tiny.cc/76bq4x] [http://tiny.cc/r7bq4x] [identifi Global Resources] [http://tiny.cc/e9bq4x] This message is intended only for the use of the person(s) to whom it is addressed. Any dissemination, distribution, copying or other use of this message or any of its contents by any person other that the intended recipient may constitute a breach of civil or criminal law and is strictly prohibited. Emails are susceptible to interference. Identifi Global Resources Limited accepts no responsibility for information, errors, omissions or corrupt files received as a result of this email. Identifi Global Resources Limited is a Company registered in the UK, Company number 9652112. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 989 bytes Desc: image001.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 766 bytes Desc: image002.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.jpg Type: image/jpeg Size: 839 bytes Desc: image003.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image004.jpg Type: image/jpeg Size: 916 bytes Desc: image004.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image005.png Type: image/png Size: 7839 bytes Desc: image005.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image007.jpg Type: image/jpeg Size: 3957 bytes Desc: image007.jpg URL: From matt at openssl.org Fri Jun 16 09:25:06 2017 From: matt at openssl.org (Matt Caswell) Date: Fri, 16 Jun 2017 10:25:06 +0100 Subject: [openssl-dev] Travis failures Message-ID: I'm seeing some weird Travis failures: LD_LIBRARY_PATH=.: clang-3.9 -DDSO_DLFCN -DHAVE_DLFCN_H -DNDEBUG -DOPENSSL_THREADS -DOPENSSL_NO_DYNAMIC_ENGINE -DOPENSSL_PIC -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -DOPENSSLDIR="/usr/local/ssl" -DENGINESDIR="/usr/local/lib/engines-1.1" -Wall -O3 -pthread -m64 -DL_ENDIAN -fsanitize=address -fno-omit-frame-pointer -g -o apps/openssl apps/app_rand.o apps/apps.o apps/asn1pars.o apps/ca.o apps/ciphers.o apps/cms.o apps/crl.o apps/crl2p7.o apps/dgst.o apps/dhparam.o apps/dsa.o apps/dsaparam.o apps/ec.o apps/ecparam.o apps/enc.o apps/engine.o apps/errstr.o apps/gendsa.o apps/genpkey.o apps/genrsa.o apps/nseq.o apps/ocsp.o apps/openssl.o apps/opt.o apps/passwd.o apps/pkcs12.o apps/pkcs7.o apps/pkcs8.o apps/pkey.o apps/pkeyparam.o apps/pkeyutl.o apps/prime.o apps/rand.o apps/rehash.o apps/req.o apps/rsa.o apps/rsautl.o apps/s_cb.o apps/s_client.o apps/s_server.o apps/s_socket.o apps/s_time.o apps/sess_id.o apps/smime.o apps/speed.o apps/spkac.o apps/srp.o apps/ts.o apps/verify.o apps/version.o apps/x509.o -L. -lssl -L. -lcrypto -ldl /usr/bin/ld: /usr/lib/llvm-3.9/bin/../lib/clang/3.9.1/lib/linux/libclang_rt.asan-x86_64.a(asan_allocator.cc.o): unrecognized relocation (0x2a) in section `.text' /usr/bin/ld: final link failed: Bad value clang: error: linker command failed with exit code 1 (use -v to see invocation) make[2]: *** [link_app.] Error 1 make[2]: Leaving directory `/home/travis/build/openssl/openssl' make[1]: *** [apps/openssl] Error 2 make[1]: Leaving directory `/home/travis/build/openssl/openssl' make: *** [tests] Error 2 It only seems to be on clang-3.9 builds. The first commit where this showed up was this one: commit 594da712ffe416e61eeedc736d0332fa4664b24f Author: Rich Salz AuthorDate: Wed Jun 14 12:05:25 2017 -0400 Commit: Rich Salz CommitDate: Wed Jun 14 12:05:25 2017 -0400 Remove OLD_STR_TO_KEY compile option This flag was added in 1992 and only documented in the CHANGES file. Reviewed-by: Andy Polyakov (Merged from https://github.com/openssl/openssl/pull/3681) Looking at that commit I find it hard to believe that it would have this effect. Also, interestingly, I note that that build only failed on the last clang-3.9 job, but all the others since then have failed in all of them. Possibly a bad clang-3.9 update?? Any ideas? Matt From matt at openssl.org Fri Jun 16 09:43:59 2017 From: matt at openssl.org (Matt Caswell) Date: Fri, 16 Jun 2017 10:43:59 +0100 Subject: [openssl-dev] Travis failures In-Reply-To: References: Message-ID: <0ab0f0bc-2b44-1897-22f2-b5b76eaf0317@openssl.org> On 16/06/17 10:25, Matt Caswell wrote:> Possibly a bad clang-3.9 update?? Any ideas? After this command in the logs: $ sudo -E apt-get -yq --no-install-suggests --no-install-recommends --force-yes install clang-3.9 Failing builds look like this: After this operation, 262 MB of additional disk space will be used. Get:1 http://us-central1.gce.archive.ubuntu.com/ubuntu/ trusty-updates/universe libllvm3.9v4 amd64 1:3.9.1-4ubuntu3~14.04.2 [10.6 MB] Get:2 http://us-central1.gce.archive.ubuntu.com/ubuntu/ trusty-updates/universe libclang1-3.9 amd64 1:3.9.1-4ubuntu3~14.04.2 [5,709 kB] Get:3 http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu/ trusty/main gcc-7-base amd64 7.1.0-5ubuntu2~14.04 [18.5 kB] Get:4 http://us-central1.gce.archive.ubuntu.com/ubuntu/ trusty-updates/universe binutils-2.26 amd64 2.26.1-1ubuntu1~14.04 [3,437 kB] Get:5 http://us-central1.gce.archive.ubuntu.com/ubuntu/ trusty-updates/universe libclang-common-3.9-dev amd64 1:3.9.1-4ubuntu3~14.04.2 [2,733 kB] Older successful builds have this: Get:1 http://apt.llvm.org/trusty/ llvm-toolchain-trusty-3.9/main libllvm3.9 amd64 1:3.9~svn288847-1~exp1 [10.6 MB] Get:2 http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu/ trusty/main gcc-7-base amd64 7.1.0-5ubuntu2~14.04 [18.5 kB] Get:3 http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu/ trusty/main libstdc++6 amd64 7.1.0-5ubuntu2~14.04 [304 kB] Get:4 http://apt.llvm.org/trusty/ llvm-toolchain-trusty-3.9/main libclang1-3.9 amd64 1:3.9~svn288847-1~exp1 [5,752 kB] Get:5 http://apt.llvm.org/trusty/ llvm-toolchain-trusty-3.9/main libclang-common-3.9-dev amd64 1:3.9~svn288847-1~exp1 [2,328 kB] Get:6 http://apt.llvm.org/trusty/ llvm-toolchain-trusty-3.9/main clang-3.9 amd64 1:3.9~svn288847-1~exp1 [35.6 MB] Note the successful builds download from llvm.org but failing builds do not. I cannot seem to connect with my browser to llvm.org today, so I suspect they are having problems. Matt From bhat.jayalakshmi at gmail.com Sun Jun 18 17:03:12 2017 From: bhat.jayalakshmi at gmail.com (Jayalakshmi bhat) Date: Sun, 18 Jun 2017 22:33:12 +0530 Subject: [openssl-dev] FIPS CAVP tests for WinCE. Message-ID: Hi All, I am using OpenSSL-FIPS-2.0.4 library on ARM7 + WinCE 6.0 with "user affirm" the validation for Y per I.G. G.5. We want to run latest CAVP test suites. We have built the *build_algvs and other executable* for the above product/build environment. However when we are trying to execute the executable with req file and resp file parameters, example fips_drbgvs CTR_DRBG.req CTR_DRBG.resp we end up in receiving error "error opening the input file". Later we found that WinCE environment cannot read simple character file name, it needs some windows specific conversion like WideCharToMultiByte. We have the below questions, 1. Is there any way to build the test suites on WinCE environment. User guide says it is incomplete? 2. As these are test files, is it OK to modify them? Regards Jayalakshmi -------------- next part -------------- An HTML attachment was scrubbed... URL: From marquess at openssl.com Sun Jun 18 20:51:09 2017 From: marquess at openssl.com (Steve Marquess) Date: Sun, 18 Jun 2017 16:51:09 -0400 Subject: [openssl-dev] FIPS CAVP tests for WinCE. In-Reply-To: References: Message-ID: On 06/18/2017 01:03 PM, Jayalakshmi bhat wrote: > Hi All, > > I am using OpenSSL-FIPS-2.0.4 library on ARM7 + WinCE 6.0 with "user > affirm" the validation for Y per I.G. G.5. > > We want to run latest CAVP test suites. We have built the *build_algvs > and other executable* for the above product/build environment. > However when we are trying to execute the executable with req file and > resp file parameters, example fips_drbgvs CTR_DRBG.req CTR_DRBG.resp > we end up in receiving error "error opening the input file". > > Later we found that WinCE environment cannot read simple character file > name, it needs some windows specific conversion like WideCharToMultiByte. > > We have the below questions, > > 1. Is there any way to build the test suites on WinCE environment. User > guide says it is incomplete? > 2. As these are test files, is it OK to modify them? WinCE (with WinEC) is one of the most painful environments to test on, and what's worse is no two such devices seem to be very similar. We've ended up hand-hacking the test suite software for each and every such platform validation, and the hacks for one WinCR device aren't necessarily useful for the next one. The test suite software is outside the ideologically significant "cryptographic module boundary", but you'll need to consult with your particular accredited test lab on just what is and isn't acceptable. Also please note that the algorithm test vectors change frequently, so even on a well-behaved target device you generally can't use the test suite software from earlier validations as-is. Usually the hacks to accommodate the latest test vector format aren't difficult, but you don't know what's actually needed until you get the new set of test vectors (which of course cost money). -Steve M. -- Steve Marquess OpenSSL Validation Services, Inc. 1829 Mount Ephraim Road Adamstown, MD 21710 USA +1 301 874 2571 marquess at openssl.com gpg/pgp key: http://openssl.com/docs/0x6D1892F5.asc From mohannadmostafa at outlook.com Mon Jun 19 18:03:19 2017 From: mohannadmostafa at outlook.com (Mohannad S. Mostafa) Date: Mon, 19 Jun 2017 18:03:19 +0000 Subject: [openssl-dev] Barrett's Reduction in OpenSSL Message-ID: Hello, I am doing a research related to Barrett's reduction and I was looking for OpenSSL implementation of Barrett's Reduction. However, I couldn't find anything close to it. I know there's a reduction algorithm implemented in this file bn_gf2m.c but I was wondering if Barrett's reduction is implemented in OpenSSL? Thank you, Mohannad -------------- next part -------------- An HTML attachment was scrubbed... URL: From matt at openssl.org Tue Jun 20 13:29:57 2017 From: matt at openssl.org (Matt Caswell) Date: Tue, 20 Jun 2017 14:29:57 +0100 Subject: [openssl-dev] Travis failures In-Reply-To: <0ab0f0bc-2b44-1897-22f2-b5b76eaf0317@openssl.org> References: <0ab0f0bc-2b44-1897-22f2-b5b76eaf0317@openssl.org> Message-ID: <04497c25-d58a-6494-94ed-8800c88b92ca@openssl.org> On 16/06/17 10:43, Matt Caswell wrote: > > > On 16/06/17 10:25, Matt Caswell wrote:> Possibly a bad clang-3.9 > update?? Any ideas? > After this command in the logs: > > $ sudo -E apt-get -yq --no-install-suggests --no-install-recommends > --force-yes install clang-3.9 > > Failing builds look like this: > > After this operation, 262 MB of additional disk space will be used. > Get:1 http://us-central1.gce.archive.ubuntu.com/ubuntu/ > trusty-updates/universe libllvm3.9v4 amd64 1:3.9.1-4ubuntu3~14.04.2 > [10.6 MB] > Get:2 http://us-central1.gce.archive.ubuntu.com/ubuntu/ > trusty-updates/universe libclang1-3.9 amd64 1:3.9.1-4ubuntu3~14.04.2 > [5,709 kB] > Get:3 http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu/ > trusty/main gcc-7-base amd64 7.1.0-5ubuntu2~14.04 [18.5 kB] > Get:4 http://us-central1.gce.archive.ubuntu.com/ubuntu/ > trusty-updates/universe binutils-2.26 amd64 2.26.1-1ubuntu1~14.04 [3,437 kB] > Get:5 http://us-central1.gce.archive.ubuntu.com/ubuntu/ > trusty-updates/universe libclang-common-3.9-dev amd64 > 1:3.9.1-4ubuntu3~14.04.2 [2,733 kB] > > > Older successful builds have this: > > Get:1 http://apt.llvm.org/trusty/ llvm-toolchain-trusty-3.9/main > libllvm3.9 amd64 1:3.9~svn288847-1~exp1 [10.6 MB] > Get:2 http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu/ > trusty/main gcc-7-base amd64 7.1.0-5ubuntu2~14.04 [18.5 kB] > Get:3 http://ppa.launchpad.net/ubuntu-toolchain-r/test/ubuntu/ > trusty/main libstdc++6 amd64 7.1.0-5ubuntu2~14.04 [304 kB] > Get:4 http://apt.llvm.org/trusty/ llvm-toolchain-trusty-3.9/main > libclang1-3.9 amd64 1:3.9~svn288847-1~exp1 [5,752 kB] > Get:5 http://apt.llvm.org/trusty/ llvm-toolchain-trusty-3.9/main > libclang-common-3.9-dev amd64 1:3.9~svn288847-1~exp1 [2,328 kB] > Get:6 http://apt.llvm.org/trusty/ llvm-toolchain-trusty-3.9/main > clang-3.9 amd64 1:3.9~svn288847-1~exp1 [35.6 MB] > > > Note the successful builds download from llvm.org but failing builds do > not. I cannot seem to connect with my browser to llvm.org today, so I > suspect they are having problems. Well llvm.org is back up, but we still seem to be failing in travis with the same issue. Any ideas? Matt From nicola.tuveri at tut.fi Sun Jun 25 21:07:29 2017 From: nicola.tuveri at tut.fi (Nicola Tuveri) Date: Mon, 26 Jun 2017 00:07:29 +0300 Subject: [openssl-dev] Dynamically adding a NID Message-ID: Hi, I'm working on ENGINE development, and I have the need to add an NID for a custom message digest, and eventually for ciphers and PKEY methods. Some of the associated object don't (and won't ever) have an associated OID, but I need to add them dynamically to avoid requiring patches to the upstream OpenSSL code before being able to use my engine. I'm currently (ab)using OBJ_create() [0], but it looks like it requires to specify a valid OID. I know it is possible to have NIDs associated with objects without OID (e.g. NID_siphash) when they are statically defined in OpenSSL source code, but I cannot find a way to declare similar objects without OID dynamically. Before 1.1.0, when structures weren't opaque, I could manipulate the contents of the created object directly and somehow work around this limitation, but in 1.1.0 this is not possible. Does anyone know of the right way to dynamically create an NID associated with an object without OID? Thanks. Nicola [0] : https://github.com/openssl/openssl/blob/master/include/ope nssl/objects.h#L157 -------------- next part -------------- An HTML attachment was scrubbed... URL: From rsalz at akamai.com Sun Jun 25 23:18:28 2017 From: rsalz at akamai.com (Salz, Rich) Date: Sun, 25 Jun 2017 23:18:28 +0000 Subject: [openssl-dev] Dynamically adding a NID In-Reply-To: References: Message-ID: You can get an OID arc of your own for free. And then you can use real OID?s which you just ?throw away? See https://en.wikipedia.org/wiki/Private_Enterprise_Number -------------- next part -------------- An HTML attachment was scrubbed... URL: From rsalz at akamai.com Mon Jun 26 12:49:08 2017 From: rsalz at akamai.com (Salz, Rich) Date: Mon, 26 Jun 2017 12:49:08 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL Message-ID: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> We are starting to work on a new cryptographically strong pseudo random number generator for OpenSSL, known as CSPRNG or PRNG or RNG. Please take a look at GitHub pull request https://github.com/openssl/openssl/pull/3758 which is the start of a series. In particular, please take a look at some detailed comments from one of our committers, at https://github.com/openssl/openssl/pull/3758#issuecomment-310938562, and the followon. We welcome your input. -- Senior Architect, Akamai Technologies Member, OpenSSL Dev Team IM: richsalz at jabber.at Twitter: RichSalz -------------- next part -------------- An HTML attachment was scrubbed... URL: From rsalz at akamai.com Mon Jun 26 13:10:08 2017 From: rsalz at akamai.com (Salz, Rich) Date: Mon, 26 Jun 2017 13:10:08 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> Message-ID: My thoughts. Entropy should be whitened. Anything feed into an entropy pool, should be mixed in and run through SHA256. pool = SHA256(pool || new-entropy) The current read and write file routines, and the current routine RAND_poll, etc., will add to that global pool. The idea of cascading pools is neat. We need at least one per thread, using our existing thread-local-storage API. The current ?lazy evaluation? will work fine, we don?t need a create-thread API. We do need fork/exec protection which is the point of https://github.com/openssl/openssl/pull/3754 Each pool should have an atomic counter that is incremented when entropy is added. Descendant pools can compare counters and mix in their parent when the counters don?t match. Then when RAND_poll is called, or perhaps a new routine RAND_poll_system, it goes into the global pool and eventually all other pools will get it (whitened with their current state). RAND_poll isn?t documented. Per-thread pools don?t need a lock. The global and other pools do. Putting a pool in the SSL_CTX is probably reasonable. I seriously doubt the SSL object needs it because the number of random bytes to generate keys is pretty small ? we?ll expose things through AES misused first :) But adding it to the SSL object is simple so we might as well. Then to generate random bytes use ChaCha. See, for example, http://gitweb.dragonflybsd.org/dragonfly.git/blob/2aa3f894bd9b5b8f58a1526adb26663405b91679:/sys/kern/subr_csprng.c My first thoughts on reading that code were, wow, is it really that easy? We want to be able to save the current global state ? write to a BIO ? and restore it ? read from a BIO. This will let us reasonably work in low-entropy situations like system boot. We want to provide a platform-neutral API that makes it?s best effort attempt to get entropy from the system and merge it into the global pool. That should be a new API; I suggested RAND_poll_system above, but don?t really care Does this make sense? Are there holes? -------------- next part -------------- An HTML attachment was scrubbed... URL: From beldmit at gmail.com Mon Jun 26 14:12:13 2017 From: beldmit at gmail.com (Dmitry Belyavsky) Date: Mon, 26 Jun 2017 17:12:13 +0300 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> Message-ID: Dear Rich, On Mon, Jun 26, 2017 at 3:49 PM, Salz, Rich via openssl-dev < openssl-dev at openssl.org> wrote: > We are starting to work on a new cryptographically strong pseudo random > number generator for OpenSSL, known as CSPRNG or PRNG or RNG. > > > > Please take a look at GitHub pull request https://github.com/openssl/ > openssl/pull/3758 which is the start of a series. In particular, please > take a look at some detailed comments from one of our committers, at > https://github.com/openssl/openssl/pull/3758#issuecomment-310938562, and > the followon. > > > > We welcome your input. > Will the new architecture still allow engine-defined RNG methods? It's a critical requirement for our products. Thank you! -- SY, Dmitry Belyavsky -------------- next part -------------- An HTML attachment was scrubbed... URL: From tmraz at redhat.com Mon Jun 26 14:24:11 2017 From: tmraz at redhat.com (Tomas Mraz) Date: Mon, 26 Jun 2017 16:24:11 +0200 Subject: [openssl-dev] Disablement of insecure hashes for digital signatures Message-ID: <1498487051.5915.17.camel@redhat.com> Just a notice for anyone interested, In Red Hat Enterprise Linux 6 and 7 we disabled support for insecure hashes for digital signatures. Basically signatures with MD5, MD4, MD2, and SHA0 will fail verification by default. We could not switch off the support for these weak hash algorithms completely due to possible legacy uses so we at least switched it off for signature verification. Regards, -- Tom?? Mr?z Red Hat No matter how far down the wrong road you've gone, turn back. ??????????????????????????????????????????????Turkish proverb [You'll know whether the road is wrong if you carefully listen to your conscience.] * Google and NSA associates, this message is none of your business. * Please leave it alone, and consider whether your actions are * authorized by the contract with Red Hat, or by the US constitution. * If you feel you're being encouraged to disregard the limits built * into them, remember Edward Snowden and Wikileaks. From ssx at av8n.com Mon Jun 26 14:30:52 2017 From: ssx at av8n.com (John Denker) Date: Mon, 26 Jun 2017 07:30:52 -0700 Subject: [openssl-dev] discussion venue policy In-Reply-To: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> Message-ID: <16530b47-e93a-bc66-17a9-933150d1589c@av8n.com> On 06/26/2017 05:49 AM, Salz, Rich wrote: > Please take a look at GitHub pull request Is the RNG topic going to be discussed on github, or on openssl-dev? What about other topics? Having some topics one place and some another seems like a Bad Idea? Having a single topic split across multiple venues seems even worse. From rsalz at akamai.com Mon Jun 26 14:41:24 2017 From: rsalz at akamai.com (Salz, Rich) Date: Mon, 26 Jun 2017 14:41:24 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> Message-ID: <6f01c47cea7b4e659aefce8f50acbad9@usma1ex-dag1mb1.msg.corp.akamai.com> > Will the new architecture still allow engine-defined RNG methods? It's a critical requirement for our products. Yes From rsalz at akamai.com Mon Jun 26 14:43:00 2017 From: rsalz at akamai.com (Salz, Rich) Date: Mon, 26 Jun 2017 14:43:00 +0000 Subject: [openssl-dev] discussion venue policy In-Reply-To: <16530b47-e93a-bc66-17a9-933150d1589c@av8n.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <16530b47-e93a-bc66-17a9-933150d1589c@av8n.com> Message-ID: <01365f97c76347aa824ace46c66f04d2@usma1ex-dag1mb1.msg.corp.akamai.com> Discussion should be here on the mailing list. From ssx at av8n.com Mon Jun 26 15:22:48 2017 From: ssx at av8n.com (John Denker) Date: Mon, 26 Jun 2017 08:22:48 -0700 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> Message-ID: <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> On 06/26/2017 05:49 AM, Salz, Rich via openssl-dev wrote: > We welcome your input. Here is an observation plus some suggestions: Using the word "entropy" in this context is unhelpful. Normally, entropy means something very specific, in which case using entropy to design and explain your RNG is a bad idea. I can exhibit a distribution that has provably infinite entropy, even though you can guess the exact output more than 25% of the time. If perhaps you mean something else, calling it "entropy" is an even worse idea. It is likely that readers will misunderstand what is written. I am quite aware that the word appears in kernel source, but that doesn't make it right. It is used inconsistently, and AFAICT none of the possible interpretations is really correct. Note: The real issue here it not the terminology. Ideas are primary and fundamental; terminology is tertiary. Terminology is only important insofar as it helps us formulate and communicate the ideas. There are at least five different ideas that need to be understood: 1) The randomness of an ideal PRNG. 2) The randomness of an ideal TRNG aka HRNG. 3) The opposite, i.e. pure determinism. 4) Squish, which is neither reliably predictable nor reliably unpredictable. 5) Combinations of the above. Suggestion: Get rid of every mention of "entropy" from openssl code, documentation, design discussions, and everywhere else. Suggestion: In the common case where exact meaning is not important, "entropy" can be replaced by a noncommittal nontechnical word such as "randomness". Even so, it should be clearly documented that this term is not meant to be quantitative. Suggestion: If you mean for something to be hard for the attacker to guess, the word "adamance" can be used. This can be quantified in terms of the R?nyi H_? functional, plus some additional attention to detail (including specifying that it is a functional of the attacker's macrostate, not anybody else's). Suggestion: In the remaining cases, which are not rare, it is important to take a step back and figure out what is the actual idea that is being (or should be) discussed. This will not be easy, but it must be done, line by line. Otherwise the whole enterprise is likely to be a waste of time. From rsalz at akamai.com Mon Jun 26 15:32:05 2017 From: rsalz at akamai.com (Salz, Rich) Date: Mon, 26 Jun 2017 15:32:05 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> Message-ID: <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> > Suggestion: Get rid of every mention of "entropy" from openssl code, > documentation, design discussions, and everywhere else. I like this and will do it. Except for our support of the EGD. > Suggestion: In the common case where exact meaning is not important, > "entropy" can be replaced by a noncommittal nontechnical word such as > "randomness". Even so, it should be clearly documented that this term is not > meant to be quantitative. Okay. > Suggestion: If you mean for something to be hard for the attacker to guess, > the word "adamance" can be used. All my attempts to look up a definition of this word came up with a noun for for adamant. Which is often appropriate, but maybe no here :) > Suggestion: In the remaining cases, which are not rare, it is important to take > a step back and figure out what is the actual idea that is being (or should be) > discussed. This will not be easy, but it must be done, line by line. Otherwise > the whole enterprise is likely to be a waste of time. We are trying hard not to waste anyone's time, a d we appreciate your input. Is it worth reposting my thoughts with your suggested wording changes? From ssx at av8n.com Mon Jun 26 15:58:21 2017 From: ssx at av8n.com (John Denker) Date: Mon, 26 Jun 2017 08:58:21 -0700 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> Message-ID: <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> In the context of: >> If you mean for something to be hard for the attacker to guess, >> the word "adamance" can be used. On 06/26/2017 08:32 AM, Salz, Rich wrote: > All my attempts to look up a definition of this word came up with a noun for for adamant. The word "adamance", meaning hardness (as in hard to guess), was coined for this purpose. The allusion to "adamance", meaning hardness (as in rheologically hard), is not a coincidence. Can anybody suggest a better term? For more on this, and a host of RNG-related issues see: https://www.av8n.com/turbid/paper/rng-intro.htm > Is it worth reposting my thoughts with your suggested wording changes? OK. Off-list or on. This stuff is important. From Brett.R.Nicholas.TH at dartmouth.edu Mon Jun 26 14:29:14 2017 From: Brett.R.Nicholas.TH at dartmouth.edu (Brett R. Nicholas) Date: Mon, 26 Jun 2017 14:29:14 +0000 Subject: [openssl-dev] How to define EVP_EncryptUpdate and EVP_EncryptFinal functions for an AES engine? (and a separate question re: padding) Message-ID: Hi there, I'm building a dynamic engine to support a custom AES hardware module that I've implemented in FPGA logic**, but after reading all available documentation, and pouring over the source code, I'm still very confused about the following two things..... 1. How and where I should define the EVP_[En/De]cryptInit_ex(..), EVP_[En/De]cryptUpdate(..), and EVP_[En/De]cryptFinal_ex(..) functions in my Engine code? Prior to this, I successfully built an engine for my sha256 accelerator, and now I'm trying to follow the same steps for AES. For sha256, the EVP_MD structure allowed me to declare pointers to my init, update, and final functions. This all worked flawlessly. Now, when I'm building the AES engine, I see that the EVP_CIPHER structure does not have these pointers (init, update, final), but instead has a pointer to init_key and do_cipher functions. However, the EVP encryption interface still has these functions defined. AFAIK (and please correct me if this is wrong) my init_key function is invoked by the EVP interface when I call the EVP_[En/De]cryptInit_ex function, and the do_cipher function is called upon EVP_[En/De]cryptUpdate. But how should I handle the EVP_[En/De]cryptFinal functions? Should I not be implementing them in my engine? Or am I missing something here.... 2. Does the EVP interface handle padding when a dynamic engine is involved? Or is it up to me to implement a padding structure within the engine itself? If the latter is the case, then I think the answer to the previous questions will help me figure out exactly where to implement it. So to recap, two questions: 1. How can I explicitly define which operations in my engine happen when the EVP_[En/De]cryptInit_ex(..), EVP_[En/De]cryptUpdate(..), and EVP_[En/De]cryptFinal_ex(..) functions are called from a driver program? 2. Does my engine need to handle padding the input data upon encryption, and stripping the padding when decrypting? Or does the EVP API handle the padding for me, and I only need to worry about the core AES algorithm on the arbitrary input data? (for reference, I'd like to just use standard PKCS padding) Thanks in advance, - Brett ** I'm using the Xilinx Zynq SoC, so I can create custom hardware in the programmable logic, and then interact with it from software running on the processor through the memory map, just like any peripheral....details irrelevant -------------- next part -------------- An HTML attachment was scrubbed... URL: From rsalz at akamai.com Mon Jun 26 16:14:40 2017 From: rsalz at akamai.com (Salz, Rich) Date: Mon, 26 Jun 2017 16:14:40 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> Message-ID: <5ba95d22a83c4cbfa0744f1b2287dc3d@usma1ex-dag1mb1.msg.corp.akamai.com> > > Is it worth reposting my thoughts with your suggested wording changes? > > OK. Off-list or on. This stuff is important. I will repost. And please also see https://github.com/openssl/openssl/pull/3773 From rsalz at akamai.com Mon Jun 26 16:17:41 2017 From: rsalz at akamai.com (Salz, Rich) Date: Mon, 26 Jun 2017 16:17:41 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> Message-ID: <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> > > Is it worth reposting my thoughts with your suggested wording changes? > > OK. Off-list or on. This stuff is important. Reposting. My thoughts. Randomness should be whitened. Anything feed into an randomness pool, should be mixed in and run through SHA256. pool = SHA256(pool || new-randomness) The current read and write file routines, and the current routine RAND_poll, etc., will add to that global pool. The idea of cascading pools is neat. We need at least one per thread, using our existing thread-local-storage API. The current ?lazy evaluation? will work fine, we don?t need a create-thread API. We do need fork/exec protection which is the point of https://github.com/openssl/openssl/pull/3754 Each pool should have an atomic counter that is incremented when randomness is added. Descendant pools can compare counters and mix in their parent when the counters don?t match. Then when RAND_poll is called, or perhaps a new routine RAND_poll_system, it goes into the global pool and eventually all other pools will get it (whitened with their current state). RAND_poll isn?t documented. Per-thread pools don?t need a lock. The global and other pools do. Putting a pool in the SSL_CTX is probably reasonable. I seriously doubt the SSL object needs it because the number of random bytes to generate keys is pretty small ? we?ll expose things through AES misused first ? But adding it to the SSL object is simple so we might as well. Then to generate random bytes use ChaCha. See, for example, http://gitweb.dragonflybsd.org/dragonfly.git/blob/2aa3f894bd9b5b8f58a1526adb26663405b91679:/sys/kern/subr_csprng.c My first thoughts on reading that code were, wow, is it really that easy? We want to be able to save the current global state ? write to a BIO ? and restore it ? read from a BIO. This will let us reasonably work in low-randomness situations like system boot. We want to provide a platform-neutral API that makes its best effort attempt to get randomness from the system and merge it into the global pool. That should be a new API; I suggested RAND_poll_system above, but don?t really care. Does this make sense? Are there holes? From kurt at roeckx.be Mon Jun 26 16:26:30 2017 From: kurt at roeckx.be (Kurt Roeckx) Date: Mon, 26 Jun 2017 18:26:30 +0200 Subject: [openssl-dev] discussion venue policy In-Reply-To: <16530b47-e93a-bc66-17a9-933150d1589c@av8n.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <16530b47-e93a-bc66-17a9-933150d1589c@av8n.com> Message-ID: <20170626162629.k3n6fbchfjmi2m4x@roeckx.be> On Mon, Jun 26, 2017 at 07:30:52AM -0700, John Denker via openssl-dev wrote: > On 06/26/2017 05:49 AM, Salz, Rich wrote: > > > Please take a look at GitHub pull request > > Is the RNG topic going to be discussed on github, or on openssl-dev? > What about other topics? > > Having some topics one place and some another seems like a Bad Idea? > Having a single topic split across multiple venues seems even worse. I think general discussion about a topic should be here, discussion about a specific patch should be on github. Kurt From kurt at roeckx.be Mon Jun 26 16:30:04 2017 From: kurt at roeckx.be (Kurt Roeckx) Date: Mon, 26 Jun 2017 18:30:04 +0200 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> Message-ID: <20170626163003.vg6gaaolg4ldeybc@roeckx.be> On Mon, Jun 26, 2017 at 08:58:21AM -0700, John Denker via openssl-dev wrote: > In the context of: > > >> If you mean for something to be hard for the attacker to guess, > >> the word "adamance" can be used. > > On 06/26/2017 08:32 AM, Salz, Rich wrote: > > > All my attempts to look up a definition of this word came up with a noun for for adamant. > > The word "adamance", meaning hardness (as in hard to guess), > was coined for this purpose. > > The allusion to "adamance", meaning hardness (as in rheologically > hard), is not a coincidence. > > Can anybody suggest a better term? unpredictable? Kurt From uri at ll.mit.edu Mon Jun 26 16:31:15 2017 From: uri at ll.mit.edu (Blumenthal, Uri - 0553 - MITLL) Date: Mon, 26 Jun 2017 16:31:15 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> Message-ID: My thoughts. Randomness should be whitened. Anything feed into an randomness pool, should be mixed in and run through SHA256. pool = SHA256(pool || new-randomness) Pseudorandomness of the output has been a design goal/requirement only in SHA-3 family. Any prior hash function?s exhibition of this property is coincidental. Therefore I suggest using SHA3 instead. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5211 bytes Desc: not available URL: From rsalz at akamai.com Mon Jun 26 17:10:46 2017 From: rsalz at akamai.com (Salz, Rich) Date: Mon, 26 Jun 2017 17:10:46 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> Message-ID: <63f66496438849a9b130f20b3897f082@usma1ex-dag1mb1.msg.corp.akamai.com> > Pseudorandomness of the output has been a design goal/requirement only > in SHA-3 family. Any prior hash function?s exhibition of this property is > coincidental. > > Therefore I suggest using SHA3 instead. Is pseudorandomness a requirement? Or is it the "50% chance of a bitflip"? From kurt at roeckx.be Mon Jun 26 17:12:46 2017 From: kurt at roeckx.be (Kurt Roeckx) Date: Mon, 26 Jun 2017 19:12:46 +0200 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> Message-ID: <20170626171246.vqno3xcsg3bldxm4@roeckx.be> On Mon, Jun 26, 2017 at 04:17:41PM +0000, Salz, Rich via openssl-dev wrote: > > > > Is it worth reposting my thoughts with your suggested wording changes? > > > > OK. Off-list or on. This stuff is important. > > Reposting. > > My thoughts. > > Randomness should be whitened. Anything feed into an randomness pool, should be mixed in and run through SHA256. > pool = SHA256(pool || new-randomness) Do you think we need to use multiple sources of randomness? I think we should only use the one source, the one provided by the kernel. All sources of randomness already go in it, there is no need for us to try add any other source that it's already using. So there should be no need to do any whitening. > Each pool should have an atomic counter that is incremented when randomness is added. Descendant pools can compare counters and mix in their parent when the counters don?t match. Then when RAND_poll is called, or perhaps a new routine RAND_poll_system, it goes into the global pool and eventually all other pools will get it (whitened with their current state). RAND_poll isn?t documented. The only thing the pool should care about is that it's been initialized or not, and if it needs to add more data to it or not. > Then to generate random bytes use ChaCha. See, for example, http://gitweb.dragonflybsd.org/dragonfly.git/blob/2aa3f894bd9b5b8f58a1526adb26663405b91679:/sys/kern/subr_csprng.c My first thoughts on reading that code were, wow, is it really that easy? You might also want to take a look at something like: https://github.com/smuellerDD/chacha20_drng/blob/master/chacha20_drng.c > We want to be able to save the current global state ? write to a BIO ? and restore it ? read from a BIO. This will let us reasonably work in low-randomness situations like system boot. Ideally we should refuse to operate in a situation where the kernel didn't initialize it's RNG yet. I only know about Linux being broken in this regard, and getrandom() / getentropy() really should be available on them by now. I don't think we should add a workaround by reading 1 byte from /dev/random if getrandom() isn't available. Kurt From rsalz at akamai.com Mon Jun 26 17:36:39 2017 From: rsalz at akamai.com (Salz, Rich) Date: Mon, 26 Jun 2017 17:36:39 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <20170626171246.vqno3xcsg3bldxm4@roeckx.be> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <20170626171246.vqno3xcsg3bldxm4@roeckx.be> Message-ID: <9bc59efbcd9a414e82263e81bd25269e@usma1ex-dag1mb1.msg.corp.akamai.com> > Do you think we need to use multiple sources of randomness? I think we > should only use the one source, the one provided by the kernel. Yes I think we will need to support it, such as systems that don't have kernel support. Always whitening doesn't hurt, so I would like to see the simpler logic of always doing it. And also for the case where existing applications mix in their own. > > Each pool should have an atomic counter that is incremented when > randomness is added. Descendant pools can compare counters and mix in > their parent when the counters don?t match. Then when RAND_poll is > called, or perhaps a new routine RAND_poll_system, it goes into the global > pool and eventually all other pools will get it (whitened with their current > state). RAND_poll isn?t documented. > > The only thing the pool should care about is that it's been initialized or not, > and if it needs to add more data to it or not. In order to maintain conceptual compatibility with the current API, I think that when someone does RAND_poll or RAND_add, they are expecting it to be be used. I think having a "added_count" field in every pool, and doing a check is simple way to ensure that changes to the global pool propagate down. > You might also want to take a look at something like: > https://github.com/smuellerDD/chacha20_drng/blob/master/chacha20_drn > g.c Is there any reason why this is better than the other mechanisms? > > We want to be able to save the current global state ? write to a BIO ? and > restore it ? read from a BIO. This will let us reasonably work in low- > randomness situations like system boot. > > Ideally we should refuse to operate in a situation where the kernel didn't > initialize it's RNG yet. I only know about Linux being broken in this regard, and > getrandom() / getentropy() really should be available on them by now. I > don't think we should add a workaround by reading 1 byte from > /dev/random if getrandom() isn't available. We're not yet in the ideal world, and all the world's not a modern Linux kernel :) I believe we need to give applications an "escape hatch" to let them read/write the global state. And also think of embedded devices where perhaps the state is in NVRAM. From uri at ll.mit.edu Mon Jun 26 18:12:19 2017 From: uri at ll.mit.edu (Blumenthal, Uri - 0553 - MITLL) Date: Mon, 26 Jun 2017 18:12:19 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <63f66496438849a9b130f20b3897f082@usma1ex-dag1mb1.msg.corp.akamai.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <63f66496438849a9b130f20b3897f082@usma1ex-dag1mb1.msg.corp.akamai.com> Message-ID: <525AD6A6-FFBA-461C-B60F-8ABB10EB585B@ll.mit.edu> > Pseudorandomness of the output has been a design goal/requirement only > in SHA-3 family. Any prior hash function?s exhibition of this property is > coincidental. > > Therefore I suggest using SHA3 instead. Is pseudorandomness a requirement? Or is it the "50% chance of a bitflip"? For [P]RNG?! In one word: yes. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5211 bytes Desc: not available URL: From ssx at av8n.com Mon Jun 26 18:29:19 2017 From: ssx at av8n.com (John Denker) Date: Mon, 26 Jun 2017 11:29:19 -0700 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> Message-ID: <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> On 06/26/2017 09:17 AM, Salz, Rich wrote: [snip] > Does this make sense? Are there holes? Even without the snipping, the proposal is very incomplete. Insofar as any hole that is not explicitly closed should be presumed open, then yes, there are many holes. What's your threat model? I know that sounds like a clich?, but it's actually important. In particular, in my world the #1 threat against any PRNG is improper seeding. -- If you trust the ambient OS to provide a seed, why not trust it for everything, and not bother to implement an openssl-specfic RNG at all? -- Conversely, if you don't trust the OS, what makes you think you can solve a problem the OS failed to solve, especially without knowing why it failed? And (!) what do you propose to do when a suitable seed is not available at the moment but might be available later? Are you designing to resist an output-text-only attack? Or do you also want "some" ability to recover from a compromise of the PRNG internal state? Is there state in a persistent file, or only in memory? > Randomness should be whitened. Anything feed into an randomness > pool, should be mixed in and run through SHA256. pool = SHA256(pool > || new-randomness) Just having a pool and a hash function is not enough. Not even close. Constructive suggestion: If you want to see what a RNG looks like when designed by cryptographers, take a look at: Elaine Barker and John Kelsey, ?Recommendation for Random Number Generation Using Deterministic Random Bit Generators? http://csrc.nist.gov/publications/nistpubs/800-90A/SP800-90A.pdf That design may look complicated, but if you think you can leave out some of the blocks in their diagram, proceed with caution. Every one of those blocks is there for a reason. > Randomness should be whitened. Whitening at the input is neither difficult nor necessary nor sufficient. The hard part is obtaining a reliable lower bound on the amount of useful randomness in the bit-blob when it appears at the input. Where did the bits come from? Where did the bound come from? Do you trust the generic openssl user, who knows nothing about cryptology, to provide either one? > The idea of cascading pools is neat. Cascading is absolutely necessary, and must be done "just so", to prevent track-and-hold attacks. One of the weaknesses in the Enigma, exploited to the hilt by Bletchley Park, was that each change in the internal state was too small. A large state space is not sufficient if the state /changes/ are small. On 06/26/2017 10:12 AM, Kurt Roeckx wrote: >> Do you think we need to use multiple sources of randomness? Quality is more important than quantity. N reliable sources is marginally better than N-1 reliable sources. One reliable source is immeasurably better than any number of unreliable sources. Combining many lousy sources and hoping that one of them will do the job is not good engineering practice. From rsalz at akamai.com Mon Jun 26 18:51:52 2017 From: rsalz at akamai.com (Salz, Rich) Date: Mon, 26 Jun 2017 18:51:52 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> Message-ID: <17baa1f8c2f24216856bb252f409f7c9@usma1ex-dag1mb1.msg.corp.akamai.com> > Insofar as any hole that is not explicitly closed should be presumed open, > then yes, there are many holes. Any hole not known -- presumed open, or presumed closed? :) It's more about > What's your threat model? I know that sounds like a clich?, but it's actually > important. We're trying to do a better job than the current stuff. We want to provide a good cross-platform API that uses the operating system when it's worth doing so. There are lots of vague words in that. Deliberately. > -- If you trust the ambient OS to provide a seed, why not > trust it for everything, and not bother to implement an > openssl-specfic RNG at all? Because the ambient OS isn't one, but is one of many possibilities. A good kernel getrandom might exist, a good /dev/random might exist, either might exists but be bad, and one might exist and be good. We want to assume the minimum set of O/S facilities, and try to develop code that has the minimum set of ifdef's or run-time code paths. Again, there are vague words there. But code that's perfect but impenetrable does not help our user base. > -- Conversely, if you don't trust the OS, what makes you > think you can solve a problem the OS failed to solve, > especially without knowing why it failed? > > And (!) what do you propose to do when a suitable seed is not available at > the moment but might be available later? Leave it up to the application to decide. It would be nice to say "fix the platform" but we run on places where that is not possible. Right now I believe (and this entire note is my opinion) that the best we can realistically do is make it possible for portable safe applications to do the right thing. > Are you designing to resist an output-text-only attack? Or do you also want > "some" ability to recover from a compromise of the PRNG internal state? Our TCB border is the process. > Is there state in a persistent file, or only in memory? Only memory. > Just having a pool and a hash function is not enough. Not even close. I wasn't suggesting just those. I was saying that's the pool, and then something like chacha. > Constructive suggestion: If you want to see what a RNG looks like when > designed by cryptographers, take a look at: > Elaine Barker and John Kelsey, > ?Recommendation for Random Number Generation Using Deterministic > Random Bit Generators? > http://csrc.nist.gov/publications/nistpubs/800-90A/SP800-90A.pdf > > That design may look complicated, but if you think you can leave out some of > the blocks in their diagram, proceed with caution. Every one of those blocks > is there for a reason. Well maybe I can ignore section 10.3? > Do you trust the generic > openssl user, who knows nothing about cryptology, to provide either one? No, which is why we'll have a mixin-from-system API that will hopefully do the right thing. Ideally on all places where openssl runs. > Combining many lousy sources and hoping that one of them will do the job is > not good engineering practice. But if there are a couple and they're both mediocre? From rsalz at akamai.com Mon Jun 26 18:57:19 2017 From: rsalz at akamai.com (Salz, Rich) Date: Mon, 26 Jun 2017 18:57:19 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <17baa1f8c2f24216856bb252f409f7c9@usma1ex-dag1mb1.msg.corp.akamai.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <17baa1f8c2f24216856bb252f409f7c9@usma1ex-dag1mb1.msg.corp.akamai.com> Message-ID: <16e37085646c43fda780c299ea3d5529@usma1ex-dag1mb1.msg.corp.akamai.com> I was asked off-list why we're doing this. A reasonable question. :) There are many complains about the OpenSSL RNG. For started: https://github.com/openssl/openssl/issues/2168 https://github.com/openssl/openssl/issues/898 https://github.com/openssl/openssl/issues/2457 https://github.com/openssl/openssl/issues/3125 Also, there's things like this: It uses MD5 It has a global pool, not per-thread so there's locking It doesn't use getrandom available on modern Linux systems It uses other bizarre private hashing and mixes in time and getpid To summarize, perhaps, let's just say that it is really really outdated. The state of the art has advanced, and we have some catching-up to do. From ssx at av8n.com Mon Jun 26 19:29:51 2017 From: ssx at av8n.com (John Denker) Date: Mon, 26 Jun 2017 12:29:51 -0700 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <17baa1f8c2f24216856bb252f409f7c9@usma1ex-dag1mb1.msg.corp.akamai.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <17baa1f8c2f24216856bb252f409f7c9@usma1ex-dag1mb1.msg.corp.akamai.com> Message-ID: <8da8e30a-1b71-6565-072f-580d09b9ebe3@av8n.com> On 06/26/2017 11:51 AM, Salz, Rich wrote: >> Combining many lousy sources and hoping that one of them will do >> the job is not good engineering practice. > But if there are a couple and they're both mediocre? There are multiple dimensions to be considered, including reliability and rate. As for reliability, I don't know what "mediocre" means. Usually security-critical code is correct or it's not. For a seed-source, either a lower bound on the amount of good "hard" randomness is available and reliable, or it's not. As for rate, seeding a PRNG places rather mild demands on the source. I'm having trouble imagining a reasonable source that has a guaranteed lower bound on the rate that is nonzero yet too small for the purpose. By "reasonable" I mean to exclude things that were designed to be facetious or perverse counterexamples. Can somebody point to a specific example of a "mediocre" source? > the ambient OS isn't one, but is one of many possibilities. That's moving the outer loop to the inside, for no good reason. I suggest asking the hard questions on a per-OS basis: -- If you trust this particular OS to provide a seed, why not trust it for everything, and not bother to implement an openssl-specfic RNG at all? -- Conversely, if you don't trust this particular OS, what makes you think you can solve a problem the OS failed to solve, especially without knowing why it failed? You can then write an outer loop over all OS colors and flavors. If the questions are unanswerable for each individual OS, it seems both impossible and pointless to try to answer them for all OSs at once. > To summarize, perhaps, let's just say that it is really really > outdated. The state of the art has advanced, and we have some > catching-up to do. The standard advice that you see on e.g. the crypto list is to use whatever the OS provides. It's unlikely you can do better ... certainly not without making a treeeeemendous multi-year R&D project out of it. In particular, if the ambient environment is not secure, it is very unlikely that anything openssl can do will make it secure. If what the OS provides isn't good enough, you should file bug reports against it. From rsalz at akamai.com Mon Jun 26 19:41:27 2017 From: rsalz at akamai.com (Salz, Rich) Date: Mon, 26 Jun 2017 19:41:27 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <8da8e30a-1b71-6565-072f-580d09b9ebe3@av8n.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <17baa1f8c2f24216856bb252f409f7c9@usma1ex-dag1mb1.msg.corp.akamai.com> <8da8e30a-1b71-6565-072f-580d09b9ebe3@av8n.com> Message-ID: <15cb4c15de2a4fecb523939589288b11@usma1ex-dag1mb1.msg.corp.akamai.com> > As for reliability, I don't know what "mediocre" means. Usually security- > critical code is correct or it's not. For a seed-source, either a lower bound on > the amount of good "hard" randomness is available and reliable, or it's not. We run in many environments, and I don't think it's reasonable to say that the RNG on someone's personal web server, perhaps on the Internet, is at the same level of criticality, say, as the same RNG running on something like a global CDN. I am not trying to back out of our responsibilities here, but rather saying that I think a justifiable case can be made for accepting vague words like mediocre at times. > That's moving the outer loop to the inside, for no good reason. That's a good way to put it, but there is a good reason for doing so. What should OpenSSL do when someone says "build for linux" Because pretty much right now that's all they can say. > -- If you trust this particular OS to provide a seed, why not > trust it for everything, and not bother to implement an > openssl-specfic RNG at all? Excellent question. The only answer I have so far is avoiding the syscall overhead when not needed. > If the questions are unanswerable for each individual OS, it seems both > impossible and pointless to try to answer them for all OSs at once. > The standard advice that you see on e.g. the crypto list is to use whatever > the OS provides. So is /dev/random to be used in the same way as getrandom(2)? That's not a rhetorical question. > In particular, if the ambient environment is not secure, it is very unlikely that > anything openssl can do will make it secure. Suppose the chip supports RDRAND but the runtime doesn't have getrandom or /dev/random? > If what the OS provides isn't good enough, you should file bug reports > against it. It's not us, it's the people using it. We don't have the wherewithal or knowledge to do so. From kurt at roeckx.be Mon Jun 26 20:18:39 2017 From: kurt at roeckx.be (Kurt Roeckx) Date: Mon, 26 Jun 2017 22:18:39 +0200 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> Message-ID: <20170626201839.3e7as7y6sxuic7dk@roeckx.be> On Mon, Jun 26, 2017 at 11:29:19AM -0700, John Denker via openssl-dev wrote: > What's your threat model? I know that sounds like a clich?, > but it's actually important. > > In particular, in my world the #1 threat against any PRNG > is improper seeding. > -- If you trust the ambient OS to provide a seed, why not > trust it for everything, and not bother to implement an > openssl-specfic RNG at all? As I understand it, the main reason to provide our own one is speed. Related to it, it might not be able to provide the amount of random data that we would like to use. I see no other reasons not to use the OS provided one. > And (!) what do you propose to do when a suitable seed is not > available at the moment but might be available later? The current API returns an error, but it would be better that it would block until it's available. > Are you designing to resist an output-text-only attack? Or do > you also want "some" ability to recover from a compromise of > the PRNG internal state? I think at least some people seem to want to deal with a compromised internal state, but I have to wonder how it would get compromised in the first place and if you can really fix it. > Is there state in a persistent file, or only in memory? We currently have a ~/.rnd file. We should probably get rid of it. > ?Recommendation for Random Number Generation Using Deterministic Random Bit Generators? > http://csrc.nist.gov/publications/nistpubs/800-90A/SP800-90A.pdf > > That design may look complicated, but if you think you can > leave out some of the blocks in their diagram, proceed with > caution. Every one of those blocks is there for a reason. SP800-90A (or revision 1) can clearly be used as reference on how to implement it, even if we don't use an approved algorithm from it. And I really think we should look at that document when implementing it. There should probably also be an option to use an RNG that conforms to it. > > Randomness should be whitened. > > Whitening at the input is neither difficult nor necessary nor sufficient. > The hard part is obtaining a reliable lower bound on the amount of > useful randomness in the bit-blob when it appears at the input. Where > did the bits come from? Where did the bound come from? Do you trust > the generic openssl user, who knows nothing about cryptology, to provide > either one? I think it should by default be provided by the OS, and I don't think any OS is documenting how much randomness it can provide. > >> Do you think we need to use multiple sources of randomness? > > Quality is more important than quantity. > > N reliable sources is marginally better than N-1 reliable sources. > One reliable source is immeasurably better than any number of > unreliable sources. > > Combining many lousy sources and hoping that one of them will do > the job is not good engineering practice. The OS should already be using all the different sources. I don't think OpenSSL also using those sources is adding anything, I think it will only make the code harder for no good reason. Kurt From ssx at av8n.com Mon Jun 26 20:18:58 2017 From: ssx at av8n.com (John Denker) Date: Mon, 26 Jun 2017 13:18:58 -0700 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <15cb4c15de2a4fecb523939589288b11@usma1ex-dag1mb1.msg.corp.akamai.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <17baa1f8c2f24216856bb252f409f7c9@usma1ex-dag1mb1.msg.corp.akamai.com> <8da8e30a-1b71-6565-072f-580d09b9ebe3@av8n.com> <15cb4c15de2a4fecb523939589288b11@usma1ex-dag1mb1.msg.corp.akamai.com> Message-ID: <155ffdb5-891e-08b2-8c1f-aadb3b18acbb@av8n.com> On 06/26/2017 12:41 PM, Salz, Rich wrote: > Suppose the chip supports RDRAND but the runtime doesn't have > getrandom or /dev/random? That's an easy one! Check the feature-test bit and then call RDRAND yourself. Code to do this exists, e.g. https://en.wikipedia.org/wiki/RdRand#Sample_x86_asm_code_to_check_upon_RDRAND_instruction A version of that for 64-bit architecture exists somewhere, too. From rsalz at akamai.com Mon Jun 26 20:20:38 2017 From: rsalz at akamai.com (Salz, Rich) Date: Mon, 26 Jun 2017 20:20:38 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <155ffdb5-891e-08b2-8c1f-aadb3b18acbb@av8n.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <17baa1f8c2f24216856bb252f409f7c9@usma1ex-dag1mb1.msg.corp.akamai.com> <8da8e30a-1b71-6565-072f-580d09b9ebe3@av8n.com> <15cb4c15de2a4fecb523939589288b11@usma1ex-dag1mb1.msg.corp.akamai.com> <155ffdb5-891e-08b2-8c1f-aadb3b18acbb@av8n.com> Message-ID: <72ffae5191ba4b388de71e887960bbc6@usma1ex-dag1mb1.msg.corp.akamai.com> > > Suppose the chip supports RDRAND but the runtime doesn't have > > getrandom or /dev/random? > > That's an easy one! > > Check the feature-test bit and then call RDRAND yourself. > Code to do this exists, e.g. Yeah, that's kinda what I meant we'd do. From kurt at roeckx.be Mon Jun 26 20:24:40 2017 From: kurt at roeckx.be (Kurt Roeckx) Date: Mon, 26 Jun 2017 22:24:40 +0200 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <155ffdb5-891e-08b2-8c1f-aadb3b18acbb@av8n.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <17baa1f8c2f24216856bb252f409f7c9@usma1ex-dag1mb1.msg.corp.akamai.com> <8da8e30a-1b71-6565-072f-580d09b9ebe3@av8n.com> <15cb4c15de2a4fecb523939589288b11@usma1ex-dag1mb1.msg.corp.akamai.com> <155ffdb5-891e-08b2-8c1f-aadb3b18acbb@av8n.com> Message-ID: <20170626202440.z2yu5pcjo45rjjbv@roeckx.be> On Mon, Jun 26, 2017 at 01:18:58PM -0700, John Denker via openssl-dev wrote: > On 06/26/2017 12:41 PM, Salz, Rich wrote: > > > Suppose the chip supports RDRAND but the runtime doesn't have > > getrandom or /dev/random? > > That's an easy one! > > Check the feature-test bit and then call RDRAND yourself. > Code to do this exists, e.g. > https://en.wikipedia.org/wiki/RdRand#Sample_x86_asm_code_to_check_upon_RDRAND_instruction > > A version of that for 64-bit architecture exists somewhere, too. We already have code to detect it. Kurt From ssx at av8n.com Tue Jun 27 04:17:22 2017 From: ssx at av8n.com (John Denker) Date: Mon, 26 Jun 2017 21:17:22 -0700 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <15cb4c15de2a4fecb523939589288b11@usma1ex-dag1mb1.msg.corp.akamai.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <17baa1f8c2f24216856bb252f409f7c9@usma1ex-dag1mb1.msg.corp.akamai.com> <8da8e30a-1b71-6565-072f-580d09b9ebe3@av8n.com> <15cb4c15de2a4fecb523939589288b11@usma1ex-dag1mb1.msg.corp.akamai.com> Message-ID: <08d9fd24-857b-4943-d91a-71c91c2cc728@av8n.com> On 06/26/2017 12:41 PM, Salz, Rich wrote: > We run in many environments, and I don't think it's reasonable to say > that the RNG on someone's personal web server, perhaps on the > Internet, is at the same level of criticality, say, as the same RNG > running on something like a global CDN. I am not trying to back out > of our responsibilities here, but rather saying that I think a > justifiable case can be made for accepting vague words like mediocre > at times. That argument cuts the other way, much more acutely. When writing a low- to mid-level library such as openssl, the problem is you *don't know* how it will be used. If you design a RNG that is good enough for a game of Go Fish, it is entirely possible that some user will turn around and use the same RNG to sign a multi-million dollar contract, or encrypt some life-and-death critical messages. The days when we could get away with mediocre security are gone, and have been for quite a while now. The idea of "provably correct" code has been around for decades now. I don't always succeed, but I try to write provably correct code, even for things that are vastly less critical than a cryptographic RNG. In particular, the idea of combining several lousy upstream sources and hoping for the best is 100% virgin serpentoleum. It violates every engineering principle known to man, except for Murphy's law. The fact that RNGs are hard to test makes it easy to fool your friends. Your enemies will not be so easily fooled. This just makes it extra-super-important to insist on sound engineering practices, top to bottom. From paul.dale at oracle.com Tue Jun 27 04:28:01 2017 From: paul.dale at oracle.com (Paul Dale) Date: Mon, 26 Jun 2017 21:28:01 -0700 (PDT) Subject: [openssl-dev] Work on a new RNG for OpenSSL Message-ID: <64072fc1-4411-4b3f-88cf-296ea19753cf@default> It looks like there are two different questions being addressed: * How should OpenSSL architect the random number generators? * How should these be seeded? I agree with John that the larger threat comes from poor seeding, but both have to be considered. A weak architecture will be just as bad from a security standpoint. The pull request referenced was about the former. It seems like the consensus is leaning toward cascaded RNGs with reseeding. There are open questions about which algorithm(s) to use, where to place the RNG nodes in the hierarchy and integration with threading. Given the variety of RNGs available, would an EVP RNG interface make sense? With a safe default in place (and no weak generators provided), the decision can be left to the user. A side benefit is that the unit tests could implement a simple fully deterministic generator and the code for producing known sequences removed from the code base. There seems to be more discussion about the seeding. The quality of the seed sources is critical. The problem is how to quantify this. Determining a true lower bound for an entropy source is a difficult problem. Turbid does this, I'm not aware of any other daemons that do and precious few HRNGs that attempt it (let alone prove it). Estimates of the minimum entropy are available in NIST's SP800-90B draft. These aren't ideal and can easily be tricked but they do represent a reputable starting point. As I mentioned, they appear to be well designed with sound statistics backing them. Defence in depth seems prudent: independent sources with agglomeration and whitening. We shouldn't trust the user to provide entropy. I've seen what is typically provided. Uninitialised buffers aren't random. User inputs (mouse and keyboard) likewise aren't very good. That both are still being suggested is frustrating. I've seen worse suggestions, some to the effect that "time(NULL) ^ getpid()" is too good and just time() is enough. As for specific questions and comments: John Denker wrote: > If you trust the ambient OS to provide a seed, why not > trust it for everything, and not bother to implement an > openssl-specfic RNG at all? I can think of a few possibilities: * Diversifying the sources provides resistance to compromise of individual sources. Although a full kernel compromise is unrecoverable, a kernel bug that leaked the internal pools in a read only manner isn't unforeseeable. * Not all operating systems have good RNGs. * Draining the kernel's entropy pools is unfriendly behaviour, other processes will typically want some randomness too. * At boot time the kernel pools are empty (low or no quality). This compounds when several things require seeding. * Performance is also a consideration, although with a gradual collection strategy this should be less of a concern. Except at start up. John Denker wrote: > Are you designing to resist an output-text-only attack? Or do you > also want "some" ability to recover from a compromise of > the PRNG internal state? Yes to both ideally. Feeding additional seed material into a PRNG could help in the latter case. It depends on how easy it is to compromise the PRNG state. If it is trivial in terms of resources, it isn't going to be recoverable. A short reseed interval can be effective against a slow attack (but not always). John Denker wrote: > Is there state in a persistent file, or only in memory? Rich Salz replied: > Only memory. I'd propose secure memory. Leaking RNG state is very bad. John Denker wrote: >> Do you think we need to use multiple sources of randomness? > Quality is more important than quantity. Given the difficulty of quantifying the quality of sources, I think that having multiple sources is prudent. Finding out that a source is low quality when it was believed to be good is less embarrassing when you've other sources in operation. Likewise, if a source fails. I agree that one good source trumps a pile of low quality sources. The aim is to have several good sources. A proven source is better than one that is believed to be good. These can fail (kill -STOP anyone) and a fall-back is not necessarily bad. John Denker wrote: > Cascading is absolutely necessary, and must be done "just so", to > prevent track-and-hold attacks. One of the weaknesses in the > Enigma, exploited to the hilt by Bletchley Park, was that each > change in the internal state was too small. A large state space > is not sufficient if the state /changes/ are small. My suggestion was to XOR the entire state space with new random bits. Some RNGs will have better reseeding procedures and they should be used. I agree completely that small additions aren't useful. Kurt Roeckx wrote: > We currently have a ~/.rnd file. We should probably get rid of it. Perhaps make it optional? Embedded systems have trouble with random state at boot and a ~/.rnd file or equivalent is beneficial here. I've implemented this to seed /dev/random a couple of times now. It isn't ideal but it is better than nothing. Pauli -- Oracle Dr Paul Dale | Cryptographer | Network Security & Encryption Phone +61 7 3031 7217 Oracle Australia From ssx at av8n.com Tue Jun 27 04:39:47 2017 From: ssx at av8n.com (John Denker) Date: Mon, 26 Jun 2017 21:39:47 -0700 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <17baa1f8c2f24216856bb252f409f7c9@usma1ex-dag1mb1.msg.corp.akamai.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <17baa1f8c2f24216856bb252f409f7c9@usma1ex-dag1mb1.msg.corp.akamai.com> Message-ID: <871f8027-630c-fdbd-316c-0583ce857307@av8n.com> In the context of >> What's your threat model? >> Are you designing to resist an output-text-only attack? Or do you also want >> "some" ability to recover from a compromise of the PRNG internal state? On 06/26/2017 11:51 AM, Salz, Rich wrote: > Our TCB border is the process. That doesn't answer the question. > then something like chacha. That doesn't answer the question either. ============ I'm not mentioning any names, but some people are *unduly* worried about recovery following compromise of the PRNG internal state, so they constantly re-seed the PRNG, to the point where it becomes a denial-of-service attack against the upstream source of randomness. This is also mostly pointless, because any attack that compromises the PRNG state will likely compromise so many other things that recovery will be very difficult. All future outputs will be suspect. So please let's not go overboard in that direction. On the other hand, it seems reasonable to insist on /forward/ secrecy. That is, we should insist that /previous/ outputs should not be compromised. This is achievable at small but not-quite-zero cost. Specifically, ChaCha (or any other cipher) in counter mode does *not* provide forward secrecy when the PRNG state is compromised. From matt at openssl.org Tue Jun 27 07:20:37 2017 From: matt at openssl.org (Matt Caswell) Date: Tue, 27 Jun 2017 08:20:37 +0100 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <16e37085646c43fda780c299ea3d5529@usma1ex-dag1mb1.msg.corp.akamai.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <17baa1f8c2f24216856bb252f409f7c9@usma1ex-dag1mb1.msg.corp.akamai.com> <16e37085646c43fda780c299ea3d5529@usma1ex-dag1mb1.msg.corp.akamai.com> Message-ID: On 26/06/17 19:57, Salz, Rich via openssl-dev wrote: > I was asked off-list why we're doing this. A reasonable question. :) > > There are many complains about the OpenSSL RNG. For started: > https://github.com/openssl/openssl/issues/2168 > https://github.com/openssl/openssl/issues/898 > https://github.com/openssl/openssl/issues/2457 > https://github.com/openssl/openssl/issues/3125 > > Also, there's things like this: > It uses MD5 Correction: it uses SHA1. Matt From matt at openssl.org Tue Jun 27 07:28:56 2017 From: matt at openssl.org (Matt Caswell) Date: Tue, 27 Jun 2017 08:28:56 +0100 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <20170626201839.3e7as7y6sxuic7dk@roeckx.be> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <20170626201839.3e7as7y6sxuic7dk@roeckx.be> Message-ID: <97aef447-9dfd-5479-c4df-08d2c62ea8f1@openssl.org> On 26/06/17 21:18, Kurt Roeckx wrote: >> ?Recommendation for Random Number Generation Using Deterministic Random Bit Generators? >> http://csrc.nist.gov/publications/nistpubs/800-90A/SP800-90A.pdf >> >> That design may look complicated, but if you think you can >> leave out some of the blocks in their diagram, proceed with >> caution. Every one of those blocks is there for a reason. > > SP800-90A (or revision 1) can clearly be used as reference on how > to implement it, even if we don't use an approved algorithm from > it. And I really think we should look at that document when > implementing it. > > There should probably also be an option to use an RNG that > conforms to it. I am strongly in favour of this approach. We should be led by standards. > >>> Randomness should be whitened. >> >> Whitening at the input is neither difficult nor necessary nor sufficient. >> The hard part is obtaining a reliable lower bound on the amount of >> useful randomness in the bit-blob when it appears at the input. Where >> did the bits come from? Where did the bound come from? Do you trust >> the generic openssl user, who knows nothing about cryptology, to provide >> either one? > > I think it should by default be provided by the OS, and I don't > think any OS is documenting how much randomness it can provide. > I also agree that, by default, using the OS provided source makes a lot of sense. Matt From gilles.vanassche at st.com Tue Jun 27 11:54:12 2017 From: gilles.vanassche at st.com (Gilles Van Assche) Date: Tue, 27 Jun 2017 13:54:12 +0200 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> Message-ID: <59524764.5080701@st.com> > We welcome your input. On this topic, I would like to point out the construction we presented at SAC 2011 [1]. It implements a reseedable pseudo-random number generator in a rather simple way. One can supply seeds, extract pseudo-random numbers and apply a ratchet mechanism at any chosen time. We implemented it in the Keccak code package. The documentation can be found there [2]. Kind regards, Gilles, for the Keccak team [1] http://eprint.iacr.org/2011/499.pdf [2] https://github.com/gvanas/KeccakCodePackage/blob/master/Modes/KeccakPRG.h From rsalz at akamai.com Tue Jun 27 12:46:11 2017 From: rsalz at akamai.com (Salz, Rich) Date: Tue, 27 Jun 2017 12:46:11 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <08d9fd24-857b-4943-d91a-71c91c2cc728@av8n.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <17baa1f8c2f24216856bb252f409f7c9@usma1ex-dag1mb1.msg.corp.akamai.com> <8da8e30a-1b71-6565-072f-580d09b9ebe3@av8n.com> <15cb4c15de2a4fecb523939589288b11@usma1ex-dag1mb1.msg.corp.akamai.com> <08d9fd24-857b-4943-d91a-71c91c2cc728@av8n.com> Message-ID: <401541e01743473ca0fe78f12b79f867@usma1ex-dag1mb1.msg.corp.akamai.com> John, Thanks for pushing. It can be a thankless task, and I wanted to say thank you :) From Matthias.St.Pierre at ncp-e.com Tue Jun 27 12:42:52 2017 From: Matthias.St.Pierre at ncp-e.com (Matthias St. Pierre) Date: Tue, 27 Jun 2017 14:42:52 +0200 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <17baa1f8c2f24216856bb252f409f7c9@usma1ex-dag1mb1.msg.corp.akamai.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <17baa1f8c2f24216856bb252f409f7c9@usma1ex-dag1mb1.msg.corp.akamai.com> Message-ID: <212d9cd2-3aa8-8917-dcdd-29a28449e322@ncp-e.com> On 26.06.2017 20:51, Salz, Rich via openssl-dev wrote: > >> Constructive suggestion: If you want to see what a RNG looks like when >> designed by cryptographers, take a look at: >> Elaine Barker and John Kelsey, >> ?Recommendation for Random Number Generation Using Deterministic >> Random Bit Generators? >> http://csrc.nist.gov/publications/nistpubs/800-90A/SP800-90A.pdf >> >> That design may look complicated, but if you think you can leave out some of >> the blocks in their diagram, proceed with caution. Every one of those blocks >> is there for a reason. > Well maybe I can ignore section 10.3? > That's a nice joke Rich, but the Dual_EC_DRBG chapter has been dropped in SP800-90Ar1, which supersedes SP800-90A: http://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-90Ar1.pdf But seriously: OpenSSL already has an implementation of the SP800-90A DRBG, but unfortunately it is only part of the FIPS object module (see reference [1] below). I always wondered why the code was never migrated to OpenSSL master, (say, replacing the FIPS_drbg_* names by e.g. RAND_drbg_*). Then the SP800-90A DRBG would be usable by everyone and could be activated by RAND_set_rand_method(RAND_drbg_method()); To me, the design and implementation of the DRBG appeals sophisticated and I like its concept for reseeding which is highly configurable using FIPS_drbg_set_reseed_interval() and FIPS_drbg_set_callbacks() In fact, we are currently using the AES-CTR DRBG in our product (see [2]) because we had the requirement that the random generator should be seeded periodically from an external entropy source, for example a smart card or a cryptographic acceleration unit. This was easily achieved using the aforementioned DRBG callback mechanism. So I have two questions: - Do you intend to continue supporting RAND_set_rand_method() or will there only be one 'perfect' random generator and no choice anymore? - Do you consider the SP800-90A DRBG outdated or will there be a chance that it will be added to the OpenSSL master as officially supported RAND method? - Will the new OpenSSL RNG support a way to configure reseed intervals and external entropy sources in a similar fashion as the FIPS DRBG did? Best regards, Matthias St. Pierre [1] Section 6.1 of the OpenSSL FIPS User Guide 2.0 https://www.openssl.org/docs/fips/UserGuide-2.0.pdf [2] We link against a FIPS capable OpenSSL 1.0.2 crypto library and use the FIPS DRBG even in the case where FIPS mode is not enabled globally: In that case, during initialization we check whether FIPS mode initialization is successfull, then and then turn FIPS mode off again and only keep the random generator by calling RAND_set_rand_method(FIPS_drbg_method()). For Windows, we had to add some FIPS_drbg_* symbols to libeay.num to make this work. From rsalz at akamai.com Tue Jun 27 12:55:14 2017 From: rsalz at akamai.com (Salz, Rich) Date: Tue, 27 Jun 2017 12:55:14 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <212d9cd2-3aa8-8917-dcdd-29a28449e322@ncp-e.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <17baa1f8c2f24216856bb252f409f7c9@usma1ex-dag1mb1.msg.corp.akamai.com> <212d9cd2-3aa8-8917-dcdd-29a28449e322@ncp-e.com> Message-ID: > > Well maybe I can ignore section 10.3? > > > > That's a nice joke Rich, but the Dual_EC_DRBG chapter has been dropped in > SP800-90Ar1, which supersedes SP800-90A: I know. I was trying to gently point out that even John makes mistakes :) > - Do you intend to continue supporting RAND_set_rand_method() or will > there only be one 'perfect' random generator and no choice anymore? This will continue to work. > - Do you consider the SP800-90A DRBG outdated or will there be a chance > that it will be added to the OpenSSL master as > officially supported RAND method? That's a great idea, I can work on that now. > - Will the new OpenSSL RNG support a way to configure reseed intervals and > external entropy sources in a similar fashion > as the FIPS DRBG did? That's three questions :) But yes, we should address that. I'm not sure if new RAND API's are the way to go or perhaps a RAND_control API that gives us a bit more flexibility. From Matthias.St.Pierre at ncp-e.com Tue Jun 27 12:59:28 2017 From: Matthias.St.Pierre at ncp-e.com (Matthias St. Pierre) Date: Tue, 27 Jun 2017 14:59:28 +0200 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <17baa1f8c2f24216856bb252f409f7c9@usma1ex-dag1mb1.msg.corp.akamai.com> <212d9cd2-3aa8-8917-dcdd-29a28449e322@ncp-e.com> Message-ID: On 27.06.2017 14:55, Salz, Rich via openssl-dev wrote: > That's three questions :) But yes, we should address that. I'm not sure if new RAND API's are the way to go or perhaps a RAND_control API that gives us a bit more flexibility. Ups, it used to be only two. That's always the problem with last minute changes ;-). Thanks for your quick reply and for considering my suggestions. Regards, Matthias From kurt at roeckx.be Tue Jun 27 16:41:41 2017 From: kurt at roeckx.be (Kurt Roeckx) Date: Tue, 27 Jun 2017 18:41:41 +0200 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <871f8027-630c-fdbd-316c-0583ce857307@av8n.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <17baa1f8c2f24216856bb252f409f7c9@usma1ex-dag1mb1.msg.corp.akamai.com> <871f8027-630c-fdbd-316c-0583ce857307@av8n.com> Message-ID: <20170627164141.oxtfy7ndva3hy3ao@roeckx.be> On Mon, Jun 26, 2017 at 09:39:47PM -0700, John Denker via openssl-dev wrote: > > I'm not mentioning any names, but some people are *unduly* > worried about recovery following compromise of the PRNG > internal state, so they constantly re-seed the PRNG, to > the point where it becomes a denial-of-service attack > against the upstream source of randomness. > > This is also mostly pointless, because any attack that > compromises the PRNG state will likely compromise so many > other things that recovery will be very difficult. All > future outputs will be suspect. > > So please let's not go overboard in that direction. > > On the other hand, it seems reasonable to insist on /forward/ > secrecy. That is, we should insist that /previous/ outputs > should not be compromised. This is achievable at small but > not-quite-zero cost. I think that's named backward secrecy? Kurt From kurt at roeckx.be Tue Jun 27 16:47:04 2017 From: kurt at roeckx.be (Kurt Roeckx) Date: Tue, 27 Jun 2017 18:47:04 +0200 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <212d9cd2-3aa8-8917-dcdd-29a28449e322@ncp-e.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <17baa1f8c2f24216856bb252f409f7c9@usma1ex-dag1mb1.msg.corp.akamai.com> <212d9cd2-3aa8-8917-dcdd-29a28449e322@ncp-e.com> Message-ID: <20170627164703.inv5d37sxfeotnjj@roeckx.be> On Tue, Jun 27, 2017 at 02:42:52PM +0200, Matthias St. Pierre wrote: > > So I have two questions: > > - Do you intend to continue supporting RAND_set_rand_method() or will there only be one 'perfect' random generator and no choice anymore? I think we should have a default one, but an option to have a different one. > - Do you consider the SP800-90A DRBG outdated or will there be a chance that it will be added to the OpenSSL master as > officially supported RAND method? I think we should have at least 1 that follows SP800-90A, it's clearly something some people will need. > - Will the new OpenSSL RNG support a way to configure reseed intervals and external entropy sources in a similar fashion > as the FIPS DRBG did? It should at least reseed by default. Having an option to change the default interval might make sense. There clearly should be a way to use a source other than the one provided by the kernel, which I think it needed for SP800-90A. Kurt From bkaduk at akamai.com Tue Jun 27 18:50:11 2017 From: bkaduk at akamai.com (Benjamin Kaduk) Date: Tue, 27 Jun 2017 13:50:11 -0500 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <97aef447-9dfd-5479-c4df-08d2c62ea8f1@openssl.org> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <20170626201839.3e7as7y6sxuic7dk@roeckx.be> <97aef447-9dfd-5479-c4df-08d2c62ea8f1@openssl.org> Message-ID: On 06/27/2017 02:28 AM, Matt Caswell wrote: > > On 26/06/17 21:18, Kurt Roeckx wrote: > >> I think it should by default be provided by the OS, and I don't >> think any OS is documenting how much randomness it can provide. >> > I also agree that, by default, using the OS provided source makes a lot > of sense. > Do you mean having openssl just pass through to getrandom()/read()-from-'/dev/random'/etc. or just using those to seed our own thing? The former seems simpler and preferable to me (perhaps modulo linux's broken idea about "running out of entropy"), but the argument presented about us being used in all sorts of environments that we can't even enumerate has basically convinced me that we will need to provide some alternative as well. (It remains unclear how such environments will be able to provide usable seed randomness, but there is only so much we can do about that.) -Ben -------------- next part -------------- An HTML attachment was scrubbed... URL: From rsalz at akamai.com Tue Jun 27 18:55:47 2017 From: rsalz at akamai.com (Salz, Rich) Date: Tue, 27 Jun 2017 18:55:47 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <20170626201839.3e7as7y6sxuic7dk@roeckx.be> <97aef447-9dfd-5479-c4df-08d2c62ea8f1@openssl.org> Message-ID: For windows RAND_bytes should just call CryptGenRandom (or its equiv). For modern Linux, probably call getrandom(2). For OpenBSD call arc4random(). Getrandom() is a syscall, and I have concerns about the syscall performance. I would rather feed getrandom (or /dev/random if that?s not available) into a FIPS DRBG generator. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ssx at av8n.com Tue Jun 27 18:56:04 2017 From: ssx at av8n.com (John Denker) Date: Tue, 27 Jun 2017 11:56:04 -0700 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <20170626201839.3e7as7y6sxuic7dk@roeckx.be> <97aef447-9dfd-5479-c4df-08d2c62ea8f1@openssl.org> Message-ID: <60e14ee0-9992-6832-61bc-ebe002bf1dde@av8n.com> On 06/27/2017 02:28 AM, Matt Caswell wrote: >> >> I also agree that, by default, using the OS provided source makes a lot >> of sense. Reality is more complicated than that; see below. On 06/27/2017 11:50 AM, Benjamin Kaduk via openssl-dev wrote: > Do you mean having openssl just pass through to > getrandom()/read()-from-'/dev/random'/etc. or just using those to seed > our own thing? > > The former seems simpler and preferable to me (perhaps modulo linux's > broken idea about "running out of entropy") That's a pretty big modulus. As I wrote over on the crypto list: The xenial 16.04 LTS manpage for getrandom(2) says quite explicitly: >> Unnecessarily reading large quantities of data will have a >> negative impact on other users of the /dev/random and /dev/urandom >> devices. And that's an understatement. Whether unnecessary or not, reading not-particularly-large quantities of data is tantamount to a denial of service attack against /dev/random and against its upstream sources of randomness. No later LTS is available. Reference: http://manpages.ubuntu.com/manpages/xenial/man2/getrandom.2.html Recently there has been some progress on this, as reflected in in the zesty 17.04 manpage: http://manpages.ubuntu.com/manpages/zesty/man2/getrandom.2.html However, in the meantime openssl needs to run on the platforms that are out there, which includes a very wide range of platforms. It could be argued that the best *long-term* strategy is to file a flurry of bug reports against the various kernel RNGs, and then at some *later* date rely on whatever the kernel provides ... but still, in the meantime openssl needs to run on the platforms that are out there. From bhat.jayalakshmi at gmail.com Tue Jun 27 19:00:19 2017 From: bhat.jayalakshmi at gmail.com (Jayalakshmi bhat) Date: Wed, 28 Jun 2017 00:30:19 +0530 Subject: [openssl-dev] OpenSSL FIPS CAVP tests throws an error iob_func while linking Message-ID: Hi All, I am trying to build CAVP test executable for WinCE. Most of the executable are built except 1-2. I am facing iob_func unresolved error. Every thing seems to be proper. Any idea or help is well appreciated. Regards Jaya -------------- next part -------------- An HTML attachment was scrubbed... URL: From rsalz at akamai.com Tue Jun 27 21:02:54 2017 From: rsalz at akamai.com (Salz, Rich) Date: Tue, 27 Jun 2017 21:02:54 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <20170627204041.h2sr3zovsnutj43y@thunk.org> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <20170626201839.3e7as7y6sxuic7dk@roeckx.be> <97aef447-9dfd-5479-c4df-08d2c62ea8f1@openssl.org> <20170627204041.h2sr3zovsnutj43y@thunk.org> Message-ID: > > Getrandom() is a syscall, and I have concerns about the syscall > > performance. I would rather feed getrandom (or /dev/random if that?s > > not available) into a FIPS DRBG generator. > > What is your concerns about syscall performance? What are your > performance requirements? I can tell you that Chrome has been using > /dev/urandom Well, Chrome ultimately works at human-scale. On the server side, thousands of connections per second and one or two syscalls per connection seems like something we should avoid. > My recommendation for Linux is to use getrandom(2) the flags field set to > zero. And for older Linux? > So if you are going to be trying to design your own RNG > for OpenSSL --- welcome to my world. We seem to have moved away from that somewhat. That's a better place to be. > find that in the end, it's impossible to make them all happy, and they will end > up questioning your intelligence, judgement, and in some cases, your > paternity. :-) I miss Usenet. :) From tytso at mit.edu Tue Jun 27 20:40:41 2017 From: tytso at mit.edu (Theodore Ts'o) Date: Tue, 27 Jun 2017 16:40:41 -0400 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <20170626201839.3e7as7y6sxuic7dk@roeckx.be> <97aef447-9dfd-5479-c4df-08d2c62ea8f1@openssl.org> Message-ID: <20170627204041.h2sr3zovsnutj43y@thunk.org> On Tue, Jun 27, 2017 at 06:55:47PM +0000, Salz, Rich via openssl-dev wrote: > Getrandom() is a syscall, and I have concerns about the syscall > performance. I would rather feed getrandom (or /dev/random if > that?s not available) into a FIPS DRBG generator. What is your concerns about syscall performance? What are your performance requirements? I can tell you that Chrome has been using /dev/urandom (which has the same performance characteristics as the getrandom system call) directly for all of its random number generation needs (e.g., it's calling each time in dees to generate a session key for TLS, etc.) and no one has complained. My recommendation for Linux is to use getrandom(2) the flags field set to zero. This will cause it to use a CRNG that will be reseeded every five minutes from environmental noise gathered primarily from interrupt timing data. For modern kernels, the CRNG is based on ChaCha20. For older kernels, it is based on SHA-1. There are a lot of people who have complained about whether or not Linux's urandom generator has met with there religious beliefs about how RNG's should be designed and implemented. One of the things you will find is that many of these people are very vocal, and in some cases, their advice will be mutually exclusive. So if you are going to be trying to design your own RNG for OpenSSL --- welcome to my world. (In other words, I do listen to many of the people who have opined on this thread. I just don't happen to agree with all of them. And I suspect you will find that in the end, it's impossible to make them all happy, and they will end up questioning your intelligence, judgement, and in some cases, your paternity. :-) - Ted From bkaduk at akamai.com Tue Jun 27 21:12:48 2017 From: bkaduk at akamai.com (Benjamin Kaduk) Date: Tue, 27 Jun 2017 16:12:48 -0500 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <20170627204041.h2sr3zovsnutj43y@thunk.org> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <20170626201839.3e7as7y6sxuic7dk@roeckx.be> <97aef447-9dfd-5479-c4df-08d2c62ea8f1@openssl.org> <20170627204041.h2sr3zovsnutj43y@thunk.org> Message-ID: Hi Ted, On 06/27/2017 03:40 PM, Theodore Ts'o wrote: > > My recommendation for Linux is to use getrandom(2) the flags field set > to zero. This will cause it to use a CRNG that will be reseeded every > five minutes from environmental noise gathered primarily from > interrupt timing data. For modern kernels, the CRNG is based on > ChaCha20. For older kernels, it is based on SHA-1. > > There are a lot of people who have complained about whether or not > Linux's urandom generator has met with there religious beliefs about > how RNG's should be designed and implemented. One of the things you > will find is that many of these people are very vocal, and in some > cases, their advice will be mutually exclusive. So if you are going > to be trying to design your own RNG for OpenSSL --- welcome to my > world. > > (In other words, I do listen to many of the people who have opined on > this thread. I just don't happen to agree with all of them. And I > suspect you will find that in the end, it's impossible to make them > all happy, and they will end up questioning your intelligence, > judgement, and in some cases, your paternity. :-) > Thanks for the input, and for reading what is being said. While you're here, would you mind confirming/denying the claim I read that the reason the linux /dev/random tracks an entropy estimate and blocks when it gets too low is to preserve backward security in the face of attacks against SHA1? I'm happy to respect that there are different opinions, but it would be nice to know the reasoning behind the behavior, even if I do not necessarily agree with it. Thanks, Ben -------------- next part -------------- An HTML attachment was scrubbed... URL: From rsalz at akamai.com Tue Jun 27 21:20:47 2017 From: rsalz at akamai.com (Salz, Rich) Date: Tue, 27 Jun 2017 21:20:47 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <212d9cd2-3aa8-8917-dcdd-29a28449e322@ncp-e.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <17baa1f8c2f24216856bb252f409f7c9@usma1ex-dag1mb1.msg.corp.akamai.com> <212d9cd2-3aa8-8917-dcdd-29a28449e322@ncp-e.com> Message-ID: <3966d59af8c9417ca9d150327e60f94b@usma1ex-dag1mb1.msg.corp.akamai.com> > RAND_set_rand_method(RAND_drbg_method()); > > FIPS_drbg_set_reseed_interval() and > FIPS_drbg_set_callbacks() Take a look at https://github.com/openssl/openssl/pull/3789 Thanks! From kurt at roeckx.be Tue Jun 27 21:51:23 2017 From: kurt at roeckx.be (Kurt Roeckx) Date: Tue, 27 Jun 2017 23:51:23 +0200 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <60e14ee0-9992-6832-61bc-ebe002bf1dde@av8n.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <20170626201839.3e7as7y6sxuic7dk@roeckx.be> <97aef447-9dfd-5479-c4df-08d2c62ea8f1@openssl.org> <60e14ee0-9992-6832-61bc-ebe002bf1dde@av8n.com> Message-ID: <20170627215123.bpevx4nygbgw3sjq@roeckx.be> On Tue, Jun 27, 2017 at 11:56:04AM -0700, John Denker via openssl-dev wrote: > On 06/27/2017 02:28 AM, Matt Caswell wrote: > >> > >> I also agree that, by default, using the OS provided source makes a lot > >> of sense. > > Reality is more complicated than that; see below. > > On 06/27/2017 11:50 AM, Benjamin Kaduk via openssl-dev wrote: > > > Do you mean having openssl just pass through to > > getrandom()/read()-from-'/dev/random'/etc. or just using those to seed > > our own thing? > > > > The former seems simpler and preferable to me (perhaps modulo linux's > > broken idea about "running out of entropy") > > That's a pretty big modulus. As I wrote over on the crypto list: > > The xenial 16.04 LTS manpage for getrandom(2) says quite explicitly: > > >> Unnecessarily reading large quantities of data will have a > >> negative impact on other users of the /dev/random and /dev/urandom > >> devices. > > And that's an understatement. Whether unnecessary or not, reading > not-particularly-large quantities of data is tantamount to a > denial of service attack against /dev/random and against its > upstream sources of randomness. > > No later LTS is available. Reference: > http://manpages.ubuntu.com/manpages/xenial/man2/getrandom.2.html > > Recently there has been some progress on this, as reflected in in > the zesty 17.04 manpage: > http://manpages.ubuntu.com/manpages/zesty/man2/getrandom.2.html > > However, in the meantime openssl needs to run on the platforms that > are out there, which includes a very wide range of platforms. And I think it's actually because of changes in the Linux RNG that the manpage has been changed, but they did not document the different behavior of the kernel versions. In case it wasn't clear, I think we should use the OS provided source as a seed. By default that should be the only source of randomness. Kurt From bkaduk at akamai.com Tue Jun 27 23:38:15 2017 From: bkaduk at akamai.com (Benjamin Kaduk) Date: Tue, 27 Jun 2017 18:38:15 -0500 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <20170627215123.bpevx4nygbgw3sjq@roeckx.be> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <20170626201839.3e7as7y6sxuic7dk@roeckx.be> <97aef447-9dfd-5479-c4df-08d2c62ea8f1@openssl.org> <60e14ee0-9992-6832-61bc-ebe002bf1dde@av8n.com> <20170627215123.bpevx4nygbgw3sjq@roeckx.be> Message-ID: <638e04b9-467c-3a6a-6921-34092028edd3@akamai.com> On 06/27/2017 04:51 PM, Kurt Roeckx wrote: > On Tue, Jun 27, 2017 at 11:56:04AM -0700, John Denker via openssl-dev wrote: >> >> On 06/27/2017 11:50 AM, Benjamin Kaduk via openssl-dev wrote: >> >>> Do you mean having openssl just pass through to >>> getrandom()/read()-from-'/dev/random'/etc. or just using those to seed >>> our own thing? >>> >>> The former seems simpler and preferable to me (perhaps modulo linux's >>> broken idea about "running out of entropy") >> That's a pretty big modulus. As I wrote over on the crypto list: >> >> The xenial 16.04 LTS manpage for getrandom(2) says quite explicitly: >> >>>> Unnecessarily reading large quantities of data will have a >>>> negative impact on other users of the /dev/random and /dev/urandom >>>> devices. >> And that's an understatement. Whether unnecessary or not, reading >> not-particularly-large quantities of data is tantamount to a >> denial of service attack against /dev/random and against its >> upstream sources of randomness. >> >> No later LTS is available. Reference: >> http://manpages.ubuntu.com/manpages/xenial/man2/getrandom.2.html >> >> Recently there has been some progress on this, as reflected in in >> the zesty 17.04 manpage: >> http://manpages.ubuntu.com/manpages/zesty/man2/getrandom.2.html >> >> However, in the meantime openssl needs to run on the platforms that >> are out there, which includes a very wide range of platforms. > And I think it's actually because of changes in the Linux RNG that > the manpage has been changed, but they did not document the > different behavior of the kernel versions. > > In case it wasn't clear, I think we should use the OS provided > source as a seed. By default that should be the only source of > randomness. > I think we can get away with using OS-provided randomness directly in many common cases. /dev/urandom suffices once we know that the kernel RNG has been properly seeded. On FreeBSD, /dev/urandom blocks until the kernel RNG is seeded; on other systems maybe we have to make one read from /dev/random to get the blocking behavior we want before switching to /dev/urandom for bulk reads. -Ben -------------- next part -------------- An HTML attachment was scrubbed... URL: From rsalz at akamai.com Wed Jun 28 00:07:01 2017 From: rsalz at akamai.com (Salz, Rich) Date: Wed, 28 Jun 2017 00:07:01 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <20170627215123.bpevx4nygbgw3sjq@roeckx.be> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <20170626201839.3e7as7y6sxuic7dk@roeckx.be> <97aef447-9dfd-5479-c4df-08d2c62ea8f1@openssl.org> <60e14ee0-9992-6832-61bc-ebe002bf1dde@av8n.com> <20170627215123.bpevx4nygbgw3sjq@roeckx.be> Message-ID: <194eb363f7e343a586cf714a4e1b594b@usma1ex-dag1mb1.msg.corp.akamai.com> > In case it wasn't clear, I think we should use the OS provided source as a > seed. By default that should be the only source of randomness. I generally agree. But some applications might save their state and restore it during boot time, for example. From rsalz at akamai.com Wed Jun 28 00:11:54 2017 From: rsalz at akamai.com (Salz, Rich) Date: Wed, 28 Jun 2017 00:11:54 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <638e04b9-467c-3a6a-6921-34092028edd3@akamai.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <20170626201839.3e7as7y6sxuic7dk@roeckx.be> <97aef447-9dfd-5479-c4df-08d2c62ea8f1@openssl.org> <60e14ee0-9992-6832-61bc-ebe002bf1dde@av8n.com> <20170627215123.bpevx4nygbgw3sjq@roeckx.be> <638e04b9-467c-3a6a-6921-34092028edd3@akamai.com> Message-ID: <47ed88cf3eda49eab994b79de2412aa1@usma1ex-dag1mb1.msg.corp.akamai.com> > I think we can get away with using OS-provided randomness directly in many common cases.? /dev/urandom suffices once we know that the kernel RNG has been properly seeded.? On FreeBSD, /dev/urandom blocks until the kernel RNG is seeded; on other systems maybe we have to make one read from /dev/random to get the blocking behavior we want before switching to /dev/urandom for bulk reads. It's not a question of "get away with." If the O/S libraries provides random bytes, like CryptGenRandom in windows or arc4random() then we should just wrap those functions and use them by default. If the O/S kernel provides random bytes, then we should use those bytes to seed (and to reseed) for a DRBG generator. We should allow applications to save/restore state, such as on reboot. From paul.dale at oracle.com Wed Jun 28 00:24:13 2017 From: paul.dale at oracle.com (Paul Dale) Date: Tue, 27 Jun 2017 17:24:13 -0700 (PDT) Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <20170626201839.3e7as7y6sxuic7dk@roeckx.be> <97aef447-9dfd-5479-c4df-08d2c62ea8f1@openssl.org> Message-ID: <47de60b0-9429-424a-bae6-ddeecd9fd8f8@default> The hierarchy of RNGs will overcome some of the performance concerns.? Only the root needs to call getrandom(). I do agree that having a DRBG at the root level is a good idea though. ? Pauli -- Oracle Dr Paul Dale | Cryptographer | Network Security & Encryption Phone +61 7 3031 7217 Oracle Australia ? From: Salz, Rich via openssl-dev [mailto:openssl-dev at openssl.org] Sent: Wednesday, 28 June 2017 4:56 AM To: Kaduk, Ben ; openssl-dev at openssl.org; Matt Caswell Subject: Re: [openssl-dev] Work on a new RNG for OpenSSL ? For windows RAND_bytes should just call CryptGenRandom (or its equiv).? For modern Linux, probably call getrandom(2).? For OpenBSD call arc4random(). ? Getrandom() is a syscall, and I have concerns about the syscall performance.? I would rather feed getrandom (or /dev/random if that?s not available) into a FIPS DRBG generator. ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From bkaduk at akamai.com Wed Jun 28 01:22:03 2017 From: bkaduk at akamai.com (Benjamin Kaduk) Date: Tue, 27 Jun 2017 20:22:03 -0500 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <47de60b0-9429-424a-bae6-ddeecd9fd8f8@default> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <20170626201839.3e7as7y6sxuic7dk@roeckx.be> <97aef447-9dfd-5479-c4df-08d2c62ea8f1@openssl.org> <47de60b0-9429-424a-bae6-ddeecd9fd8f8@default> Message-ID: <8b325b45-1b15-d172-fdfa-939668eeca9e@akamai.com> On 06/27/2017 07:24 PM, Paul Dale wrote: > > The hierarchy of RNGs will overcome some of the performance concerns. > Only the root needs to call getrandom(). > > I do agree that having a DRBG at the root level is a good idea though. > > > Just to check my understanding, the claim is that adding more layers of hashing and/or encryption will still be faster than a larger number of syscalls? -Ben -------------- next part -------------- An HTML attachment was scrubbed... URL: From tytso at mit.edu Wed Jun 28 01:22:39 2017 From: tytso at mit.edu (Theodore Ts'o) Date: Tue, 27 Jun 2017 21:22:39 -0400 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: References: <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <20170626201839.3e7as7y6sxuic7dk@roeckx.be> <97aef447-9dfd-5479-c4df-08d2c62ea8f1@openssl.org> <20170627204041.h2sr3zovsnutj43y@thunk.org> Message-ID: <20170628012239.qh365mfpq52465x5@thunk.org> On Tue, Jun 27, 2017 at 09:02:54PM +0000, Salz, Rich wrote: > > What is your concerns about syscall performance? What are your > > performance requirements? I can tell you that Chrome has been using > > /dev/urandom > > Well, Chrome ultimately works at human-scale. On the server side, thousands of connections per second and one or two syscalls per connection seems like something we should avoid. > > > My recommendation for Linux is to use getrandom(2) the flags field set to > > zero. > > And for older Linux? The Chacha20 based CRNG is available since Linux 4.9. So Debian Stable and Ubuntu LTS 16.04.3 (out in August) will be using a kernel with the ChaCha20 based CRNG. Using the attached benchmark program, which calls getrandom() a million times to generate a 32 byte (256 bit) session key, the latency of getrandom(2) is approximately 2.9 microseconds, when run on a n1-standard-1 GCE VM. For older Linux systems, where you are using a SHA1 based CRNG, the time to run getrandom(2) is approximately 5.5 microseconds on the same class of VM. It really is not that much worse. And I'll gently suggest that the public key operations will probably dominate the cost of calling getrandom(2) or reading from /dev/urandom. The reason why I moved to the ChaCha20 based algorithm was only partially about the speed. It also had the significant bonus that I didn't have keep on explaining that theoretical and demonstrated attacks against SHA-1 weren't applicable to how it was being used in Linux's RNG. (See also previous comments about how a lot of complaints about RNG's are more religious than real.) It's also very unlikely that Enterprise distributions will bother updating to a newer version of OpenSSL, since in general they don't want to break shared library ABI's, and the version of OpenSSL that they shipped with originally will almosrt certainly have an incompatible ABI with a modern version of OpenSSL. So I doubt this is really a significant issue. By the time a new version OpenSSL which uses getrandom(2) is picked up by Linux distributions, it is extremely likely that they will be using a 4.9 or newer kernel --- and unless you have a crypto accelerator to speed up your symmetric key operations, I really doubt performance is going to be an issue when you are using getrandom(2), regardless of which kernel version you are using. Cheers, - Ted P.S. The version of the getrandom(2) manpage in Ubuntu LTS's are very much obsolete. Even after Ubuntu LTS updates to a 4.9 or newer kernel when 16.04.3 gets released in August 2017, it's unlikely they will bother to update the manpages, and the 16.04 getrandom(2) man page was outdated when the distribution was released. So arguments that getrandom(2) should not be used based on the manpage are not valid. From pwalten at au1.ibm.com Wed Jun 28 01:41:11 2017 From: pwalten at au1.ibm.com (Peter Waltenberg) Date: Wed, 28 Jun 2017 11:41:11 +1000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <638e04b9-467c-3a6a-6921-34092028edd3@akamai.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <20170626201839.3e7as7y6sxuic7dk@roeckx.be> <97aef447-9dfd-5479-c4df-08d2c62ea8f1@openssl.org> <60e14ee0-9992-6832-61bc-ebe002bf1dde@av8n.com> <20170627215123.bpevx4nygbgw3sjq@roeckx.be> <638e04b9-467c-3a6a-6921-34092028edd3@akamai.com> Message-ID: The next question you should be asking is: does our proposed design mitigate known issues ?. For example this: http://www.pcworld.com/article/2886432/tens-of-thousands-of-home-routers-at-risk-with-duplicate-ssh-keys.html Consider most of the worlds compute is now done on VM's where images are cloned, duplicated and restarted as a matter of course. Not vastly different from an embedded system where the clock powers up as 00:00 1-Jan, 1970 on each image. If you can trust the OS to come up with unique state each time you can rely solely on the OS RNG - well provided you reseed often enough anyway, i.e. before key generation. That's also why seeding a chain of PRNG's once at startup is probably not sufficient here. And FYI. On systems not backed with hardware RNG's /dev/random is extremely slow. 1-2 bytes/second is a DOS attack on it's own without any other effort required. This isn't solely a matter of good software design. And yes, I know, hard problem. If it wasn't a hard problem you probably wouldn't be dealing with it now. Peter From: Benjamin Kaduk via openssl-dev To: openssl-dev at openssl.org, Kurt Roeckx , John Denker Date: 28/06/2017 09:38 Subject: Re: [openssl-dev] Work on a new RNG for OpenSSL Sent by: "openssl-dev" On 06/27/2017 04:51 PM, Kurt Roeckx wrote: On Tue, Jun 27, 2017 at 11:56:04AM -0700, John Denker via openssl-dev wrote: On 06/27/2017 11:50 AM, Benjamin Kaduk via openssl-dev wrote: Do you mean having openssl just pass through to getrandom()/read()-from-'/dev/random'/etc. or just using those to seed our own thing? The former seems simpler and preferable to me (perhaps modulo linux's broken idea about "running out of entropy") That's a pretty big modulus. As I wrote over on the crypto list: The xenial 16.04 LTS manpage for getrandom(2) says quite explicitly: Unnecessarily reading large quantities of data will have a negative impact on other users of the /dev/random and /dev/urandom devices. And that's an understatement. Whether unnecessary or not, reading not-particularly-large quantities of data is tantamount to a denial of service attack against /dev/random and against its upstream sources of randomness. No later LTS is available. Reference: http://manpages.ubuntu.com/manpages/xenial/man2/getrandom.2.html Recently there has been some progress on this, as reflected in in the zesty 17.04 manpage: http://manpages.ubuntu.com/manpages/zesty/man2/getrandom.2.html However, in the meantime openssl needs to run on the platforms that are out there, which includes a very wide range of platforms. And I think it's actually because of changes in the Linux RNG that the manpage has been changed, but they did not document the different behavior of the kernel versions. In case it wasn't clear, I think we should use the OS provided source as a seed. By default that should be the only source of randomness. I think we can get away with using OS-provided randomness directly in many common cases. /dev/urandom suffices once we know that the kernel RNG has been properly seeded. On FreeBSD, /dev/urandom blocks until the kernel RNG is seeded; on other systems maybe we have to make one read from /dev/random to get the blocking behavior we want before switching to /dev/urandom for bulk reads. -Ben-- openssl-dev mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From rsalz at akamai.com Wed Jun 28 01:50:49 2017 From: rsalz at akamai.com (Salz, Rich) Date: Wed, 28 Jun 2017 01:50:49 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <20170628012239.qh365mfpq52465x5@thunk.org> References: <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <20170626201839.3e7as7y6sxuic7dk@roeckx.be> <97aef447-9dfd-5479-c4df-08d2c62ea8f1@openssl.org> <20170627204041.h2sr3zovsnutj43y@thunk.org> <20170628012239.qh365mfpq52465x5@thunk.org> Message-ID: That's interesting info Ted, thanks. But we don't currently know anything about which kernel or glibc version we're building for; maybe that has to change. (We barely, and rarely, look at the toolchain version.) And we need to be able to build a version which runs across a whole mess of kernels and x86 CPU's. From rsalz at akamai.com Wed Jun 28 02:04:16 2017 From: rsalz at akamai.com (Salz, Rich) Date: Wed, 28 Jun 2017 02:04:16 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <8b325b45-1b15-d172-fdfa-939668eeca9e@akamai.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <20170626201839.3e7as7y6sxuic7dk@roeckx.be> <97aef447-9dfd-5479-c4df-08d2c62ea8f1@openssl.org> <47de60b0-9429-424a-bae6-ddeecd9fd8f8@default> <8b325b45-1b15-d172-fdfa-939668eeca9e@akamai.com> Message-ID: -- Senior Architect, Akamai Technologies Member, OpenSSL Dev Team IM: richsalz at jabber.at Twitter: RichSalz From: Benjamin Kaduk via openssl-dev [mailto:openssl-dev at openssl.org] Sent: Tuesday, June 27, 2017 9:22 PM To: openssl-dev at openssl.org; Paul Dale Subject: Re: [openssl-dev] Work on a new RNG for OpenSSL On 06/27/2017 07:24 PM, Paul Dale wrote: The hierarchy of RNGs will overcome some of the performance concerns. Only the root needs to call getrandom(). I do agree that having a DRBG at the root level is a good idea though. Just to check my understanding, the claim is that adding more layers of hashing and/or encryption will still be faster than a larger number of syscalls? -Ben -------------- next part -------------- An HTML attachment was scrubbed... URL: From rsalz at akamai.com Wed Jun 28 02:04:47 2017 From: rsalz at akamai.com (Salz, Rich) Date: Wed, 28 Jun 2017 02:04:47 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <8b325b45-1b15-d172-fdfa-939668eeca9e@akamai.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <20170626201839.3e7as7y6sxuic7dk@roeckx.be> <97aef447-9dfd-5479-c4df-08d2c62ea8f1@openssl.org> <47de60b0-9429-424a-bae6-ddeecd9fd8f8@default> <8b325b45-1b15-d172-fdfa-939668eeca9e@akamai.com> Message-ID: <37bbcf7ac6b14fed8127922e7cafdfda@usma1ex-dag1mb1.msg.corp.akamai.com> > Just to check my understanding, the claim is that adding more layers of hashing and/or encryption will still be faster than a larger number of syscalls? In terms of overall system performance? Yes. From tytso at mit.edu Wed Jun 28 02:17:22 2017 From: tytso at mit.edu (Theodore Ts'o) Date: Tue, 27 Jun 2017 22:17:22 -0400 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: References: <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <20170626201839.3e7as7y6sxuic7dk@roeckx.be> <97aef447-9dfd-5479-c4df-08d2c62ea8f1@openssl.org> <20170627204041.h2sr3zovsnutj43y@thunk.org> Message-ID: <20170628021722.qcmbkimkgr5f5mmd@thunk.org> On Tue, Jun 27, 2017 at 04:12:48PM -0500, Benjamin Kaduk wrote: > > While you're here, would you mind confirming/denying the claim I read > that the reason the linux /dev/random tracks an entropy estimate and > blocks when it gets too low is to preserve backward security in the face > of attacks against SHA1? The explanation is a bit more complicated than that. Linux was the first OS to have a built-in random number generator suitable for use in cryptographic applications. Linux's /dev/random dates back to the days when we still had export control (this is why a cryptographic hash was used instead of an encryption algorithm), and when we didn't have nearly as much understanding about cryptoanalysis as we do now. As a result, just about all of the RNG's from that era (such the original PGP's RNG, designed by Colin Plumb, and with whom I consulted extensively when I designed Linux's /dev/random system) were much more focused on collecting environmental noise and using cryptographic primitives for mixing. We were much less comfortable "back in the day" in being willing to trust in the strength of any cryptographic primitives. Given the weaknesses that were found in MD4 and MD5 (which date from that era), I'd say that our caution was justified. The original entropy estimation and accounting philosophy comes from these historical reasons. The reason why it is still preserved is mainly because the shift from trusting in cryptographic primitives versus entropy estimates was gradual, and by the time we had reached general consensus, just about everyone was using /dev/urandom anyway. And there was no real point in changing /dev/random. In addition, as it turns out, we still need an entropy estimate anyway. First of all, you still need to decide when you have gathered enough environmental "noise" that you are willing to consider the CRNG to be fully initialized. Using some kind of entropy estimate is good way to do this which allows us to have some kind of standard which is portable across wide variety of CPU architecture and hardware configurations. Secondly, if you are using certain types of hardware random number generators, they may be expensive to use them continuously, so using the entropy accounting to control how often to read from the hwrng can be useful. This also allows us to assign a "derating percentage" if we don't want to invest complete trust in the quality of the output from said hardware RNG. In any case, at this point what I recommend is that people just use getrandom(2) and be happy. Getrandom will block until CRNG is initialized (which is only really an issue during early boot on most systems), and eliminates a lot of the other potential pitfalls with using /dev/urandom (for example, it's not vulnerable to a file descriptor exhaustion attack combined with the universal problem that most userspace application programmers don't bother to check error returns). If your libc doesn't support getrandom(2) for whatever reason, open and read from /dev/urandom instead. Unless you are running during early boot (and doing something really silly like generating long term public keys a few seconds after the box is powered on, fresh from the factory), you should be fine so long as do proper error checking of all of your system calls. Cheers, - Ted From ssx at av8n.com Wed Jun 28 02:19:11 2017 From: ssx at av8n.com (John Denker) Date: Tue, 27 Jun 2017 19:19:11 -0700 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <20170626201839.3e7as7y6sxuic7dk@roeckx.be> <97aef447-9dfd-5479-c4df-08d2c62ea8f1@openssl.org> <60e14ee0-9992-6832-61bc-ebe002bf1dde@av8n.com> <20170627215123.bpevx4nygbgw3sjq@roeckx.be> <638e04b9-467c-3a6a-6921-34092028edd3@akamai.com> Message-ID: <2fae4ad9-9ab1-f9a5-deb5-f8e327b45105@av8n.com> On 06/27/2017 06:41 PM, Peter Waltenberg wrote: > Consider most of the worlds compute is now done on VM's where images are > cloned, duplicated and restarted as a matter of course. Not vastly > different from an embedded system where the clock powers up as 00:00 > 1-Jan, 1970 on each image. If you can trust the OS to come up with unique > state each time you can rely solely on the OS RNG - well provided you > reseed often enough anyway, i.e. before key generation. That's also why > seeding a chain of PRNG's once at startup is probably not sufficient here. That is approximately the last thing openssl should be fussing over. There is a set of problems there, with a set of solutions, none of which openssl has any say over. ===> The VM setup should provide a virtual /dev/hwrng <=== Trying to secure a virtual machine without a virtual hwrng (or the equivalent) is next to impossible. There may be workarounds, but they tend to be exceedingly locale-specific, and teaching openssl to try to discover them would be a tremendous waste of resources. So stop trying to operate without /dev/hwrng already. It reminds me of the old Smith & Dale shtick: -- Doctor, doctor, it hurts when I do *this*. -- So don't do that. From paul.dale at oracle.com Wed Jun 28 02:34:47 2017 From: paul.dale at oracle.com (Paul Dale) Date: Tue, 27 Jun 2017 19:34:47 -0700 (PDT) Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <8b325b45-1b15-d172-fdfa-939668eeca9e@akamai.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <20170626201839.3e7as7y6sxuic7dk@roeckx.be> <97aef447-9dfd-5479-c4df-08d2c62ea8f1@openssl.org> <47de60b0-9429-424a-bae6-ddeecd9fd8f8@default> <8b325b45-1b15-d172-fdfa-939668eeca9e@akamai.com> Message-ID: <0d5059e4-4c6c-4d51-b673-c3d1f6391fb4@default> Ben wrote: > On 06/27/2017 07:24 PM, Paul Dale wrote: >> The hierarchy of RNGs will overcome some of the >> performance concerns. Only the root needs to call getrandom(). >> I do agree that having a DRBG at the root level is a good idea though. > Just to check my understanding, the claim is that adding more layers of hashing and/or encryption will still be faster than a larger number of syscalls? I'm not sure if it will be faster or not, although it seems likely. The kernel will have to do the same cryptographic operations so using it adds a syscall overhead. If the kernel is doing different cryptographic operations, then it could be faster. However, I'm more interested in separation of the random sources. I'd prefer to not be sharing my RNG with others if possible. A compromise is unlikely but if one happens it would be nice to limit the damage. Pauli -- Oracle Dr Paul Dale | Cryptographer | Network Security & Encryption Phone +61 7 3031 7217 Oracle Australia From: Benjamin Kaduk [mailto:bkaduk at akamai.com] Sent: Wednesday, 28 June 2017 11:22 AM To: openssl-dev at openssl.org; Paul Dale Subject: Re: [openssl-dev] Work on a new RNG for OpenSSL -Ben From tytso at mit.edu Wed Jun 28 02:39:03 2017 From: tytso at mit.edu (Theodore Ts'o) Date: Tue, 27 Jun 2017 22:39:03 -0400 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: References: <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <20170626201839.3e7as7y6sxuic7dk@roeckx.be> <97aef447-9dfd-5479-c4df-08d2c62ea8f1@openssl.org> <20170627204041.h2sr3zovsnutj43y@thunk.org> <20170628012239.qh365mfpq52465x5@thunk.org> Message-ID: <20170628023903.3pg5raxavax2xwuf@thunk.org> On Wed, Jun 28, 2017 at 01:50:49AM +0000, Salz, Rich wrote: > That's interesting info Ted, thanks. But we don't currently know > anything about which kernel or glibc version we're building for; > maybe that has to change. (We barely, and rarely, look at the > toolchain version.) And we need to be able to build a version which > runs across a whole mess of kernels and x86 CPU's. The performance numbers for /dev/urandom and getrandom pre-4.9 should be largely consistent until you get back to when we were using MD5 instead of SHA-1 for the CRNG. When you get back to the days when /dev/urandom used MD5, the performance will be (obviously) faster, and given the way we use the cryptographic hash algorithms, the currently known attacks to MD5 and SHA won't be an issue. The much *LARGER* problem is that those ancient kernels are almost certainly filled with unpatched security holes (e.g., the kind that might be exploitable remotely using a malformed ICMP packet.) Why should the attacker bother with some complex cryptographic attack with a simple buffer overrun will get them all that they need, much more easily? If you are using an older version of glibc, that doesn't have getrandom(2) support, you can either attempt to call getrandom(2) directly via the syscall interface, or you can just fall back to opening /dev/urandom and reading from it. As long as you do proper error checking to catch things like the fd exhaustion attack, you should be fine from a security perspective, and the performance of reading from /dev/urandom should be no different from using getrandom(2), assuming you keep the fd to /dev/urandom open during the life of the process. If you have a pre-3.17 kernel that doesn't have getrandom(2) support, the same logic applies above. You won't have a performance problem, and the potential security concerns (IMHO, of a purely certificational nature) of /dev/random from that era will be the *least* of your problems. Cheers, - Ted From tytso at mit.edu Wed Jun 28 02:41:10 2017 From: tytso at mit.edu (Theodore Ts'o) Date: Tue, 27 Jun 2017 22:41:10 -0400 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: References: <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <20170626201839.3e7as7y6sxuic7dk@roeckx.be> <97aef447-9dfd-5479-c4df-08d2c62ea8f1@openssl.org> <60e14ee0-9992-6832-61bc-ebe002bf1dde@av8n.com> <20170627215123.bpevx4nygbgw3sjq@roeckx.be> <638e04b9-467c-3a6a-6921-34092028edd3@akamai.com> Message-ID: <20170628024110.4aukgmdjianyisjv@thunk.org> On Wed, Jun 28, 2017 at 11:41:11AM +1000, Peter Waltenberg wrote: > And FYI. On systems not backed with hardware RNG's /dev/random is > extremely slow. 1-2 bytes/second is a DOS attack on it's own without any > other effort required. Please, stop suggesting the use /dev/random. The right answer is /dev/urandom or getrandom(2). - Ted From pwalten at au1.ibm.com Wed Jun 28 02:55:05 2017 From: pwalten at au1.ibm.com (Peter Waltenberg) Date: Wed, 28 Jun 2017 12:55:05 +1000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <2fae4ad9-9ab1-f9a5-deb5-f8e327b45105@av8n.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <20170626201839.3e7as7y6sxuic7dk@roeckx.be> <97aef447-9dfd-5479-c4df-08d2c62ea8f1@openssl.org> <60e14ee0-9992-6832-61bc-ebe002bf1dde@av8n.com> <20170627215123.bpevx4nygbgw3sjq@roeckx.be> <638e04b9-467c-3a6a-6921-34092028edd3@akamai.com> <2fae4ad9-9ab1-f9a5-deb5-f8e327b45105@av8n.com> Message-ID: If the desired outcome is security you must generate instance unique keys and elegant software design alone is simply not enough to achieve that. And I didn't say solve below I said mitigate. You can't solve the problem of someone using already created keys in multiple VM's. But you can and should reduce the chances that someone will create them from a fresh keygen because that simply can't be mitigated anywhere else but in your code. Simillar issues exist with fork(), and again, you should make efforts to mitigate that risk because the user can't. Magic fairy dust like (/dev/hwrng) undoubtedly helps where it exists, but you still have to apply it correctly to achieve the desired outcome. Peter From: John Denker via openssl-dev To: "openssl-dev at openssl.org" Date: 28/06/2017 12:19 Subject: Re: [openssl-dev] Work on a new RNG for OpenSSL Sent by: "openssl-dev" On 06/27/2017 06:41 PM, Peter Waltenberg wrote: > Consider most of the worlds compute is now done on VM's where images are > cloned, duplicated and restarted as a matter of course. Not vastly > different from an embedded system where the clock powers up as 00:00 > 1-Jan, 1970 on each image. If you can trust the OS to come up with unique > state each time you can rely solely on the OS RNG - well provided you > reseed often enough anyway, i.e. before key generation. That's also why > seeding a chain of PRNG's once at startup is probably not sufficient here. That is approximately the last thing openssl should be fussing over. There is a set of problems there, with a set of solutions, none of which openssl has any say over. ===> The VM setup should provide a virtual /dev/hwrng <=== Trying to secure a virtual machine without a virtual hwrng (or the equivalent) is next to impossible. There may be workarounds, but they tend to be exceedingly locale-specific, and teaching openssl to try to discover them would be a tremendous waste of resources. So stop trying to operate without /dev/hwrng already. It reminds me of the old Smith & Dale shtick: -- Doctor, doctor, it hurts when I do *this*. -- So don't do that. -- openssl-dev mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.dale at oracle.com Wed Jun 28 03:00:59 2017 From: paul.dale at oracle.com (Paul Dale) Date: Tue, 27 Jun 2017 20:00:59 -0700 (PDT) Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <20170626201839.3e7as7y6sxuic7dk@roeckx.be> <97aef447-9dfd-5479-c4df-08d2c62ea8f1@openssl.org> <60e14ee0-9992-6832-61bc-ebe002bf1dde@av8n.com> <20170627215123.bpevx4nygbgw3sjq@roeckx.be> <638e04b9-467c-3a6a-6921-34092028edd3@akamai.com> Message-ID: Peter Waltenberg wrote: > The next question you should be asking is: does our proposed design mitigate known issues ?. > For example this: > http://www.pcworld.com/article/2886432/tens-of-thousands-of-home-routers-at-risk-with-duplicate-ssh-keys.html Using the OS RNG won't fix the lack of boot time randomness unless there is a HRNG present. For VMs, John's suggestion that /dev/hwrng should be installed is reasonable. For embedded devices, a HRNG is often not possible. Here getrandom() (or /dev/random since old kernels are common) should be used. Often /dev/urandom is used instead and the linked article is the result. There are possible mitigations that some manufacturers include (usually with downsides). The question is should a security toolkit try to do a good job regardless? I've seen more cases than I care to count where long term cryptographic material is generated on first boot out of the factory. I've even seen some cases where this was done during the factory test. Pauli -- Oracle Dr Paul Dale | Cryptographer | Network Security & Encryption Phone +61 7 3031 7217 Oracle Australia From cory at lukasa.co.uk Wed Jun 28 07:15:20 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Wed, 28 Jun 2017 08:15:20 +0100 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <20170626201839.3e7as7y6sxuic7dk@roeckx.be> <97aef447-9dfd-5479-c4df-08d2c62ea8f1@openssl.org> <60e14ee0-9992-6832-61bc-ebe002bf1dde@av8n.com> <20170627215123.bpevx4nygbgw3sjq@roeckx.be> <638e04b9-467c-3a6a-6921-34092028edd3@akamai.com> Message-ID: > On 28 Jun 2017, at 04:00, Paul Dale wrote: > > > Peter Waltenberg wrote: >> The next question you should be asking is: does our proposed design mitigate known issues ?. >> For example this: >> http://www.pcworld.com/article/2886432/tens-of-thousands-of-home-routers-at-risk-with-duplicate-ssh-keys.html > > Using the OS RNG won't fix the lack of boot time randomness unless there is a HRNG present. > > For VMs, John's suggestion that /dev/hwrng should be installed is reasonable. > > For embedded devices, a HRNG is often not possible. Here getrandom() (or /dev/random since old kernels are common) should be used. Often /dev/urandom is used instead and the linked article is the result. There are possible mitigations that some manufacturers include (usually with downsides). When you say ?the linked article?, do you mean the PCWorld one? Because that article doesn?t provide any suggestion that /dev/urandom has anything to do with it. It is at least as likely that the SSH key is hard-coded into the machine image. The flaw here is not ?using /dev/urandom?, it?s ?exposing your router?s SSH access on the external side of the router?, plus the standard level of poor configuration done by shovelware router manufacturers. Cory From matt at openssl.org Wed Jun 28 09:29:22 2017 From: matt at openssl.org (Matt Caswell) Date: Wed, 28 Jun 2017 10:29:22 +0100 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <20170626201839.3e7as7y6sxuic7dk@roeckx.be> <97aef447-9dfd-5479-c4df-08d2c62ea8f1@openssl.org> Message-ID: <407d8771-9cfc-26a7-b906-0734ac99de2c@openssl.org> On 27/06/17 19:50, Benjamin Kaduk wrote: > On 06/27/2017 02:28 AM, Matt Caswell wrote: >> >> On 26/06/17 21:18, Kurt Roeckx wrote: >> >>> I think it should by default be provided by the OS, and I don't >>> think any OS is documenting how much randomness it can provide. >>> >> I also agree that, by default, using the OS provided source makes a lot >> of sense. >> > > Do you mean having openssl just pass through to > getrandom()/read()-from-'/dev/random'/etc. or just using those to seed > our own thing? I meant the former. Matt From tytso at mit.edu Wed Jun 28 13:02:32 2017 From: tytso at mit.edu (Theodore Ts'o) Date: Wed, 28 Jun 2017 09:02:32 -0400 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: References: <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <20170626201839.3e7as7y6sxuic7dk@roeckx.be> <97aef447-9dfd-5479-c4df-08d2c62ea8f1@openssl.org> <60e14ee0-9992-6832-61bc-ebe002bf1dde@av8n.com> <20170627215123.bpevx4nygbgw3sjq@roeckx.be> <638e04b9-467c-3a6a-6921-34092028edd3@akamai.com> Message-ID: <20170628130231.mth7ajhvmqptw6fq@thunk.org> On Wed, Jun 28, 2017 at 08:15:20AM +0100, Cory Benfield wrote: > When you say ?the linked article?, do you mean the PCWorld one? > Because that article doesn?t provide any suggestion that > /dev/urandom has anything to do with it. It is at least as likely > that the SSH key is hard-coded into the machine image. The flaw here > is not ?using /dev/urandom?, it?s ?exposing your router?s SSH access > on the external side of the router?, plus the standard level of poor > configuration done by shovelware router manufacturers. The PC World article was clearly caused by hard-coding the RSA private key in the flashed firmware and sharing it across fast numbers of routers. The reference which I believe Paul Dale was looking for dates back to five years ago, in the "Minding your P's and Q's paper", published in Usenix Security: https://factorable.net/ The researchers gave me early access to the paper in early July and I spent most of July 4th on the rooftop of the Museum of Science rooftop, reworking Linux's random driver during the day (so we could watch the fireworks from an advantageous position that evening). We had patches released before the P's and Q's paper was published, and these fixes are in 3.6 and newer kernels. There may still be issues with silly programs which initialize symmetric keys the first time the system boots, instead of doing it on an on-demand basis the first time someone tries connecting to the SSH port (when there would be plenty of entropy mixed in). And of course, things have gotten more challenging because systemd starts the sshd service much more quickly, and if you are on an embedded system which does not have a high resolution clock, and very little entropy when the box is first booted (for example, if the user plugs in the router without having first attached any of the LAN or WAN cables), it may still be a problem on some systems. Unfortunately, though, this ends up being a system design problem. There only so much you can do just at the kernel level, or just at the openssl level. My recommendation, again, is to use getrandom(2) if it is available, and if it isn't (on kernels older than 3.17) fall back to using /dev/urandom. On kernels older than 3.17, you're probably totally screwed because the kernel is very likely going to be missing large numbers of security fixes, so if someone is unfortunate enough to be using a kernel which is older than 3.17, they're probably totally screwed anyway. (The only exception to this rule is if you're getting monthly security updates from your cell phone provider, but if you're not using an Google-supported android releases, and you're using a ARM SOC with a 3.10 BSP kernel, you might as well give up now. And if the leader of the free world, is using such a device to tweet messages in lieu of press conferences --- again, you've got bigger problems. :-) If you are using a system which is 3.17 or newer, but your libc doesn't have support for getrandom(2), I would recommend trying to call getrandom(2) directly using the syscall(2) interface. This will mean hardcoding Linux's syscall number for different architectures, which is painful and ugly, but that's going to be the best way to block until you are sure the CRNG has been initialized, in case you do have silly programs trying to generate long-term secrets a second or so after the box is first powered on. Cheers, - Ted P.S. Short summry of kernel versions from an upstream perspective 3.17 -- first kernel version with getrandom(2) 3.18 -- Long Term Stable kernel 4.1 -- Long Term Stable kernel 4.4 -- Long Term Stable kernel 4.8 -- first kernel version using a ChaCha20 CRNG 4.9 -- Long Term Stable kernel 4.12 -- Next Long Term Stable kernel (to be released in a week or two) (Long Term Stable kernels are ones which get upstream security fixes backported by Greg K-H and his team. Just because your kernel has a LTS version may not mean you are getting the latest bug fixes. SOC vendors are notorious for not bothering to update their kernels with the latest bug fixes, and many embedded and mobile device vendors aren't bothering to keep up as well.) From Matthias.St.Pierre at ncp-e.com Wed Jun 28 14:42:25 2017 From: Matthias.St.Pierre at ncp-e.com (Matthias St. Pierre) Date: Wed, 28 Jun 2017 16:42:25 +0200 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <407d8771-9cfc-26a7-b906-0734ac99de2c@openssl.org> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <20170626201839.3e7as7y6sxuic7dk@roeckx.be> <97aef447-9dfd-5479-c4df-08d2c62ea8f1@openssl.org> <407d8771-9cfc-26a7-b906-0734ac99de2c@openssl.org> Message-ID: Hello Matt, I am not quite sure what your current favourite solution for the upcoming default OpenSSL random generator is. Are you favouring - a DRBG (following SP800-90Ar1) which is using the OS RNG as entropy source for (re-)seeding or - simply passing all generate requests over to the OS RNG? It looks like you made two votes for the first and one vote for the second variant (see below). Could you please clarify your preference? Regards, Matthias St. Pierre Vote 1: On 27.06.2017 09:28, Matt Caswell wrote: > On 26/06/17 21:18, Kurt Roeckx wrote: >>> ?Recommendation for Random Number Generation Using Deterministic Random Bit Generators? >>> http://csrc.nist.gov/publications/nistpubs/800-90A/SP800-90A.pdf >>> >>> That design may look complicated, but if you think you can >>> leave out some of the blocks in their diagram, proceed with >>> caution. Every one of those blocks is there for a reason. >> SP800-90A (or revision 1) can clearly be used as reference on how >> to implement it, even if we don't use an approved algorithm from >> it. And I really think we should look at that document when >> implementing it. >> >> There should probably also be an option to use an RNG that >> conforms to it. > I am strongly in favour of this approach. We should be led by standards. > Vote 2: (comment on PR #3789: WIP: Add DRBG random method) https://github.com/openssl/openssl/pull/3789#issuecomment-311494544 Vote 3: On 28.06.2017 11:29, Matt Caswell wrote: > On 27/06/17 19:50, Benjamin Kaduk wrote: >> On 06/27/2017 02:28 AM, Matt Caswell wrote: >>> On 26/06/17 21:18, Kurt Roeckx wrote: >>> >>>> I think it should by default be provided by the OS, and I don't >>>> think any OS is documenting how much randomness it can provide. >>>> >>> I also agree that, by default, using the OS provided source makes a lot >>> of sense. >>> >> Do you mean having openssl just pass through to >> getrandom()/read()-from-'/dev/random'/etc. or just using those to seed >> our own thing? > I meant the former. > > Matt From matt at openssl.org Wed Jun 28 14:46:31 2017 From: matt at openssl.org (Matt Caswell) Date: Wed, 28 Jun 2017 15:46:31 +0100 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <20170626201839.3e7as7y6sxuic7dk@roeckx.be> <97aef447-9dfd-5479-c4df-08d2c62ea8f1@openssl.org> <407d8771-9cfc-26a7-b906-0734ac99de2c@openssl.org> Message-ID: <4bbd95fd-78d2-87cf-ec3a-18d3bd9d2dc4@openssl.org> On 28/06/17 15:42, Matthias St. Pierre wrote: > Hello Matt, > > I am not quite sure what your current favourite solution for the upcoming default OpenSSL random generator is. Are you favouring > > - a DRBG (following SP800-90Ar1) which is using the OS RNG as entropy source for (re-)seeding or > > - simply passing all generate requests over to the OS RNG? > > It looks like you made two votes for the first and one vote for the second variant (see below). Could you please clarify your preference? Both :-) i.e. both should be available as an option. I don't think we will necessary be able to do the latter on all platforms that we support. As for which of the two is the default: where it is available - the latter. Where it isn't the former. Matt > > Regards, > > Matthias St. Pierre > > > Vote 1: > > On 27.06.2017 09:28, Matt Caswell wrote: >> On 26/06/17 21:18, Kurt Roeckx wrote: >>>> ?Recommendation for Random Number Generation Using Deterministic Random Bit Generators? >>>> http://csrc.nist.gov/publications/nistpubs/800-90A/SP800-90A.pdf >>>> >>>> That design may look complicated, but if you think you can >>>> leave out some of the blocks in their diagram, proceed with >>>> caution. Every one of those blocks is there for a reason. >>> SP800-90A (or revision 1) can clearly be used as reference on how >>> to implement it, even if we don't use an approved algorithm from >>> it. And I really think we should look at that document when >>> implementing it. >>> >>> There should probably also be an option to use an RNG that >>> conforms to it. >> I am strongly in favour of this approach. We should be led by standards. >> > > Vote 2: (comment on PR #3789: WIP: Add DRBG random method) > > https://github.com/openssl/openssl/pull/3789#issuecomment-311494544 > > > Vote 3: > > > On 28.06.2017 11:29, Matt Caswell wrote: >> On 27/06/17 19:50, Benjamin Kaduk wrote: >>> On 06/27/2017 02:28 AM, Matt Caswell wrote: >>>> On 26/06/17 21:18, Kurt Roeckx wrote: >>>> >>>>> I think it should by default be provided by the OS, and I don't >>>>> think any OS is documenting how much randomness it can provide. >>>>> >>>> I also agree that, by default, using the OS provided source makes a lot >>>> of sense. >>>> >>> Do you mean having openssl just pass through to >>> getrandom()/read()-from-'/dev/random'/etc. or just using those to seed >>> our own thing? >> I meant the former. >> >> Matt > From Matthias.St.Pierre at ncp-e.com Wed Jun 28 15:09:43 2017 From: Matthias.St.Pierre at ncp-e.com (Matthias St. Pierre) Date: Wed, 28 Jun 2017 17:09:43 +0200 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <4bbd95fd-78d2-87cf-ec3a-18d3bd9d2dc4@openssl.org> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <20170626201839.3e7as7y6sxuic7dk@roeckx.be> <97aef447-9dfd-5479-c4df-08d2c62ea8f1@openssl.org> <407d8771-9cfc-26a7-b906-0734ac99de2c@openssl.org> <4bbd95fd-78d2-87cf-ec3a-18d3bd9d2dc4@openssl.org> Message-ID: <2dd86c32-1d15-c759-cebd-29da2e86fad9@ncp-e.com> Thanks for the quick reply. It sounds reasonable to make the default choice depending on the os environment. For me it is not a religious question what OpenSSL's default choice should be. I trust that you will find a sensible solution. And if OpenSSL supports both methods I can always make my own choice if I need to. Regards, Matthias On 28.06.2017 16:46, Matt Caswell wrote: > > On 28/06/17 15:42, Matthias St. Pierre wrote: >> Hello Matt, >> >> I am not quite sure what your current favourite solution for the upcoming default OpenSSL random generator is. Are you favouring >> >> - a DRBG (following SP800-90Ar1) which is using the OS RNG as entropy source for (re-)seeding or >> >> - simply passing all generate requests over to the OS RNG? >> >> It looks like you made two votes for the first and one vote for the second variant (see below). Could you please clarify your preference? > Both :-) > > i.e. both should be available as an option. > > I don't think we will necessary be able to do the latter on all > platforms that we support. > > As for which of the two is the default: where it is available - the > latter. Where it isn't the former. > > Matt From bkaduk at akamai.com Wed Jun 28 17:01:29 2017 From: bkaduk at akamai.com (Benjamin Kaduk) Date: Wed, 28 Jun 2017 12:01:29 -0500 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <64072fc1-4411-4b3f-88cf-296ea19753cf@default> References: <64072fc1-4411-4b3f-88cf-296ea19753cf@default> Message-ID: On 06/26/2017 11:28 PM, Paul Dale wrote: > Given the variety of RNGs available, would an EVP RNG interface make sense? With a safe default in place (and no weak generators provided), the decision can be left to the user. > A side benefit is that the unit tests could implement a simple fully deterministic generator and the code for producing known sequences removed from the code base. There are some benefits to this idea, as you note, but it does not seem like a clear "immediate win" to me. Maybe this is just some emotional response that has not fully absorbed "no weak generators provided", as I can't really articulate any reason to oppose it other than "randomness is so low-level that we should just provide it and not options for it". > > Defence in depth seems prudent: independent sources with agglomeration and whitening. As Kurt noted, [on modern OSes,] it is really unclear what sources are available to us that are not already being used by the kernel. Rich had commented about the dragonfly (kernel) implementation "wow, is it really that easy?". To large extent, yes, a secure RNG can present as being that simple/easy -- if you're writing it in the kernel! The kernel has easy and direct access to lots of interrupt-driven entropy sources, any hardware generators present, etc., as well as rdrand/etc. It doesn't have to worry about fork-safety or syscall overhead, and can basically just implement the raw crypto needed for whitening/mixing/stretching. So, [on these same modern OSes,] what benefit do we really get from using multiple "independent" sources? They are unlikely to actually be independent if the kernel is consuming them as well and we consume the kernel. Now, of course OpenSSL runs on OSes that do not provide a modern kernel RNG and we will need some solution for them, which will likely look as you describe. I'm just not convinced there is much value in duplicating what the kernel is doing in the cases that the kernel does it well. > > We shouldn't trust the user to provide entropy. I've seen what is typically provided. Uninitialised buffers aren't random. User inputs (mouse and keyboard) likewise aren't very good. That both are still being suggested is frustrating. I've seen worse suggestions, some to the effect that "time(NULL) ^ getpid()" is too good and just time() is enough. Definitely. But, as we're not the kernel, finding good sources of real randomness as a generic userspace process is quite hard. > > As for specific questions and comments: > > John Denker wrote: >> If you trust the ambient OS to provide a seed, why not >> trust it for everything, and not bother to implement an >> openssl-specfic RNG at all? > I can think of a few possibilities: Ah, preemptive replies to my comments above, excellent. > * Diversifying the sources provides resistance to compromise of individual sources. Although a full kernel compromise is unrecoverable, a kernel bug that leaked the internal pools in a read only manner isn't unforeseeable. It is not unforseeable, sure, but so are lots of things. Spewing the contents of the openssl process-local randomness pool on the network isn't unforseeable, either; do we have any reason to think there is substantially more risk from one unknown than the other? > * Not all operating systems have good RNGs. Sure, and we need to support the ones that don't have good RNGs. But on the ones that do, what do we gain from duplicating the effort? > > * Draining the kernel's entropy pools is unfriendly behaviour, other processes will typically want some randomness too. > > * At boot time the kernel pools are empty (low or no quality). This compounds when several things require seeding. I'm not sure what you mean by "draining the kernel's entropy pools". That is, if you are adhering to the belief that taking random bits out of a generator removes entropy from it that must be replenished, does that not apply also to any generator/pool we write for ourselves? Or maybe you just refer to the behavior of linux /dev/random, in which case I would point out Ted (the author/maintainer of linux /dev/random)'s suggestion to just use (getrandom or) /dev/random and tacit agreement that the behavior of reducing the entropy count on reads from /dev/random is not really needed anymore. At boot time *all* pools are empty. FreeBSD has a random seed file on disk to be loaded on next boot that helps with this (I didn't check linux), and openssl has/can use ~/.rnd or similar, but those are not immune from compromise out-of-band. In order to be properly confident of good randomness, new randomness needs to be collected from the environment and added to the pool, and the kernel is in a much better position to do so (and know when it has enough!) than we are. > * Performance is also a consideration, although with a gradual collection strategy this should be less of a concern. Except at start up. Given that we're going to be implementing several variants anyway, I propose to defer decisions based on performance considerations until hard data are actually available. > > John Denker wrote: >> Are you designing to resist an output-text-only attack? Or do you >> also want "some" ability to recover from a compromise of >> the PRNG internal state? > Yes to both ideally. Feeding additional seed material into a PRNG could help in the latter case. It depends on how easy it is to compromise the PRNG state. If it is trivial in terms of resources, it isn't going to be recoverable. A short reseed interval can be effective against a slow attack (but not always). Of course, if we just use the kernel RNG's output directly, we have no internal state to compromise (though the kernel does, but the kernel also does its own reseeding). > > > John Denker wrote: >>> Do you think we need to use multiple sources of randomness? >> Quality is more important than quantity. > Given the difficulty of quantifying the quality of sources, I think that having multiple sources is prudent. Finding out that a source is low quality when it was believed to be good is less embarrassing when you've other sources in operation. Likewise, if a source fails. More sources are better, sure. But don't count the same source twice! (That is, when the kernel uses it already and then we want to use it again.) -Ben -------------- next part -------------- An HTML attachment was scrubbed... URL: From kurt at roeckx.be Wed Jun 28 17:20:44 2017 From: kurt at roeckx.be (Kurt Roeckx) Date: Wed, 28 Jun 2017 19:20:44 +0200 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: References: <64072fc1-4411-4b3f-88cf-296ea19753cf@default> Message-ID: <20170628172044.hssuq3kutsqr2fqj@roeckx.be> On Wed, Jun 28, 2017 at 12:01:29PM -0500, Benjamin Kaduk via openssl-dev wrote: > > I'm not sure what you mean by "draining the kernel's entropy pools". > That is, if you are adhering to the belief that taking random bits out > of a generator removes entropy from it that must be replenished, does > that not apply also to any generator/pool we write for ourselves? Or > maybe you just refer to the behavior of linux /dev/random, in which case > I would point out Ted (the author/maintainer of linux /dev/random)'s > suggestion to just use (getrandom or) /dev/random and tacit agreement > that the behavior of reducing the entropy count on reads from > /dev/random is not really needed anymore. Replace all /dev/random with /dev/urandom. > At boot time *all* pools are empty. FreeBSD has a random seed file on > disk to be loaded on next boot that helps with this (I didn't check > linux), It depends on the distro, but they should all be doing this. On systems using systemd that file is probably /var/lib/systemd/random-seed. Kurt From uri at ll.mit.edu Wed Jun 28 17:28:26 2017 From: uri at ll.mit.edu (Blumenthal, Uri - 0553 - MITLL) Date: Wed, 28 Jun 2017 17:28:26 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: References: <64072fc1-4411-4b3f-88cf-296ea19753cf@default> Message-ID: <8F220926-A6C7-44A5-A5FE-464E8416078A@ll.mit.edu> Defence in depth seems prudent: independent sources with agglomeration and whitening. As Kurt noted, [on modern OSes,] it is really unclear what sources are available to us that are not already being used by the kernel. I would turn to hardware. Since OpenSSL already has assembly-level optimization for various CPU types, accessing instructions like RDSEED and RDRAND (when available) doesn?t sound too hard. Mix their output into the seed. It can only improve the result. So, [on these same modern OSes,] what benefit do we really get from using multiple "independent" sources? They are unlikely to actually be independent if the kernel is consuming them as well and we consume the kernel. Depends on what you mean by ?independent?. A completely different mechanism ? probably not. A mechanism whose output bits/bytes are not (tractably) correlated? Probably yes. We shouldn't trust the user to provide entropy. Definitely. No. ?We shouldn?t trust the user to provide all entropy ? but should welcome user?s contribution to the entropy pool. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5211 bytes Desc: not available URL: From ben at links.org Wed Jun 28 20:33:09 2017 From: ben at links.org (Ben Laurie) Date: Wed, 28 Jun 2017 21:33:09 +0100 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <63f66496438849a9b130f20b3897f082@usma1ex-dag1mb1.msg.corp.akamai.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <63f66496438849a9b130f20b3897f082@usma1ex-dag1mb1.msg.corp.akamai.com> Message-ID: On 26 June 2017 at 18:10, Salz, Rich via openssl-dev < openssl-dev at openssl.org> wrote: > > Pseudorandomness of the output has been a design goal/requirement only > > in SHA-3 family. Any prior hash function?s exhibition of this property is > > coincidental. > > > > Therefore I suggest using SHA3 instead. > > Is pseudorandomness a requirement? Or is it the "50% chance of a bitflip"? > You are asking if a pseudorandom number generator requires pseudorandomness? -------------- next part -------------- An HTML attachment was scrubbed... URL: From pwalten at au1.ibm.com Wed Jun 28 22:13:20 2017 From: pwalten at au1.ibm.com (Peter Waltenberg) Date: Thu, 29 Jun 2017 08:13:20 +1000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <20170626201839.3e7as7y6sxuic7dk@roeckx.be> <97aef447-9dfd-5479-c4df-08d2c62ea8f1@openssl.org> <60e14ee0-9992-6832-61bc-ebe002bf1dde@av8n.com> <20170627215123.bpevx4nygbgw3sjq@roeckx.be> <638e04b9-467c-3a6a-6921-34092028edd3@akamai.com> Message-ID: Debian also screwed up here at one point and the SSH keys for Debian installs came from a very small subset of keys. This CLASS of problem is common and it's something you need to make efforts to avoid. And again, it is something you need to address as far as you can because you simply can't rely on the users of your software to be able to do better. Seeding is a hard problem as is using the seed material correctly. The overall objective is security, security requires instance unique keys, keys that aren't trivially guessed. Quite a few of the suggestions made so far would compromise that. It's a very different problem from generating good pseudo-random sequences and by it's nature doesn't lend itself well to clean and elegant solutions. Peter From: Cory Benfield To: openssl-dev at openssl.org Date: 28/06/2017 17:15 Subject: Re: [openssl-dev] Work on a new RNG for OpenSSL Sent by: "openssl-dev" > On 28 Jun 2017, at 04:00, Paul Dale wrote: > > > Peter Waltenberg wrote: >> The next question you should be asking is: does our proposed design mitigate known issues ?. >> For example this: >> http://www.pcworld.com/article/2886432/tens-of-thousands-of-home-routers-at-risk-with-duplicate-ssh-keys.html > > Using the OS RNG won't fix the lack of boot time randomness unless there is a HRNG present. > > For VMs, John's suggestion that /dev/hwrng should be installed is reasonable. > > For embedded devices, a HRNG is often not possible. Here getrandom() (or /dev/random since old kernels are common) should be used. Often /dev/urandom is used instead and the linked article is the result. There are possible mitigations that some manufacturers include (usually with downsides). When you say ?the linked article?, do you mean the PCWorld one? Because that article doesn?t provide any suggestion that /dev/urandom has anything to do with it. It is at least as likely that the SSH key is hard-coded into the machine image. The flaw here is not ?using /dev/urandom?, it?s ?exposing your router?s SSH access on the external side of the router?, plus the standard level of poor configuration done by shovelware router manufacturers. Cory -- openssl-dev mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.dale at oracle.com Thu Jun 29 01:20:33 2017 From: paul.dale at oracle.com (Paul Dale) Date: Wed, 28 Jun 2017 18:20:33 -0700 (PDT) Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <20170626201839.3e7as7y6sxuic7dk@roeckx.be> <97aef447-9dfd-5479-c4df-08d2c62ea8f1@openssl.org> <60e14ee0-9992-6832-61bc-ebe002bf1dde@av8n.com> <20170627215123.bpevx4nygbgw3sjq@roeckx.be> <638e04b9-467c-3a6a-6921-34092028edd3@akamai.com> Message-ID: Cory asked: > When you say ?the linked article?, do you mean the PCWorld one? My apologies I meant the one Ted referred to soon after. Pauli -- Oracle Dr Paul Dale | Cryptographer | Network Security & Encryption Phone +61 7 3031 7217 Oracle Australia -----Original Message----- From: Cory Benfield [mailto:cory at lukasa.co.uk] Sent: Wednesday, 28 June 2017 5:15 PM To: openssl-dev at openssl.org Subject: Re: [openssl-dev] Work on a new RNG for OpenSSL > On 28 Jun 2017, at 04:00, Paul Dale wrote: > > > Peter Waltenberg wrote: >> The next question you should be asking is: does our proposed design mitigate known issues ?. >> For example this: >> http://www.pcworld.com/article/2886432/tens-of-thousands-of-home-routers-at-risk-with-duplicate-ssh-keys.html > > Using the OS RNG won't fix the lack of boot time randomness unless there is a HRNG present. > > For VMs, John's suggestion that /dev/hwrng should be installed is reasonable. > > For embedded devices, a HRNG is often not possible. Here getrandom() (or /dev/random since old kernels are common) should be used. Often /dev/urandom is used instead and the linked article is the result. There are possible mitigations that some manufacturers include (usually with downsides). When you say ?the linked article?, do you mean the PCWorld one? Because that article doesn?t provide any suggestion that /dev/urandom has anything to do with it. It is at least as likely that the SSH key is hard-coded into the machine image. The flaw here is not ?using /dev/urandom?, it?s ?exposing your router?s SSH access on the external side of the router?, plus the standard level of poor configuration done by shovelware router manufacturers. Cory -- openssl-dev mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev From ben at links.org Thu Jun 29 04:03:23 2017 From: ben at links.org (Ben Laurie) Date: Thu, 29 Jun 2017 05:03:23 +0100 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <20170628024110.4aukgmdjianyisjv@thunk.org> References: <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <20170626201839.3e7as7y6sxuic7dk@roeckx.be> <97aef447-9dfd-5479-c4df-08d2c62ea8f1@openssl.org> <60e14ee0-9992-6832-61bc-ebe002bf1dde@av8n.com> <20170627215123.bpevx4nygbgw3sjq@roeckx.be> <638e04b9-467c-3a6a-6921-34092028edd3@akamai.com> <20170628024110.4aukgmdjianyisjv@thunk.org> Message-ID: On 28 June 2017 at 03:41, Theodore Ts'o wrote: > On Wed, Jun 28, 2017 at 11:41:11AM +1000, Peter Waltenberg wrote: > > And FYI. On systems not backed with hardware RNG's /dev/random is > > extremely slow. 1-2 bytes/second is a DOS attack on it's own without any > > other effort required. > > Please, stop suggesting the use /dev/random. The right answer is > /dev/urandom or getrandom(2). > a) On Linux. b) If its the right answer, why is there a difference between /dev/random and /dev/urandom? > > - Ted > -- > openssl-dev mailing list > To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dimitry at andric.com Thu Jun 29 11:03:00 2017 From: dimitry at andric.com (Dimitry Andric) Date: Thu, 29 Jun 2017 13:03:00 +0200 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: References: <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <20170626201839.3e7as7y6sxuic7dk@roeckx.be> <97aef447-9dfd-5479-c4df-08d2c62ea8f1@openssl.org> <60e14ee0-9992-6832-61bc-ebe002bf1dde@av8n.com> <20170627215123.bpevx4nygbgw3sjq@roeckx.be> <638e04b9-467c-3a6a-6921-34092028edd3@akamai.com> <20170628024110.4aukgmdjianyisjv@thunk.org> Message-ID: <2E42F140-B6A8-4C64-9062-1EF0DD89AA08@andric.com> On 29 Jun 2017, at 06:03, Ben Laurie wrote: > > On 28 June 2017 at 03:41, Theodore Ts'o wrote: > On Wed, Jun 28, 2017 at 11:41:11AM +1000, Peter Waltenberg wrote: > > And FYI. On systems not backed with hardware RNG's /dev/random is > > extremely slow. 1-2 bytes/second is a DOS attack on it's own without any > > other effort required. > > Please, stop suggesting the use /dev/random. The right answer is > /dev/urandom or getrandom(2). > > a) On Linux. > > b) If its the right answer, why is there a difference between /dev/random and /dev/urandom? The Linux random(4) manpage says: The /dev/random device is a legacy interface which dates back to a time where the cryptographic primitives used in the imple? mentation of /dev/urandom were not widely trusted. It will return random bytes only within the estimated number of bits of fresh noise in the entropy pool, blocking if necessary. /dev/random is suitable for applications that need high quality randomness, and can afford indeterminate delays. and: Usage The /dev/random interface is considered a legacy interface, and /dev/urandom is preferred and sufficient in all use cases, with the exception of applications which require randomness during early boot time; for these applications, getrandom(2) must be used instead, because it will block until the entropy pool is initialized. In short, almost everybody should use /dev/urandom, and /dev/random is kept alive for old programs. -Dimitry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 194 bytes Desc: Message signed with OpenPGP URL: From ssx at av8n.com Thu Jun 29 15:01:11 2017 From: ssx at av8n.com (John Denker) Date: Thu, 29 Jun 2017 08:01:11 -0700 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <20170627204041.h2sr3zovsnutj43y@thunk.org> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <20170626201839.3e7as7y6sxuic7dk@roeckx.be> <97aef447-9dfd-5479-c4df-08d2c62ea8f1@openssl.org> <20170627204041.h2sr3zovsnutj43y@thunk.org> Message-ID: <2314c104-e850-e629-f692-4f3f6aebb5e8@av8n.com> Executive summary: As has been said many times before, what we need (but do not have) is /one/ source of randomness that never blocks and never returns bits that are guessable by the adversary. In favorable cases, using getrandom(,,0) \*\ is appropriate for openssl. There are problems with that, but switching to getrandom(,,GRND_RANDOM) \**\ would not solve the problems. \*\ Reading /dev/urandom is almost the same. \**\ Reading /dev/random is essentially the same. In cases where getrandom() is not good enough, the problems tend to be highly platform-dependent. Many of these problems would be quite difficult for openssl to detect (much less solve). Some platforms are not secure and cannot be made secure. On 06/27/2017 01:40 PM, Theodore Ts'o wrote: >> My recommendation for Linux is to use getrandom(2) the flags field set >> to zero. >> [...] /dev/urandom (which has the same performance characteristics as the >> getrandom system call) Similarly, on 06/29/2017 04:03 AM, Dimitry Andric gave what might be considered the usually-correct answer for the wrong reasons: > In short, almost everybody should use /dev/urandom OK. There's also getrandom(). > and /dev/random is kept alive for old programs. [...] > The Linux random(4) manpage says: > > The /dev/random device is a legacy interface which dates back > to a time where the cryptographic primitives used in the imple? > mentation of /dev/urandom were not widely trusted. It will > return random bytes only within the estimated number of bits of > fresh noise in the entropy pool, blocking if necessary. > /dev/random is suitable for applications that need high quality > randomness, and can afford indeterminate delays. That's what the manpage says ... but does anybody believe it? On 06/27/2017 06:22 PM, Ted told us not to trust what it says in the man pages. Oddly enough, all the advice given above (including the list traffic and the man pages) is flatly contradicted by what it says in the most up-to-date kernel source, namely: >>> /dev/random is suitable for use when very high >>> quality randomness is desired (for example, for key generation Reference: https://git.kernel.org/pub/scm/linux/kernel/git/tytso/random.git/tree/drivers/char/random.c?id=e2682130931f#n111 All it all, it's hardly surprising that users are confused. ================================== When it was introduced, the random / urandom split was advertised as a way of solving certain problems with the old approach. To block or not to block, that is the question..... The problems didn't actually get solved, just shifted. The split requires users (rather than the RNG designers) to deal with the problems. The fact that the recently- introduced getrandom(2) call has flags such as GRND_RANDOM and GRND_NONBLOCK means that users are still on the hook for problems they almost certainly cannot understand, much less solve. The conclusion remains the same: What we need (but do not have) is /one/ source of randomness that never blocks and never returns bits that are guessable by the adversary. ========== In fact there are profound distinctions between an ideal HRNG and an ideal PRNG. AFAICT neither one exists in the real world, in the same sense that ideal spheres and planes do not exist, but still the idealizations are meaningful and helpful. It seems likely that /dev/random was intended, at the time of the split, to serve as approximate HRNG, while /dev/urandom was intended to be a PRNG of some kind. Using terms like "legacy interface" is an astonishing mischaracterization of the distinction. ==== Similarly, it is strange to talk about > a time where the cryptographic primitives used in the imple? > mentation of /dev/urandom were not widely trusted In fact, ++ Improper seeding is, and has always been, the #1 threat to both /dev/random and /dev/urandom. ++ Compromise of the internal state is a threat to /dev/urandom. It is better to prevent this than to try to cure it. If the PRNG is compromised, probably a lot of other things are too. ++ Lousy architectural design is always a threat. ++ Coding errors are always a threat. ++ etc. ++ etc. -- Cryptanalytic attack against the outputs is way, Way, WAY down on the list, and always has been, assuming the crypto primitives are halfway decent, assuming the architecture and implementation are sound. From ssx at av8n.com Thu Jun 29 18:28:24 2017 From: ssx at av8n.com (John Denker) Date: Thu, 29 Jun 2017 11:28:24 -0700 Subject: [openssl-dev] what's possible and what's not ... including RNGs In-Reply-To: <2314c104-e850-e629-f692-4f3f6aebb5e8@av8n.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <20170626201839.3e7as7y6sxuic7dk@roeckx.be> <97aef447-9dfd-5479-c4df-08d2c62ea8f1@openssl.org> <20170627204041.h2sr3zovsnutj43y@thunk.org> <2314c104-e850-e629-f692-4f3f6aebb5e8@av8n.com> Message-ID: <9056490c-34c2-e946-f482-2416e7b82597@av8n.com> On 06/29/2017 08:01 AM, I wrote: > Some platforms are not secure and cannot be made secure. This is relevant to the RNG discussion, and lots of other stuff besides. *) For example, you can use openssl rsa to generate a so-called private key, but it will not actually be private (in any practical sense) if the operating system's RNG has been compromised. There is little that openssl can do to detect this problem. RNGs are infamously hard to test. *) As another example, a VM guest is at the mercy of the host. *) Don't enable remote administration using "admin" as the root password. *) Etc. etc. etc. All this "should" be obvious. However, there are lots of people who use openssl but don't know much about security in general or crypto in particular. Therefore the limitations need to be prominently mentioned in the openssl documentation. That's not sufficient by itself, but it counts as a step in the right direction. Series metaphor: A chain is only as strong as its weakest link. Parallel metaphor: A fence is only as strong as its weakest picket. One of the defining properties of modern civilization is division of labor. You simply cannot have everybody be responsible for everything. We make jokes when the division is done improperly: https://www.av8n.com/physics/not-my-job.htm but even so, a great many divisions are necessary. -- If you hire a locksmith to install a deadbolt on your front door, that does not make him responsible for the fact that the back door is made of balsa wood, and the side door is standing wide open. ++ If you hire a security consultant to look at the big picture, that's different. +- The locksmith MAY remark on security problems that he notices, but he is not obliged to fix them, or even to search for them. In this parable, openssl is the locksmith. The project does not have the resources to do much beyond basic locksmithing. ======== There are deep issues here that I don't know how to solve. For example, what should be done when the situation is discovered to be insecure? a) Ignore the discovery and soldier on? b) Print a warning? c) Block? Blocking generally infuriates the users. They replace the offending software with something that doesn't block, even if it's less secure. Warnings aren't much better. It does little good to print a warning about a problem that the user does not understand and could not solve. Ignorance is not bliss. Into the valley of death rode the six hundred. Asking the user a question that has no good answers, e.g. to choose between /dev/random and /dev/urandom, does not solve the problem at all. ================================= On 06/29/2017 10:07 AM, Theo de Raadt wrote: >> As has been said many times before, what we need (but do not have) >> is /one/ source of randomness that never blocks and never returns >> bits that are guessable by the adversary. > I've been preaching this for more than a decade, and that is exactly > what I built in OpenBSD. > It isn't very hard to do this properly in the kernel. Sometimes it's not very hard ... but sometimes it's provably impossible, depending on what sort of support is available from the hardware. I stand by the assertion that some platforms are not secure and cannot be made secure. RNGs are one manifestation of this, among others. From uri at ll.mit.edu Thu Jun 29 18:45:40 2017 From: uri at ll.mit.edu (Blumenthal, Uri - 0553 - MITLL) Date: Thu, 29 Jun 2017 18:45:40 +0000 Subject: [openssl-dev] what's possible and what's not ... including RNGs In-Reply-To: <9056490c-34c2-e946-f482-2416e7b82597@av8n.com> References: <105446b8af7a40baad217032ab0a1b82@usma1ex-dag1mb1.msg.corp.akamai.com> <2e17a0bf-5dc0-5ce5-e5ce-62948c501a3b@av8n.com> <84c27de9ab124c52a618a0873b6ad765@usma1ex-dag1mb1.msg.corp.akamai.com> <7246d73f-bb02-6730-3d9a-67b76b9ec296@av8n.com> <03b76cffb35042cc9c325b7f5dda1c8d@usma1ex-dag1mb1.msg.corp.akamai.com> <2571d92b-65b5-b587-a20b-c9bee040a98e@av8n.com> <20170626201839.3e7as7y6sxuic7dk@roeckx.be> <97aef447-9dfd-5479-c4df-08d2c62ea8f1@openssl.org> <20170627204041.h2sr3zovsnutj43y@thunk.org> <2314c104-e850-e629-f692-4f3f6aebb5e8@av8n.com> <9056490c-34c2-e946-f482-2416e7b82597@av8n.com> Message-ID: Knowledge of the platform is a required part of the OpenSSL configuration. If the platform supports HRNG (usually in the form of CPU instructions), use it: let OpenSSL mix its output with whatever other randomness sources it picks on that platform/system. IMHO that?s the best strategy. Thankfully, many of the newer platforms support those instructions. For those that don?t ? you?d have to either rely on the OS, or try to play OS (which is difficult if the OS is not friendly, and impossible if the OS is hostile). PGP used to collect randomness from the user keyboard input. That may be fine for some applications ? but a no-go for a library, IMHO. -- Regards, Uri Blumenthal -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5211 bytes Desc: not available URL: