From mtstickney at gmail.com Tue Aug 1 01:07:07 2017 From: mtstickney at gmail.com (Matthew Stickney) Date: Mon, 31 Jul 2017 21:07:07 -0400 Subject: [openssl-dev] Build issue In-Reply-To: References: <20170731.161705.1568825245335180776.levitte@openssl.org> Message-ID: I swear I sent my last reply to the list; either my mail client is messing with me, or I'm going crazy. There's no reason for this to be off-list, and that wasn't my intention in the first place. Anyhow: util/libcrypto.num doesn't seem to have anything unusual in it associated with '_num', so that's fine, but then neither does crypto.def, so it seems like it's not an issue with util/mkdef.pl. I mean, obviously the error is /somewhere/.... -Matt Stickney On Mon, Jul 31, 2017 at 10:22 AM, Matthew Stickney wrote: > Thanks for the suggestion, and my apologies: I didn't intend to take > this off-list. I'll check libcrypto.num when I get back on the machine > tonight. If it helps, I am building this with 64-bit mingw, but I > don't see how that would have an effect (there are build warnings > earlier in the process, but from what I've seen they look harmless). > > -Matt Stickney > > On Mon, Jul 31, 2017 at 10:17 AM, Richard Levitte wrote: >> util/mkdef.pl picks up all the data from configdata.pm, and regarding >> options, it looks at $config{options}, which includes everything >> that's disabled by default. >> >> This all is an issue that I'm currently labeling *weird*, 'cause it >> makes no sense at all. Most of all, this line really has me gawking, >> scratching my head and generally looking silly: >> >> Error: _num does not have a number assigned >> >> Basically, it looks like *something* is getting ba-a-a-a-dly parsed, >> and I have no idea if that's in util/libcrypto.num (could you try and >> grep for '_num' and possibly '^_num' to see if you can find something >> weird?) or if it's a serious corner case cock up in util/mkdef.pl. >> >> Worst thing is, I haven't been able to reproduce... I am going to try >> again just now, with the hope that 32-bit mingw doesn't make a >> difference... >> >> Cheers, >> Richard >> >> In message on Mon, 31 Jul 2017 09:02:15 -0500, Benjamin Kaduk said: >> >> bkaduk> Whoops, I missed that this was unicast when skimming mail over the weekend. >> bkaduk> Looping back in Richard, at least, though if you don't mind, putting the list back would be best. >> bkaduk> >> bkaduk> Richard, my (extremely) quick skim of mkdef.pl is that it is only picking up explicitly set options >> bkaduk> and not things disabled by default; does that seem plausible? >> bkaduk> >> bkaduk> -Ben >> bkaduk> >> bkaduk> On 07/28/2017 10:38 PM, Matthew Stickney wrote: >> bkaduk> >> bkaduk> [1] has the contents of my crypto.def file, although I don't see >> bkaduk> anything especially odd in there (not that I would know, really). >> bkaduk> >> bkaduk> I removed the offending symbols from crypto.def, and that caused the >> bkaduk> previously-failing command to succeed. I added some print statements >> bkaduk> to mkdef.pl and configured with "./configure mingw64 no-rc5 no-md2", >> bkaduk> and mkdef.pl appeared to be adding them to its list of disabled >> bkaduk> algorithms, but the MD2 and RC5 symbols are still in crypto.def. I'm >> bkaduk> no perl hacker, but I'll play around with it a bit. >> bkaduk> >> bkaduk> If I hand-remove the undefined symbols, building libssl fails with >> bkaduk> undefined references if I run make again, and if I run make again >> bkaduk> after that, building engines/e_capi.o fails with similar errors >> bkaduk> (probably related, but seems strange that libssl is ignored after the >> bkaduk> first failure). Some logs below. >> bkaduk> >> bkaduk> -Matt Stickney >> bkaduk> >> bkaduk> [1] https://gist.github.com/mtstickney/eac60bb98cafc9b978407cbb0a466cdc >> bkaduk> [2] http://openssl.6102.n7.nabble.com/Issues-with-latest-snapshot-td36148.html >> bkaduk> >> bkaduk> perl ./util/mkrc.pl libssl-1_1-x64.dll | windres --target=pe-x86-64 -o rc.o >> bkaduk> LD_LIBRARY_PATH=.: gcc -DDSO_WIN32 -DNDEBUG -DOPENSSL_THREADS -DOPENSSL_NO_STATI >> bkaduk> C_ENGINE -DOPENSSL_PIC -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_AS >> bkaduk> M_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DRC4_ASM -DM >> bkaduk> D5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DPADLOC >> bkaduk> K_ASM -DPOLY1305_ASM -DOPENSSLDIR="/usr/local/ssl" -DENGINESDIR="/usr/local/lib/ >> bkaduk> engines-1_1" -DL_ENDIAN -DWIN32_LEAN_AND_MEAN -DUNICODE -D_UNICODE -m64 -Wall -O >> bkaduk> 3 -D_MT -D_WINDLL -static-libgcc -shared -Wl,-Bsymbolic -Wl,--out-implib,libssl. >> bkaduk> dll.a ssl.def rc.o -o libssl-1_1-x64.dll -Wl,--whole-archive libssl.a -Wl,--no-w >> bkaduk> hole-archive -L. -lcrypto -lws2_32 -lgdi32 -lcrypt32 >> bkaduk> libssl.a(ssl_init.o):ssl_init.c:(.text+0x276): undefined reference to `err_free_ >> bkaduk> strings_int' >> bkaduk> libssl.a(ssl_lib.o):ssl_lib.c:(.text+0x1d6c): undefined reference to `d2i_PUBKEY >> bkaduk> ' >> bkaduk> libssl.a(ssl_lib.o):ssl_lib.c:(.text+0x1ef9): undefined reference to `d2i_X509' >> bkaduk> libssl.a(ssl_lib.o):ssl_lib.c:(.text+0x7c80): undefined reference to `d2i_PUBKEY >> bkaduk> ' >> bkaduk> >> bkaduk> >> bkaduk> >> bkaduk> LD_LIBRARY_PATH=.: gcc -DDSO_WIN32 -DNDEBUG -DOPENSSL_THREADS >> bkaduk> -DOPENSSL_NO_STATIC_ENGINE -DOPENSSL_PIC -DOPENSSL_IA32_SSE2 >> bkaduk> -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m >> bkaduk> -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DRC4_ASM -DMD5_ASM -DAES_ASM >> bkaduk> -DVPAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DPADLOCK_ASM >> bkaduk> -DPOLY1305_ASM -DOPENSSLDIR="/usr/local/ssl" >> bkaduk> -DENGINESDIR="/usr/local/lib/engines-1_1" -DL_ENDIAN >> bkaduk> -DWIN32_LEAN_AND_MEAN -DUNICODE -D_UNICODE -m64 -Wall -O3 -D_MT >> bkaduk> -D_WINDLL -static-libgcc -shared -Wl,--enable-auto-image-base >> bkaduk> -Wl,-Bsymbolic -o engines/capi.dll engines/e_capi.o -L. -lcrypto >> bkaduk> -lws2_32 -lgdi32 -lcrypt32 >> bkaduk> engines/e_capi.o:e_capi.c:(.text+0x1d2e): undefined reference to `d2i_X509' >> bkaduk> engines/e_capi.o:e_capi.c:(.text+0x2e9a): undefined reference to `d2i_X509' >> bkaduk> engines/e_capi.o:e_capi.c:(.text+0x3283): undefined reference to `d2i_X509' >> bkaduk> >> bkaduk> >> bkaduk> On Fri, Jul 28, 2017 at 8:29 AM, Benjamin Kaduk wrote: >> bkaduk> >> bkaduk> On 07/28/2017 01:22 AM, Matthew Stickney wrote: >> bkaduk> >> bkaduk> With a make distclean, ./config, make depend (didn't appear to do >> bkaduk> anything), and a make, I'm getting the essentially the same thing: >> bkaduk> >> bkaduk> Error: _num does not have a number assigned >> bkaduk> /usr/bin/perl ./util/mkrc.pl libcrypto-1_1-x64.dll | windres >> bkaduk> --target=pe-x86-64 >> bkaduk> -o rc.o >> bkaduk> LD_LIBRARY_PATH=: gcc -DDSO_WIN32 -DNDEBUG -DOPENSSL_THREADS >> bkaduk> -DOPENSSL_NO_STATIC >> bkaduk> _ENGINE -DOPENSSL_PIC -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT >> bkaduk> -DOPENSSL_BN_ASM >> bkaduk> _MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DRC4_ASM >> bkaduk> -DMD >> bkaduk> 5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM >> bkaduk> -DPADLOCK >> bkaduk> _ASM -DPOLY1305_ASM -DOPENSSLDIR="/usr/local/ssl" >> bkaduk> -DENGINESDIR="/usr/local/lib/e >> bkaduk> ngines-1_1" -DL_ENDIAN -DWIN32_LEAN_AND_MEAN -DUNICODE -D_UNICODE -m64 -Wall >> bkaduk> -O3 >> bkaduk> -D_MT -D_WINDLL -static-libgcc -shared -Wl,-Bsymbolic >> bkaduk> -Wl,--out-implib,libcrypt >> bkaduk> o.dll.a crypto.def rc.o -o libcrypto-1_1-x64.dll -Wl,--whole-archive >> bkaduk> libcrypto.a >> bkaduk> -Wl,--no-whole-archive -lws2_32 -lgdi32 -lcrypt32 >> bkaduk> Cannot export MD2: symbol not defined >> bkaduk> Cannot export MD2_Final: symbol not defined >> bkaduk> Cannot export MD2_Init: symbol not defined >> bkaduk> Cannot export MD2_Update: symbol not defined >> bkaduk> Cannot export MD2_options: symbol not defined >> bkaduk> Cannot export RC5_32_cbc_encrypt: symbol not defined >> bkaduk> Cannot export RC5_32_cfb64_encrypt: symbol not defined >> bkaduk> Cannot export RC5_32_decrypt: symbol not defined >> bkaduk> Cannot export RC5_32_ecb_encrypt: symbol not defined >> bkaduk> Cannot export RC5_32_encrypt: symbol not defined >> bkaduk> Cannot export RC5_32_ofb64_encrypt: symbol not defined >> bkaduk> Cannot export RC5_32_set_key: symbol not defined >> bkaduk> collect2.exe: error: ld returned 1 exit status >> bkaduk> >> bkaduk> >> bkaduk> MD2 and RC5 are disabled by default, so it is expected that they will not be >> bkaduk> defined. It is hard to say whether those messages are the source of the >> bkaduk> error exit status or just warnings, though. >> bkaduk> >> bkaduk> It's certainly plausible that there are further mkdef.pl issues responsible, >> bkaduk> though. Since mkdef.pl generates the crypto.def file referenced on your link >> bkaduk> line, maybe you could post that somewhere and link to it? (mkdef.pl can >> bkaduk> also be used to generate .map files, but it seems like the .def file is the >> bkaduk> relevant one at the moment.) >> bkaduk> >> bkaduk> But I may have to defer to Richard for the workings of mkdef.pl itself... >> bkaduk> >> bkaduk> -Ben >> bkaduk> From levitte at openssl.org Tue Aug 1 07:40:48 2017 From: levitte at openssl.org (Richard Levitte) Date: Tue, 01 Aug 2017 09:40:48 +0200 (CEST) Subject: [openssl-dev] Build issue In-Reply-To: References: <20170731.161705.1568825245335180776.levitte@openssl.org> Message-ID: <20170801.094048.630068859635955346.levitte@openssl.org> Ok, I have still not been able to reproduce this. We have already established that the perl you use is the mingw one, haven't we? (if we haven't, that really needs to be checked. Matching perl and all that) A test to figure out is this: perl util/mkdef.pl 32 crypto debug 2> mkdef-debug.txt > /dev/null Then let someone who knows mkdef look at the output. I would usually volunteer, but my time is currently limited (on vacation), so I won't be able to look at it before mid August. The debug output is messy, fair warning. Cheers, Richard In message on Mon, 31 Jul 2017 21:07:07 -0400, Matthew Stickney said: mtstickney> I swear I sent my last reply to the list; either my mail client is mtstickney> messing with me, or I'm going crazy. There's no reason for this to be mtstickney> off-list, and that wasn't my intention in the first place. mtstickney> mtstickney> Anyhow: util/libcrypto.num doesn't seem to have anything unusual in it mtstickney> associated with '_num', so that's fine, but then neither does mtstickney> crypto.def, so it seems like it's not an issue with util/mkdef.pl. I mtstickney> mean, obviously the error is /somewhere/.... mtstickney> mtstickney> -Matt Stickney mtstickney> mtstickney> On Mon, Jul 31, 2017 at 10:22 AM, Matthew Stickney wrote: mtstickney> > Thanks for the suggestion, and my apologies: I didn't intend to take mtstickney> > this off-list. I'll check libcrypto.num when I get back on the machine mtstickney> > tonight. If it helps, I am building this with 64-bit mingw, but I mtstickney> > don't see how that would have an effect (there are build warnings mtstickney> > earlier in the process, but from what I've seen they look harmless). mtstickney> > mtstickney> > -Matt Stickney mtstickney> > mtstickney> > On Mon, Jul 31, 2017 at 10:17 AM, Richard Levitte wrote: mtstickney> >> util/mkdef.pl picks up all the data from configdata.pm, and regarding mtstickney> >> options, it looks at $config{options}, which includes everything mtstickney> >> that's disabled by default. mtstickney> >> mtstickney> >> This all is an issue that I'm currently labeling *weird*, 'cause it mtstickney> >> makes no sense at all. Most of all, this line really has me gawking, mtstickney> >> scratching my head and generally looking silly: mtstickney> >> mtstickney> >> Error: _num does not have a number assigned mtstickney> >> mtstickney> >> Basically, it looks like *something* is getting ba-a-a-a-dly parsed, mtstickney> >> and I have no idea if that's in util/libcrypto.num (could you try and mtstickney> >> grep for '_num' and possibly '^_num' to see if you can find something mtstickney> >> weird?) or if it's a serious corner case cock up in util/mkdef.pl. mtstickney> >> mtstickney> >> Worst thing is, I haven't been able to reproduce... I am going to try mtstickney> >> again just now, with the hope that 32-bit mingw doesn't make a mtstickney> >> difference... mtstickney> >> mtstickney> >> Cheers, mtstickney> >> Richard mtstickney> >> mtstickney> >> In message on Mon, 31 Jul 2017 09:02:15 -0500, Benjamin Kaduk said: mtstickney> >> mtstickney> >> bkaduk> Whoops, I missed that this was unicast when skimming mail over the weekend. mtstickney> >> bkaduk> Looping back in Richard, at least, though if you don't mind, putting the list back would be best. mtstickney> >> bkaduk> mtstickney> >> bkaduk> Richard, my (extremely) quick skim of mkdef.pl is that it is only picking up explicitly set options mtstickney> >> bkaduk> and not things disabled by default; does that seem plausible? mtstickney> >> bkaduk> mtstickney> >> bkaduk> -Ben mtstickney> >> bkaduk> mtstickney> >> bkaduk> On 07/28/2017 10:38 PM, Matthew Stickney wrote: mtstickney> >> bkaduk> mtstickney> >> bkaduk> [1] has the contents of my crypto.def file, although I don't see mtstickney> >> bkaduk> anything especially odd in there (not that I would know, really). mtstickney> >> bkaduk> mtstickney> >> bkaduk> I removed the offending symbols from crypto.def, and that caused the mtstickney> >> bkaduk> previously-failing command to succeed. I added some print statements mtstickney> >> bkaduk> to mkdef.pl and configured with "./configure mingw64 no-rc5 no-md2", mtstickney> >> bkaduk> and mkdef.pl appeared to be adding them to its list of disabled mtstickney> >> bkaduk> algorithms, but the MD2 and RC5 symbols are still in crypto.def. I'm mtstickney> >> bkaduk> no perl hacker, but I'll play around with it a bit. mtstickney> >> bkaduk> mtstickney> >> bkaduk> If I hand-remove the undefined symbols, building libssl fails with mtstickney> >> bkaduk> undefined references if I run make again, and if I run make again mtstickney> >> bkaduk> after that, building engines/e_capi.o fails with similar errors mtstickney> >> bkaduk> (probably related, but seems strange that libssl is ignored after the mtstickney> >> bkaduk> first failure). Some logs below. mtstickney> >> bkaduk> mtstickney> >> bkaduk> -Matt Stickney mtstickney> >> bkaduk> mtstickney> >> bkaduk> [1] https://gist.github.com/mtstickney/eac60bb98cafc9b978407cbb0a466cdc mtstickney> >> bkaduk> [2] http://openssl.6102.n7.nabble.com/Issues-with-latest-snapshot-td36148.html mtstickney> >> bkaduk> mtstickney> >> bkaduk> perl ./util/mkrc.pl libssl-1_1-x64.dll | windres --target=pe-x86-64 -o rc.o mtstickney> >> bkaduk> LD_LIBRARY_PATH=.: gcc -DDSO_WIN32 -DNDEBUG -DOPENSSL_THREADS -DOPENSSL_NO_STATI mtstickney> >> bkaduk> C_ENGINE -DOPENSSL_PIC -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_AS mtstickney> >> bkaduk> M_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DRC4_ASM -DM mtstickney> >> bkaduk> D5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DPADLOC mtstickney> >> bkaduk> K_ASM -DPOLY1305_ASM -DOPENSSLDIR="/usr/local/ssl" -DENGINESDIR="/usr/local/lib/ mtstickney> >> bkaduk> engines-1_1" -DL_ENDIAN -DWIN32_LEAN_AND_MEAN -DUNICODE -D_UNICODE -m64 -Wall -O mtstickney> >> bkaduk> 3 -D_MT -D_WINDLL -static-libgcc -shared -Wl,-Bsymbolic -Wl,--out-implib,libssl. mtstickney> >> bkaduk> dll.a ssl.def rc.o -o libssl-1_1-x64.dll -Wl,--whole-archive libssl.a -Wl,--no-w mtstickney> >> bkaduk> hole-archive -L. -lcrypto -lws2_32 -lgdi32 -lcrypt32 mtstickney> >> bkaduk> libssl.a(ssl_init.o):ssl_init.c:(.text+0x276): undefined reference to `err_free_ mtstickney> >> bkaduk> strings_int' mtstickney> >> bkaduk> libssl.a(ssl_lib.o):ssl_lib.c:(.text+0x1d6c): undefined reference to `d2i_PUBKEY mtstickney> >> bkaduk> ' mtstickney> >> bkaduk> libssl.a(ssl_lib.o):ssl_lib.c:(.text+0x1ef9): undefined reference to `d2i_X509' mtstickney> >> bkaduk> libssl.a(ssl_lib.o):ssl_lib.c:(.text+0x7c80): undefined reference to `d2i_PUBKEY mtstickney> >> bkaduk> ' mtstickney> >> bkaduk> mtstickney> >> bkaduk> mtstickney> >> bkaduk> mtstickney> >> bkaduk> LD_LIBRARY_PATH=.: gcc -DDSO_WIN32 -DNDEBUG -DOPENSSL_THREADS mtstickney> >> bkaduk> -DOPENSSL_NO_STATIC_ENGINE -DOPENSSL_PIC -DOPENSSL_IA32_SSE2 mtstickney> >> bkaduk> -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m mtstickney> >> bkaduk> -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DRC4_ASM -DMD5_ASM -DAES_ASM mtstickney> >> bkaduk> -DVPAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DPADLOCK_ASM mtstickney> >> bkaduk> -DPOLY1305_ASM -DOPENSSLDIR="/usr/local/ssl" mtstickney> >> bkaduk> -DENGINESDIR="/usr/local/lib/engines-1_1" -DL_ENDIAN mtstickney> >> bkaduk> -DWIN32_LEAN_AND_MEAN -DUNICODE -D_UNICODE -m64 -Wall -O3 -D_MT mtstickney> >> bkaduk> -D_WINDLL -static-libgcc -shared -Wl,--enable-auto-image-base mtstickney> >> bkaduk> -Wl,-Bsymbolic -o engines/capi.dll engines/e_capi.o -L. -lcrypto mtstickney> >> bkaduk> -lws2_32 -lgdi32 -lcrypt32 mtstickney> >> bkaduk> engines/e_capi.o:e_capi.c:(.text+0x1d2e): undefined reference to `d2i_X509' mtstickney> >> bkaduk> engines/e_capi.o:e_capi.c:(.text+0x2e9a): undefined reference to `d2i_X509' mtstickney> >> bkaduk> engines/e_capi.o:e_capi.c:(.text+0x3283): undefined reference to `d2i_X509' mtstickney> >> bkaduk> mtstickney> >> bkaduk> mtstickney> >> bkaduk> On Fri, Jul 28, 2017 at 8:29 AM, Benjamin Kaduk wrote: mtstickney> >> bkaduk> mtstickney> >> bkaduk> On 07/28/2017 01:22 AM, Matthew Stickney wrote: mtstickney> >> bkaduk> mtstickney> >> bkaduk> With a make distclean, ./config, make depend (didn't appear to do mtstickney> >> bkaduk> anything), and a make, I'm getting the essentially the same thing: mtstickney> >> bkaduk> mtstickney> >> bkaduk> Error: _num does not have a number assigned mtstickney> >> bkaduk> /usr/bin/perl ./util/mkrc.pl libcrypto-1_1-x64.dll | windres mtstickney> >> bkaduk> --target=pe-x86-64 mtstickney> >> bkaduk> -o rc.o mtstickney> >> bkaduk> LD_LIBRARY_PATH=: gcc -DDSO_WIN32 -DNDEBUG -DOPENSSL_THREADS mtstickney> >> bkaduk> -DOPENSSL_NO_STATIC mtstickney> >> bkaduk> _ENGINE -DOPENSSL_PIC -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT mtstickney> >> bkaduk> -DOPENSSL_BN_ASM mtstickney> >> bkaduk> _MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DRC4_ASM mtstickney> >> bkaduk> -DMD mtstickney> >> bkaduk> 5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM mtstickney> >> bkaduk> -DPADLOCK mtstickney> >> bkaduk> _ASM -DPOLY1305_ASM -DOPENSSLDIR="/usr/local/ssl" mtstickney> >> bkaduk> -DENGINESDIR="/usr/local/lib/e mtstickney> >> bkaduk> ngines-1_1" -DL_ENDIAN -DWIN32_LEAN_AND_MEAN -DUNICODE -D_UNICODE -m64 -Wall mtstickney> >> bkaduk> -O3 mtstickney> >> bkaduk> -D_MT -D_WINDLL -static-libgcc -shared -Wl,-Bsymbolic mtstickney> >> bkaduk> -Wl,--out-implib,libcrypt mtstickney> >> bkaduk> o.dll.a crypto.def rc.o -o libcrypto-1_1-x64.dll -Wl,--whole-archive mtstickney> >> bkaduk> libcrypto.a mtstickney> >> bkaduk> -Wl,--no-whole-archive -lws2_32 -lgdi32 -lcrypt32 mtstickney> >> bkaduk> Cannot export MD2: symbol not defined mtstickney> >> bkaduk> Cannot export MD2_Final: symbol not defined mtstickney> >> bkaduk> Cannot export MD2_Init: symbol not defined mtstickney> >> bkaduk> Cannot export MD2_Update: symbol not defined mtstickney> >> bkaduk> Cannot export MD2_options: symbol not defined mtstickney> >> bkaduk> Cannot export RC5_32_cbc_encrypt: symbol not defined mtstickney> >> bkaduk> Cannot export RC5_32_cfb64_encrypt: symbol not defined mtstickney> >> bkaduk> Cannot export RC5_32_decrypt: symbol not defined mtstickney> >> bkaduk> Cannot export RC5_32_ecb_encrypt: symbol not defined mtstickney> >> bkaduk> Cannot export RC5_32_encrypt: symbol not defined mtstickney> >> bkaduk> Cannot export RC5_32_ofb64_encrypt: symbol not defined mtstickney> >> bkaduk> Cannot export RC5_32_set_key: symbol not defined mtstickney> >> bkaduk> collect2.exe: error: ld returned 1 exit status mtstickney> >> bkaduk> mtstickney> >> bkaduk> mtstickney> >> bkaduk> MD2 and RC5 are disabled by default, so it is expected that they will not be mtstickney> >> bkaduk> defined. It is hard to say whether those messages are the source of the mtstickney> >> bkaduk> error exit status or just warnings, though. mtstickney> >> bkaduk> mtstickney> >> bkaduk> It's certainly plausible that there are further mkdef.pl issues responsible, mtstickney> >> bkaduk> though. Since mkdef.pl generates the crypto.def file referenced on your link mtstickney> >> bkaduk> line, maybe you could post that somewhere and link to it? (mkdef.pl can mtstickney> >> bkaduk> also be used to generate .map files, but it seems like the .def file is the mtstickney> >> bkaduk> relevant one at the moment.) mtstickney> >> bkaduk> mtstickney> >> bkaduk> But I may have to defer to Richard for the workings of mkdef.pl itself... mtstickney> >> bkaduk> mtstickney> >> bkaduk> -Ben mtstickney> >> bkaduk> mtstickney> -- mtstickney> openssl-dev mailing list mtstickney> To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev mtstickney> From sfhacker at hotmail.com Tue Aug 1 12:54:11 2017 From: sfhacker at hotmail.com (Sergio NNX) Date: Tue, 1 Aug 2017 12:54:11 +0000 Subject: [openssl-dev] Build issue In-Reply-To: <20170801.094048.630068859635955346.levitte@openssl.org> References: <20170731.161705.1568825245335180776.levitte@openssl.org> , <20170801.094048.630068859635955346.levitte@openssl.org> Message-ID: > Ok, I have still not been able to reproduce this. Neither have I! I've just downloaded OpenSSL 1.1.0f and built it from source. This is my configuration: Microsoft Windows [Version 10.0.15063] This is perl 5, version 24, subversion 1 (v5.24.1) GCC v6.3.0 (64-bit) ./config mingw64 ........ make depend make all After a while, I get the executable and libraries. OpenSSL 1.1.0f 25 May 2017 built on: reproducible build, date unspecified platform: mingw64 perl util/mkdef.pl 32 crypto debug 2> mkdef-debug.txt > /dev/null [It generates a huge file (125MB)] Cheers. Ser/ P.S.: Try to use a more specific subject next time! -------------- next part -------------- An HTML attachment was scrubbed... URL: From mtstickney at gmail.com Wed Aug 2 02:19:20 2017 From: mtstickney at gmail.com (Matthew Stickney) Date: Tue, 1 Aug 2017 22:19:20 -0400 Subject: [openssl-dev] Build issue In-Reply-To: References: <20170731.161705.1568825245335180776.levitte@openssl.org> <20170801.094048.630068859635955346.levitte@openssl.org> Message-ID: Ok, progress (sort of)! It turns out I was indeed using the wrong version of perl -- I was using the perl that was installed as part of the msys2 base-devel group, not the mingw-w64--perl package, which is a whole separate thing. However, having installed the mingw-w64 version of perl, the configure script is failing because it thinks this is a unix-like platform, but perl is producing windows-style paths: > This perl implementation doesn't produce Unix like paths (with forward slash > directory separators). Please use an implementation that matches your > building platform. > This Perl version: 5.22.0 for MSWin32-x64-multi-thread I can't follow the configure script well enough to tell where it's detecting the relevant bits from -- the build_scheme key seems to be responsible for triggering the unix-checker.pm script that fails, but grep only turns up two references to it: the readme, and the configure script. Is this an issue with msys2+mingw-w64 environments? -Matt Stickney On Tue, Aug 1, 2017 at 8:54 AM, Sergio NNX wrote: >> Ok, I have still not been able to reproduce this. > > Neither have I! > > > I've just downloaded OpenSSL 1.1.0f and built it from source. > > This is my configuration: > > > Microsoft Windows [Version 10.0.15063] > > This is perl 5, version 24, subversion 1 (v5.24.1) > > GCC v6.3.0 (64-bit) > > > ./config mingw64 ........ > > make depend > > make all > > > After a while, I get the executable and libraries. > > > OpenSSL 1.1.0f 25 May 2017 > built on: reproducible build, date unspecified > platform: mingw64 > > > perl util/mkdef.pl 32 crypto debug 2> mkdef-debug.txt > /dev/null > [It generates a huge file (125MB)] > > Cheers. > > Ser/ > > > P.S.: Try to use a more specific subject next time! From levitte at openssl.org Wed Aug 2 06:53:01 2017 From: levitte at openssl.org (Richard Levitte) Date: Wed, 02 Aug 2017 08:53:01 +0200 (CEST) Subject: [openssl-dev] Build issue In-Reply-To: References: <20170801.094048.630068859635955346.levitte@openssl.org> Message-ID: <20170802.085301.116349862633576695.levitte@openssl.org> In message on Tue, 1 Aug 2017 22:19:20 -0400, Matthew Stickney said: mtstickney> However, having installed the mingw-w64 version of perl, the configure mtstickney> script is failing because it thinks this is a unix-like platform, but mtstickney> perl is producing windows-style paths: mtstickney> mtstickney> > This perl implementation doesn't produce Unix like paths (with forward slash mtstickney> > directory separators). Please use an implementation that matches your mtstickney> > building platform. mtstickney> mtstickney> > This Perl version: 5.22.0 for MSWin32-x64-multi-thread mtstickney> mtstickney> I can't follow the configure script well enough to tell where it's mtstickney> detecting the relevant bits from -- the build_scheme key seems to be mtstickney> responsible for triggering the unix-checker.pm script that fails, but mtstickney> grep only turns up two references to it: the readme, and the configure mtstickney> script. Is this an issue with msys2+mingw-w64 environments? OR unix-checker.pem might be buggy. May I suggest insert this line before the check, try again and see what it says? print STDERR "Current directory: ", rel2abs('.'), "\n"; -- Richard Levitte levitte at openssl.org OpenSSL Project http://www.openssl.org/~levitte/ From zlm2006 at gmail.com Wed Aug 2 07:10:34 2017 From: zlm2006 at gmail.com (Lingmo Zhu) Date: Wed, 2 Aug 2017 15:10:34 +0800 Subject: [openssl-dev] Fix session resumption issue when using external cache Message-ID: Hi, I found that the session resumption process may become corrupted when using external cache for version >= 1.1.0. It is almost ensure that all valid SSL_SESSION returned by get_session_cb() will cause session resumption failing unless SSL_SESS_CACHE_NO_INTERNAL_STORE is set. A PR has been filed to solve this problem: https://github.com/openssl/openssl/pull/4014 . The description of the PR should be enough. Thank you. Lingmo Zhu From matt at openssl.org Wed Aug 2 08:07:18 2017 From: matt at openssl.org (Matt Caswell) Date: Wed, 2 Aug 2017 09:07:18 +0100 Subject: [openssl-dev] Build issue In-Reply-To: References: <20170731.161705.1568825245335180776.levitte@openssl.org> <20170801.094048.630068859635955346.levitte@openssl.org> Message-ID: On 02/08/17 03:19, Matthew Stickney wrote: > Ok, progress (sort of)! It turns out I was indeed using the wrong > version of perl -- I was using the perl that was installed as part of > the msys2 base-devel group, not the mingw-w64--perl package, > which is a whole separate thing. Errr no. You were right the first time - you should be using the msys2 perl. This is my perl --version output: This is perl 5, version 22, subversion 1 (v5.22.1) built for x86_64-msys-thread-multi Do NOT use the mingw perl (I think Richard misspoke when he said this before - the "matching perl" requirement means that using msys2 shell requires msys2 perl). Just to check - you are using the msys2 shell right? Matt > > However, having installed the mingw-w64 version of perl, the configure > script is failing because it thinks this is a unix-like platform, but > perl is producing windows-style paths: > >> This perl implementation doesn't produce Unix like paths (with forward slash >> directory separators). Please use an implementation that matches your >> building platform. > >> This Perl version: 5.22.0 for MSWin32-x64-multi-thread > > I can't follow the configure script well enough to tell where it's > detecting the relevant bits from -- the build_scheme key seems to be > responsible for triggering the unix-checker.pm script that fails, but > grep only turns up two references to it: the readme, and the configure > script. Is this an issue with msys2+mingw-w64 environments? > > -Matt Stickney > > On Tue, Aug 1, 2017 at 8:54 AM, Sergio NNX wrote: >>> Ok, I have still not been able to reproduce this. >> >> Neither have I! >> >> >> I've just downloaded OpenSSL 1.1.0f and built it from source. >> >> This is my configuration: >> >> >> Microsoft Windows [Version 10.0.15063] >> >> This is perl 5, version 24, subversion 1 (v5.24.1) >> >> GCC v6.3.0 (64-bit) >> >> >> ./config mingw64 ........ >> >> make depend >> >> make all >> >> >> After a while, I get the executable and libraries. >> >> >> OpenSSL 1.1.0f 25 May 2017 >> built on: reproducible build, date unspecified >> platform: mingw64 >> >> >> perl util/mkdef.pl 32 crypto debug 2> mkdef-debug.txt > /dev/null >> [It generates a huge file (125MB)] >> >> Cheers. >> >> Ser/ >> >> >> P.S.: Try to use a more specific subject next time! From rsalz at akamai.com Wed Aug 2 12:41:21 2017 From: rsalz at akamai.com (Salz, Rich) Date: Wed, 2 Aug 2017 12:41:21 +0000 Subject: [openssl-dev] Code health tuesday is back! Message-ID: <3c01f8ac264a4e68b2793607c84208ae@usma1ex-dag1mb1.msg.corp.akamai.com> After a short summer vacation, our biweekly code health Tuesday is back! Our topic this time is ... documentation. There have been many updates to the manpages in the past few weeks, typo fixes, additional clarifications, and so on. We hope that folks will be emboldened to help fill in the gaps, but any PR to make things better will help. Please submit your fixes by Tuesday; if you can't add a label, put 'code health' somewhere in the commit message. Please have a CLA onfile; if your commit is trivial and not copyrightable, put "CLA: trivial" in the commit message. If you have a whole bunch of trivial fixes, put them in one PR (separate commits if you want). Make sure any changes pass find-doc-nits (a script in util). You can also use that script to list places where documentation is missing: ; ./util/find-doc-nits -u | fgrep '#' # Found 4373 in util/libcrypto.num # Found 1724 missing from util/libcrypto.num # Found 464 in util/libssl.num # Found 64 missing from util/libssl.num # Checking macros (approximate) # Found 246 macros missing (not all should be documnted) Thanks for all your help in improving OpenSSL! -------------- next part -------------- An HTML attachment was scrubbed... URL: From mtstickney at gmail.com Thu Aug 3 02:56:48 2017 From: mtstickney at gmail.com (Matthew Stickney) Date: Wed, 2 Aug 2017 22:56:48 -0400 Subject: [openssl-dev] Build issue In-Reply-To: References: <20170731.161705.1568825245335180776.levitte@openssl.org> <20170801.094048.630068859635955346.levitte@openssl.org> Message-ID: Aha, victory! Turns out this was a line-ending problem: after removing the mingw perl, the mkdef.pl debug output had (among many, many other things), the following when I went looking for MD2 references: > DEBUG: include/openssl/md2.h: found tag OPENSSL_NO_MD2^M = -1 I checked, and sure enough the md2 header had windows line endings. core.autocrlf was false, and core.eol was lf, so I'm not sure how only some headers had the wrong line endings, but an "rm -rf * && git checkout ." produced a working build. Thanks for the time and assistance, everybody. -Matt Stickney On Wed, Aug 2, 2017 at 4:07 AM, Matt Caswell wrote: > > > On 02/08/17 03:19, Matthew Stickney wrote: >> Ok, progress (sort of)! It turns out I was indeed using the wrong >> version of perl -- I was using the perl that was installed as part of >> the msys2 base-devel group, not the mingw-w64--perl package, >> which is a whole separate thing. > > > Errr no. You were right the first time - you should be using the msys2 > perl. This is my perl --version output: > > This is perl 5, version 22, subversion 1 (v5.22.1) built for > x86_64-msys-thread-multi > > Do NOT use the mingw perl (I think Richard misspoke when he said this > before - the "matching perl" requirement means that using msys2 shell > requires msys2 perl). > > Just to check - you are using the msys2 shell right? > > Matt > > >> >> However, having installed the mingw-w64 version of perl, the configure >> script is failing because it thinks this is a unix-like platform, but >> perl is producing windows-style paths: >> >>> This perl implementation doesn't produce Unix like paths (with forward slash >>> directory separators). Please use an implementation that matches your >>> building platform. >> >>> This Perl version: 5.22.0 for MSWin32-x64-multi-thread >> >> I can't follow the configure script well enough to tell where it's >> detecting the relevant bits from -- the build_scheme key seems to be >> responsible for triggering the unix-checker.pm script that fails, but >> grep only turns up two references to it: the readme, and the configure >> script. Is this an issue with msys2+mingw-w64 environments? >> >> -Matt Stickney >> >> On Tue, Aug 1, 2017 at 8:54 AM, Sergio NNX wrote: >>>> Ok, I have still not been able to reproduce this. >>> >>> Neither have I! >>> >>> >>> I've just downloaded OpenSSL 1.1.0f and built it from source. >>> >>> This is my configuration: >>> >>> >>> Microsoft Windows [Version 10.0.15063] >>> >>> This is perl 5, version 24, subversion 1 (v5.24.1) >>> >>> GCC v6.3.0 (64-bit) >>> >>> >>> ./config mingw64 ........ >>> >>> make depend >>> >>> make all >>> >>> >>> After a while, I get the executable and libraries. >>> >>> >>> OpenSSL 1.1.0f 25 May 2017 >>> built on: reproducible build, date unspecified >>> platform: mingw64 >>> >>> >>> perl util/mkdef.pl 32 crypto debug 2> mkdef-debug.txt > /dev/null >>> [It generates a huge file (125MB)] >>> >>> Cheers. >>> >>> Ser/ >>> >>> >>> P.S.: Try to use a more specific subject next time! > -- > openssl-dev mailing list > To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev From uri at ll.mit.edu Thu Aug 3 19:09:35 2017 From: uri at ll.mit.edu (Blumenthal, Uri - 0553 - MITLL) Date: Thu, 3 Aug 2017 15:09:35 -0400 Subject: [openssl-dev] Current master fails to build Message-ID: <53666239-905E-4101-ACA4-F0E8CEBD48B8@ll.mit.edu> See https://github.com/openssl/openssl/issues/4080 First, ssl/s3_lib.c file misses ?#include ? in the beginning. This prevents successful linking of libssl. After that is fixed, test 80-test_new_ssl.t fails (wstat 256, 0x100). ? Regards, Uri -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5734 bytes Desc: not available URL: From rsalz at akamai.com Thu Aug 3 19:46:53 2017 From: rsalz at akamai.com (Salz, Rich) Date: Thu, 3 Aug 2017 19:46:53 +0000 Subject: [openssl-dev] Current master fails to build In-Reply-To: <53666239-905E-4101-ACA4-F0E8CEBD48B8@ll.mit.edu> References: <53666239-905E-4101-ACA4-F0E8CEBD48B8@ll.mit.edu> Message-ID: <7417869a2b5d46cdbce6a5e6418faede@usma1ex-dag1mb1.msg.corp.akamai.com> > After that is fixed, test 80-test_new_ssl.t fails (wstat 256, 0x100). Do you have core file? The failure is not repeatable, at least on my system :( From uri at ll.mit.edu Thu Aug 3 20:14:04 2017 From: uri at ll.mit.edu (Blumenthal, Uri - 0553 - MITLL) Date: Thu, 3 Aug 2017 16:14:04 -0400 Subject: [openssl-dev] Current master fails to build In-Reply-To: <7417869a2b5d46cdbce6a5e6418faede@usma1ex-dag1mb1.msg.corp.akamai.com> References: <53666239-905E-4101-ACA4-F0E8CEBD48B8@ll.mit.edu> <7417869a2b5d46cdbce6a5e6418faede@usma1ex-dag1mb1.msg.corp.akamai.com> Message-ID: <20DB0F7B-ED8D-4645-9FD7-4345EBB0A718@ll.mit.edu> On Aug 3, 2017, at 15:46, Salz, Rich via openssl-dev wrote: > >> After that is fixed, test 80-test_new_ssl.t fails (wstat 256, 0x100). > > Do you have core file? The failure is not repeatable, at least on my system :( Alas, I don?t. But Travis CI fails this master 4 out of 14 jobs, see this for example: https://travis-ci.org/mouse07410/openssl/jobs/260690849 -------------- next part -------------- An HTML attachment was scrubbed... URL: From rsalz at akamai.com Mon Aug 7 12:36:58 2017 From: rsalz at akamai.com (Salz, Rich) Date: Mon, 7 Aug 2017 12:36:58 +0000 Subject: [openssl-dev] FW: Code health tuesday is back! In-Reply-To: <3c01f8ac264a4e68b2793607c84208ae@usma1ex-dag1mb1.msg.corp.akamai.com> References: <3c01f8ac264a4e68b2793607c84208ae@usma1ex-dag1mb1.msg.corp.akamai.com> Message-ID: <2bb4a3d76ad44a61903e11a17f9a64bf@usma1ex-dag1mb1.msg.corp.akamai.com> A reminder: After a short summer vacation, our biweekly code health Tuesday is back! Our topic this time is ? documentation. There have been many updates to the manpages in the past few weeks, typo fixes, additional clarifications, and so on.? We hope that folks will be emboldened to help fill in the gaps, but any PR to make things better will help. Please submit your fixes by Tuesday; if you can?t add a label, put ?code health? somewhere in the commit message.? Please have a CLA on file; if your commit is trivial and not copyrightable, put ?CLA: trivial? in the commit message.? If you have a whole bunch of trivial fixes, put them in one PR (separate commits if you want).? Make sure any changes pass find-doc-nits (a script in util).? You can also use that script to list places where documentation is missing: ; ./util/find-doc-nits -u | fgrep '#' # Found 4373 in util/libcrypto.num # Found 1724 missing from util/libcrypto.num # Found 464 in util/libssl.num # Found 64 missing from util/libssl.num # Checking macros (approximate) # Found 246 macros missing (not all should be documnted) Thanks for all your help in improving OpenSSL! From loganaden at gmail.com Wed Aug 9 13:03:47 2017 From: loganaden at gmail.com (Loganaden Velvindron) Date: Wed, 9 Aug 2017 17:03:47 +0400 Subject: [openssl-dev] draft-21 status Message-ID: Dear OpenSSL folks, I was wondering if there is a branch for draft-21 ? Kind regards, //Logan From bkaduk at akamai.com Wed Aug 9 13:05:29 2017 From: bkaduk at akamai.com (Benjamin Kaduk) Date: Wed, 9 Aug 2017 08:05:29 -0500 Subject: [openssl-dev] draft-21 status In-Reply-To: References: Message-ID: <8ba1ad02-e3cd-44d5-441c-487d9cd6510d@akamai.com> On 08/09/2017 08:03 AM, Loganaden Velvindron wrote: > Dear OpenSSL folks, > > I was wondering if there is a branch for draft-21 ? > draft-21 support is on master at the moment; there's no need for a separate branch until there is a draft-22 document to support. -Ben -------------- next part -------------- An HTML attachment was scrubbed... URL: From David.von.Oheimb at siemens.com Wed Aug 9 16:26:19 2017 From: David.von.Oheimb at siemens.com (David von Oheimb) Date: Wed, 9 Aug 2017 18:26:19 +0200 Subject: [openssl-dev] [openssl.org #4620] OCSP_basic_verify() question/comment In-Reply-To: <721270cb-d175-cb4c-2760-f54ba54c2729@siemens.com> References: <721270cb-d175-cb4c-2760-f54ba54c2729@siemens.com> Message-ID: It seems that OCSP_basic_verify(bs, certs, st, flags) unfortunately is not documented, but from its code it becomes clear that the "certs" parameter is meant to be a set of untrusted certificates, which is first used (together with bs->certs) to determine the signer cert of the OCSP response "bs" and then is partly(!) used to construct the chain of certs towards a trusted (root) cert in the store passed in the "st" parameter. >> OCSP responses do not seem to include the intermediate certificates so they >> have to be acquired in other ways. I have been doing this and adding them >> to the certificate stack handed to OCSP_basic_verify(). > > Perhaps adding them to X509_STORE or STORE_CTX directly? This does not work because OCSP_basic_verify(bs, certs, st, flags) produces its own internal X509_STORE_CTX: init_res = X509_STORE_CTX_init(ctx, st, signer, untrusted); where the "st" parameter is taken as the trusted store, while the set of untrusted certs cannot be directly set by the caller. >> I am relatively new to this so I may be incorrect; however, it seems to me >> that the certificates in the cert argument should be added to the >> X509_STORE_CTX. > > If you need to add certificates to validate a chain, it seems safer to explicitly add them to the store, not implicitly. As long as the OCSP response pointed to by "bs" includes a non-NULL bs->certs field, OCSP_basic_verify() takes the union of any certs in the "certs" parameter and in bs->certs as untrusted certs for chain construction, but if bs->certs is NULL, i.e. when the OCSP responder did not include any certs its response, for some reason OCSP_basic_verify() does not take "certs" but bs->certs, which corresponds to the empty set. I consider this a bug, for which I've just submitted a pull request: https://github.com/openssl/openssl/pull/4124 This fix takes "certs" as the set of untrusted certs in case bs->certs is NULL. As long as a fix is not yet available, one can use the following workaround: { X509 *dummy = X509_new(); (void)OCSP_basic_add1_cert(bs, dummy); X509_free(dummy); } before calling OCSP_basic_verify(bs, certs, st, flags); From lullajd at yahoo.com Mon Aug 14 07:20:40 2017 From: lullajd at yahoo.com (Jitendra Lulla) Date: Mon, 14 Aug 2017 07:20:40 +0000 (UTC) Subject: [openssl-dev] afalg with OpenSSL 1.1.0f 25 May 2017 References: <209232057.1092600.1502695240022.ref@mail.yahoo.com> Message-ID: <209232057.1092600.1502695240022@mail.yahoo.com> Hi, I am trying to use afalg on Linux 4.9.37 with OpenSSL 1.1.0f. I am facing 2 issues: ONE: when I issue the speed command, I see the following: [root at localhost apps]# ./openssl speed -evp aes-128-cbc -engine afalg invalid engine "afalg" 139853452924736:error:2506406A:DSO support routines:dlfcn_bind_func:could not bind to the requested symbol name:crypto/dso/dso_dlfcn.c:178:symname(bind_engine): /usr/local/lib64/engines-1.1/afalg.so: undefined symbol: bind_engine 139853452924736:error:2506C06A:DSO support routines:DSO_bind_func:could not bind to the requested symbol name:crypto/dso/dso_lib.c:185: 139853452924736:error:260B6068:engine routines:dynamic_load:DSO failure:crypto/engine/eng_dyn.c:427: 139853452924736:error:2606A074:engine routines:ENGINE_by_id:no such engine:crypto/engine/eng_list.c:339:id=afalg 139853452924736:error:25066067:DS nm afalg.so doesn't show bind_engine When I modify the openssl.cnf file with the engine name and the CIPHERS, still I dont get it working. The command output and the change in the openssl.cnf pasted at the end of the mail. TWO: I had to create a softlink to libcrypto.so.1.1 and libssl.so.1.1 like the following to make openssl command work: ln -s /usr/local/lib64/libssl.so.1.1 /lib64/libssl.so.1.1 ln -s /usr/local/lib64/libcrypto.so.1.1 /lib64/libcrypto.so.1.1 Is creating the softlinks a known issue and will be fixed? I have pasted the complete information about the OS/distro environment and installation commands I ran at the bottom. Could you please suggest what wrong I am doing to make afalg work. Thanks Jitendra Lulla ======== BEFORE INSTALLATION: [root at localhost jlulla]# rpm -qa |grep openssl openssl-1.0.1e-60.el7.x86_64 openssl-devel-1.0.1e-60.el7.x86_64 openssl-libs-1.0.1e-60.el7.x86_64 [root at localhost jlulla]# openssl version OpenSSL 1.0.1e-fips 11 Feb 2013 PLEASE SEE FROM HERE PLEASE SEE FROM HERE PLEASE SEE FROM HERE---------------------------- STEP 1 : SOURCE TAKEN FROM https://www.openssl.org/source/openssl-1.1.0f.tar.gz 2017-May-25 13:09:51 [root at localhost jlulla]# uname -a Linux localhost.localdomain 4.9.37 #1 SMP Fri Jul 21 04:52:46 PDT 2017 x86_64 x86_64 x86_64 GNU/Linux [root at localhost jlulla]# cat /etc/redhat-release Red Hat Enterprise Linux Server release 7.3 (Maipo) [root at localhost openssl-1.1.0f]# pwd /home/jlulla/openssl-1.1.0f STEP 2: [root at localhost openssl-1.1.0f]# ./config shared enable-engine enable-dso enable-afalgeng Operating system: x86_64-whatever-linux2 Configuring for linux-x86_64 Configuring OpenSSL version 1.1.0f (0x1010006fL) no-asan [default] OPENSSL_NO_ASAN no-crypto-mdebug [default] OPENSSL_NO_CRYPTO_MDEBUG no-crypto-mdebug-backtrace [default] OPENSSL_NO_CRYPTO_MDEBUG_BACKTRACE no-ec_nistp_64_gcc_128 [default] OPENSSL_NO_EC_NISTP_64_GCC_128 no-egd [default] OPENSSL_NO_EGD no-fuzz-afl [default] OPENSSL_NO_FUZZ_AFL no-fuzz-libfuzzer [default] OPENSSL_NO_FUZZ_LIBFUZZER no-heartbeats [default] OPENSSL_NO_HEARTBEATS no-md2 [default] OPENSSL_NO_MD2 (skip dir) no-msan [default] OPENSSL_NO_MSAN no-rc5 [default] OPENSSL_NO_RC5 (skip dir) no-sctp [default] OPENSSL_NO_SCTP no-ssl-trace [default] OPENSSL_NO_SSL_TRACE no-ssl3 [default] OPENSSL_NO_SSL3 no-ssl3-method [default] OPENSSL_NO_SSL3_METHOD no-ubsan [default] OPENSSL_NO_UBSAN no-unit-test [default] OPENSSL_NO_UNIT_TEST no-weak-ssl-ciphers [default] OPENSSL_NO_WEAK_SSL_CIPHERS no-zlib [default] no-zlib-dynamic [default] Configuring for linux-x86_64 CC =gcc CFLAG =-Wall -O3 -pthread -m64 -DL_ENDIAN -Wa,--noexecstack SHARED_CFLAG =-fPIC -DOPENSSL_USE_NODELETE DEFINES =DSO_DLFCN HAVE_DLFCN_H NDEBUG OPENSSL_THREADS OPENSSL_NO_STATIC_ENGINE OPENSSL_PIC OPENSSL_IA32_SSE2 OPENSSL_BN_ASM_MONT OPENSSL_BN_ASM_MONT5 OPENSSL_BN_ASM_GF2m SHA1_ASM SHA256_ASM SHA512_ASM RC4_ASM MD5_ASM AES_ASM VPAES_ASM BSAES_ASM GHASH_ASM ECP_NISTZ256_ASM PADLOCK_ASM POLY1305_ASM LFLAG = PLIB_LFLAG = EX_LIBS =-ldl APPS_OBJ = CPUID_OBJ =x86_64cpuid.o UPLINK_OBJ = BN_ASM =asm/x86_64-gcc.o x86_64-mont.o x86_64-mont5.o x86_64-gf2m.o rsaz_exp.o rsaz-x86_64.o rsaz-avx2.o EC_ASM =ecp_nistz256.o ecp_nistz256-x86_64.o DES_ENC =des_enc.o fcrypt_b.o AES_ENC =aes-x86_64.o vpaes-x86_64.o bsaes-x86_64.o aesni-x86_64.o aesni-sha1-x86_64.o aesni-sha256-x86_64.o aesni-mb-x86_64.o BF_ENC =bf_enc.o CAST_ENC =c_enc.o RC4_ENC =rc4-x86_64.o rc4-md5-x86_64.o RC5_ENC =rc5_enc.o MD5_OBJ_ASM =md5-x86_64.o SHA1_OBJ_ASM =sha1-x86_64.o sha256-x86_64.o sha512-x86_64.o sha1-mb-x86_64.o sha256-mb-x86_64.o RMD160_OBJ_ASM= CMLL_ENC =cmll-x86_64.o cmll_misc.o MODES_OBJ =ghash-x86_64.o aesni-gcm-x86_64.o PADLOCK_OBJ =e_padlock-x86_64.o CHACHA_ENC =chacha-x86_64.o POLY1305_OBJ =poly1305-x86_64.o BLAKE2_OBJ = PROCESSOR = RANLIB =ranlib ARFLAGS = PERL =/usr/bin/perl SIXTY_FOUR_BIT_LONG mode Configured for linux-x86_64. [root at localhost openssl-1.1.0f]# make didnt give any error. make install didnt give any error updatedb STEP 2: [root at localhost openssl-1.1.0f]# md5sum /home/jlulla/openssl-1.1.0f/libssl.so.1.1 /usr/local/lib64/libssl.so.1.1 5a7264a3ebda22989c831794d30677c5 /home/jlulla/openssl-1.1.0f/libssl.so.1.1 5a7264a3ebda22989c831794d30677c5 /usr/local/lib64/libssl.so.1.1 [root at localhost openssl-1.1.0f]# ls -l /home/jlulla/openssl-1.1.0f/libssl.so lrwxrwxrwx. 1 root root 13 Aug 10 23:45 /home/jlulla/openssl-1.1.0f/libssl.so -> libssl.so.1.1 [root at localhost apps]# pwd /home/jlulla/openssl-1.1.0f/apps [root at localhost apps]# ./openssl ./openssl: error while loading shared libraries: libssl.so.1.1: cannot open shared object file: No such file or directory [root at localhost apps]# ldd ./openssl linux-vdso.so.1 => (0x00007ffd02f3c000) libssl.so.1.1 => not found libcrypto.so.1.1 => not found libdl.so.2 => /lib64/libdl.so.2 (0x00007f0439fda000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f0439dbe000) libc.so.6 => /lib64/libc.so.6 (0x00007f04399fc000) /lib64/ld-linux-x86-64.so.2 (0x000055d618530000) [root at localhost apps]# ls -l /lib64/libssl.so.1.1 ls: cannot access /lib64/libssl.so.1.1: No such file or directory ln -s /usr/local/lib64/libssl.so.1.1 /lib64/libssl.so.1.1 STEP 3: [root at localhost apps]# ln -s /usr/local/lib64/libssl.so.1.1 /lib64/libssl.so.1.1 [root at localhost apps]# ls -l /lib64/libssl.so.1.1 lrwxrwxrwx. 1 root root 30 Aug 10 23:57 /lib64/libssl.so.1.1 -> /usr/local/lib64/libssl.so.1.1 [root at localhost apps]# ln -s /usr/local/lib64/libcrypto.so.1.1 /lib64/libcrypto.so.1.1 [root at localhost apps]# ls -l /lib64/libcrypto.so.1.1 lrwxrwxrwx. 1 root root 33 Aug 10 23:58 /lib64/libcrypto.so.1.1 -> /usr/local/lib64/libcrypto.so.1.1 [root at localhost apps]# ldd ./openssl linux-vdso.so.1 => (0x00007ffd37b5e000) libssl.so.1.1 => /lib64/libssl.so.1.1 (0x00007f82881f8000) libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007f8287d73000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f8287b6e000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f8287952000) libc.so.6 => /lib64/libc.so.6 (0x00007f8287591000) /lib64/ld-linux-x86-64.so.2 (0x000055f06bc67000) [root at localhost apps]# ./openssl version OpenSSL 1.1.0f 25 May 2017 STEP 4: [root at localhost afalg]# pwd /home/jlulla/openssl-1.1.0f/engines/afalg [root at localhost afalg]# ls afalg.so build.info e_afalg.c e_afalg.d e_afalg.ec e_afalg_err.c e_afalg_err.d e_afalg_err.h e_afalg_err.o e_afalg.h e_afalg.o [root at localhost afalg]# ldd afalg.so linux-vdso.so.1 => (0x00007fff1d7ef000) libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007f3a80166000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f3a7ff62000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f3a7fd45000) libc.so.6 => /lib64/libc.so.6 (0x00007f3a7f984000) /lib64/ld-linux-x86-64.so.2 (0x0000560991976000) [root at localhost afalg]# nm afalg.so 0000000000202060 d AFALG_error_init 0000000000202220 b AFALG_lib_error_code 0000000000202160 d AFALG_str_functs 0000000000202080 d AFALG_str_reasons 0000000000202210 B __bss_start 0000000000202210 b completed.6344 w __cxa_finalize@@GLIBC_2.2.5 0000000000000a00 t deregister_tm_clones 0000000000000a70 t __do_global_dtors_aux 0000000000201da0 t __do_global_dtors_aux_fini_array_entry 0000000000201db0 d __dso_handle 0000000000201db8 d _DYNAMIC 0000000000202210 D _edata 0000000000202228 B _end 0000000000000af0 T engine_load_afalg_int 0000000000000bb0 T ERR_AFALG_error U ERR_get_next_error_library@@OPENSSL_1_1_0 0000000000000b00 T ERR_load_AFALG_strings U ERR_load_strings@@OPENSSL_1_1_0 U ERR_put_error@@OPENSSL_1_1_0 0000000000000b60 T ERR_unload_AFALG_strings U ERR_unload_strings@@OPENSSL_1_1_0 0000000000000bf8 T _fini 0000000000000ab0 t frame_dummy 0000000000201d98 t __frame_dummy_init_array_entry 0000000000000eb8 r __FRAME_END__ 0000000000202000 d _GLOBAL_OFFSET_TABLE_ w __gmon_start__ 0000000000000970 T _init w _ITM_deregisterTMCloneTable w _ITM_registerTMCloneTable 0000000000201da8 d __JCR_END__ 0000000000201da8 d __JCR_LIST__ w _Jv_RegisterClasses 0000000000000a30 t register_tm_clones 0000000000202210 d __TMC_END__ [root at localhost openssl-1.1.0f]# locate afalg.so /home/jlulla/openssl-1.1.0f/engines/afalg/afalg.so /usr/local/lib64/engines-1.1/afalg.so [root at localhost openssl-1.1.0f]# md5sum /home/jlulla/openssl-1.1.0f/engines/afalg/afalg.so /usr/local/lib64/engines-1.1/afalg.so ec273faa6eb6e25f55e44a7d37adf195 /home/jlulla/openssl-1.1.0f/engines/afalg/afalg.so ec273faa6eb6e25f55e44a7d37adf195 /usr/local/lib64/engines-1.1/afalg.so [root at localhost apps]# date Fri Aug 11 00:01:10 PDT 2017 [root at localhost apps]# ls -l /usr/local/ssl/openssl.cnf -rw-r--r--. 1 root root 10771 Aug 10 23:47 /usr/local/ssl/openssl.cnf [root at localhost apps]# md5sum /usr/local/ssl/openssl.cnf /home/jlulla/openssl-1.1.0f/apps/openssl.cnf f697ef5df0d006882e6326606e8dbf4a /usr/local/ssl/openssl.cnf f697ef5df0d006882e6326606e8dbf4a /home/jlulla/openssl-1.1.0f/apps/openssl.cnf openssl.cnf has the following: .. .. # Extra OBJECT IDENTIFIER info: #oid_file = $ENV::HOME/.oid oid_section = new_oids # To use this configuration file with the "-extfile" option of the # "openssl x509" utility, name here the section containing the # X.509v3 extensions to use: # extensions = # (Alternatively, use a configuration file that has only # X.509v3 extensions in its main [= default] section.) .. .. [root at localhost apps]# ./openssl speed -evp aes-128-cbc -engine afalg invalid engine "afalg" 139853452924736:error:2506406A:DSO support routines:dlfcn_bind_func:could not bind to the requested symbol name:crypto/dso/dso_dlfcn.c:178:symname(bind_engine): /usr/local/lib64/engines-1.1/afalg.so: undefined symbol: bind_engine 139853452924736:error:2506C06A:DSO support routines:DSO_bind_func:could not bind to the requested symbol name:crypto/dso/dso_lib.c:185: 139853452924736:error:260B6068:engine routines:dynamic_load:DSO failure:crypto/engine/eng_dyn.c:427: 139853452924736:error:2606A074:engine routines:ENGINE_by_id:no such engine:crypto/engine/eng_list.c:339:id=afalg 139853452924736:error:25066067:DSO support routines:dlfcn_load:could not load the shared library:crypto/dso/dso_dlfcn.c:113:filename(libafalg.so): libafalg.so: cannot open shared object file: No such file or directory 139853452924736:error:25070067:DSO support routines:DSO_load:could not load the shared library:crypto/dso/dso_lib.c:161: 139853452924736:error:260B6084:engine routines:dynamic_load:dso not found:crypto/engine/eng_dyn.c:414: ^C Modified both the files as the follwing: /usr/local/ssl/openssl.cnf /home/jlulla/openssl-1.1.0f/apps/openssl.cnf # Extra OBJECT IDENTIFIER info: #oid_file = $ENV::HOME/.oid oid_section = new_oids openssl_conf = openssl_def [openssl_def] engines = openssl_engines [openssl_engines] afalg = afalg_engine [afalg_engine] CIPHERS=aes-128-cbc With the above, the errors are more: [root at localhost apps]# ./openssl speed -evp aes-128-cbc -engine afalg invalid engine "afalg" 140473780422464:error:2506406A:DSO support routines:dlfcn_bind_func:could not bind to the requested symbol name:crypto/dso/dso_dlfcn.c:178:symname(bind_engine): /usr/local/lib64/engines-1.1/afalg.so: undefined symbol: bind_engine 140473780422464:error:2506C06A:DSO support routines:DSO_bind_func:could not bind to the requested symbol name:crypto/dso/dso_lib.c:185: 140473780422464:error:260B6068:engine routines:dynamic_load:DSO failure:crypto/engine/eng_dyn.c:427: 140473780422464:error:2606A074:engine routines:ENGINE_by_id:no such engine:crypto/engine/eng_list.c:339:id=afalg 140473780422464:error:260BC066:engine routines:int_engine_configure:engine configuration error:crypto/engine/eng_cnf.c:141:section=afalg_engine, name=CIPHERS, value=aes-128-cbc 140473780422464:error:0E07606D:configuration file routines:module_run:module initialization error:crypto/conf/conf_mod.c:173:module=engines, value=openssl_engines, retcode=-1 140473780422464:error:2506406A:DSO support routines:dlfcn_bind_func:could not bind to the requested symbol name:crypto/dso/dso_dlfcn.c:178:symname(bind_engine): /usr/local/lib64/engines-1.1/afalg.so: undefined symbol: bind_engine 140473780422464:error:2506C06A:DSO support routines:DSO_bind_func:could not bind to the requested symbol name:crypto/dso/dso_lib.c:185: 140473780422464:error:260B6068:engine routines:dynamic_load:DSO failure:crypto/engine/eng_dyn.c:427: 140473780422464:error:2606A074:engine routines:ENGINE_by_id:no such engine:crypto/engine/eng_list.c:339:id=afalg 140473780422464:error:25066067:DSO support routines:dlfcn_load:could not load the shared library:crypto/dso/dso_dlfcn.c:113:filename(libafalg.so): libafalg.so: cannot open shared object file: No such file or directory 140473780422464:error:25070067:DSO support routines:DSO_load:could not load the shared library:crypto/dso/dso_lib.c:161: 140473780422464:error:260B6084:engine routines:dynamic_load:dso not found:crypto/engine/eng_dyn.c:414: ^C From matt at openssl.org Mon Aug 14 15:44:18 2017 From: matt at openssl.org (Matt Caswell) Date: Mon, 14 Aug 2017 16:44:18 +0100 Subject: [openssl-dev] afalg with OpenSSL 1.1.0f 25 May 2017 In-Reply-To: <209232057.1092600.1502695240022@mail.yahoo.com> References: <209232057.1092600.1502695240022.ref@mail.yahoo.com> <209232057.1092600.1502695240022@mail.yahoo.com> Message-ID: <1c3ab9e3-ff33-f16d-c3a7-450ce8de26a7@openssl.org> Comments inserted. On 14/08/17 08:20, Jitendra Lulla wrote: > Hi, > > I am trying to use afalg on Linux 4.9.37 with OpenSSL 1.1.0f. > > I am facing 2 issues: > > ONE: when I issue the speed command, I see the following: > > [root at localhost apps]# ./openssl speed -evp aes-128-cbc -engine afalg > invalid engine "afalg" > 139853452924736:error:2506406A:DSO support routines:dlfcn_bind_func:could not bind to the requested symbol name:crypto/dso/dso_dlfcn.c:178:symname(bind_engine): /usr/local/lib64/engines-1.1/afalg.so: undefined symbol: bind_engine > 139853452924736:error:2506C06A:DSO support routines:DSO_bind_func:could not bind to the requested symbol name:crypto/dso/dso_lib.c:185: > 139853452924736:error:260B6068:engine routines:dynamic_load:DSO failure:crypto/engine/eng_dyn.c:427: > 139853452924736:error:2606A074:engine routines:ENGINE_by_id:no such engine:crypto/engine/eng_list.c:339:id=afalg > 139853452924736:error:25066067:DS > > > nm afalg.so doesn't show bind_engine > Assuming you have already successfully built OpenSSL using "make", from the "test" subdir of the directory where you downloaded the source, what happens if you execute: OPENSSL_ENGINES=../engines/afalg ../util/shlib_wrap.sh ./afalgtest Another thing to try is (from the top level source dir) touch engines/afalg/e_afalg.c make Check to see if there are any warnings generated during the compilation of the engine. > > When I modify the openssl.cnf file with the engine name and the CIPHERS, still I dont get it working. The command output and the change in the openssl.cnf pasted at the end of the mail. > > > TWO: I had to create a softlink to libcrypto.so.1.1 and libssl.so.1.1 like the following to make openssl command work: > ln -s /usr/local/lib64/libssl.so.1.1 /lib64/libssl.so.1.1 > ln -s /usr/local/lib64/libcrypto.so.1.1 /lib64/libcrypto.so.1.1 > > Is creating the softlinks a known issue and will be fixed? No, this will not be fixed and may not be the most appropriate thing to do on all systems. Matt > > I have pasted the complete information about the OS/distro environment and installation commands I ran at the bottom. > Could you please suggest what wrong I am doing to make afalg work. > > Thanks > Jitendra Lulla > > ======== > > > BEFORE INSTALLATION: > > [root at localhost jlulla]# rpm -qa |grep openssl > openssl-1.0.1e-60.el7.x86_64 > openssl-devel-1.0.1e-60.el7.x86_64 > openssl-libs-1.0.1e-60.el7.x86_64 > > [root at localhost jlulla]# openssl version > OpenSSL 1.0.1e-fips 11 Feb 2013 > > > > PLEASE SEE FROM HERE PLEASE SEE FROM HERE PLEASE SEE FROM HERE---------------------------- > > STEP 1 : SOURCE TAKEN FROM https://www.openssl.org/source/openssl-1.1.0f.tar.gz 2017-May-25 13:09:51 > > [root at localhost jlulla]# uname -a > Linux localhost.localdomain 4.9.37 #1 SMP Fri Jul 21 04:52:46 PDT 2017 x86_64 x86_64 x86_64 GNU/Linux > > [root at localhost jlulla]# cat /etc/redhat-release > Red Hat Enterprise Linux Server release 7.3 (Maipo) > > > > [root at localhost openssl-1.1.0f]# pwd > /home/jlulla/openssl-1.1.0f > > STEP 2: [root at localhost openssl-1.1.0f]# ./config shared enable-engine enable-dso enable-afalgeng > Operating system: x86_64-whatever-linux2 > Configuring for linux-x86_64 > Configuring OpenSSL version 1.1.0f (0x1010006fL) > no-asan [default] OPENSSL_NO_ASAN > no-crypto-mdebug [default] OPENSSL_NO_CRYPTO_MDEBUG > no-crypto-mdebug-backtrace [default] OPENSSL_NO_CRYPTO_MDEBUG_BACKTRACE > no-ec_nistp_64_gcc_128 [default] OPENSSL_NO_EC_NISTP_64_GCC_128 > no-egd [default] OPENSSL_NO_EGD > no-fuzz-afl [default] OPENSSL_NO_FUZZ_AFL > no-fuzz-libfuzzer [default] OPENSSL_NO_FUZZ_LIBFUZZER > no-heartbeats [default] OPENSSL_NO_HEARTBEATS > no-md2 [default] OPENSSL_NO_MD2 (skip dir) > no-msan [default] OPENSSL_NO_MSAN > no-rc5 [default] OPENSSL_NO_RC5 (skip dir) > no-sctp [default] OPENSSL_NO_SCTP > no-ssl-trace [default] OPENSSL_NO_SSL_TRACE > no-ssl3 [default] OPENSSL_NO_SSL3 > no-ssl3-method [default] OPENSSL_NO_SSL3_METHOD > no-ubsan [default] OPENSSL_NO_UBSAN > no-unit-test [default] OPENSSL_NO_UNIT_TEST > no-weak-ssl-ciphers [default] OPENSSL_NO_WEAK_SSL_CIPHERS > no-zlib [default] > no-zlib-dynamic [default] > Configuring for linux-x86_64 > CC =gcc > CFLAG =-Wall -O3 -pthread -m64 -DL_ENDIAN -Wa,--noexecstack > SHARED_CFLAG =-fPIC -DOPENSSL_USE_NODELETE > DEFINES =DSO_DLFCN HAVE_DLFCN_H NDEBUG OPENSSL_THREADS OPENSSL_NO_STATIC_ENGINE OPENSSL_PIC OPENSSL_IA32_SSE2 OPENSSL_BN_ASM_MONT OPENSSL_BN_ASM_MONT5 OPENSSL_BN_ASM_GF2m SHA1_ASM SHA256_ASM SHA512_ASM RC4_ASM MD5_ASM AES_ASM VPAES_ASM BSAES_ASM GHASH_ASM ECP_NISTZ256_ASM PADLOCK_ASM POLY1305_ASM > LFLAG = > PLIB_LFLAG = > EX_LIBS =-ldl > APPS_OBJ = > CPUID_OBJ =x86_64cpuid.o > UPLINK_OBJ = > BN_ASM =asm/x86_64-gcc.o x86_64-mont.o x86_64-mont5.o x86_64-gf2m.o rsaz_exp.o rsaz-x86_64.o rsaz-avx2.o > EC_ASM =ecp_nistz256.o ecp_nistz256-x86_64.o > DES_ENC =des_enc.o fcrypt_b.o > AES_ENC =aes-x86_64.o vpaes-x86_64.o bsaes-x86_64.o aesni-x86_64.o aesni-sha1-x86_64.o aesni-sha256-x86_64.o aesni-mb-x86_64.o > BF_ENC =bf_enc.o > CAST_ENC =c_enc.o > RC4_ENC =rc4-x86_64.o rc4-md5-x86_64.o > RC5_ENC =rc5_enc.o > MD5_OBJ_ASM =md5-x86_64.o > SHA1_OBJ_ASM =sha1-x86_64.o sha256-x86_64.o sha512-x86_64.o sha1-mb-x86_64.o sha256-mb-x86_64.o > RMD160_OBJ_ASM= > CMLL_ENC =cmll-x86_64.o cmll_misc.o > MODES_OBJ =ghash-x86_64.o aesni-gcm-x86_64.o > PADLOCK_OBJ =e_padlock-x86_64.o > CHACHA_ENC =chacha-x86_64.o > POLY1305_OBJ =poly1305-x86_64.o > BLAKE2_OBJ = > PROCESSOR = > RANLIB =ranlib > ARFLAGS = > PERL =/usr/bin/perl > > SIXTY_FOUR_BIT_LONG mode > > Configured for linux-x86_64. > [root at localhost openssl-1.1.0f]# > > make didnt give any error. > > make install didnt give any error > > updatedb > > STEP 2: > [root at localhost openssl-1.1.0f]# md5sum /home/jlulla/openssl-1.1.0f/libssl.so.1.1 /usr/local/lib64/libssl.so.1.1 > 5a7264a3ebda22989c831794d30677c5 /home/jlulla/openssl-1.1.0f/libssl.so.1.1 > 5a7264a3ebda22989c831794d30677c5 /usr/local/lib64/libssl.so.1.1 > > [root at localhost openssl-1.1.0f]# ls -l /home/jlulla/openssl-1.1.0f/libssl.so > lrwxrwxrwx. 1 root root 13 Aug 10 23:45 /home/jlulla/openssl-1.1.0f/libssl.so -> libssl.so.1.1 > > > [root at localhost apps]# pwd > /home/jlulla/openssl-1.1.0f/apps > [root at localhost apps]# ./openssl > ./openssl: error while loading shared libraries: libssl.so.1.1: cannot open shared object file: No such file or directory > > [root at localhost apps]# ldd ./openssl > linux-vdso.so.1 => (0x00007ffd02f3c000) > libssl.so.1.1 => not found > libcrypto.so.1.1 => not found > libdl.so.2 => /lib64/libdl.so.2 (0x00007f0439fda000) > libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f0439dbe000) > libc.so.6 => /lib64/libc.so.6 (0x00007f04399fc000) > /lib64/ld-linux-x86-64.so.2 (0x000055d618530000) > > > [root at localhost apps]# ls -l /lib64/libssl.so.1.1 > ls: cannot access /lib64/libssl.so.1.1: No such file or directory > > ln -s /usr/local/lib64/libssl.so.1.1 /lib64/libssl.so.1.1 > > STEP 3: > [root at localhost apps]# ln -s /usr/local/lib64/libssl.so.1.1 /lib64/libssl.so.1.1 > [root at localhost apps]# ls -l /lib64/libssl.so.1.1 > lrwxrwxrwx. 1 root root 30 Aug 10 23:57 /lib64/libssl.so.1.1 -> /usr/local/lib64/libssl.so.1.1 > > > [root at localhost apps]# ln -s /usr/local/lib64/libcrypto.so.1.1 /lib64/libcrypto.so.1.1 > [root at localhost apps]# ls -l /lib64/libcrypto.so.1.1 > lrwxrwxrwx. 1 root root 33 Aug 10 23:58 /lib64/libcrypto.so.1.1 -> /usr/local/lib64/libcrypto.so.1.1 > > [root at localhost apps]# ldd ./openssl > linux-vdso.so.1 => (0x00007ffd37b5e000) > libssl.so.1.1 => /lib64/libssl.so.1.1 (0x00007f82881f8000) > libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007f8287d73000) > libdl.so.2 => /lib64/libdl.so.2 (0x00007f8287b6e000) > libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f8287952000) > libc.so.6 => /lib64/libc.so.6 (0x00007f8287591000) > /lib64/ld-linux-x86-64.so.2 (0x000055f06bc67000) > > > [root at localhost apps]# ./openssl version > OpenSSL 1.1.0f 25 May 2017 > > > STEP 4: > [root at localhost afalg]# pwd > /home/jlulla/openssl-1.1.0f/engines/afalg > > [root at localhost afalg]# ls > afalg.so build.info e_afalg.c e_afalg.d e_afalg.ec e_afalg_err.c e_afalg_err.d e_afalg_err.h e_afalg_err.o e_afalg.h e_afalg.o > > [root at localhost afalg]# ldd afalg.so > linux-vdso.so.1 => (0x00007fff1d7ef000) > libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007f3a80166000) > libdl.so.2 => /lib64/libdl.so.2 (0x00007f3a7ff62000) > libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f3a7fd45000) > libc.so.6 => /lib64/libc.so.6 (0x00007f3a7f984000) > /lib64/ld-linux-x86-64.so.2 (0x0000560991976000) > > > [root at localhost afalg]# nm afalg.so > 0000000000202060 d AFALG_error_init > 0000000000202220 b AFALG_lib_error_code > 0000000000202160 d AFALG_str_functs > 0000000000202080 d AFALG_str_reasons > 0000000000202210 B __bss_start > 0000000000202210 b completed.6344 > w __cxa_finalize@@GLIBC_2.2.5 > 0000000000000a00 t deregister_tm_clones > 0000000000000a70 t __do_global_dtors_aux > 0000000000201da0 t __do_global_dtors_aux_fini_array_entry > 0000000000201db0 d __dso_handle > 0000000000201db8 d _DYNAMIC > 0000000000202210 D _edata > 0000000000202228 B _end > 0000000000000af0 T engine_load_afalg_int > 0000000000000bb0 T ERR_AFALG_error > U ERR_get_next_error_library@@OPENSSL_1_1_0 > 0000000000000b00 T ERR_load_AFALG_strings > U ERR_load_strings@@OPENSSL_1_1_0 > U ERR_put_error@@OPENSSL_1_1_0 > 0000000000000b60 T ERR_unload_AFALG_strings > U ERR_unload_strings@@OPENSSL_1_1_0 > 0000000000000bf8 T _fini > 0000000000000ab0 t frame_dummy > 0000000000201d98 t __frame_dummy_init_array_entry > 0000000000000eb8 r __FRAME_END__ > 0000000000202000 d _GLOBAL_OFFSET_TABLE_ > w __gmon_start__ > 0000000000000970 T _init > w _ITM_deregisterTMCloneTable > w _ITM_registerTMCloneTable > 0000000000201da8 d __JCR_END__ > 0000000000201da8 d __JCR_LIST__ > w _Jv_RegisterClasses > 0000000000000a30 t register_tm_clones > 0000000000202210 d __TMC_END__ > > > [root at localhost openssl-1.1.0f]# locate afalg.so > /home/jlulla/openssl-1.1.0f/engines/afalg/afalg.so > /usr/local/lib64/engines-1.1/afalg.so > > > [root at localhost openssl-1.1.0f]# md5sum /home/jlulla/openssl-1.1.0f/engines/afalg/afalg.so /usr/local/lib64/engines-1.1/afalg.so > ec273faa6eb6e25f55e44a7d37adf195 /home/jlulla/openssl-1.1.0f/engines/afalg/afalg.so > ec273faa6eb6e25f55e44a7d37adf195 /usr/local/lib64/engines-1.1/afalg.so > > > [root at localhost apps]# date > Fri Aug 11 00:01:10 PDT 2017 > > > [root at localhost apps]# ls -l /usr/local/ssl/openssl.cnf > -rw-r--r--. 1 root root 10771 Aug 10 23:47 /usr/local/ssl/openssl.cnf > > [root at localhost apps]# md5sum /usr/local/ssl/openssl.cnf /home/jlulla/openssl-1.1.0f/apps/openssl.cnf > f697ef5df0d006882e6326606e8dbf4a /usr/local/ssl/openssl.cnf > f697ef5df0d006882e6326606e8dbf4a /home/jlulla/openssl-1.1.0f/apps/openssl.cnf > > > openssl.cnf has the following: > .. > .. > # Extra OBJECT IDENTIFIER info: > #oid_file = $ENV::HOME/.oid > oid_section = new_oids > > # To use this configuration file with the "-extfile" option of the > # "openssl x509" utility, name here the section containing the > # X.509v3 extensions to use: > # extensions = > # (Alternatively, use a configuration file that has only > # X.509v3 extensions in its main [= default] section.) > > .. > .. > > [root at localhost apps]# ./openssl speed -evp aes-128-cbc -engine afalg > invalid engine "afalg" > 139853452924736:error:2506406A:DSO support routines:dlfcn_bind_func:could not bind to the requested symbol name:crypto/dso/dso_dlfcn.c:178:symname(bind_engine): /usr/local/lib64/engines-1.1/afalg.so: undefined symbol: bind_engine > 139853452924736:error:2506C06A:DSO support routines:DSO_bind_func:could not bind to the requested symbol name:crypto/dso/dso_lib.c:185: > 139853452924736:error:260B6068:engine routines:dynamic_load:DSO failure:crypto/engine/eng_dyn.c:427: > 139853452924736:error:2606A074:engine routines:ENGINE_by_id:no such engine:crypto/engine/eng_list.c:339:id=afalg > 139853452924736:error:25066067:DSO support routines:dlfcn_load:could not load the shared library:crypto/dso/dso_dlfcn.c:113:filename(libafalg.so): libafalg.so: cannot open shared object file: No such file or directory > 139853452924736:error:25070067:DSO support routines:DSO_load:could not load the shared library:crypto/dso/dso_lib.c:161: > 139853452924736:error:260B6084:engine routines:dynamic_load:dso not found:crypto/engine/eng_dyn.c:414: > ^C > > > Modified both the files as the follwing: > /usr/local/ssl/openssl.cnf > /home/jlulla/openssl-1.1.0f/apps/openssl.cnf > > # Extra OBJECT IDENTIFIER info: > #oid_file = $ENV::HOME/.oid > oid_section = new_oids > > openssl_conf = openssl_def > > [openssl_def] > engines = openssl_engines > > [openssl_engines] > afalg = afalg_engine > > [afalg_engine] > CIPHERS=aes-128-cbc > > With the above, the errors are more: > [root at localhost apps]# ./openssl speed -evp aes-128-cbc -engine afalg > invalid engine "afalg" > 140473780422464:error:2506406A:DSO support routines:dlfcn_bind_func:could not bind to the requested symbol name:crypto/dso/dso_dlfcn.c:178:symname(bind_engine): /usr/local/lib64/engines-1.1/afalg.so: undefined symbol: bind_engine > 140473780422464:error:2506C06A:DSO support routines:DSO_bind_func:could not bind to the requested symbol name:crypto/dso/dso_lib.c:185: > 140473780422464:error:260B6068:engine routines:dynamic_load:DSO failure:crypto/engine/eng_dyn.c:427: > 140473780422464:error:2606A074:engine routines:ENGINE_by_id:no such engine:crypto/engine/eng_list.c:339:id=afalg > 140473780422464:error:260BC066:engine routines:int_engine_configure:engine configuration error:crypto/engine/eng_cnf.c:141:section=afalg_engine, name=CIPHERS, value=aes-128-cbc > 140473780422464:error:0E07606D:configuration file routines:module_run:module initialization error:crypto/conf/conf_mod.c:173:module=engines, value=openssl_engines, retcode=-1 > 140473780422464:error:2506406A:DSO support routines:dlfcn_bind_func:could not bind to the requested symbol name:crypto/dso/dso_dlfcn.c:178:symname(bind_engine): /usr/local/lib64/engines-1.1/afalg.so: undefined symbol: bind_engine > 140473780422464:error:2506C06A:DSO support routines:DSO_bind_func:could not bind to the requested symbol name:crypto/dso/dso_lib.c:185: > 140473780422464:error:260B6068:engine routines:dynamic_load:DSO failure:crypto/engine/eng_dyn.c:427: > 140473780422464:error:2606A074:engine routines:ENGINE_by_id:no such engine:crypto/engine/eng_list.c:339:id=afalg > 140473780422464:error:25066067:DSO support routines:dlfcn_load:could not load the shared library:crypto/dso/dso_dlfcn.c:113:filename(libafalg.so): libafalg.so: cannot open shared object file: No such file or directory > 140473780422464:error:25070067:DSO support routines:DSO_load:could not load the shared library:crypto/dso/dso_lib.c:161: > 140473780422464:error:260B6084:engine routines:dynamic_load:dso not found:crypto/engine/eng_dyn.c:414: > ^C > From rsalz at akamai.com Mon Aug 14 15:45:08 2017 From: rsalz at akamai.com (Salz, Rich) Date: Mon, 14 Aug 2017 15:45:08 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL Message-ID: <97E7DB38-D94A-4CED-BF11-F8AAF2511D81@akamai.com> Thanks everyone for the discussion (mainly in June) about this. There?s a blog post describing what we?ve done for the 1.1.1 release: https://www.openssl.org/blog/blog/2017/08/12/random/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From uri at ll.mit.edu Mon Aug 14 16:06:59 2017 From: uri at ll.mit.edu (Blumenthal, Uri - 0553 - MITLL) Date: Mon, 14 Aug 2017 16:06:59 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL Message-ID: <37FF22D2-5FCA-4043-A789-EED78D79BA9B@ll.mit.edu> Thanks everyone for the discussion (mainly in June) about this. There?s a blog post describing what we?ve done for the 1.1.1 release: https://www.openssl.org/blog/blog/2017/08/12/random/ Nice. But some important things could be made clearer. We added a new configuration parameter, --with-rand-seed, which takes a comma-separated list of values for seed sources. Each method is tried in turn, stopping when enough bits of randomness have been collected. What?s the default if ?with-rand-seed? was not provided? All of the listed supported types? None of them? Some of them?? What is the order in which the seed sources are tried (both when ?with-random-seed? was and was not given)? What should I do if I want a given source to be used in addition to the other sources, regardless of whether openssl thinks it got ?enough bits? of randomness or not? -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5211 bytes Desc: not available URL: From rsalz at akamai.com Mon Aug 14 16:27:07 2017 From: rsalz at akamai.com (Salz, Rich) Date: Mon, 14 Aug 2017 16:27:07 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <37FF22D2-5FCA-4043-A789-EED78D79BA9B@ll.mit.edu> References: <37FF22D2-5FCA-4043-A789-EED78D79BA9B@ll.mit.edu> Message-ID: > 1. What?s the default if ?with-rand-seed? was not provided? All of the listed supported types? None of them? Some of them?? As the first bullet says, it?s ?os?. As for the second part of your question, it is deliberately not answered. If you care, you?ll have to read the source. (It?s clean and easy to do so, now.) We?re not documenting everything. >2. What is the order in which the seed sources are tried (both when ?with-random-seed? was and was not given)? Read the source. > 3. What should I do if I want a given source to be used in addition to the other sources, regardless of whether openssl thinks it got ?enough bits? of randomness or not? Modify the source :) For a few reasons, we?re deliberately not documenting all the details. Interested parties will have to read the source. From uri at ll.mit.edu Mon Aug 14 16:43:58 2017 From: uri at ll.mit.edu (Blumenthal, Uri - 0553 - MITLL) Date: Mon, 14 Aug 2017 16:43:58 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: References: <37FF22D2-5FCA-4043-A789-EED78D79BA9B@ll.mit.edu> Message-ID: >> 1. What?s the default if ?with-rand-seed? was not provided? All of the listed supported types? None of them? Some of them?? > > As the first bullet says, it?s ?os?. OK, thanks. > As for the second part of your question, it is deliberately not answered. If you care, you?ll have to read the source. (It?s clean and easy to do so, now.) We?re not documenting everything. I think it?s a bad approach to not document this important behavior. It has to be security-analyzed, then frozen & published. >>2. What is the order in which the seed sources are tried (both when ?with-random-seed? was and was not given)? > > Read the source. Likewise. It has to be published, and the developers need to figure out the right way here and commit to it (no pun intended). >> 3. What should I do if I want a given source to be used in addition to the other sources, regardless of whether openssl thinks it got ?enough bits? of randomness or not? > > Modify the source :) Very bad answer. When randomness is involved, adding more of diverse sources can only improve the outcome. Therefore there must be a way to tell OpenSSL to *not* stop when it got what it believe to be ?enough? but mix in more from other sources. And that mechanism must *not* be an individual hack ? but a standard reviewed maintained access method. > For a few reasons, we?re deliberately not documenting all the details. Interested parties will have to read the source. I have no problem reading the source code. I do have a problem with (a) important decisions like this not ?formalized? and documented, and (b) mechanisms to tune the RNG seeding not provided and clearly and comprehensively documented. I urge the developers to reconsider. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5211 bytes Desc: not available URL: From rsalz at akamai.com Mon Aug 14 16:49:34 2017 From: rsalz at akamai.com (Salz, Rich) Date: Mon, 14 Aug 2017 16:49:34 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: References: <37FF22D2-5FCA-4043-A789-EED78D79BA9B@ll.mit.edu> Message-ID: <0B81DA14-C636-4586-B781-18A25FEFAE88@akamai.com> >>> 3. What should I do if I want a given source to be used in addition to the other sources, regardless of whether openssl thinks it got ?enough bits? of randomness or not? >> Modify the source :) > Very bad answer. And also a wrong one. Your application can always call RAND_add(). Sorry for mistake. > I have no problem reading the source code. I do have a problem with (a) important decisions like this not ?formalized? and documented, and (b) mechanisms to tune the RNG seeding not provided and clearly and comprehensively documented. This is a mostly volunteer open source project. We are unlikely to commit to something that requires so much effort when, frankly, most of the consumers aren?t interested, or qualified, to make an assessment. I am sorry if that sounds obnoxious or conceited. It shouldn?t; there are many things that I know I?m not qualified to comment on :) And also, we reserve the right to make changes. I expect that the FIPS project, just starting, will be of interest to you. From uri at ll.mit.edu Mon Aug 14 17:44:58 2017 From: uri at ll.mit.edu (Blumenthal, Uri - 0553 - MITLL) Date: Mon, 14 Aug 2017 17:44:58 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <0B81DA14-C636-4586-B781-18A25FEFAE88@akamai.com> References: <37FF22D2-5FCA-4043-A789-EED78D79BA9B@ll.mit.edu> <0B81DA14-C636-4586-B781-18A25FEFAE88@akamai.com> Message-ID: >>> Modify the source :) >> >> Very bad answer. > > And also a wrong one. Your application can always call RAND_add(). Sorry for mistake. And this is a very good answer. Perhaps this guidance deserves being documented somewhere besides this mailing list? Something along the lines of ?RNG Seed Sources can be set by --with-rand-seed. ?os? is the default source. Sources are tried until enough bits of randomness have been collected. If you want to mix data from a particular source into the seed, but don?t want to make that source exclusive ? use RAND_add() method.? > This is a mostly volunteer open source project. Yeah, I realize and appreciate that. > We are unlikely to commit to something that requires so much effort I?m not sure I agree here. What effort are you talking about? In order to select an order in which available sources are queried, the developers had to think (hopefully :). Those thought could be documented in a few lines of text. > when, frankly, most of the consumers aren?t interested, or qualified, to make an assessment. So they?ll be happy with the default. Fine with me. ;-) > I am sorry if that sounds obnoxious or conceited. It shouldn?t; there are many things that I know I?m not qualified to comment on :) And also, we reserve the right to make changes. No offense taken. But you ?freeze? interface to and behavior of ciphers and cipher modes, for example. This (how you seed RNG that provides keys to those) is at least equally important. It?s not a minute detail that nobody should care about or nose in. So while the team clearly has the right to make changes (especially before the interface became public ;), but I?d rather that such changes are guided by an informed consent from the public (such as yours truly ;). > I expect that the FIPS project, just starting, will be of interest to you. Thank you ? indeed it is of interest. (Though I see FIPS more as a curse than as a blessing ;-). One important thing I missed earlier: > We also added a separate global DRBG for private key generation and added API?s to use it. > This object isn?t reachable directly, but it is used by the new BN_priv_rand and BN_priv_rand_range API?s. > Those API?s, in turn, are used by all private-key generating functions. I think it is *imperative* for a user to be able to RAND_add() to the DRBG that gnerates private keys. I cannot emphasize enough how critical this is. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5211 bytes Desc: not available URL: From rsalz at akamai.com Mon Aug 14 18:57:47 2017 From: rsalz at akamai.com (Salz, Rich) Date: Mon, 14 Aug 2017 18:57:47 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: References: <37FF22D2-5FCA-4043-A789-EED78D79BA9B@ll.mit.edu> <0B81DA14-C636-4586-B781-18A25FEFAE88@akamai.com> Message-ID: <23E5D32D-59E5-46C2-8E56-80C687EA09A5@akamai.com> And this is a very good answer. Perhaps this guidance deserves being documented somewhere besides this mailing list? Something along the lines of It is documented in the RAND_add manpage. ? I?m not sure I agree here. What effort are you talking about? In order to select an order in which available sources are queried, the developers had to think (hopefully :). Those thought could be documented in a few lines of text. And what would be the point? Why should someone trust the documentation, which can get out of date, more than the source? ? So while the team clearly has the right to make changes (especially before the interface became public ;), but I?d rather that such changes are guided by an informed consent from the public (such as yours truly ;). We aren?t cutting off any avenues of participation. Email discussion and pull requests are always welcome. But yes, the barrier to useful participation is that someone has to first read and understand the source. ? I think it is *imperative* for a user to be able to RAND_add() to the DRBG that gnerates private keys. I cannot emphasize enough how critical this is. I am curious to know your justification for this. It seems to me that if you accept the DRBG document, which we do, then the way we do the seeding is fine. If you don?t accept the document, then modify the source. From lullajd at yahoo.com Wed Aug 16 06:30:42 2017 From: lullajd at yahoo.com (Jitendra Lulla) Date: Wed, 16 Aug 2017 06:30:42 +0000 (UTC) Subject: [openssl-dev] afalg with OpenSSL 1.1.0f 25 May 2017 References: <1624014880.2585224.1502865042816.ref@mail.yahoo.com> Message-ID: <1624014880.2585224.1502865042816@mail.yahoo.com> Hi Matt, I have linux 4.9.37 on RHEL7.3. [root at localhost jlulla]# uname -a Linux localhost.localdomain 4.9.37 #1 SMP Fri Jul 21 04:52:46 PDT 2017 x86_64 x86_64 x86_64 GNU/Linux [root at localhost test]# OPENSSL_ENGINES=../engines/afalg ../util/shlib_wrap.sh ./afalgtest AFALG not supported - skipping AFALG tests PASS [root at localhost test]# I am getting here: # if LINUX_VERSION_CODE <= KERNEL_VERSION(K_MAJ, K_MIN1, K_MIN2) /* * If we get here then it looks like there is a mismatch between the linux * headers and the actual kernel version, so we have tried to compile with * afalg support, but then skipped it in e_afalg.c. As far as this test is * concerned we behave as if we had been configured without support */ # define OPENSSL_NO_AFALGENG # endif Following is the value for KERNEL_VERSION for me: [root at localhost jlulla]# ./kernelversion (program at the bottom of this mail) KERNEL_VERSION: 262400 LINUX_VERSION_CODE 199168 condition:1 Where should I look to fix it? Thanks Jitrendra [root at localhost jlulla]# cat kernelversion.c #define LINUX_VERSION_CODE 199168 #define KERNEL_VERSION(a,b,c) (((a) << 16) + ((b) << 8) + (c)) #define RHEL_MAJOR 7 #define RHEL_MINOR 3 #define RHEL_RELEASE_VERSION(a,b) (((a) << 8) + (b)) #define RHEL_RELEASE_CODE 1795 #define RHEL_RELEASE "514" # define K_MAJ 4 # define K_MIN1 1 # define K_MIN2 0 #include int main() { printf("KERNEL_VERSION: %d\n", KERNEL_VERSION(K_MAJ, K_MIN1, K_MIN2)); printf("LINUX_VERSION_CODE %d\n", LINUX_VERSION_CODE); printf("condition:%d\n", (LINUX_VERSION_CODE <= KERNEL_VERSION(K_MAJ, K_MIN1, K_MIN2))); } -------------------------------------------- On Mon, 8/14/17, Matt Caswell wrote: Subject: Re: afalg with OpenSSL 1.1.0f 25 May 2017 To: "openssl-dev at openssl.org" Cc: "Jitendra Lulla" Date: Monday, August 14, 2017, 3:44 PM Comments inserted. On 14/08/17 08:20, Jitendra Lulla wrote: > Hi, > > I am trying to use afalg on Linux 4.9.37 with OpenSSL 1.1.0f. > > I am facing 2 issues: > > ONE: when I issue the speed command, I see the following: > > [root at localhost apps]# ./openssl speed -evp aes-128-cbc -engine afalg > invalid engine "afalg" > 139853452924736:error:2506406A:DSO support routines:dlfcn_bind_func:could not bind to the requested symbol name:crypto/dso/dso_dlfcn.c:178:symname(bind_engine): /usr/local/lib64/engines-1.1/afalg.so: undefined symbol: bind_engine > 139853452924736:error:2506C06A:DSO support routines:DSO_bind_func:could not bind to the requested symbol name:crypto/dso/dso_lib.c:185: > 139853452924736:error:260B6068:engine routines:dynamic_load:DSO failure:crypto/engine/eng_dyn.c:427: > 139853452924736:error:2606A074:engine routines:ENGINE_by_id:no such engine:crypto/engine/eng_list.c:339:id=afalg > 139853452924736:error:25066067:DS > > > nm afalg.so doesn't show bind_engine > Assuming you have already successfully built OpenSSL using "make", from the "test" subdir of the directory where you downloaded the source, what happens if you execute: OPENSSL_ENGINES=../engines/afalg ../util/shlib_wrap.sh ./afalgtest Another thing to try is (from the top level source dir) touch engines/afalg/e_afalg.c make Check to see if there are any warnings generated during the compilation of the engine. > > When I modify the openssl.cnf file with the engine name and the CIPHERS, still I dont get it working. The command output and the change in the openssl.cnf pasted at the end of the mail. > > > TWO: I had to create a softlink to libcrypto.so.1.1 and libssl.so.1.1 like the following to make openssl command work: > ln -s /usr/local/lib64/libssl.so.1.1 /lib64/libssl.so.1.1 > ln -s /usr/local/lib64/libcrypto.so.1.1 /lib64/libcrypto.so.1.1 > > Is creating the softlinks a known issue and will be fixed? No, this will not be fixed and may not be the most appropriate thing to do on all systems. Matt > > I have pasted the complete information about the OS/distro environment and installation commands I ran at the bottom. > Could you please suggest what wrong I am doing to make afalg work. > > Thanks > Jitendra Lulla > > ======== > > > BEFORE INSTALLATION: > > [root at localhost jlulla]# rpm -qa? |grep openssl > openssl-1.0.1e-60.el7.x86_64 > openssl-devel-1.0.1e-60.el7.x86_64 > openssl-libs-1.0.1e-60.el7.x86_64 > > [root at localhost jlulla]# openssl version > OpenSSL 1.0.1e-fips 11 Feb 2013 > > > > PLEASE SEE FROM HERE PLEASE SEE FROM HERE PLEASE SEE FROM HERE---------------------------- > > STEP 1 : SOURCE TAKEN FROM https://www.openssl.org/source/openssl-1.1.0f.tar.gz 2017-May-25 13:09:51 > > [root at localhost jlulla]# uname -a > Linux localhost.localdomain 4.9.37 #1 SMP Fri Jul 21 04:52:46 PDT 2017 x86_64 x86_64 x86_64 GNU/Linux > > [root at localhost jlulla]# cat /etc/redhat-release > Red Hat Enterprise Linux Server release 7.3 (Maipo) > > > > [root at localhost openssl-1.1.0f]# pwd > /home/jlulla/openssl-1.1.0f > > STEP 2: [root at localhost openssl-1.1.0f]# ./config shared enable-engine enable-dso enable-afalgeng > Operating system: x86_64-whatever-linux2 > Configuring for linux-x86_64 > Configuring OpenSSL version 1.1.0f (0x1010006fL) >? ? no-asan? ? ? ? [default]? OPENSSL_NO_ASAN >? ? no-crypto-mdebug [default]? OPENSSL_NO_CRYPTO_MDEBUG >? ? no-crypto-mdebug-backtrace [default]? OPENSSL_NO_CRYPTO_MDEBUG_BACKTRACE >? ? no-ec_nistp_64_gcc_128 [default]? OPENSSL_NO_EC_NISTP_64_GCC_128 >? ? no-egd? ? ? ? ? [default]? OPENSSL_NO_EGD >? ? no-fuzz-afl? ? [default]? OPENSSL_NO_FUZZ_AFL >? ? no-fuzz-libfuzzer [default]? OPENSSL_NO_FUZZ_LIBFUZZER >? ? no-heartbeats? [default]? OPENSSL_NO_HEARTBEATS >? ? no-md2? ? ? ? ? [default]? OPENSSL_NO_MD2 (skip dir) >? ? no-msan? ? ? ? [default]? OPENSSL_NO_MSAN >? ? no-rc5? ? ? ? ? [default]? OPENSSL_NO_RC5 (skip dir) >? ? no-sctp? ? ? ? [default]? OPENSSL_NO_SCTP >? ? no-ssl-trace? ? [default]? OPENSSL_NO_SSL_TRACE >? ? no-ssl3? ? ? ? [default]? OPENSSL_NO_SSL3 >? ? no-ssl3-method? [default]? OPENSSL_NO_SSL3_METHOD >? ? no-ubsan? ? ? ? [default]? OPENSSL_NO_UBSAN >? ? no-unit-test? ? [default]? OPENSSL_NO_UNIT_TEST >? ? no-weak-ssl-ciphers [default]? OPENSSL_NO_WEAK_SSL_CIPHERS >? ? no-zlib? ? ? ? [default] >? ? no-zlib-dynamic [default] > Configuring for linux-x86_64 > CC? ? ? ? ? ? =gcc > CFLAG? ? ? ? =-Wall -O3 -pthread -m64 -DL_ENDIAN? -Wa,--noexecstack > SHARED_CFLAG? =-fPIC -DOPENSSL_USE_NODELETE > DEFINES? ? ? =DSO_DLFCN HAVE_DLFCN_H NDEBUG OPENSSL_THREADS OPENSSL_NO_STATIC_ENGINE OPENSSL_PIC OPENSSL_IA32_SSE2 OPENSSL_BN_ASM_MONT OPENSSL_BN_ASM_MONT5 OPENSSL_BN_ASM_GF2m SHA1_ASM SHA256_ASM SHA512_ASM RC4_ASM MD5_ASM AES_ASM VPAES_ASM BSAES_ASM GHASH_ASM ECP_NISTZ256_ASM PADLOCK_ASM POLY1305_ASM > LFLAG? ? ? ? = > PLIB_LFLAG? ? = > EX_LIBS? ? ? =-ldl > APPS_OBJ? ? ? = > CPUID_OBJ? ? =x86_64cpuid.o > UPLINK_OBJ? ? = > BN_ASM? ? ? ? =asm/x86_64-gcc.o x86_64-mont.o x86_64-mont5.o x86_64-gf2m.o rsaz_exp.o rsaz-x86_64.o rsaz-avx2.o > EC_ASM? ? ? ? =ecp_nistz256.o ecp_nistz256-x86_64.o > DES_ENC? ? ? =des_enc.o fcrypt_b.o > AES_ENC? ? ? =aes-x86_64.o vpaes-x86_64.o bsaes-x86_64.o aesni-x86_64.o aesni-sha1-x86_64.o aesni-sha256-x86_64.o aesni-mb-x86_64.o > BF_ENC? ? ? ? =bf_enc.o > CAST_ENC? ? ? =c_enc.o > RC4_ENC? ? ? =rc4-x86_64.o rc4-md5-x86_64.o > RC5_ENC? ? ? =rc5_enc.o > MD5_OBJ_ASM? =md5-x86_64.o > SHA1_OBJ_ASM? =sha1-x86_64.o sha256-x86_64.o sha512-x86_64.o sha1-mb-x86_64.o sha256-mb-x86_64.o > RMD160_OBJ_ASM= > CMLL_ENC? ? ? =cmll-x86_64.o cmll_misc.o > MODES_OBJ? ? =ghash-x86_64.o aesni-gcm-x86_64.o > PADLOCK_OBJ? =e_padlock-x86_64.o > CHACHA_ENC? ? =chacha-x86_64.o > POLY1305_OBJ? =poly1305-x86_64.o > BLAKE2_OBJ? ? = > PROCESSOR? ? = > RANLIB? ? ? ? =ranlib > ARFLAGS? ? ? = > PERL? ? ? ? ? =/usr/bin/perl > > SIXTY_FOUR_BIT_LONG mode > > Configured for linux-x86_64. > [root at localhost openssl-1.1.0f]# > > make didnt give any error. > > make install didnt give any error > > updatedb > > STEP 2: > [root at localhost openssl-1.1.0f]# md5sum /home/jlulla/openssl-1.1.0f/libssl.so.1.1 /usr/local/lib64/libssl.so.1.1 > 5a7264a3ebda22989c831794d30677c5? /home/jlulla/openssl-1.1.0f/libssl.so.1.1 > 5a7264a3ebda22989c831794d30677c5? /usr/local/lib64/libssl.so.1.1 > > [root at localhost openssl-1.1.0f]# ls -l /home/jlulla/openssl-1.1.0f/libssl.so > lrwxrwxrwx. 1 root root 13 Aug 10 23:45 /home/jlulla/openssl-1.1.0f/libssl.so -> libssl.so.1.1 > > > [root at localhost apps]# pwd > /home/jlulla/openssl-1.1.0f/apps > [root at localhost apps]# ./openssl > ./openssl: error while loading shared libraries: libssl.so.1.1: cannot open shared object file: No such file or directory > > [root at localhost apps]# ldd ./openssl >? ? ? ? linux-vdso.so.1 =>? (0x00007ffd02f3c000) >? ? ? ? libssl.so.1.1 => not found >? ? ? ? libcrypto.so.1.1 => not found >? ? ? ? libdl.so.2 => /lib64/libdl.so.2 (0x00007f0439fda000) >? ? ? ? libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f0439dbe000) >? ? ? ? libc.so.6 => /lib64/libc.so.6 (0x00007f04399fc000) >? ? ? ? /lib64/ld-linux-x86-64.so.2 (0x000055d618530000) > > > [root at localhost apps]# ls -l /lib64/libssl.so.1.1 > ls: cannot access /lib64/libssl.so.1.1: No such file or directory > > ln -s /usr/local/lib64/libssl.so.1.1 /lib64/libssl.so.1.1 > > STEP 3: > [root at localhost apps]# ln -s /usr/local/lib64/libssl.so.1.1 /lib64/libssl.so.1.1 > [root at localhost apps]# ls -l /lib64/libssl.so.1.1 > lrwxrwxrwx. 1 root root 30 Aug 10 23:57 /lib64/libssl.so.1.1 -> /usr/local/lib64/libssl.so.1.1 > > > [root at localhost apps]# ln -s /usr/local/lib64/libcrypto.so.1.1 /lib64/libcrypto.so.1.1 > [root at localhost apps]# ls -l /lib64/libcrypto.so.1.1 > lrwxrwxrwx. 1 root root 33 Aug 10 23:58 /lib64/libcrypto.so.1.1 -> /usr/local/lib64/libcrypto.so.1.1 > > [root at localhost apps]# ldd ./openssl >? ? ? ? linux-vdso.so.1 =>? (0x00007ffd37b5e000) >? ? ? ? libssl.so.1.1 => /lib64/libssl.so.1.1 (0x00007f82881f8000) >? ? ? ? libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007f8287d73000) >? ? ? ? libdl.so.2 => /lib64/libdl.so.2 (0x00007f8287b6e000) >? ? ? ? libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f8287952000) >? ? ? ? libc.so.6 => /lib64/libc.so.6 (0x00007f8287591000) >? ? ? ? /lib64/ld-linux-x86-64.so.2 (0x000055f06bc67000) > > > [root at localhost apps]# ./openssl version > OpenSSL 1.1.0f? 25 May 2017 > > > STEP 4: > [root at localhost afalg]# pwd > /home/jlulla/openssl-1.1.0f/engines/afalg > > [root at localhost afalg]# ls > afalg.so? build.info? e_afalg.c? e_afalg.d? e_afalg.ec? e_afalg_err.c? e_afalg_err.d? e_afalg_err.h? e_afalg_err.o? e_afalg.h? e_afalg.o > > [root at localhost afalg]# ldd afalg.so >? ? ? ? linux-vdso.so.1 =>? (0x00007fff1d7ef000) >? ? ? ? libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007f3a80166000) >? ? ? ? libdl.so.2 => /lib64/libdl.so.2 (0x00007f3a7ff62000) >? ? ? ? libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f3a7fd45000) >? ? ? ? libc.so.6 => /lib64/libc.so.6 (0x00007f3a7f984000) >? ? ? ? /lib64/ld-linux-x86-64.so.2 (0x0000560991976000) > > > [root at localhost afalg]# nm afalg.so > 0000000000202060 d AFALG_error_init > 0000000000202220 b AFALG_lib_error_code > 0000000000202160 d AFALG_str_functs > 0000000000202080 d AFALG_str_reasons > 0000000000202210 B __bss_start > 0000000000202210 b completed.6344 >? ? ? ? ? ? ? ? w __cxa_finalize@@GLIBC_2.2.5 > 0000000000000a00 t deregister_tm_clones > 0000000000000a70 t __do_global_dtors_aux > 0000000000201da0 t __do_global_dtors_aux_fini_array_entry > 0000000000201db0 d __dso_handle > 0000000000201db8 d _DYNAMIC > 0000000000202210 D _edata > 0000000000202228 B _end > 0000000000000af0 T engine_load_afalg_int > 0000000000000bb0 T ERR_AFALG_error >? ? ? ? ? ? ? ? U ERR_get_next_error_library@@OPENSSL_1_1_0 > 0000000000000b00 T ERR_load_AFALG_strings >? ? ? ? ? ? ? ? U ERR_load_strings@@OPENSSL_1_1_0 >? ? ? ? ? ? ? ? U ERR_put_error@@OPENSSL_1_1_0 > 0000000000000b60 T ERR_unload_AFALG_strings >? ? ? ? ? ? ? ? U ERR_unload_strings@@OPENSSL_1_1_0 > 0000000000000bf8 T _fini > 0000000000000ab0 t frame_dummy > 0000000000201d98 t __frame_dummy_init_array_entry > 0000000000000eb8 r __FRAME_END__ > 0000000000202000 d _GLOBAL_OFFSET_TABLE_ >? ? ? ? ? ? ? ? w __gmon_start__ > 0000000000000970 T _init >? ? ? ? ? ? ? ? w _ITM_deregisterTMCloneTable >? ? ? ? ? ? ? ? w _ITM_registerTMCloneTable > 0000000000201da8 d __JCR_END__ > 0000000000201da8 d __JCR_LIST__ >? ? ? ? ? ? ? ? w _Jv_RegisterClasses > 0000000000000a30 t register_tm_clones > 0000000000202210 d __TMC_END__ > > > [root at localhost openssl-1.1.0f]# locate afalg.so > /home/jlulla/openssl-1.1.0f/engines/afalg/afalg.so > /usr/local/lib64/engines-1.1/afalg.so > > > [root at localhost openssl-1.1.0f]# md5sum /home/jlulla/openssl-1.1.0f/engines/afalg/afalg.so /usr/local/lib64/engines-1.1/afalg.so > ec273faa6eb6e25f55e44a7d37adf195? /home/jlulla/openssl-1.1.0f/engines/afalg/afalg.so > ec273faa6eb6e25f55e44a7d37adf195? /usr/local/lib64/engines-1.1/afalg.so > > > [root at localhost apps]# date > Fri Aug 11 00:01:10 PDT 2017 > > > [root at localhost apps]# ls -l /usr/local/ssl/openssl.cnf > -rw-r--r--. 1 root root 10771 Aug 10 23:47 /usr/local/ssl/openssl.cnf > > [root at localhost apps]# md5sum /usr/local/ssl/openssl.cnf /home/jlulla/openssl-1.1.0f/apps/openssl.cnf > f697ef5df0d006882e6326606e8dbf4a? /usr/local/ssl/openssl.cnf > f697ef5df0d006882e6326606e8dbf4a? /home/jlulla/openssl-1.1.0f/apps/openssl.cnf > > > openssl.cnf has the following: > .. > .. > # Extra OBJECT IDENTIFIER info: > #oid_file? ? ? ? ? ? ? = $ENV::HOME/.oid > oid_section? ? ? ? ? ? = new_oids > > # To use this configuration file with the "-extfile" option of the > # "openssl x509" utility, name here the section containing the > # X.509v3 extensions to use: > # extensions? ? ? ? ? ? = > # (Alternatively, use a configuration file that has only > # X.509v3 extensions in its main [= default] section.) > > .. > .. > > [root at localhost apps]# ./openssl speed -evp aes-128-cbc -engine afalg > invalid engine "afalg" > 139853452924736:error:2506406A:DSO support routines:dlfcn_bind_func:could not bind to the requested symbol name:crypto/dso/dso_dlfcn.c:178:symname(bind_engine): /usr/local/lib64/engines-1.1/afalg.so: undefined symbol: bind_engine > 139853452924736:error:2506C06A:DSO support routines:DSO_bind_func:could not bind to the requested symbol name:crypto/dso/dso_lib.c:185: > 139853452924736:error:260B6068:engine routines:dynamic_load:DSO failure:crypto/engine/eng_dyn.c:427: > 139853452924736:error:2606A074:engine routines:ENGINE_by_id:no such engine:crypto/engine/eng_list.c:339:id=afalg > 139853452924736:error:25066067:DSO support routines:dlfcn_load:could not load the shared library:crypto/dso/dso_dlfcn.c:113:filename(libafalg.so): libafalg.so: cannot open shared object file: No such file or directory > 139853452924736:error:25070067:DSO support routines:DSO_load:could not load the shared library:crypto/dso/dso_lib.c:161: > 139853452924736:error:260B6084:engine routines:dynamic_load:dso not found:crypto/engine/eng_dyn.c:414: > ^C > > > Modified both the files as the follwing: > /usr/local/ssl/openssl.cnf > /home/jlulla/openssl-1.1.0f/apps/openssl.cnf > > # Extra OBJECT IDENTIFIER info: > #oid_file? ? ? ? ? ? ? = $ENV::HOME/.oid > oid_section? ? ? ? ? ? = new_oids > > openssl_conf = openssl_def > > [openssl_def] > engines = openssl_engines > > [openssl_engines] > afalg = afalg_engine > > [afalg_engine] > CIPHERS=aes-128-cbc > > With the above, the errors are more: > [root at localhost apps]# ./openssl speed -evp aes-128-cbc -engine afalg > invalid engine "afalg" > 140473780422464:error:2506406A:DSO support routines:dlfcn_bind_func:could not bind to the requested symbol name:crypto/dso/dso_dlfcn.c:178:symname(bind_engine): /usr/local/lib64/engines-1.1/afalg.so: undefined symbol: bind_engine > 140473780422464:error:2506C06A:DSO support routines:DSO_bind_func:could not bind to the requested symbol name:crypto/dso/dso_lib.c:185: > 140473780422464:error:260B6068:engine routines:dynamic_load:DSO failure:crypto/engine/eng_dyn.c:427: > 140473780422464:error:2606A074:engine routines:ENGINE_by_id:no such engine:crypto/engine/eng_list.c:339:id=afalg > 140473780422464:error:260BC066:engine routines:int_engine_configure:engine configuration error:crypto/engine/eng_cnf.c:141:section=afalg_engine, name=CIPHERS, value=aes-128-cbc > 140473780422464:error:0E07606D:configuration file routines:module_run:module initialization error:crypto/conf/conf_mod.c:173:module=engines, value=openssl_engines, retcode=-1 > 140473780422464:error:2506406A:DSO support routines:dlfcn_bind_func:could not bind to the requested symbol name:crypto/dso/dso_dlfcn.c:178:symname(bind_engine): /usr/local/lib64/engines-1.1/afalg.so: undefined symbol: bind_engine > 140473780422464:error:2506C06A:DSO support routines:DSO_bind_func:could not bind to the requested symbol name:crypto/dso/dso_lib.c:185: > 140473780422464:error:260B6068:engine routines:dynamic_load:DSO failure:crypto/engine/eng_dyn.c:427: > 140473780422464:error:2606A074:engine routines:ENGINE_by_id:no such engine:crypto/engine/eng_list.c:339:id=afalg > 140473780422464:error:25066067:DSO support routines:dlfcn_load:could not load the shared library:crypto/dso/dso_dlfcn.c:113:filename(libafalg.so): libafalg.so: cannot open shared object file: No such file or directory > 140473780422464:error:25070067:DSO support routines:DSO_load:could not load the shared library:crypto/dso/dso_lib.c:161: > 140473780422464:error:260B6084:engine routines:dynamic_load:dso not found:crypto/engine/eng_dyn.c:414: > ^C > From lullajd at yahoo.com Wed Aug 16 08:00:56 2017 From: lullajd at yahoo.com (Jitendra Lulla) Date: Wed, 16 Aug 2017 08:00:56 +0000 (UTC) Subject: [openssl-dev] afalg with OpenSSL 1.1.0f 25 May 2017 References: <736159717.2581176.1502870456604.ref@mail.yahoo.com> Message-ID: <736159717.2581176.1502870456604@mail.yahoo.com> Hi Matt, Thanks, I could find that the /usr/include/linux/version.h has #define LINUX_VERSION_CODE 199168 for my booted kernel 4.9.37. Which is why I see the following warnings also: gcc -Iinclude -DDSO_DLFCN -DHAVE_DLFCN_H -DNDEBUG -DOPENSSL_THREADS -DOPENSSL_NO_STATIC_ENGINE -DOPENSSL_PIC -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DRC4_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DPADLOCK_ASM -DPOLY1305_ASM -DOPENSSLDIR="\"/usr/local/ssl\"" -DENGINESDIR="\"/usr/local/lib64/engines-1.1\"" -Wall -O3 -pthread -m64 -DL_ENDIAN -Wa,--noexecstack -fPIC -DOPENSSL_USE_NODELETE -MMD -MF engines/afalg/e_afalg.d.tmp -MT engines/afalg/e_afalg.o -c -o engines/afalg/e_afalg.o engines/afalg/e_afalg.c engines/afalg/e_afalg.c:30:4: warning: #warning "AFALG ENGINE requires Kernel Headers >= 4.1.0" [-Wcpp] # warning "AFALG ENGINE requires Kernel Headers >= 4.1.0" ^ engines/afalg/e_afalg.c:31:4: warning: #warning "Skipping Compilation of AFALG engine" [-Wcpp] # warning "Skipping Compilation of AFALG engine" I will fix this problem now by having proper setup. Will update if I face any more issues. Thanks Jitendra -------------------------------------------- On Wed, 8/16/17, Jitendra Lulla wrote: Subject: Re: afalg with OpenSSL 1.1.0f 25 May 2017 To: "openssl-dev at openssl.org" , "Matt Caswell" Cc: "Jitendra Lulla" Date: Wednesday, August 16, 2017, 6:30 AM Hi Matt, I have linux 4.9.37 on RHEL7.3. [root at localhost jlulla]# uname -a Linux localhost.localdomain 4.9.37 #1 SMP Fri Jul 21 04:52:46 PDT 2017 x86_64 x86_64 x86_64 GNU/Linux [root at localhost test]# OPENSSL_ENGINES=../engines/afalg ../util/shlib_wrap.sh ./afalgtest AFALG not supported - skipping AFALG tests PASS [root at localhost test]# I am getting here: # if LINUX_VERSION_CODE <= KERNEL_VERSION(K_MAJ, K_MIN1, K_MIN2) /* * If we get here then it looks like there is a mismatch between the linux * headers and the actual kernel version, so we have tried to compile with * afalg support, but then skipped it in e_afalg.c. As far as this test is * concerned we behave as if we had been configured without support */ #? define OPENSSL_NO_AFALGENG # endif Following is the value for KERNEL_VERSION for me: [root at localhost jlulla]# ./kernelversion (program at the bottom of this mail) KERNEL_VERSION: 262400 LINUX_VERSION_CODE 199168 condition:1 Where should I look to fix it? Thanks Jitrendra [root at localhost jlulla]# cat kernelversion.c #define LINUX_VERSION_CODE 199168 #define KERNEL_VERSION(a,b,c) (((a) << 16) + ((b) << 8) + (c)) #define RHEL_MAJOR 7 #define RHEL_MINOR 3 #define RHEL_RELEASE_VERSION(a,b) (((a) << 8) + (b)) #define RHEL_RELEASE_CODE 1795 #define RHEL_RELEASE "514" # define K_MAJ? 4 # define K_MIN1? 1 # define K_MIN2? 0 #include int main() { ? ? ? ? printf("KERNEL_VERSION: %d\n",? KERNEL_VERSION(K_MAJ, K_MIN1, K_MIN2)); ? ? ? ? printf("LINUX_VERSION_CODE %d\n", LINUX_VERSION_CODE); ? ? ? ? printf("condition:%d\n", ? ? ? ? ? ? ? ? ? ? ? ? (LINUX_VERSION_CODE <= KERNEL_VERSION(K_MAJ, K_MIN1, K_MIN2))); } -------------------------------------------- On Mon, 8/14/17, Matt Caswell wrote: Subject: Re: afalg with OpenSSL 1.1.0f 25 May 2017 To: "openssl-dev at openssl.org" Cc: "Jitendra Lulla" Date: Monday, August 14, 2017, 3:44 PM Comments inserted. On 14/08/17 08:20, Jitendra Lulla wrote: > Hi, > > I am trying to use afalg on Linux 4.9.37 with OpenSSL 1.1.0f. > > I am facing 2 issues: > > ONE: when I issue the speed command, I see the following: > > [root at localhost apps]# ./openssl speed -evp aes-128-cbc -engine afalg > invalid engine "afalg" > 139853452924736:error:2506406A:DSO support routines:dlfcn_bind_func:could not bind to the requested symbol name:crypto/dso/dso_dlfcn.c:178:symname(bind_engine): /usr/local/lib64/engines-1.1/afalg.so: undefined symbol: bind_engine > 139853452924736:error:2506C06A:DSO support routines:DSO_bind_func:could not bind to the requested symbol name:crypto/dso/dso_lib.c:185: > 139853452924736:error:260B6068:engine routines:dynamic_load:DSO failure:crypto/engine/eng_dyn.c:427: > 139853452924736:error:2606A074:engine routines:ENGINE_by_id:no such engine:crypto/engine/eng_list.c:339:id=afalg > 139853452924736:error:25066067:DS > > > nm afalg.so doesn't show bind_engine > Assuming you have already successfully built OpenSSL using "make", from the "test" subdir of the directory where you downloaded the source, what happens if you execute: OPENSSL_ENGINES=../engines/afalg ../util/shlib_wrap.sh ./afalgtest Another thing to try is (from the top level source dir) touch engines/afalg/e_afalg.c make Check to see if there are any warnings generated during the compilation of the engine. > > When I modify the openssl.cnf file with the engine name and the CIPHERS, still I dont get it working. The command output and the change in the openssl.cnf pasted at the end of the mail. > > > TWO: I had to create a softlink to libcrypto.so.1.1 and libssl.so.1.1 like the following to make openssl command work: > ln -s /usr/local/lib64/libssl.so.1.1 /lib64/libssl.so.1.1 > ln -s /usr/local/lib64/libcrypto.so.1.1 /lib64/libcrypto.so.1.1 > > Is creating the softlinks a known issue and will be fixed? No, this will not be fixed and may not be the most appropriate thing to do on all systems. Matt > > I have pasted the complete information about the OS/distro environment and installation commands I ran at the bottom. > Could you please suggest what wrong I am doing to make afalg work. > > Thanks > Jitendra Lulla > > ======== > > > BEFORE INSTALLATION: > > [root at localhost jlulla]# rpm -qa? |grep openssl > openssl-1.0.1e-60.el7.x86_64 > openssl-devel-1.0.1e-60.el7.x86_64 > openssl-libs-1.0.1e-60.el7.x86_64 > > [root at localhost jlulla]# openssl version > OpenSSL 1.0.1e-fips 11 Feb 2013 > > > > PLEASE SEE FROM HERE PLEASE SEE FROM HERE PLEASE SEE FROM HERE---------------------------- > > STEP 1 : SOURCE TAKEN FROM https://www.openssl.org/source/openssl-1.1.0f.tar.gz 2017-May-25 13:09:51 > > [root at localhost jlulla]# uname -a > Linux localhost.localdomain 4.9.37 #1 SMP Fri Jul 21 04:52:46 PDT 2017 x86_64 x86_64 x86_64 GNU/Linux > > [root at localhost jlulla]# cat /etc/redhat-release > Red Hat Enterprise Linux Server release 7.3 (Maipo) > > > > [root at localhost openssl-1.1.0f]# pwd > /home/jlulla/openssl-1.1.0f > > STEP 2: [root at localhost openssl-1.1.0f]# ./config shared enable-engine enable-dso enable-afalgeng > Operating system: x86_64-whatever-linux2 > Configuring for linux-x86_64 > Configuring OpenSSL version 1.1.0f (0x1010006fL) >? ? no-asan? ? ? ? [default]? OPENSSL_NO_ASAN >? ?? no-crypto-mdebug [default]? OPENSSL_NO_CRYPTO_MDEBUG >? ? no-crypto-mdebug-backtrace [default]? OPENSSL_NO_CRYPTO_MDEBUG_BACKTRACE >? ? ? no-ec_nistp_64_gcc_128 [default]? OPENSSL_NO_EC_NISTP_64_GCC_128 >? ? no-egd? ? ? ? ? [default]? OPENSSL_NO_EGD >? ?? no-fuzz-afl? ? [default]? OPENSSL_NO_FUZZ_AFL >? ? no-fuzz-libfuzzer [default]? OPENSSL_NO_FUZZ_LIBFUZZER >? ?? no-heartbeats? [default]? OPENSSL_NO_HEARTBEATS >? ?? no-md2? ? ? ? ? [default]? OPENSSL_NO_MD2 (skip dir) >? ?? no-msan? ? ? ? [default]? OPENSSL_NO_MSAN >? ?? no-rc5? ? ? ? ? [default]? OPENSSL_NO_RC5 (skip dir) >? ?? no-sctp? ? ? ? [default]? OPENSSL_NO_SCTP >? ?? no-ssl-trace? ? [default]? OPENSSL_NO_SSL_TRACE >? ? no-ssl3? ? ? ? [default]? OPENSSL_NO_SSL3 >? ?? no-ssl3-method? [default]? OPENSSL_NO_SSL3_METHOD >? ?? no-ubsan? ? ? ? [default]? OPENSSL_NO_UBSAN >? ?? no-unit-test? ? [default]? OPENSSL_NO_UNIT_TEST >? ?? no-weak-ssl-ciphers [default]? OPENSSL_NO_WEAK_SSL_CIPHERS >? ? no-zlib? ? ? ? [default] >? ? no-zlib-dynamic [default] > Configuring for linux-x86_64 > CC? ? ? ? ? ? =gcc > CFLAG? ? ? ? =-Wall -O3 -pthread -m64 -DL_ENDIAN? -Wa,--noexecstack > SHARED_CFLAG? =-fPIC -DOPENSSL_USE_NODELETE > DEFINES? ? ? =DSO_DLFCN HAVE_DLFCN_H NDEBUG OPENSSL_THREADS OPENSSL_NO_STATIC_ENGINE OPENSSL_PIC OPENSSL_IA32_SSE2 OPENSSL_BN_ASM_MONT OPENSSL_BN_ASM_MONT5 OPENSSL_BN_ASM_GF2m SHA1_ASM SHA256_ASM SHA512_ASM RC4_ASM MD5_ASM AES_ASM VPAES_ASM BSAES_ASM GHASH_ASM ECP_NISTZ256_ASM PADLOCK_ASM POLY1305_ASM > LFLAG? ? ? ? = > PLIB_LFLAG? ? = > EX_LIBS? ? ? =-ldl > APPS_OBJ? ? ? = > CPUID_OBJ? ? =x86_64cpuid.o > UPLINK_OBJ? ? = > BN_ASM? ? ? ? =asm/x86_64-gcc.o x86_64-mont.o x86_64-mont5.o x86_64-gf2m.o rsaz_exp.o rsaz-x86_64.o rsaz-avx2.o > EC_ASM? ? ? ? =ecp_nistz256.o ecp_nistz256-x86_64.o > DES_ENC? ? ? =des_enc.o fcrypt_b.o > AES_ENC? ? ? =aes-x86_64.o vpaes-x86_64.o bsaes-x86_64.o aesni-x86_64.o aesni-sha1-x86_64.o aesni-sha256-x86_64.o aesni-mb-x86_64.o > BF_ENC? ? ? ? =bf_enc.o > CAST_ENC? ? ? =c_enc.o > RC4_ENC? ? ? =rc4-x86_64.o rc4-md5-x86_64.o > RC5_ENC? ? ? =rc5_enc.o > MD5_OBJ_ASM? =md5-x86_64.o > SHA1_OBJ_ASM? =sha1-x86_64.o sha256-x86_64.o sha512-x86_64.o sha1-mb-x86_64.o sha256-mb-x86_64.o > RMD160_OBJ_ASM= > CMLL_ENC? ? ? =cmll-x86_64.o cmll_misc.o > MODES_OBJ? ? =ghash-x86_64.o aesni-gcm-x86_64.o > PADLOCK_OBJ? =e_padlock-x86_64.o > CHACHA_ENC? ? =chacha-x86_64.o > POLY1305_OBJ? =poly1305-x86_64.o > BLAKE2_OBJ? ? = > PROCESSOR? ? = > RANLIB? ? ? ? =ranlib > ARFLAGS? ? ? = > PERL? ? ? ? ? =/usr/bin/perl > > SIXTY_FOUR_BIT_LONG mode > > Configured for linux-x86_64. > [root at localhost openssl-1.1.0f]# > > make didnt give any error. > > make install didnt give any error > > updatedb > > STEP 2: > [root at localhost openssl-1.1.0f]# md5sum /home/jlulla/openssl-1.1.0f/libssl.so.1.1 /usr/local/lib64/libssl.so.1.1 > 5a7264a3ebda22989c831794d30677c5? /home/jlulla/openssl-1.1.0f/libssl.so.1.1 > 5a7264a3ebda22989c831794d30677c5? /usr/local/lib64/libssl.so.1.1 > > [root at localhost openssl-1.1.0f]# ls -l /home/jlulla/openssl-1.1.0f/libssl.so > lrwxrwxrwx. 1 root root 13 Aug 10 23:45 /home/jlulla/openssl-1.1.0f/libssl.so -> libssl.so.1.1 > > > [root at localhost apps]# pwd > /home/jlulla/openssl-1.1.0f/apps > [root at localhost apps]# ./openssl > ./openssl: error while loading shared libraries: libssl.so.1.1: cannot open shared object file: No such file or directory > > [root at localhost apps]# ldd ./openssl >? ? ? ? linux-vdso.so.1 =>? (0x00007ffd02f3c000) >? ? ? ?? libssl.so.1.1 => not found >? ? ? ?? libcrypto.so.1.1 => not found >? ? ? ?? libdl.so.2 => /lib64/libdl.so.2 (0x00007f0439fda000) >? ? ? ?? libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f0439dbe000) >? ? ? ?? libc.so.6 => /lib64/libc.so.6 (0x00007f04399fc000) >? ? ? ?? /lib64/ld-linux-x86-64.so.2 (0x000055d618530000) > > > [root at localhost apps]# ls -l /lib64/libssl.so.1.1 > ls: cannot access /lib64/libssl.so.1.1: No such file or directory > > ln -s /usr/local/lib64/libssl.so.1.1 /lib64/libssl.so.1.1 > > STEP 3: > [root at localhost apps]# ln -s /usr/local/lib64/libssl.so.1.1 /lib64/libssl.so.1.1 > [root at localhost apps]# ls -l /lib64/libssl.so.1.1 > lrwxrwxrwx. 1 root root 30 Aug 10 23:57 /lib64/libssl.so.1.1 -> /usr/local/lib64/libssl.so.1.1 > > > [root at localhost apps]# ln -s /usr/local/lib64/libcrypto.so.1.1 /lib64/libcrypto.so.1.1 > [root at localhost apps]# ls -l /lib64/libcrypto.so.1.1 > lrwxrwxrwx. 1 root root 33 Aug 10 23:58 /lib64/libcrypto.so.1.1 -> /usr/local/lib64/libcrypto.so.1.1 > > [root at localhost apps]# ldd ./openssl >? ? ? ? linux-vdso.so.1 =>? (0x00007ffd37b5e000) >? ? ? ?? libssl.so.1.1 => /lib64/libssl.so.1.1 (0x00007f82881f8000) >? ? ? ?? libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007f8287d73000) >? ? ? ?? libdl.so.2 => /lib64/libdl.so.2 (0x00007f8287b6e000) >? ? ? ?? libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f8287952000) >? ? ? ?? libc.so.6 => /lib64/libc.so.6 (0x00007f8287591000) >? ? ? ?? /lib64/ld-linux-x86-64.so.2 (0x000055f06bc67000) > > > [root at localhost apps]# ./openssl version > OpenSSL 1.1.0f? 25 May 2017 > > > STEP 4: > [root at localhost afalg]# pwd > /home/jlulla/openssl-1.1.0f/engines/afalg > > [root at localhost afalg]# ls > afalg.so? build.info? e_afalg.c? e_afalg.d? e_afalg.ec? e_afalg_err.c? e_afalg_err.d? e_afalg_err.h? e_afalg_err.o? e_afalg.h? e_afalg.o > > [root at localhost afalg]# ldd afalg.so >? ? ? ? linux-vdso.so.1 =>? (0x00007fff1d7ef000) >? ? ? ?? libcrypto.so.1.1 => /lib64/libcrypto.so.1.1 (0x00007f3a80166000) >? ? ? ?? libdl.so.2 => /lib64/libdl.so.2 (0x00007f3a7ff62000) >? ? ? ?? libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f3a7fd45000) >? ? ? ?? libc.so.6 => /lib64/libc.so.6 (0x00007f3a7f984000) >? ? ? ?? /lib64/ld-linux-x86-64.so.2 (0x0000560991976000) > > > [root at localhost afalg]# nm afalg.so > 0000000000202060 d AFALG_error_init > 0000000000202220 b AFALG_lib_error_code > 0000000000202160 d AFALG_str_functs > 0000000000202080 d AFALG_str_reasons > 0000000000202210 B __bss_start > 0000000000202210 b completed.6344 >? ? ? ? ? ? ? ? w __cxa_finalize@@GLIBC_2.2.5 > 0000000000000a00 t deregister_tm_clones > 0000000000000a70 t __do_global_dtors_aux > 0000000000201da0 t __do_global_dtors_aux_fini_array_entry > 0000000000201db0 d __dso_handle > 0000000000201db8 d _DYNAMIC > 0000000000202210 D _edata > 0000000000202228 B _end > 0000000000000af0 T engine_load_afalg_int > 0000000000000bb0 T ERR_AFALG_error >? ? ? ? ? ? ? ?? U ERR_get_next_error_library@@OPENSSL_1_1_0 > 0000000000000b00 T ERR_load_AFALG_strings >? ? ? ? ? ? ? ?? U ERR_load_strings@@OPENSSL_1_1_0 >? ? ? ? ? ? ? ?? U ERR_put_error@@OPENSSL_1_1_0 > 0000000000000b60 T ERR_unload_AFALG_strings >? ? ? ? ? ? ? ?? U ERR_unload_strings@@OPENSSL_1_1_0 > 0000000000000bf8 T _fini > 0000000000000ab0 t frame_dummy > 0000000000201d98 t __frame_dummy_init_array_entry > 0000000000000eb8 r __FRAME_END__ > 0000000000202000 d _GLOBAL_OFFSET_TABLE_ >? ? ? ? ? ? ? ?? w __gmon_start__ > 0000000000000970 T _init >? ? ? ? ? ? ? ?? w _ITM_deregisterTMCloneTable >? ? ? ? ? ? ? ?? w _ITM_registerTMCloneTable > 0000000000201da8 d __JCR_END__ > 0000000000201da8 d __JCR_LIST__ >? ? ? ? ? ? ? ?? w _Jv_RegisterClasses > 0000000000000a30 t register_tm_clones > 0000000000202210 d __TMC_END__ > > > [root at localhost openssl-1.1.0f]# locate afalg.so > /home/jlulla/openssl-1.1.0f/engines/afalg/afalg.so > /usr/local/lib64/engines-1.1/afalg.so > > > [root at localhost openssl-1.1.0f]# md5sum /home/jlulla/openssl-1.1.0f/engines/afalg/afalg.so /usr/local/lib64/engines-1.1/afalg.so > ec273faa6eb6e25f55e44a7d37adf195? /home/jlulla/openssl-1.1.0f/engines/afalg/afalg.so > ec273faa6eb6e25f55e44a7d37adf195? /usr/local/lib64/engines-1.1/afalg.so > > > [root at localhost apps]# date > Fri Aug 11 00:01:10 PDT 2017 > > > [root at localhost apps]# ls -l /usr/local/ssl/openssl.cnf > -rw-r--r--. 1 root root 10771 Aug 10 23:47 /usr/local/ssl/openssl.cnf > > [root at localhost apps]# md5sum /usr/local/ssl/openssl.cnf /home/jlulla/openssl-1.1.0f/apps/openssl.cnf > f697ef5df0d006882e6326606e8dbf4a? /usr/local/ssl/openssl.cnf > f697ef5df0d006882e6326606e8dbf4a? /home/jlulla/openssl-1.1.0f/apps/openssl.cnf > > > openssl.cnf has the following: > .. > .. > # Extra OBJECT IDENTIFIER info: > #oid_file? ? ? ? ? ? ? = $ENV::HOME/.oid > oid_section? ? ? ? ? ? = new_oids > > # To use this configuration file with the "-extfile" option of the > # "openssl x509" utility, name here the section containing the > # X.509v3 extensions to use: > # extensions? ? ? ? ? ? = > # (Alternatively, use a configuration file that has only > # X.509v3 extensions in its main [= default] section.) > > .. > .. > > [root at localhost apps]# ./openssl speed -evp aes-128-cbc -engine afalg > invalid engine "afalg" > 139853452924736:error:2506406A:DSO support routines:dlfcn_bind_func:could not bind to the requested symbol name:crypto/dso/dso_dlfcn.c:178:symname(bind_engine): /usr/local/lib64/engines-1.1/afalg.so: undefined symbol: bind_engine > 139853452924736:error:2506C06A:DSO support routines:DSO_bind_func:could not bind to the requested symbol name:crypto/dso/dso_lib.c:185: > 139853452924736:error:260B6068:engine routines:dynamic_load:DSO failure:crypto/engine/eng_dyn.c:427: > 139853452924736:error:2606A074:engine routines:ENGINE_by_id:no such engine:crypto/engine/eng_list.c:339:id=afalg > 139853452924736:error:25066067:DSO support routines:dlfcn_load:could not load the shared library:crypto/dso/dso_dlfcn.c:113:filename(libafalg.so): libafalg.so: cannot open shared object file: No such file or directory > 139853452924736:error:25070067:DSO support routines:DSO_load:could not load the shared library:crypto/dso/dso_lib.c:161: > 139853452924736:error:260B6084:engine routines:dynamic_load:dso not found:crypto/engine/eng_dyn.c:414: > ^C > > > Modified both the files as the follwing: > /usr/local/ssl/openssl.cnf > /home/jlulla/openssl-1.1.0f/apps/openssl.cnf > > # Extra OBJECT IDENTIFIER info: > #oid_file? ? ? ? ? ? ? = $ENV::HOME/.oid > oid_section? ? ? ? ? ? = new_oids > > openssl_conf = openssl_def > > [openssl_def] > engines = openssl_engines > > [openssl_engines] > afalg = afalg_engine > > [afalg_engine] > CIPHERS=aes-128-cbc > > With the above, the errors are more: > [root at localhost apps]# ./openssl speed -evp aes-128-cbc -engine afalg > invalid engine "afalg" > 140473780422464:error:2506406A:DSO support routines:dlfcn_bind_func:could not bind to the requested symbol name:crypto/dso/dso_dlfcn.c:178:symname(bind_engine): /usr/local/lib64/engines-1.1/afalg.so: undefined symbol: bind_engine > 140473780422464:error:2506C06A:DSO support routines:DSO_bind_func:could not bind to the requested symbol name:crypto/dso/dso_lib.c:185: > 140473780422464:error:260B6068:engine routines:dynamic_load:DSO failure:crypto/engine/eng_dyn.c:427: > 140473780422464:error:2606A074:engine routines:ENGINE_by_id:no such engine:crypto/engine/eng_list.c:339:id=afalg > 140473780422464:error:260BC066:engine routines:int_engine_configure:engine configuration error:crypto/engine/eng_cnf.c:141:section=afalg_engine, name=CIPHERS, value=aes-128-cbc > 140473780422464:error:0E07606D:configuration file routines:module_run:module initialization error:crypto/conf/conf_mod.c:173:module=engines, value=openssl_engines, retcode=-1 > 140473780422464:error:2506406A:DSO support routines:dlfcn_bind_func:could not bind to the requested symbol name:crypto/dso/dso_dlfcn.c:178:symname(bind_engine): /usr/local/lib64/engines-1.1/afalg.so: undefined symbol: bind_engine > 140473780422464:error:2506C06A:DSO support routines:DSO_bind_func:could not bind to the requested symbol name:crypto/dso/dso_lib.c:185: > 140473780422464:error:260B6068:engine routines:dynamic_load:DSO failure:crypto/engine/eng_dyn.c:427: > 140473780422464:error:2606A074:engine routines:ENGINE_by_id:no such engine:crypto/engine/eng_list.c:339:id=afalg > 140473780422464:error:25066067:DSO support routines:dlfcn_load:could not load the shared library:crypto/dso/dso_dlfcn.c:113:filename(libafalg.so): libafalg.so: cannot open shared object file: No such file or directory > 140473780422464:error:25070067:DSO support routines:DSO_load:could not load the shared library:crypto/dso/dso_lib.c:161: > 140473780422464:error:260B6084:engine routines:dynamic_load:dso not found:crypto/engine/eng_dyn.c:414: > ^C > From matt at openssl.org Wed Aug 16 08:15:48 2017 From: matt at openssl.org (Matt Caswell) Date: Wed, 16 Aug 2017 09:15:48 +0100 Subject: [openssl-dev] afalg with OpenSSL 1.1.0f 25 May 2017 In-Reply-To: <736159717.2581176.1502870456604@mail.yahoo.com> References: <736159717.2581176.1502870456604.ref@mail.yahoo.com> <736159717.2581176.1502870456604@mail.yahoo.com> Message-ID: On 16/08/17 09:00, Jitendra Lulla wrote: > Hi Matt, > > Thanks, I could find that the /usr/include/linux/version.h has #define LINUX_VERSION_CODE 199168 for my booted kernel 4.9.37. Which is why I see the following warnings also: > > gcc -Iinclude -DDSO_DLFCN -DHAVE_DLFCN_H -DNDEBUG -DOPENSSL_THREADS -DOPENSSL_NO_STATIC_ENGINE -DOPENSSL_PIC -DOPENSSL_IA32_SSE2 -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DRC4_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM -DBSAES_ASM -DGHASH_ASM -DECP_NISTZ256_ASM -DPADLOCK_ASM -DPOLY1305_ASM -DOPENSSLDIR="\"/usr/local/ssl\"" -DENGINESDIR="\"/usr/local/lib64/engines-1.1\"" -Wall -O3 -pthread -m64 -DL_ENDIAN -Wa,--noexecstack -fPIC -DOPENSSL_USE_NODELETE -MMD -MF engines/afalg/e_afalg.d.tmp -MT engines/afalg/e_afalg.o -c -o engines/afalg/e_afalg.o engines/afalg/e_afalg.c > engines/afalg/e_afalg.c:30:4: warning: #warning "AFALG ENGINE requires Kernel Headers >= 4.1.0" [-Wcpp] > # warning "AFALG ENGINE requires Kernel Headers >= 4.1.0" > ^ > engines/afalg/e_afalg.c:31:4: warning: #warning "Skipping Compilation of AFALG engine" [-Wcpp] > # warning "Skipping Compilation of AFALG engine" > > > I will fix this problem now by having proper setup. Will update if I face any more issues. Yes - your kernel headers must be at least at 4.1.0 or better for afalg to work. Matt > > Thanks > Jitendra > > > > > -------------------------------------------- > On Wed, 8/16/17, Jitendra Lulla wrote: > > Subject: Re: afalg with OpenSSL 1.1.0f 25 May 2017 > To: "openssl-dev at openssl.org" , "Matt Caswell" > Cc: "Jitendra Lulla" > Date: Wednesday, August 16, 2017, 6:30 AM > > Hi Matt, > > > I have linux 4.9.37 on RHEL7.3. > [root at localhost > jlulla]# uname -a > Linux localhost.localdomain 4.9.37 #1 > SMP Fri Jul 21 04:52:46 PDT 2017 x86_64 x86_64 x86_64 > GNU/Linux > > > [root at localhost > test]# OPENSSL_ENGINES=../engines/afalg > ../util/shlib_wrap.sh ./afalgtest > AFALG not supported - skipping AFALG > tests > PASS > [root at localhost > test]# > > > I am getting here: > # if LINUX_VERSION_CODE <= > KERNEL_VERSION(K_MAJ, K_MIN1, K_MIN2) > /* > * If we get here then it looks like > there is a mismatch between the linux > * headers and the actual kernel > version, so we have tried to compile with > * afalg support, but then skipped it > in e_afalg.c. As far as this test is > * concerned we behave as if we had > been configured without support > */ > # define OPENSSL_NO_AFALGENG > # endif > > > Following is the value for > KERNEL_VERSION for me: > > [root at localhost > jlulla]# ./kernelversion (program at the bottom of this > mail) > KERNEL_VERSION: 262400 > LINUX_VERSION_CODE 199168 > condition:1 > > > Where should I look to fix it? > > Thanks > Jitrendra > > > [root at localhost > jlulla]# cat kernelversion.c > #define LINUX_VERSION_CODE 199168 > #define KERNEL_VERSION(a,b,c) (((a) > << 16) + ((b) << 8) + (c)) > #define RHEL_MAJOR 7 > #define RHEL_MINOR 3 > #define RHEL_RELEASE_VERSION(a,b) (((a) > << 8) + (b)) > #define RHEL_RELEASE_CODE 1795 > #define RHEL_RELEASE "514" > > # define K_MAJ 4 > # define K_MIN1 1 > # define K_MIN2 0 > #include > > int main() > { > > printf("KERNEL_VERSION: %d\n", KERNEL_VERSION(K_MAJ, > K_MIN1, K_MIN2)); > > printf("LINUX_VERSION_CODE %d\n", LINUX_VERSION_CODE); > > printf("condition:%d\n", > > > (LINUX_VERSION_CODE <= KERNEL_VERSION(K_MAJ, K_MIN1, > K_MIN2))); > } > > > > -------------------------------------------- > On Mon, 8/14/17, Matt Caswell > wrote: > > Subject: Re: afalg with OpenSSL 1.1.0f > 25 May 2017 > To: "openssl-dev at openssl.org" > > Cc: "Jitendra Lulla" > Date: Monday, August 14, 2017, 3:44 > PM > > Comments inserted. > > On 14/08/17 08:20, Jitendra > Lulla wrote: > > Hi, > > > > > I am trying to use afalg on > Linux > 4.9.37 with OpenSSL 1.1.0f. > > > > I am facing 2 issues: > > > > > ONE: when I issue the speed > command, I > see the following: > > > > [root at localhost > apps]# ./openssl speed -evp > aes-128-cbc -engine afalg > > invalid engine "afalg" > > > 139853452924736:error:2506406A:DSO support > routines:dlfcn_bind_func:could not > bind to the requested > symbol > name:crypto/dso/dso_dlfcn.c:178:symname(bind_engine): > /usr/local/lib64/engines-1.1/afalg.so: > undefined symbol: > bind_engine > > > 139853452924736:error:2506C06A:DSO > support > routines:DSO_bind_func:could not bind > to the requested > symbol name:crypto/dso/dso_lib.c:185: > > > 139853452924736:error:260B6068:engine > routines:dynamic_load:DSO > failure:crypto/engine/eng_dyn.c:427: > > > 139853452924736:error:2606A074:engine > routines:ENGINE_by_id:no such > > engine:crypto/engine/eng_list.c:339:id=afalg > > > 139853452924736:error:25066067:DS > > > > > > nm afalg.so doesn't show > bind_engine > > > Assuming > you have already successfully built > OpenSSL using > "make", from > the "test" > subdir of the directory where you > downloaded the source, > what > happens if you execute: > > OPENSSL_ENGINES=../engines/afalg > ../util/shlib_wrap.sh ./afalgtest > > Another thing to try is (from the top > level > source dir) > > touch > engines/afalg/e_afalg.c > make > > Check to see if there are any > warnings generated during the > compilation > of > the engine. > > > > > When I modify the openssl.cnf > file with > the engine name and the CIPHERS, still > I dont get it > working. The command output and the > change in the > openssl.cnf pasted at the end of the > mail. > > > > > > TWO: I had to create a softlink > to > libcrypto.so.1.1 and libssl.so.1.1 > like the following to > make openssl command work: > > ln -s > /usr/local/lib64/libssl.so.1.1 > /lib64/libssl.so.1.1 > > ln -s > /usr/local/lib64/libcrypto.so.1.1 > /lib64/libcrypto.so.1.1 > > > > Is creating the softlinks a known > issue > and will be fixed? > No, this will not be > fixed and may not be the most > appropriate thing to > do on all systems. > > > Matt > > > > > > I have pasted the > complete information about the > OS/distro environment and > installation commands I ran at the > bottom. > > Could you please suggest what > wrong I am > doing to make afalg work. > > > > Thanks > > Jitendra > Lulla > > > > ======== > > > > > > BEFORE INSTALLATION: > > > > > [root at localhost > jlulla]# rpm -qa |grep openssl > > > openssl-1.0.1e-60.el7.x86_64 > > > openssl-devel-1.0.1e-60.el7.x86_64 > > > openssl-libs-1.0.1e-60.el7.x86_64 > > > > [root at localhost > jlulla]# openssl version > > OpenSSL > 1.0.1e-fips 11 Feb 2013 > > > > > > > > PLEASE SEE FROM HERE PLEASE SEE > FROM HERE > PLEASE SEE FROM > HERE---------------------------- > > > > STEP 1 : SOURCE > TAKEN FROM https://www.openssl.org/source/openssl-1.1.0f.tar.gz > 2017-May-25 13:09:51 > > > > [root at localhost > jlulla]# uname -a > > Linux > localhost.localdomain 4.9.37 #1 SMP > Fri Jul 21 04:52:46 PDT > 2017 x86_64 x86_64 x86_64 GNU/Linux > > > > [root at localhost > jlulla]# cat /etc/redhat-release > > Red > Hat Enterprise Linux Server release > 7.3 (Maipo) > > > > > > > > [root at localhost > openssl-1.1.0f]# pwd > > > /home/jlulla/openssl-1.1.0f > > > > STEP 2: [root at localhost > openssl-1.1.0f]# ./config shared > enable-engine enable-dso > enable-afalgeng > > Operating system: > x86_64-whatever-linux2 > > Configuring for > linux-x86_64 > > Configuring OpenSSL > version 1.1.0f (0x1010006fL) > > > no-asan [default] > OPENSSL_NO_ASAN > > no-crypto-mdebug > [default] > OPENSSL_NO_CRYPTO_MDEBUG > > > no-crypto-mdebug-backtrace > [default] > OPENSSL_NO_CRYPTO_MDEBUG_BACKTRACE > > > no-ec_nistp_64_gcc_128 > [default] > OPENSSL_NO_EC_NISTP_64_GCC_128 > > > no-egd [default] > OPENSSL_NO_EGD > > no-fuzz-afl > [default] > OPENSSL_NO_FUZZ_AFL > > > no-fuzz-libfuzzer [default] > OPENSSL_NO_FUZZ_LIBFUZZER > > no-heartbeats > [default] > OPENSSL_NO_HEARTBEATS > > no-md2 > [default] OPENSSL_NO_MD2 > (skip dir) > > no-msan > [default] > OPENSSL_NO_MSAN > > no-rc5 > [default] OPENSSL_NO_RC5 (skip > dir) > > no-sctp > [default] > OPENSSL_NO_SCTP > > no-ssl-trace > [default] OPENSSL_NO_SSL_TRACE > > > no-ssl3 [default] > OPENSSL_NO_SSL3 > > no-ssl3-method > [default] > OPENSSL_NO_SSL3_METHOD > > no-ubsan > [default] OPENSSL_NO_UBSAN > > > no-unit-test [default] > OPENSSL_NO_UNIT_TEST > > no-weak-ssl-ciphers > [default] > OPENSSL_NO_WEAK_SSL_CIPHERS > > > no-zlib [default] > > > no-zlib-dynamic [default] > > Configuring > for linux-x86_64 > > CC > =gcc > > CFLAG =-Wall -O3 > -pthread -m64 -DL_ENDIAN > -Wa,--noexecstack > > SHARED_CFLAG =-fPIC > -DOPENSSL_USE_NODELETE > > DEFINES > =DSO_DLFCN HAVE_DLFCN_H NDEBUG > OPENSSL_THREADS > OPENSSL_NO_STATIC_ENGINE OPENSSL_PIC > OPENSSL_IA32_SSE2 > OPENSSL_BN_ASM_MONT > OPENSSL_BN_ASM_MONT5 OPENSSL_BN_ASM_GF2m > SHA1_ASM SHA256_ASM SHA512_ASM RC4_ASM > MD5_ASM AES_ASM > VPAES_ASM BSAES_ASM GHASH_ASM > ECP_NISTZ256_ASM PADLOCK_ASM > POLY1305_ASM > > LFLAG = > > PLIB_LFLAG = > > > EX_LIBS =-ldl > > APPS_OBJ > = > > CPUID_OBJ =x86_64cpuid.o > > UPLINK_OBJ = > > > BN_ASM =asm/x86_64-gcc.o > x86_64-mont.o > x86_64-mont5.o x86_64-gf2m.o > rsaz_exp.o rsaz-x86_64.o > rsaz-avx2.o > > EC_ASM > =ecp_nistz256.o ecp_nistz256-x86_64.o > > > DES_ENC =des_enc.o fcrypt_b.o > > > AES_ENC =aes-x86_64.o > vpaes-x86_64.o bsaes-x86_64.o > aesni-x86_64.o aesni-sha1-x86_64.o > aesni-sha256-x86_64.o > aesni-mb-x86_64.o > > BF_ENC > =bf_enc.o > > CAST_ENC =c_enc.o > > RC4_ENC =rc4-x86_64.o > rc4-md5-x86_64.o > > RC5_ENC > =rc5_enc.o > > MD5_OBJ_ASM > =md5-x86_64.o > > SHA1_OBJ_ASM > =sha1-x86_64.o sha256-x86_64.o > sha512-x86_64.o > sha1-mb-x86_64.o sha256-mb-x86_64.o > > > RMD160_OBJ_ASM= > > CMLL_ENC > =cmll-x86_64.o cmll_misc.o > > MODES_OBJ > =ghash-x86_64.o aesni-gcm-x86_64.o > > > PADLOCK_OBJ =e_padlock-x86_64.o > > > CHACHA_ENC =chacha-x86_64.o > > > POLY1305_OBJ =poly1305-x86_64.o > > > BLAKE2_OBJ = > > PROCESSOR = > > RANLIB =ranlib > > ARFLAGS = > > > PERL =/usr/bin/perl > > > > SIXTY_FOUR_BIT_LONG mode > > > > Configured for > linux-x86_64. > > [root at localhost > openssl-1.1.0f]# > > > > > make didnt give any error. > > > > make install didnt give any > error > > > > updatedb > > > > STEP 2: > > [root at localhost > openssl-1.1.0f]# md5sum > > /home/jlulla/openssl-1.1.0f/libssl.so.1.1 > /usr/local/lib64/libssl.so.1.1 > > > 5a7264a3ebda22989c831794d30677c5 > > /home/jlulla/openssl-1.1.0f/libssl.so.1.1 > > > 5a7264a3ebda22989c831794d30677c5 > /usr/local/lib64/libssl.so.1.1 > > > > [root at localhost > openssl-1.1.0f]# ls -l > /home/jlulla/openssl-1.1.0f/libssl.so > > > lrwxrwxrwx. 1 root root 13 Aug 10 > 23:45 > /home/jlulla/openssl-1.1.0f/libssl.so > -> libssl.so.1.1 > > > > > > [root at localhost > apps]# pwd > > > /home/jlulla/openssl-1.1.0f/apps > > [root at localhost > apps]# ./openssl > > ./openssl: error while > loading shared libraries: > libssl.so.1.1: cannot open shared > object file: No such file or > directory > > > > > [root at localhost > apps]# ldd ./openssl > > > linux-vdso.so.1 => > (0x00007ffd02f3c000) > > libssl.so.1.1 > => not > found > > > libcrypto.so.1.1 > => not found > > libdl.so.2 > => /lib64/libdl.so.2 > (0x00007f0439fda000) > > libpthread.so.0 > => > /lib64/libpthread.so.0 > (0x00007f0439dbe000) > > libc.so.6 => > /lib64/libc.so.6 (0x00007f04399fc000) > > > > /lib64/ld-linux-x86-64.so.2 > (0x000055d618530000) > > > > > > [root at localhost > apps]# ls -l /lib64/libssl.so.1.1 > > ls: > cannot access /lib64/libssl.so.1.1: No > such file or > directory > > > > ln -s > /usr/local/lib64/libssl.so.1.1 > /lib64/libssl.so.1.1 > > > > STEP 3: > > [root at localhost > apps]# ln -s > /usr/local/lib64/libssl.so.1.1 > /lib64/libssl.so.1.1 > > [root at localhost > apps]# ls -l /lib64/libssl.so.1.1 > > > lrwxrwxrwx. 1 root root 30 Aug 10 > 23:57 /lib64/libssl.so.1.1 > -> /usr/local/lib64/libssl.so.1.1 > > > > > > > [root at localhost > apps]# ln -s > /usr/local/lib64/libcrypto.so.1.1 > /lib64/libcrypto.so.1.1 > > [root at localhost > apps]# ls -l /lib64/libcrypto.so.1.1 > > > lrwxrwxrwx. 1 root root 33 Aug 10 > 23:58 > /lib64/libcrypto.so.1.1 -> > /usr/local/lib64/libcrypto.so.1.1 > > > > [root at localhost > apps]# ldd ./openssl > > > linux-vdso.so.1 => > (0x00007ffd37b5e000) > > libssl.so.1.1 > => > /lib64/libssl.so.1.1 > (0x00007f82881f8000) > > libcrypto.so.1.1 > => > /lib64/libcrypto.so.1.1 > (0x00007f8287d73000) > > libdl.so.2 > => > /lib64/libdl.so.2 > (0x00007f8287b6e000) > > libpthread.so.0 > => > /lib64/libpthread.so.0 > (0x00007f8287952000) > > libc.so.6 => > /lib64/libc.so.6 (0x00007f8287591000) > > > > /lib64/ld-linux-x86-64.so.2 > (0x000055f06bc67000) > > > > > > [root at localhost > apps]# ./openssl version > > OpenSSL > 1.1.0f 25 May 2017 > > > > > > STEP 4: > > [root at localhost > afalg]# pwd > > > > /home/jlulla/openssl-1.1.0f/engines/afalg > > > > [root at localhost > afalg]# ls > > afalg.so build.info > e_afalg.c e_afalg.d e_afalg.ec > e_afalg_err.c > e_afalg_err.d e_afalg_err.h > e_afalg_err.o e_afalg.h > e_afalg.o > > > > [root at localhost > afalg]# ldd afalg.so > > > linux-vdso.so.1 => > (0x00007fff1d7ef000) > > libcrypto.so.1.1 > => > /lib64/libcrypto.so.1.1 > (0x00007f3a80166000) > > libdl.so.2 > => > /lib64/libdl.so.2 > (0x00007f3a7ff62000) > > libpthread.so.0 > => > /lib64/libpthread.so.0 > (0x00007f3a7fd45000) > > libc.so.6 => > /lib64/libc.so.6 (0x00007f3a7f984000) > > > > /lib64/ld-linux-x86-64.so.2 > (0x0000560991976000) > > > > > > [root at localhost > afalg]# nm afalg.so > > 0000000000202060 d > AFALG_error_init > > 0000000000202220 b > AFALG_lib_error_code > > 0000000000202160 d > AFALG_str_functs > > 0000000000202080 d > AFALG_str_reasons > > 0000000000202210 B > __bss_start > > 0000000000202210 b > completed.6344 > > > w __cxa_finalize@@GLIBC_2.2.5 > > > 0000000000000a00 t > deregister_tm_clones > > > 0000000000000a70 t > __do_global_dtors_aux > > 0000000000201da0 t > > __do_global_dtors_aux_fini_array_entry > > > 0000000000201db0 d __dso_handle > > > 0000000000201db8 d _DYNAMIC > > > 0000000000202210 D _edata > > > 0000000000202228 B _end > > > 0000000000000af0 T > engine_load_afalg_int > > 0000000000000bb0 T > ERR_AFALG_error > > U > > ERR_get_next_error_library@@OPENSSL_1_1_0 > > 0000000000000b00 T > ERR_load_AFALG_strings > > > U > ERR_load_strings@@OPENSSL_1_1_0 > > U > ERR_put_error@@OPENSSL_1_1_0 > > > 0000000000000b60 T > ERR_unload_AFALG_strings > > U > ERR_unload_strings@@OPENSSL_1_1_0 > > > 0000000000000bf8 T _fini > > > 0000000000000ab0 t frame_dummy > > > 0000000000201d98 t > __frame_dummy_init_array_entry > > 0000000000000eb8 r __FRAME_END__ > > 0000000000202000 d > _GLOBAL_OFFSET_TABLE_ > > > w __gmon_start__ > > > 0000000000000970 T _init > > > w > _ITM_deregisterTMCloneTable > > w > _ITM_registerTMCloneTable > > > 0000000000201da8 d __JCR_END__ > > > 0000000000201da8 d __JCR_LIST__ > > > w > _Jv_RegisterClasses > > > 0000000000000a30 t register_tm_clones > > > 0000000000202210 d __TMC_END__ > > > > > > [root at localhost > openssl-1.1.0f]# locate afalg.so > > > > /home/jlulla/openssl-1.1.0f/engines/afalg/afalg.so > > > /usr/local/lib64/engines-1.1/afalg.so > > > > > > [root at localhost > openssl-1.1.0f]# md5sum > > /home/jlulla/openssl-1.1.0f/engines/afalg/afalg.so > /usr/local/lib64/engines-1.1/afalg.so > > > ec273faa6eb6e25f55e44a7d37adf195 > > /home/jlulla/openssl-1.1.0f/engines/afalg/afalg.so > > > ec273faa6eb6e25f55e44a7d37adf195 > /usr/local/lib64/engines-1.1/afalg.so > > > > > > > [root at localhost > apps]# date > > Fri Aug 11 00:01:10 PDT > 2017 > > > > > > [root at localhost > apps]# ls -l > /usr/local/ssl/openssl.cnf > > > -rw-r--r--. 1 root root 10771 Aug 10 > 23:47 > /usr/local/ssl/openssl.cnf > > > > [root at localhost > apps]# md5sum > /usr/local/ssl/openssl.cnf > > /home/jlulla/openssl-1.1.0f/apps/openssl.cnf > > > f697ef5df0d006882e6326606e8dbf4a > /usr/local/ssl/openssl.cnf > > > f697ef5df0d006882e6326606e8dbf4a > > /home/jlulla/openssl-1.1.0f/apps/openssl.cnf > > > > > > openssl.cnf has the following: > > .. > > .. > > # Extra OBJECT IDENTIFIER info: > > #oid_file = > $ENV::HOME/.oid > > oid_section > = new_oids > > > > > # To use this configuration file with > the > "-extfile" option of the > > # > "openssl x509" utility, name here the > section > containing the > > # X.509v3 extensions to > use: > > # extensions = > > # (Alternatively, use a > configuration file > that has only > > # X.509v3 extensions in > its main [= default] section.) > > > > .. > > .. > > > > [root at localhost > apps]# ./openssl speed -evp > aes-128-cbc -engine afalg > > invalid engine "afalg" > > > 139853452924736:error:2506406A:DSO support > routines:dlfcn_bind_func:could not > bind to the requested > symbol > name:crypto/dso/dso_dlfcn.c:178:symname(bind_engine): > /usr/local/lib64/engines-1.1/afalg.so: > undefined symbol: > bind_engine > > > 139853452924736:error:2506C06A:DSO > support > routines:DSO_bind_func:could not bind > to the requested > symbol name:crypto/dso/dso_lib.c:185: > > > 139853452924736:error:260B6068:engine > routines:dynamic_load:DSO > failure:crypto/engine/eng_dyn.c:427: > > > 139853452924736:error:2606A074:engine > routines:ENGINE_by_id:no such > > engine:crypto/engine/eng_list.c:339:id=afalg > > > 139853452924736:error:25066067:DSO support > routines:dlfcn_load:could not load the > shared > > library:crypto/dso/dso_dlfcn.c:113:filename(libafalg.so): > libafalg.so: cannot open shared object > file: No such file or > directory > > > 139853452924736:error:25070067:DSO > support > routines:DSO_load:could not load the > shared > library:crypto/dso/dso_lib.c:161: > > > 139853452924736:error:260B6084:engine > routines:dynamic_load:dso not > found:crypto/engine/eng_dyn.c:414: > > > ^C > > > > > > Modified both the files as the > follwing: > > /usr/local/ssl/openssl.cnf > > > > /home/jlulla/openssl-1.1.0f/apps/openssl.cnf > > > > # Extra OBJECT > IDENTIFIER info: > > #oid_file > = $ENV::HOME/.oid > > > oid_section = > new_oids > > > > > openssl_conf = openssl_def > > > > [openssl_def] > > engines = openssl_engines > > > > [openssl_engines] > > afalg = afalg_engine > > > > > [afalg_engine] > > > CIPHERS=aes-128-cbc > > > > With the above, the errors are > more: > > [root at localhost > apps]# ./openssl speed -evp > aes-128-cbc -engine afalg > > invalid engine "afalg" > > > 140473780422464:error:2506406A:DSO support > routines:dlfcn_bind_func:could not > bind to the requested > symbol > name:crypto/dso/dso_dlfcn.c:178:symname(bind_engine): > /usr/local/lib64/engines-1.1/afalg.so: > undefined symbol: > bind_engine > > > 140473780422464:error:2506C06A:DSO > support > routines:DSO_bind_func:could not bind > to the requested > symbol name:crypto/dso/dso_lib.c:185: > > > 140473780422464:error:260B6068:engine > routines:dynamic_load:DSO > failure:crypto/engine/eng_dyn.c:427: > > > 140473780422464:error:2606A074:engine > routines:ENGINE_by_id:no such > > engine:crypto/engine/eng_list.c:339:id=afalg > > > 140473780422464:error:260BC066:engine > routines:int_engine_configure:engine > configuration > > error:crypto/engine/eng_cnf.c:141:section=afalg_engine, > name=CIPHERS, value=aes-128-cbc > > > > 140473780422464:error:0E07606D:configuration file > routines:module_run:module > initialization > > error:crypto/conf/conf_mod.c:173:module=engines, > value=openssl_engines, retcode=-1 > > > 140473780422464:error:2506406A:DSO > support > routines:dlfcn_bind_func:could not > bind to the requested > symbol > name:crypto/dso/dso_dlfcn.c:178:symname(bind_engine): > /usr/local/lib64/engines-1.1/afalg.so: > undefined symbol: > bind_engine > > > 140473780422464:error:2506C06A:DSO > support > routines:DSO_bind_func:could not bind > to the requested > symbol name:crypto/dso/dso_lib.c:185: > > > 140473780422464:error:260B6068:engine > routines:dynamic_load:DSO > failure:crypto/engine/eng_dyn.c:427: > > > 140473780422464:error:2606A074:engine > routines:ENGINE_by_id:no such > > engine:crypto/engine/eng_list.c:339:id=afalg > > > 140473780422464:error:25066067:DSO support > routines:dlfcn_load:could not load the > shared > > library:crypto/dso/dso_dlfcn.c:113:filename(libafalg.so): > libafalg.so: cannot open shared object > file: No such file or > directory > > > 140473780422464:error:25070067:DSO > support > routines:DSO_load:could not load the > shared > library:crypto/dso/dso_lib.c:161: > > > 140473780422464:error:260B6084:engine > routines:dynamic_load:dso not > found:crypto/engine/eng_dyn.c:414: > > > ^C > > > > From paul.l.kehrer at gmail.com Wed Aug 16 23:41:44 2017 From: paul.l.kehrer at gmail.com (Paul Kehrer) Date: Wed, 16 Aug 2017 18:41:44 -0500 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <97E7DB38-D94A-4CED-BF11-F8AAF2511D81@akamai.com> References: <97E7DB38-D94A-4CED-BF11-F8AAF2511D81@akamai.com> Message-ID: Great news and congratulations to everyone on landing this work. I see that the RNG is now capable of automatically reseeding itself on fork, which will be a huge win for applications that aren't rigorous about doing so themselves (read: most of them). However, it appears that OPENSSL_INIT_ATFORK is not set as an option when OpenSSL calls OPENSSL_init_crypto. Would it be possible to make this default? This would be a large improvement in terms of protecting applications linking against OpenSSL. -Paul Kehrer (reaperhulk) On Mon, Aug 14, 2017 at 10:45 AM, Salz, Rich via openssl-dev wrote: > Thanks everyone for the discussion (mainly in June) about this. There?s a > blog post describing what we?ve done for the 1.1.1 release: > https://www.openssl.org/blog/blog/2017/08/12/random/ > > > > > -- > openssl-dev mailing list > To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev > From rsalz at akamai.com Thu Aug 17 12:22:07 2017 From: rsalz at akamai.com (Salz, Rich) Date: Thu, 17 Aug 2017 12:22:07 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <1c213baf-07b5-173c-7355-bd424ff0e8f6@python.org> References: <97E7DB38-D94A-4CED-BF11-F8AAF2511D81@akamai.com> <1c213baf-07b5-173c-7355-bd424ff0e8f6@python.org> Message-ID: I understand the concern. The issue I am wrestling with is strict compatibility with the existing code. Does anyone really *want* the RNG?s to not reseed on fork? It?s hard to imagine, but maybe somewhere someone is. And then it?s not about just reseeding, but what about when (if) we add other things, like whether or not the secure arena gets zero?d in a child? So let me phrase it this way: does anyone object to changing the default so NO_ATFORK must be used to avoid the reseeding and other things we might add later? By the way I noticed that openssl_init_fork_handlers() is not guarded by RUN_ONCE(). This should be fixed, too. Yeah, I?ll fix that; thanks. From matt at openssl.org Thu Aug 17 12:33:41 2017 From: matt at openssl.org (Matt Caswell) Date: Thu, 17 Aug 2017 13:33:41 +0100 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: References: <97E7DB38-D94A-4CED-BF11-F8AAF2511D81@akamai.com> <1c213baf-07b5-173c-7355-bd424ff0e8f6@python.org> Message-ID: <110a4b17-94b2-2e64-4e7a-96dda31015b4@openssl.org> On 17/08/17 13:22, Salz, Rich via openssl-dev wrote: > I understand the concern. The issue I am wrestling with is strict > compatibility with the existing code. Does anyone really *want* the > RNG?s to not reseed on fork? It?s hard to imagine, but maybe > somewhere someone is. And then it?s not about just reseeding, but > what about when (if) we add other things, like whether or not the > secure arena gets zero?d in a child? > > So let me phrase it this way: does anyone object to changing the > default so NO_ATFORK must be used to avoid the reseeding and other > things we might add later? It's difficult to think of what circumstances this might break existing code? What scenario did you have in mind? Even if it does break something obscure, I think this is a case where security-by-default takes precedence. Matt From tmraz at redhat.com Thu Aug 17 12:34:49 2017 From: tmraz at redhat.com (Tomas Mraz) Date: Thu, 17 Aug 2017 14:34:49 +0200 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: References: <97E7DB38-D94A-4CED-BF11-F8AAF2511D81@akamai.com> <1c213baf-07b5-173c-7355-bd424ff0e8f6@python.org> Message-ID: <1502973289.8546.10.camel@redhat.com> On Thu, 2017-08-17 at 12:22 +0000, Salz, Rich via openssl-dev wrote: > I understand the concern.??The issue I am wrestling with is strict > compatibility with the existing code.??Does anyone really *want* the > RNG?s to not reseed on fork???It?s hard to imagine, but maybe > somewhere someone is.??And then it?s not about just reseeding, but > what about when (if) we add other things, like whether or not the > secure arena gets zero?d in a child? > > So let me phrase it this way:??does anyone object to changing the > default so NO_ATFORK must be used to avoid the reseeding and other > things we might add later? I can hardly see anyone would be broken if the default is to reseed RNG on fork. However that might not be true for other atfork functionalities so perhaps there is a need to make each of these future atfork functions configurable and either on or off by default individually and not as a whole. > ????By the way I noticed that openssl_init_fork_handlers() is not > guarded by > ????RUN_ONCE(). This should be fixed, too. > ???? > Yeah, I?ll fix that; thanks. > -- Tom?? Mr?z Red Hat No matter how far down the wrong road you've gone, turn back. ??????????????????????????????????????????????Turkish proverb [You'll know whether the road is wrong if you carefully listen to your conscience.] * Google and NSA associates, this message is none of your business. * Please leave it alone, and consider whether your actions are * authorized by the contract with Red Hat, or by the US constitution. * If you feel you're being encouraged to disregard the limits built * into them, remember Edward Snowden and Wikileaks. From rsalz at akamai.com Thu Aug 17 13:17:49 2017 From: rsalz at akamai.com (Salz, Rich) Date: Thu, 17 Aug 2017 13:17:49 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <110a4b17-94b2-2e64-4e7a-96dda31015b4@openssl.org> References: <97E7DB38-D94A-4CED-BF11-F8AAF2511D81@akamai.com> <1c213baf-07b5-173c-7355-bd424ff0e8f6@python.org> <110a4b17-94b2-2e64-4e7a-96dda31015b4@openssl.org> Message-ID: <13FBB55A-2768-483E-BC81-21CD88EF075D@akamai.com> \ > somewhere someone is. And then it?s not about just reseeding, but > what about when (if) we add other things, like whether or not the > secure arena gets zero?d in a child? It's difficult to think of what circumstances this might break existing code? What scenario did you have in mind? As excerpted above in my post. From uri at ll.mit.edu Thu Aug 17 13:31:52 2017 From: uri at ll.mit.edu (Blumenthal, Uri - 0553 - MITLL) Date: Thu, 17 Aug 2017 13:31:52 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <13FBB55A-2768-483E-BC81-21CD88EF075D@akamai.com> References: <97E7DB38-D94A-4CED-BF11-F8AAF2511D81@akamai.com> <1c213baf-07b5-173c-7355-bd424ff0e8f6@python.org> <110a4b17-94b2-2e64-4e7a-96dda31015b4@openssl.org> <13FBB55A-2768-483E-BC81-21CD88EF075D@akamai.com> Message-ID: <91B28E8C-D79E-40B6-B6F1-437E81242EFF@ll.mit.edu> I have only one issue with the current design: the apparent absence of RAND_add() interface for the "private" RNG. I request that it is added (no pun intended, really :). Regards, Uri Sent from my iPhone > On Aug 17, 2017, at 09:18, Salz, Rich via openssl-dev wrote: > > \ >> somewhere someone is. And then it?s not about just reseeding, but >> what about when (if) we add other things, like whether or not the >> secure arena gets zero?d in a child? > > It's difficult to think of what circumstances this might break existing > code? What scenario did you have in mind? > > As excerpted above in my post. > > -- > openssl-dev mailing list > To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4223 bytes Desc: not available URL: From kurt at roeckx.be Thu Aug 17 20:11:08 2017 From: kurt at roeckx.be (Kurt Roeckx) Date: Thu, 17 Aug 2017 22:11:08 +0200 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <1502973289.8546.10.camel@redhat.com> References: <97E7DB38-D94A-4CED-BF11-F8AAF2511D81@akamai.com> <1c213baf-07b5-173c-7355-bd424ff0e8f6@python.org> <1502973289.8546.10.camel@redhat.com> Message-ID: <20170817201108.fnwg2cf3cy7mqdr3@roeckx.be> On Thu, Aug 17, 2017 at 02:34:49PM +0200, Tomas Mraz wrote: > On Thu, 2017-08-17 at 12:22 +0000, Salz, Rich via openssl-dev wrote: > > I understand the concern.??The issue I am wrestling with is strict > > compatibility with the existing code.??Does anyone really *want* the > > RNG?s to not reseed on fork???It?s hard to imagine, but maybe > > somewhere someone is.??And then it?s not about just reseeding, but > > what about when (if) we add other things, like whether or not the > > secure arena gets zero?d in a child? > > > > So let me phrase it this way:??does anyone object to changing the > > default so NO_ATFORK must be used to avoid the reseeding and other > > things we might add later? > > I can hardly see anyone would be broken if the default is to reseed > RNG on fork. However that might not be true for other atfork > functionalities so perhaps there is a need to make each of these future > atfork functions configurable and either on or off by default > individually and not as a whole. There might be cases where after fork you're not able to get to /dev/urandom anymore. Kurt From tmraz at redhat.com Fri Aug 18 07:22:48 2017 From: tmraz at redhat.com (Tomas Mraz) Date: Fri, 18 Aug 2017 09:22:48 +0200 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <20170817201108.fnwg2cf3cy7mqdr3@roeckx.be> References: <97E7DB38-D94A-4CED-BF11-F8AAF2511D81@akamai.com> <1c213baf-07b5-173c-7355-bd424ff0e8f6@python.org> <1502973289.8546.10.camel@redhat.com> <20170817201108.fnwg2cf3cy7mqdr3@roeckx.be> Message-ID: <1503040968.22810.2.camel@redhat.com> On Thu, 2017-08-17 at 22:11 +0200, Kurt Roeckx wrote: > On Thu, Aug 17, 2017 at 02:34:49PM +0200, Tomas Mraz wrote: > > On Thu, 2017-08-17 at 12:22 +0000, Salz, Rich via openssl-dev > > wrote: > > > I understand the concern.??The issue I am wrestling with is > > > strict > > > compatibility with the existing code.??Does anyone really *want* > > > the > > > RNG?s to not reseed on fork???It?s hard to imagine, but maybe > > > somewhere someone is.??And then it?s not about just reseeding, > > > but > > > what about when (if) we add other things, like whether or not the > > > secure arena gets zero?d in a child? > > > > > > So let me phrase it this way:??does anyone object to changing the > > > default so NO_ATFORK must be used to avoid the reseeding and > > > other > > > things we might add later? > > > > I can hardly see anyone would be broken if the default is to reseed > > RNG on fork. However that might not be true for other atfork > > functionalities so perhaps there is a need to make each of these > > future > > atfork functions configurable and either on or off by default > > individually and not as a whole. > > There might be cases where after fork you're not able to get to > /dev/urandom anymore. I do not think so. Which particular cases do you have on mind? Yes, after fork+exec you could for example switch SELinux domain and won't be able to access something but immediately after fork it should not be so. Also perhaps the reseeding after fork can be made less strict in regards to failures reading /dev/urandom or so. -- Tom?? Mr?z Red Hat No matter how far down the wrong road you've gone, turn back. ??????????????????????????????????????????????Turkish proverb [You'll know whether the road is wrong if you carefully listen to your conscience.] * Google and NSA associates, this message is none of your business. * Please leave it alone, and consider whether your actions are * authorized by the contract with Red Hat, or by the US constitution. * If you feel you're being encouraged to disregard the limits built * into them, remember Edward Snowden and Wikileaks. From kurt at roeckx.be Fri Aug 18 17:42:57 2017 From: kurt at roeckx.be (Kurt Roeckx) Date: Fri, 18 Aug 2017 19:42:57 +0200 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <1503040968.22810.2.camel@redhat.com> References: <97E7DB38-D94A-4CED-BF11-F8AAF2511D81@akamai.com> <1c213baf-07b5-173c-7355-bd424ff0e8f6@python.org> <1502973289.8546.10.camel@redhat.com> <20170817201108.fnwg2cf3cy7mqdr3@roeckx.be> <1503040968.22810.2.camel@redhat.com> Message-ID: <20170818174256.22dskehi7qs32ezg@roeckx.be> On Fri, Aug 18, 2017 at 09:22:48AM +0200, Tomas Mraz wrote: > On Thu, 2017-08-17 at 22:11 +0200, Kurt Roeckx wrote: > > On Thu, Aug 17, 2017 at 02:34:49PM +0200, Tomas Mraz wrote: > > > On Thu, 2017-08-17 at 12:22 +0000, Salz, Rich via openssl-dev > > > wrote: > > > > I understand the concern.??The issue I am wrestling with is > > > > strict > > > > compatibility with the existing code.??Does anyone really *want* > > > > the > > > > RNG?s to not reseed on fork???It?s hard to imagine, but maybe > > > > somewhere someone is.??And then it?s not about just reseeding, > > > > but > > > > what about when (if) we add other things, like whether or not the > > > > secure arena gets zero?d in a child? > > > > > > > > So let me phrase it this way:??does anyone object to changing the > > > > default so NO_ATFORK must be used to avoid the reseeding and > > > > other > > > > things we might add later? > > > > > > I can hardly see anyone would be broken if the default is to reseed > > > RNG on fork. However that might not be true for other atfork > > > functionalities so perhaps there is a need to make each of these > > > future > > > atfork functions configurable and either on or off by default > > > individually and not as a whole. > > > > There might be cases where after fork you're not able to get to > > /dev/urandom anymore. > > I do not think so. Which particular cases do you have on mind? Yes, > after fork+exec you could for example switch SELinux domain and won't > be able to access something but immediately after fork it should not be > so. Also perhaps the reseeding after fork can be made less strict in > regards to failures reading /dev/urandom or so. It seems to me this all depends on the order of things you do to create a daemon. You could make sure the RNG is inited, chroot, and then fork for instance. And I suspect there are actually programs that do it in that order. Kurt From uri at ll.mit.edu Fri Aug 18 17:54:17 2017 From: uri at ll.mit.edu (Blumenthal, Uri - 0553 - MITLL) Date: Fri, 18 Aug 2017 17:54:17 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <20170818174256.22dskehi7qs32ezg@roeckx.be> References: <97E7DB38-D94A-4CED-BF11-F8AAF2511D81@akamai.com> <1c213baf-07b5-173c-7355-bd424ff0e8f6@python.org> <1502973289.8546.10.camel@redhat.com> <20170817201108.fnwg2cf3cy7mqdr3@roeckx.be> <1503040968.22810.2.camel@redhat.com> <20170818174256.22dskehi7qs32ezg@roeckx.be> Message-ID: <17AA9277-52E0-4CBE-80D3-AF1128D5B15C@ll.mit.edu> All the items discussed are important. But I?d like the development team to comment on (and ideally ? accept) my request to add RAND_add() method to the RNG that is used in generation of private keys. Reason/justification: to be able to improve the available randomness by mixing in output from hardware-based TRNG(s). -- Regards, Uri Blumenthal -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5211 bytes Desc: not available URL: From rsalz at akamai.com Fri Aug 18 18:47:47 2017 From: rsalz at akamai.com (Salz, Rich) Date: Fri, 18 Aug 2017 18:47:47 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <20170818174256.22dskehi7qs32ezg@roeckx.be> References: <97E7DB38-D94A-4CED-BF11-F8AAF2511D81@akamai.com> <1c213baf-07b5-173c-7355-bd424ff0e8f6@python.org> <1502973289.8546.10.camel@redhat.com> <20170817201108.fnwg2cf3cy7mqdr3@roeckx.be> <1503040968.22810.2.camel@redhat.com> <20170818174256.22dskehi7qs32ezg@roeckx.be> Message-ID: It seems to me this all depends on the order of things you do to create a daemon. You could make sure the RNG is inited, chroot, and then fork for instance. And I suspect there are actually programs that do it in that order. Yes. I think the safest thing is for us to not change the default. Programs that know they are going to fork can do the right/safe thing. It would be nicer if we could automatically always do the right thing, but I don?t think it?s possible. From christian at python.org Fri Aug 18 20:07:42 2017 From: christian at python.org (Christian Heimes) Date: Fri, 18 Aug 2017 22:07:42 +0200 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <20170818174256.22dskehi7qs32ezg@roeckx.be> References: <97E7DB38-D94A-4CED-BF11-F8AAF2511D81@akamai.com> <1c213baf-07b5-173c-7355-bd424ff0e8f6@python.org> <1502973289.8546.10.camel@redhat.com> <20170817201108.fnwg2cf3cy7mqdr3@roeckx.be> <1503040968.22810.2.camel@redhat.com> <20170818174256.22dskehi7qs32ezg@roeckx.be> Message-ID: <6b1ca962-a265-f479-9bf4-76c6187ee338@python.org> On 2017-08-18 19:42, Kurt Roeckx wrote: > On Fri, Aug 18, 2017 at 09:22:48AM +0200, Tomas Mraz wrote: >> On Thu, 2017-08-17 at 22:11 +0200, Kurt Roeckx wrote: >>> On Thu, Aug 17, 2017 at 02:34:49PM +0200, Tomas Mraz wrote: >>>> On Thu, 2017-08-17 at 12:22 +0000, Salz, Rich via openssl-dev >>>> wrote: >>>>> I understand the concern. The issue I am wrestling with is >>>>> strict >>>>> compatibility with the existing code. Does anyone really *want* >>>>> the >>>>> RNG?s to not reseed on fork? It?s hard to imagine, but maybe >>>>> somewhere someone is. And then it?s not about just reseeding, >>>>> but >>>>> what about when (if) we add other things, like whether or not the >>>>> secure arena gets zero?d in a child? >>>>> >>>>> So let me phrase it this way: does anyone object to changing the >>>>> default so NO_ATFORK must be used to avoid the reseeding and >>>>> other >>>>> things we might add later? >>>> >>>> I can hardly see anyone would be broken if the default is to reseed >>>> RNG on fork. However that might not be true for other atfork >>>> functionalities so perhaps there is a need to make each of these >>>> future >>>> atfork functions configurable and either on or off by default >>>> individually and not as a whole. >>> >>> There might be cases where after fork you're not able to get to >>> /dev/urandom anymore. >> >> I do not think so. Which particular cases do you have on mind? Yes, >> after fork+exec you could for example switch SELinux domain and won't >> be able to access something but immediately after fork it should not be >> so. Also perhaps the reseeding after fork can be made less strict in >> regards to failures reading /dev/urandom or so. > > It seems to me this all depends on the order of things you do to > create a daemon. You could make sure the RNG is inited, chroot, > and then fork for instance. And I suspect there are actually > programs that do it in that order. The problem with /dev/urandom will go away sooner or later. All major platforms either have a CPRNG syscall for years or introduced one recently. BSD has getentropy(2) for a while, Windows has CryptGenRandom() and Linux has getrandom(2) since Kernel 3.2 and glibc 2.15. Christian From rsalz at akamai.com Fri Aug 18 20:10:02 2017 From: rsalz at akamai.com (Salz, Rich) Date: Fri, 18 Aug 2017 20:10:02 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <6b1ca962-a265-f479-9bf4-76c6187ee338@python.org> References: <97E7DB38-D94A-4CED-BF11-F8AAF2511D81@akamai.com> <1c213baf-07b5-173c-7355-bd424ff0e8f6@python.org> <1502973289.8546.10.camel@redhat.com> <20170817201108.fnwg2cf3cy7mqdr3@roeckx.be> <1503040968.22810.2.camel@redhat.com> <20170818174256.22dskehi7qs32ezg@roeckx.be> <6b1ca962-a265-f479-9bf4-76c6187ee338@python.org> Message-ID: <277FDACE-E939-4D17-8005-31F87892974E@akamai.com> The problem with /dev/urandom will go away sooner or later. All major platforms either have a CPRNG syscall for years or introduced one recently. BSD has getentropy(2) for a while, Windows has CryptGenRandom() and Linux has getrandom(2) since Kernel 3.2 and glibc 2.15. Agreed. And when we can make ?with-rand-seed=syscall the default, then it will be a happier place ? From rsalz at akamai.com Fri Aug 18 23:42:07 2017 From: rsalz at akamai.com (Salz, Rich) Date: Fri, 18 Aug 2017 23:42:07 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <17AA9277-52E0-4CBE-80D3-AF1128D5B15C@ll.mit.edu> References: <97E7DB38-D94A-4CED-BF11-F8AAF2511D81@akamai.com> <1c213baf-07b5-173c-7355-bd424ff0e8f6@python.org> <1502973289.8546.10.camel@redhat.com> <20170817201108.fnwg2cf3cy7mqdr3@roeckx.be> <1503040968.22810.2.camel@redhat.com> <20170818174256.22dskehi7qs32ezg@roeckx.be> <17AA9277-52E0-4CBE-80D3-AF1128D5B15C@ll.mit.edu> Message-ID: <01BCDAFE-3A62-49A0-9005-AC5D6FCD3A82@akamai.com> ? But I?d like the development team to comment on (and ideally ? accept) my request to add RAND_add() method to the RNG that is used in generation of private keys. Well, I?ve been thinking about this for a bit, since you first raised it. I am still not sure of the need. And as the blog post says, we?re not convinced that the current DRBG arrangement is something that will never change. But I think a new API, RAND_add_ex that took a flag that had values like RAND_ADD_GLOBAL, RAND_ADD_LOCAL, RAND_ADD_PRIVATE, RAND_LOCAL_PRIVATE indicating which to seed. Thoughts? From steffen at sdaoden.eu Sat Aug 19 14:31:07 2017 From: steffen at sdaoden.eu (Steffen Nurpmeso) Date: Sat, 19 Aug 2017 16:31:07 +0200 Subject: [openssl-dev] Work on a new RNG for OpenSSL References: <97E7DB38-D94A-4CED-BF11-F8AAF2511D81@akamai.com> <1c213baf-07b5-173c-7355-bd424ff0e8f6@python.org> <1502973289.8546.10.camel@redhat.com> <20170817201108.fnwg2cf3cy7mqdr3@roeckx.be> <1503040968.22810.2.camel@redhat.com> <20170818174256.22dskehi7qs32ezg@roeckx.be> <17AA9277-52E0-4CBE-80D3-AF1128D5B15C@ll.mit.edu> <01BCDAFE-3A62-49A0-9005-AC5D6FCD3A82@akamai.com> Message-ID: <20170819143107.j5AUF%steffen@sdaoden.eu> "Salz, Rich via openssl-dev" wrote: |? But I?d like the development team to comment on (and ideally ? accept) \ |my request to add RAND_add() method to the RNG that is used in generation \ |of private keys. | |Well, I?ve been thinking about this for a bit, since you first raised \ |it. I am still not sure of the need. And as the blog post says, we?re \ |not convinced that the current DRBG arrangement is something that will \ |never change. But I think a new API, RAND_add_ex that took a flag \ |that had values like RAND_ADD_GLOBAL, RAND_ADD_LOCAL, RAND_ADD_PRIVATE, \ |RAND_LOCAL_PRIVATE indicating which to seed. Thoughts? Is this new RNG object available to user programs, or do they need to reinvent the wheel even though they definitely link against the OpenSSL library? --steffen | |Der Kragenbaer, The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt) From rsalz at akamai.com Sat Aug 19 14:39:32 2017 From: rsalz at akamai.com (Salz, Rich) Date: Sat, 19 Aug 2017 14:39:32 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <20170819143107.j5AUF%steffen@sdaoden.eu> References: <97E7DB38-D94A-4CED-BF11-F8AAF2511D81@akamai.com> <1c213baf-07b5-173c-7355-bd424ff0e8f6@python.org> <1502973289.8546.10.camel@redhat.com> <20170817201108.fnwg2cf3cy7mqdr3@roeckx.be> <1503040968.22810.2.camel@redhat.com> <20170818174256.22dskehi7qs32ezg@roeckx.be> <17AA9277-52E0-4CBE-80D3-AF1128D5B15C@ll.mit.edu> <01BCDAFE-3A62-49A0-9005-AC5D6FCD3A82@akamai.com> <20170819143107.j5AUF%steffen@sdaoden.eu> Message-ID: <37A92473-D69D-4BFF-93E9-0B76E824E709@akamai.com> Is this new RNG object available to user programs, or do they need to reinvent the wheel even though they definitely link against the OpenSSL library? You don?t have to re-invent the wheel, but you might have to modify the source ? Did you read the blog posting? What wasn?t clear? From steffen at sdaoden.eu Sat Aug 19 19:06:14 2017 From: steffen at sdaoden.eu (Steffen Nurpmeso) Date: Sat, 19 Aug 2017 21:06:14 +0200 Subject: [openssl-dev] Work on a new RNG for OpenSSL References: <97E7DB38-D94A-4CED-BF11-F8AAF2511D81@akamai.com> <1c213baf-07b5-173c-7355-bd424ff0e8f6@python.org> <1502973289.8546.10.camel@redhat.com> <20170817201108.fnwg2cf3cy7mqdr3@roeckx.be> <1503040968.22810.2.camel@redhat.com> <20170818174256.22dskehi7qs32ezg@roeckx.be> <17AA9277-52E0-4CBE-80D3-AF1128D5B15C@ll.mit.edu> <01BCDAFE-3A62-49A0-9005-AC5D6FCD3A82@akamai.com> <20170819143107.j5AUF%steffen@sdaoden.eu> <37A92473-D69D-4BFF-93E9-0B76E824E709@akamai.com> Message-ID: <20170819190614.g9feX%steffen@sdaoden.eu> "Salz, Rich" wrote: | Is this new RNG object available to user programs, or do they need | to reinvent the wheel even though they definitely link against the | OpenSSL library? | |You don?t have to re-invent the wheel, but you might have to modify \ |the source ? Did you read the blog posting? What wasn?t clear? Ok. Yes, and the linked thread on the crypto ML. Nothing, sir. --steffen | |Der Kragenbaer, The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt) From uri at ll.mit.edu Sun Aug 20 03:10:53 2017 From: uri at ll.mit.edu (Blumenthal, Uri - 0553 - MITLL) Date: Sun, 20 Aug 2017 03:10:53 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <01BCDAFE-3A62-49A0-9005-AC5D6FCD3A82@akamai.com> References: <97E7DB38-D94A-4CED-BF11-F8AAF2511D81@akamai.com> <1c213baf-07b5-173c-7355-bd424ff0e8f6@python.org> <1502973289.8546.10.camel@redhat.com> <20170817201108.fnwg2cf3cy7mqdr3@roeckx.be> <1503040968.22810.2.camel@redhat.com> <20170818174256.22dskehi7qs32ezg@roeckx.be> <17AA9277-52E0-4CBE-80D3-AF1128D5B15C@ll.mit.edu> <01BCDAFE-3A62-49A0-9005-AC5D6FCD3A82@akamai.com> Message-ID: <365FF22A-8D78-4874-BEF4-EF9E56A80208@ll.mit.edu> Offhand, I'd say it's a perfect solution. It allows me to mix in additional randomness when I want to the RNG that I think may need it. Exactly what I need. Thanks! P.S. I wonder if it's feasible to have a configuration parameter that would allow me to tell the TLS code to invoke RAND_add_ex() before generating session keys? Regards, Uri Sent from my iPhone > On Aug 18, 2017, at 19:42, Salz, Rich via openssl-dev wrote: > > ? But I?d like the development team to comment on (and ideally ? accept) my request to add RAND_add() method to the RNG that is used in generation of private keys. > > Well, I?ve been thinking about this for a bit, since you first raised it. I am still not sure of the need. And as the blog post says, we?re not convinced that the current DRBG arrangement is something that will never change. But I think a new API, RAND_add_ex that took a flag that had values like RAND_ADD_GLOBAL, RAND_ADD_LOCAL, RAND_ADD_PRIVATE, RAND_LOCAL_PRIVATE indicating which to seed. Thoughts? > > -- > openssl-dev mailing list > To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4223 bytes Desc: not available URL: From rsalz at akamai.com Sun Aug 20 13:43:48 2017 From: rsalz at akamai.com (Salz, Rich) Date: Sun, 20 Aug 2017 13:43:48 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <365FF22A-8D78-4874-BEF4-EF9E56A80208@ll.mit.edu> References: <97E7DB38-D94A-4CED-BF11-F8AAF2511D81@akamai.com> <1c213baf-07b5-173c-7355-bd424ff0e8f6@python.org> <1502973289.8546.10.camel@redhat.com> <20170817201108.fnwg2cf3cy7mqdr3@roeckx.be> <1503040968.22810.2.camel@redhat.com> <20170818174256.22dskehi7qs32ezg@roeckx.be> <17AA9277-52E0-4CBE-80D3-AF1128D5B15C@ll.mit.edu> <01BCDAFE-3A62-49A0-9005-AC5D6FCD3A82@akamai.com> <365FF22A-8D78-4874-BEF4-EF9E56A80208@ll.mit.edu> Message-ID: <34101F20-8DD8-4C2E-9150-8FE6FC76D922@akamai.com> ? P.S. I wonder if it's feasible to have a configuration parameter that would allow me to tell the TLS code to invoke RAND_add_ex() before generating session keys? At this point, you might as well just change the code to use getrandom() and pass it through. Either you accept that NIST SP 90A is right, or you just bypass it completely. We?re in the first camp. But it?s open source, do what fits your needs. From uri at ll.mit.edu Mon Aug 21 15:56:29 2017 From: uri at ll.mit.edu (Blumenthal, Uri - 0553 - MITLL) Date: Mon, 21 Aug 2017 15:56:29 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <34101F20-8DD8-4C2E-9150-8FE6FC76D922@akamai.com> References: <97E7DB38-D94A-4CED-BF11-F8AAF2511D81@akamai.com> <1c213baf-07b5-173c-7355-bd424ff0e8f6@python.org> <1502973289.8546.10.camel@redhat.com> <20170817201108.fnwg2cf3cy7mqdr3@roeckx.be> <1503040968.22810.2.camel@redhat.com> <20170818174256.22dskehi7qs32ezg@roeckx.be> <17AA9277-52E0-4CBE-80D3-AF1128D5B15C@ll.mit.edu> <01BCDAFE-3A62-49A0-9005-AC5D6FCD3A82@akamai.com> <365FF22A-8D78-4874-BEF4-EF9E56A80208@ll.mit.edu> <34101F20-8DD8-4C2E-9150-8FE6FC76D922@akamai.com> Message-ID: >> P.S. I wonder if it's feasible to have a configuration parameter that would allow me to tell the TLS code to invoke RAND_add_ex() before generating session keys? > > Either you accept that NIST SP 90A is right, or you just bypass it completely. We?re in the first camp. You mean NIST SP 800-90A, released Jan 2012 and withdrawn Jun 2015? With Rev 1 *draft* currently available (released Jun 2015)? ;-) I?m glad you agree that ?it is right?, because in our argument it supports my side over yours. Let?s go through the 90A Rev 1 draft http://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-90Ar1.pdf: Page 11 Section 7 provides a functional model of a DRBG (Figure 1), clearly showing ?additional input? for both the Reseed Function and the Generate Function. The text says ?? and may include additional optional sources, including ? additional input.? Page 12 Sec 7.2: ?Additional input may also be provided during reseeding and when pseudorandom bits are requested.? Page 22 Sec 8.7.2: ?Additional input may optionally be provided to the reseed and generate functions during requests. The additional input may be obtained from inside or outside a cryptographic module, and may include secret or public information. ... Additional input is optional for both the DRBG and the consuming application, and the ability to enter additional input may or may not be included in an implementation.? Same place: ?The use of additional input may be a means of providing more entropy for the DRBG internal state that will increase assurance that the entropy requirements are met. If the additional input is kept secret and has sufficient entropy, the input can provide more assurance when recovering from the compromise of the entropy input, the seed or one or more DRBG internal states.? The last quote explains exactly why people (myself included) may want that interface. The draft does not *mandate* making this interface available, but blesses it nonetheless. I?m one of those who falls into the category of ?need to increase assurance that the entropy requirements are met?. In other words, the standard clearly does allow the implementer to *include* additional input. The standard also allows the implementer to ?weasel out? of it, but I cannot imagine why a security-cautious implementer would want to. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5211 bytes Desc: not available URL: From kurt at roeckx.be Mon Aug 21 16:12:16 2017 From: kurt at roeckx.be (Kurt Roeckx) Date: Mon, 21 Aug 2017 18:12:16 +0200 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: References: <1502973289.8546.10.camel@redhat.com> <20170817201108.fnwg2cf3cy7mqdr3@roeckx.be> <1503040968.22810.2.camel@redhat.com> <20170818174256.22dskehi7qs32ezg@roeckx.be> <17AA9277-52E0-4CBE-80D3-AF1128D5B15C@ll.mit.edu> <01BCDAFE-3A62-49A0-9005-AC5D6FCD3A82@akamai.com> <365FF22A-8D78-4874-BEF4-EF9E56A80208@ll.mit.edu> <34101F20-8DD8-4C2E-9150-8FE6FC76D922@akamai.com> Message-ID: <20170821161216.c3lj6emzrddchsvs@roeckx.be> On Mon, Aug 21, 2017 at 03:56:29PM +0000, Blumenthal, Uri - 0553 - MITLL wrote: > >> P.S. I wonder if it's feasible to have a configuration parameter that would allow me to tell the TLS code to invoke RAND_add_ex() before generating session keys? > > > > Either you accept that NIST SP 90A is right, or you just bypass it completely. We?re in the first camp. > > You mean NIST SP 800-90A, released Jan 2012 and withdrawn Jun 2015? With Rev 1 *draft* currently available (released Jun 2015)? ;-) > > I?m glad you agree that ?it is right?, because in our argument it supports my side over yours. Let?s go through the 90A Rev 1 draft http://nvlpubs.nist.gov/nistpubs/SpecialPublications/NIST.SP.800-90Ar1.pdf: > > Page 11 Section 7 provides a functional model of a DRBG (Figure 1), clearly showing ?additional input? for both the Reseed Function and the Generate Function. The text says ?? and may include additional optional sources, including ? additional input.? I at least have a plan to add additional data, but probably not in the current idea was probably not the way you would like to see it. My idea was to query at least various sources that we don't attribute any entropy to, like getpid(), gettimeofday(), clock_gettime(), the TSC, ... It might also use things like RDRAND / RDSEED which we don't trust. So I guess you want an interface that can both add things to the "entropy" pool, and to the "additional data" pool? It shouldn't be that hard, I'll try to come up with some proposal soon. Kurt From uri at ll.mit.edu Mon Aug 21 16:20:50 2017 From: uri at ll.mit.edu (Blumenthal, Uri - 0553 - MITLL) Date: Mon, 21 Aug 2017 16:20:50 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <20170821161216.c3lj6emzrddchsvs@roeckx.be> References: <1502973289.8546.10.camel@redhat.com> <20170817201108.fnwg2cf3cy7mqdr3@roeckx.be> <1503040968.22810.2.camel@redhat.com> <20170818174256.22dskehi7qs32ezg@roeckx.be> <17AA9277-52E0-4CBE-80D3-AF1128D5B15C@ll.mit.edu> <01BCDAFE-3A62-49A0-9005-AC5D6FCD3A82@akamai.com> <365FF22A-8D78-4874-BEF4-EF9E56A80208@ll.mit.edu> <34101F20-8DD8-4C2E-9150-8FE6FC76D922@akamai.com> <20170821161216.c3lj6emzrddchsvs@roeckx.be> Message-ID: <05B0FAB2-033D-4DD3-BAE7-3530378B1F46@ll.mit.edu> > I at least have a plan to add additional data, but probably not in > the current idea was probably not the way you would like to see it. :-) > My idea was to query at least various sources that we don't > attribute any entropy to, like getpid(), gettimeofday(), > clock_gettime(), the TSC, ... >From my point of view ? adding these doesn?t add a whole lot, but it doesn?t hurt. IMHO ? add away. ;-) > It might also use things like RDRAND / RDSEED which we don't trust. Some don?t trust these, some think that they would add a good amount of entropy. I for one would certainly like to see the output of these mixed in. >From cryptography point of view, it cannot hurt, but may help a lot. Consider it as a lottery ticket you don?t have to pay for. ;-) > So I guess you want an interface that can both add things to the > "entropy" pool, and to the "additional data" pool? That is correct. Especially because some of us have ?real? nice/fancy hardware RNG (TRNG) available, and some like to mix in the output from RNGs on hardware tokens - maybe not as impressive as a ?real? fancy TRNG, but as they say, every bit helps ? in this case literally. > It shouldn't be that hard, I'll try to come up with some proposal soon. Thank you!! -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5211 bytes Desc: not available URL: From kurt at roeckx.be Mon Aug 21 18:47:05 2017 From: kurt at roeckx.be (Kurt Roeckx) Date: Mon, 21 Aug 2017 20:47:05 +0200 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <20170821161216.c3lj6emzrddchsvs@roeckx.be> References: <1502973289.8546.10.camel@redhat.com> <20170817201108.fnwg2cf3cy7mqdr3@roeckx.be> <1503040968.22810.2.camel@redhat.com> <20170818174256.22dskehi7qs32ezg@roeckx.be> <17AA9277-52E0-4CBE-80D3-AF1128D5B15C@ll.mit.edu> <01BCDAFE-3A62-49A0-9005-AC5D6FCD3A82@akamai.com> <365FF22A-8D78-4874-BEF4-EF9E56A80208@ll.mit.edu> <34101F20-8DD8-4C2E-9150-8FE6FC76D922@akamai.com> <20170821161216.c3lj6emzrddchsvs@roeckx.be> Message-ID: <20170821184705.5ewjpr43nvhvzgrm@roeckx.be> On Mon, Aug 21, 2017 at 06:12:16PM +0200, Kurt Roeckx wrote: > So I guess you want an interface that can both add things to the > "entropy" pool, and to the "additional data" pool? It shouldn't > be that hard, I'll try to come up with some proposal soon. I was thinking about adding 2 callbacks. One that is called when we want to have entropy, the other that is called when we can use additional data. The first would only be called by the global DRBGs, the second by all DRBGs. The DRBG now actually already uses callbacks to get entropy (and a nonce), but none of that is exposed. So it would require additional callbacks for additional data. We should probably just add functions to set the defaults or something. Kurt From rsalz at akamai.com Mon Aug 21 18:56:05 2017 From: rsalz at akamai.com (Salz, Rich) Date: Mon, 21 Aug 2017 18:56:05 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <20170821184705.5ewjpr43nvhvzgrm@roeckx.be> References: <1502973289.8546.10.camel@redhat.com> <20170817201108.fnwg2cf3cy7mqdr3@roeckx.be> <1503040968.22810.2.camel@redhat.com> <20170818174256.22dskehi7qs32ezg@roeckx.be> <17AA9277-52E0-4CBE-80D3-AF1128D5B15C@ll.mit.edu> <01BCDAFE-3A62-49A0-9005-AC5D6FCD3A82@akamai.com> <365FF22A-8D78-4874-BEF4-EF9E56A80208@ll.mit.edu> <34101F20-8DD8-4C2E-9150-8FE6FC76D922@akamai.com> <20170821161216.c3lj6emzrddchsvs@roeckx.be> <20170821184705.5ewjpr43nvhvzgrm@roeckx.be> Message-ID: <46E156E1-B709-42B8-BD5D-B89355B7EA08@akamai.com> We should think carefully about what API?s we are exposing, and might want to wait for 1.1.2 From paul.dale at oracle.com Tue Aug 22 00:06:05 2017 From: paul.dale at oracle.com (Paul Dale) Date: Mon, 21 Aug 2017 17:06:05 -0700 (PDT) Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <05B0FAB2-033D-4DD3-BAE7-3530378B1F46@ll.mit.edu> References: <1502973289.8546.10.camel@redhat.com> <20170817201108.fnwg2cf3cy7mqdr3@roeckx.be> <1503040968.22810.2.camel@redhat.com> <20170818174256.22dskehi7qs32ezg@roeckx.be> <17AA9277-52E0-4CBE-80D3-AF1128D5B15C@ll.mit.edu> <01BCDAFE-3A62-49A0-9005-AC5D6FCD3A82@akamai.com> <365FF22A-8D78-4874-BEF4-EF9E56A80208@ll.mit.edu> <34101F20-8DD8-4C2E-9150-8FE6FC76D922@akamai.com> <20170821161216.c3lj6emzrddchsvs@roeckx.be> <05B0FAB2-033D-4DD3-BAE7-3530378B1F46@ll.mit.edu> Message-ID: <9ea2c328-2e6f-4efc-b2d5-f887dd08f4e7@default> Uri wrote: >> It might also use things like RDRAND / RDSEED which we don't trust. > ... > From cryptography point of view, it cannot hurt, but may help a lot There is a scenario where it does hurt: https://www.lvh.io/posts/2013/10/thoughts-on-rdrand-in-linux.html This attack wouldn't be difficult to implement given all the out of order execution and look ahead that CPUs do. It requires a compromised RDRAND instruction changing the behaviour of a subsequent XOR into a copy. Not only would it not be producing random bits but it would remove any randomness from the bits you already have. Pauli -- Oracle Dr Paul Dale | Cryptographer | Network Security & Encryption Phone +61 7 3031 7217 Oracle Australia From uri at ll.mit.edu Tue Aug 22 00:44:38 2017 From: uri at ll.mit.edu (Blumenthal, Uri - 0553 - MITLL) Date: Tue, 22 Aug 2017 00:44:38 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <9ea2c328-2e6f-4efc-b2d5-f887dd08f4e7@default> References: <1502973289.8546.10.camel@redhat.com> <20170817201108.fnwg2cf3cy7mqdr3@roeckx.be> <1503040968.22810.2.camel@redhat.com> <20170818174256.22dskehi7qs32ezg@roeckx.be> <17AA9277-52E0-4CBE-80D3-AF1128D5B15C@ll.mit.edu> <01BCDAFE-3A62-49A0-9005-AC5D6FCD3A82@akamai.com> <365FF22A-8D78-4874-BEF4-EF9E56A80208@ll.mit.edu> <34101F20-8DD8-4C2E-9150-8FE6FC76D922@akamai.com> <20170821161216.c3lj6emzrddchsvs@roeckx.be> <05B0FAB2-033D-4DD3-BAE7-3530378B1F46@ll.mit.edu> <9ea2c328-2e6f-4efc-b2d5-f887dd08f4e7@default> Message-ID: <1C38B975-BFB2-4791-96E6-7B449F25100C@ll.mit.edu> My offhand knee-jerk reaction would be that if you have a CPU compromised to that extent - the battle has been lost with no reason to continue. Going into more details, I checked that post, and disagree with the author (and I'm in a good company, as Linus seems to hold the same opinion). According to the author (as I understood him), not only RDRAND must be compromised (i.e., produce bad output), but the CPU would have to switch to a "special" processing mode when RDRAND instruction is detected. And this malicious CPU would have to understand how RNG is implemented in Linux, Mac, Windows, FreeBSD (at what point it calls RDRAND and what instructions follow)... And it's betting on RDRAND being the last in the chain of RNG inputs (otherwise it would wipe out previous entropy, but would in turn be replaced by something good from the next source). Also, usually mixing = XOR, but some prefer arithmetic addition - so RDRAND would be crafted to replace any subsequent operation to copy. Plus, it would have to track a bunch of the following instructions because it's common but not certain that the RDRAND output is mixed in immediately (in the very next instruction)... To summarize, I don't have enough tin-foil for a hat of that size. But regardless, a good solution IMHO is to offer an interface to mix in (add to the pool) whatever data the user wants as an additional input (in terms of SP 800-90A that defined "additional input" to mix during reseeding and generation). That data could be an output of RDRAND, of an external RNG (including hardware TRNG), or... It would be up to the user to pick the additional source that he trusts. Also, nobody forces the user to employ that interface at all (just like on a decent OS nobody now is forced to use RAND_add()) - but it would be there for those who need/want it. Regards, Uri Sent from my iPhone > On Aug 21, 2017, at 20:06, Paul Dale wrote: > > Uri wrote: >>> It might also use things like RDRAND / RDSEED which we don't trust. >> ... >> From cryptography point of view, it cannot hurt, but may help a lot > > There is a scenario where it does hurt: https://www.lvh.io/posts/2013/10/thoughts-on-rdrand-in-linux.html > > This attack wouldn't be difficult to implement given all the out of order execution and look ahead that CPUs do. It requires a compromised RDRAND instruction changing the behaviour of a subsequent XOR into a copy. Not only would it not be producing random bits but it would remove any randomness from the bits you already have. > > > Pauli > -- > Oracle > Dr Paul Dale | Cryptographer | Network Security & Encryption > Phone +61 7 3031 7217 > Oracle Australia > -- > openssl-dev mailing list > To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4223 bytes Desc: not available URL: From uri at ll.mit.edu Tue Aug 22 00:49:13 2017 From: uri at ll.mit.edu (Blumenthal, Uri - 0553 - MITLL) Date: Tue, 22 Aug 2017 00:49:13 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <9ea2c328-2e6f-4efc-b2d5-f887dd08f4e7@default> References: <1502973289.8546.10.camel@redhat.com> <20170817201108.fnwg2cf3cy7mqdr3@roeckx.be> <1503040968.22810.2.camel@redhat.com> <20170818174256.22dskehi7qs32ezg@roeckx.be> <17AA9277-52E0-4CBE-80D3-AF1128D5B15C@ll.mit.edu> <01BCDAFE-3A62-49A0-9005-AC5D6FCD3A82@akamai.com> <365FF22A-8D78-4874-BEF4-EF9E56A80208@ll.mit.edu> <34101F20-8DD8-4C2E-9150-8FE6FC76D922@akamai.com> <20170821161216.c3lj6emzrddchsvs@roeckx.be> <05B0FAB2-033D-4DD3-BAE7-3530378B1F46@ll.mit.edu> <9ea2c328-2e6f-4efc-b2d5-f887dd08f4e7@default> Message-ID: <5ACBCA7B-2DCD-4B47-A18B-0156B985BF9D@ll.mit.edu> Forgot to add that the adversary would have to compromise not only Intel but also AMD CPUs. Not sure about ARM - but if it implements RDRAND then it must be compromised too, otherwise the enemy victory wouldn be incomplete. ;-) And think of the chips powering mobile devices... Regards, Uri Sent from my iPhone > On Aug 21, 2017, at 20:06, Paul Dale wrote: > > Uri wrote: >>> It might also use things like RDRAND / RDSEED which we don't trust. >> ... >> From cryptography point of view, it cannot hurt, but may help a lot > > There is a scenario where it does hurt: https://www.lvh.io/posts/2013/10/thoughts-on-rdrand-in-linux.html > > This attack wouldn't be difficult to implement given all the out of order execution and look ahead that CPUs do. It requires a compromised RDRAND instruction changing the behaviour of a subsequent XOR into a copy. Not only would it not be producing random bits but it would remove any randomness from the bits you already have. > > > Pauli > -- > Oracle > Dr Paul Dale | Cryptographer | Network Security & Encryption > Phone +61 7 3031 7217 > Oracle Australia > -- > openssl-dev mailing list > To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4223 bytes Desc: not available URL: From l.kostyra at samsung.com Tue Aug 22 10:01:56 2017 From: l.kostyra at samsung.com (Lukasz Kostyra) Date: Tue, 22 Aug 2017 12:01:56 +0200 Subject: [openssl-dev] GCM tag in manual and examples References: Message-ID: <001901d31b2d$b0899040$119cb0c0$@samsung.com> Hello, I've been trying recently to work with OpenSSL and use it to encrypt and decrypt data with AES cipher in GCM mode. While reading the documentation, I noticed an inconsistency between example code and manual. My concern is the function used to set GCM tag when decrypting some data. In current version of the manual[1] it is written that EVP_CIPHER_CTX_ctrl function with EVP_CTRL_GCM_SET_TAG argument can be legally used only before any data is processed by OpenSSL - "Sets the expected tag to taglen bytes from tag. This call is only legal when decrypting data and must be made before any data is processed (e.g. before any EVP_DecryptUpdate() call). However, looking at an example code on wiki[2] it appears that user can set a tag after calls to EVP_DecryptUpdate. The tag must be set only before calling EVP_DecryptFinal, which according how to GCM mode works should be a correct behavior. Running an example code confirms, that user doesn't have to set the tag before any processing calls, only before EVP_DecryptFinal. This inconsistency was checked in 1.0.2, but appearently it appears on 1.1.0 and on master documentation as well (with EVP_CTRL macro being different, as it also involves OCB mode now). Is it just a case of missing correction in documentation? Or maybe the documentation is correct, but there is a bug in OpenSSL? Best Regards, Lukasz Kostyra ------------------ [1] - https://www.openssl.org/docs/man1.0.2/crypto/EVP_CIPHER_CTX_ctrl.html [2] - https://wiki.openssl.org/index.php/EVP_Authenticated_Encryption_and_Decrypti on#Authenticated_Decryption_using_GCM_mode From steve at openssl.org Tue Aug 22 11:50:41 2017 From: steve at openssl.org (Dr. Stephen Henson) Date: Tue, 22 Aug 2017 11:50:41 +0000 Subject: [openssl-dev] GCM tag in manual and examples In-Reply-To: <001901d31b2d$b0899040$119cb0c0$@samsung.com> References: <001901d31b2d$b0899040$119cb0c0$@samsung.com> Message-ID: <20170822115041.GA23438@openssl.org> On Tue, Aug 22, 2017, Lukasz Kostyra wrote: > Hello, > > I've been trying recently to work with OpenSSL and use it to encrypt and > decrypt data with AES cipher in GCM mode. While reading the documentation, I > noticed an inconsistency between example code and manual. > > My concern is the function used to set GCM tag when decrypting some data. In > current version of the manual[1] it is written that EVP_CIPHER_CTX_ctrl > function with EVP_CTRL_GCM_SET_TAG argument can be legally used only before > any data is processed by OpenSSL - "Sets the expected tag to taglen bytes > from tag. This call is only legal when decrypting data and must be made > before any data is processed (e.g. before any EVP_DecryptUpdate() call). > > However, looking at an example code on wiki[2] it appears that user can set > a tag after calls to EVP_DecryptUpdate. The tag must be set only before > calling EVP_DecryptFinal, which according how to GCM mode works should be a > correct behavior. Running an example code confirms, that user doesn't have > to set the tag before any processing calls, only before EVP_DecryptFinal. > > This inconsistency was checked in 1.0.2, but appearently it appears on 1.1.0 > and on master documentation as well (with EVP_CTRL macro being different, as > it also involves OCB mode now). Is it just a case of missing correction in > documentation? Or maybe the documentation is correct, but there is a bug in > OpenSSL? > It's a bug in the documentation. The code used to require the tag first but that was fixed but the documentation wasn't. Steve. -- Dr Stephen N. Henson. OpenSSL project core developer. Commercial tech support now available see: http://www.openssl.org From Brett.R.Nicholas.TH at dartmouth.edu Wed Aug 23 01:50:04 2017 From: Brett.R.Nicholas.TH at dartmouth.edu (Brett R. Nicholas) Date: Wed, 23 Aug 2017 01:50:04 +0000 Subject: [openssl-dev] confusion with rsa_meth_st in a custom RSA engine Message-ID: I am trying to develop a engine for a custom RSA hardware accelerator, and have a few questions about the RSA_METHOD stucture implementation. Some context: for encryption, my accelerator takes as inputs the base, public exponent, and modulus, and returns the resulting ciphertext. For decryption, it takes as inputs the base, and modulus. It does not need a private key, as this is stored in hardware, and can only be configured through an out-of-band channel. I already have a kernel module that exposes an API to userspace programs to use the accelerator. Now I just need to integrate it into openSSL. I've already created a similar engine for AES and SHA256, however I'm struggling with RSA. Ideally, I'd like to not have to worry about anything other than just performing the modular exponentiation on a pre-padded and prepared chunk of data. For SHA and AES, this is straightforward: all that was taken care of by the EVP interface, so all I needed to worry about was getting the data two and from my accelerators. But it doesn't appear to be as simple for RSA (pls correct me if I'm wrong). I'm confused as to which RSA_METHOD function pointers that my engine needs to implement. I show the structure below for reference: struct rsa_meth_st { char *name; int (*rsa_pub_enc) (int flen, const unsigned char *from, unsigned char *to, RSA *rsa, int padding); int (*rsa_pub_dec) (int flen, const unsigned char *from, unsigned char *to, RSA *rsa, int padding); int (*rsa_priv_enc) (int flen, const unsigned char *from, unsigned char *to, RSA *rsa, int padding); int (*rsa_priv_dec) (int flen, const unsigned char *from, unsigned char *to, RSA *rsa, int padding); int (*rsa_mod_exp) (BIGNUM *r0, const BIGNUM *I, RSA *rsa, BN_CTX *ctx); int (*bn_mod_exp) (BIGNUM *r, const BIGNUM *a, const BIGNUM *p, const BIGNUM *m, BN_CTX *ctx, BN_MONT_CTX *m_ctx); /* ....stuff.... */ int flags; /* .... stuff ... */ }; // TYPEDEF'ED TO RSA_METHOD in include/ossl_typ.h So, three questions: 1. Is it possible for the standard OpenSSL RSA implementation to use my engine's "modular exponentiation" function, without having to rewrite the RSA_[public|private]_[encrypt|decrypt] family of functions from /include/openssl/rsa.h? 2. If so, does it suffice to only implement the rsa_mod_exp function? Or must I implement both public_enc/dec and private_enc/dec functions as well? I ask, because the source code for the old Intel RSAX engine (https://gist.github.com/bigbrett/91903f773f9d150b7329c7d462cd220a) does this, but I can't figure out how and when in the "RSA flow" the engine's function gets invoked. 3. In /include/openssl/rsa.h, I saw the following macro for the RSA_METHOD flag field (line 55): /* * This flag means the private key operations will be handled by rsa_mod_exp * and that they do not depend on the private key components being present: * for example a key stored in external hardware. Without this flag * bn_mod_exp gets called when private key components are absent. */ # define RSA_FLAG_EXT_PKEY 0x0020 Does this mean that if I use this flag in the "flags" field of RSA_METHOD, that I DO NOT need to implement rsa_pub_enc/dec and friends? I guess I'm just confused as to at what point in the RSA encryption/decryption process my engine should be invoked at. FWIW I'm planning on releasing a comprehensive engine tutorial and some documentation for all this stuff once I finish (will already be written up in my thesis), so I want to make sure I get these details ironed out! Any words of wisdom would be greatly appreciated. Best, Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From leonb at parsec.co.za Wed Aug 23 09:52:28 2017 From: leonb at parsec.co.za (Leon Brits) Date: Wed, 23 Aug 2017 09:52:28 +0000 Subject: [openssl-dev] Upgrading OpenSSL Message-ID: <13e455db80e548888c9cd15424bb146a@EXCHANGESRV.PARSEC.local> Hi all, I am task to update two machines from v0.9.8z to v1.0.2 (since it is LTS). With the minimal changes, I've been able to get the application on the machines to compile with the newer version and generate RSA 4096 key pairs. The applications are able to successfully use their respective private keys and certificates to establish TLS connection between them. However, when I used the CLI to check a dumped privatekey i got the following output: % openssl rsa -check -in privkey.pem unable to load Private Key 1995859152:error:0D078079:asn1 encoding routines:ASN1_ITEM_EX_D2I:field missing:tasn_dec.c:489:Field=d, Type=RSA 1995859152:error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121: 1995859152:error:0606F091:digital envelope routines:EVP_PKCS82PKEY:private key decode error:evp_pkey.c:92: 1995859152:error:0907B00D:PEM routines:PEM_READ_BIO_PRIVATEKEY:ASN1 lib:pem_pkey.c:141: Any suggestions at what is wrong with the key? Note that an ID is stored in the RSA extended data since the private key may be stored in HSM. Thanks for your time LJB -------------- next part -------------- An HTML attachment was scrubbed... URL: From deengert at gmail.com Wed Aug 23 14:15:47 2017 From: deengert at gmail.com (Douglas E Engert) Date: Wed, 23 Aug 2017 09:15:47 -0500 Subject: [openssl-dev] confusion with rsa_meth_st in a custom RSA engine In-Reply-To: References: Message-ID: You can use RSA_meth_dup to copy the default RSA_METHOD,then use the RSA_meth_set* routines to point to your own routines. it only overrides RSA_meth_set1_name, RSA_meth_set_priv_enc RSA_meth_set_priv_dec RSA_meth_set_finish see: https://github.com/OpenSC/libp11/blob/master/src/p11_rsa.c But You will have to populate the key with the public parts of the key. On 8/22/2017 8:50 PM, Brett R. Nicholas wrote: > I am trying to develop a engine for a custom RSA hardware accelerator, and have a few questions about the RSA_METHOD stucture implementation. > > > Some context: for encryption, my accelerator takes as inputs the base, public exponent, and modulus, and returns the resulting ciphertext. For decryption, it takes as inputs the base, and modulus. It > does not need a private key, as this is stored in hardware, and can only be configured through an out-of-band channel. I already have a kernel module that exposes an API to userspace programs to use > the accelerator. Now I just need to integrate it into openSSL. > > > I've already created a similar engine for AES and SHA256, however I'm struggling with RSA. Ideally, I'd like to not have to worry about anything other than just performing the modular exponentiation > on a pre-padded and prepared chunk of data. For SHA and AES, this is straightforward: all that was taken care of by the EVP interface, so all I needed to worry about was getting the data two and from > my accelerators. But it doesn't appear to be as simple for RSA (pls correct me if I'm wrong). > > > I'm confused as to which RSA_METHOD function pointers that my engine needs to implement. I show the structure below for reference: > > struct rsa_meth_st { > char *name; > int (*rsa_pub_enc) (int flen, const unsigned char *from, > unsigned char *to, RSA *rsa, int padding); > int (*rsa_pub_dec) (int flen, const unsigned char *from, > unsigned char *to, RSA *rsa, int padding); > int (*rsa_priv_enc) (int flen, const unsigned char *from, > unsigned char *to, RSA *rsa, int padding); > int (*rsa_priv_dec) (int flen, const unsigned char *from, > unsigned char *to, RSA *rsa, int padding); > > int (*rsa_mod_exp) (BIGNUM *r0, const BIGNUM *I, RSA *rsa, BN_CTX *ctx); > > int (*bn_mod_exp) (BIGNUM *r, const BIGNUM *a, const BIGNUM *p, > const BIGNUM *m, BN_CTX *ctx, BN_MONT_CTX *m_ctx); > /* ....stuff.... */ > int flags; > /* .... stuff ... */ > }; // TYPEDEF'ED TO RSA_METHOD in include/ossl_typ.h > > > *So, three questions: > * > > * > * > > 1. *Is it possible for the standard OpenSSL RSA implementation to use my engine's "modular exponentiation" function, without having to rewrite the RSA_[public|private]_[encrypt|decrypt] family of > functions from /include/openssl/rsa.h? > * > 2. *If so, does it suffice to only implement the rsa_mod_exp function? *Or must I implement both public_enc/dec and private_enc/dec functions as well? I ask, because the source code for the old Intel > RSAX engine (https://gist.github.com/bigbrett/91903f773f9d150b7329c7d462cd220a) does this, but I can't figure out how and when in the "RSA flow" the engine's function gets invoked. > 3. In /include/openssl/rsa.h, I saw the following macro for the RSA_METHOD flag field (line 55): > > /* > * This flag means the private key operations will be handled by rsa_mod_exp > * and that they do not depend on the private key components being present: > * for example a key stored in external hardware. Without this flag > * bn_mod_exp gets called when private key components are absent. > */ > # define RSA_FLAG_EXT_PKEY 0x0020 > > *Does this mean that if I use this flag in the "flags" field of RSA_METHOD, that I DO NOT need to implement rsa_pub_enc/dec and friends? *I guess I'm just confused as to at what point in the RSA > encryption/decryption process my engine should be invoked at. > > > FWIW I'm planning on releasing a comprehensive engine tutorial and some documentation for all this stuff once I finish (will already be written up in my thesis), so I want to make sure I get these > details ironed out! > > > Any words of wisdom would be greatly appreciated. > > Best, > > Brett > > > > -- Douglas E. Engert From uri at ll.mit.edu Wed Aug 23 16:27:49 2017 From: uri at ll.mit.edu (Blumenthal, Uri - 0553 - MITLL) Date: Wed, 23 Aug 2017 16:27:49 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <20170821161216.c3lj6emzrddchsvs@roeckx.be> References: <1502973289.8546.10.camel@redhat.com> <20170817201108.fnwg2cf3cy7mqdr3@roeckx.be> <1503040968.22810.2.camel@redhat.com> <20170818174256.22dskehi7qs32ezg@roeckx.be> <17AA9277-52E0-4CBE-80D3-AF1128D5B15C@ll.mit.edu> <01BCDAFE-3A62-49A0-9005-AC5D6FCD3A82@akamai.com> <365FF22A-8D78-4874-BEF4-EF9E56A80208@ll.mit.edu> <34101F20-8DD8-4C2E-9150-8FE6FC76D922@akamai.com> <20170821161216.c3lj6emzrddchsvs@roeckx.be> Message-ID: <95605C33-9954-4726-B398-BAE394EA468C@ll.mit.edu> > So I guess you want an interface that can both add things to the > "entropy" pool, and to the "additional data" pool? It shouldn't > be that hard, I'll try to come up with some proposal soon. I?d say the interface that Rich Salz proposed would be good enough: > ? But I think a new API, RAND_add_ex() that took a flag that had values like > RAND_ADD_GLOBAL, RAND_ADD_LOCAL, RAND_ADD_PRIVATE, > RAND_LOCAL_PRIVATE indicating which to seed. Thoughts? It exposes what?s necessary, but nothing more. Another benefit ? malicious input would not compromise the entropy pool. > We should think carefully about what API?s we are exposing, and might want to wait for 1.1.2 Nothing wrong with thinking about what API to expose, and how. Since 1.1.1 is what?s currently being shaped ? there?s no reason to postpone that thinking. Especially since the RNG/DRBG work is being done on 1.1.1 now, and this is a part of it. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5211 bytes Desc: not available URL: From paul.l.kehrer at gmail.com Wed Aug 23 21:12:56 2017 From: paul.l.kehrer at gmail.com (Paul Kehrer) Date: Wed, 23 Aug 2017 17:12:56 -0400 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: References: <97E7DB38-D94A-4CED-BF11-F8AAF2511D81@akamai.com> <1c213baf-07b5-173c-7355-bd424ff0e8f6@python.org> <1502973289.8546.10.camel@redhat.com> <20170817201108.fnwg2cf3cy7mqdr3@roeckx.be> <1503040968.22810.2.camel@redhat.com> <20170818174256.22dskehi7qs32ezg@roeckx.be> Message-ID: On August 19, 2017 at 2:48:19 AM, Salz, Rich via openssl-dev ( openssl-dev at openssl.org) wrote: I think the safest thing is for us to not change the default. Programs that know they are going to fork can do the right/safe thing. It would be nicer if we could automatically always do the right thing, but I don?t think it?s possible. It appears the current position is that since there will be edge cases where a reseed would fail (thus either halting the RNG or silently not reseeding it) that we should not attempt to reseed? I would argue it is better to attempt to reseed and document that edge cases may need to reseed themselves. This dramatically narrows the window from "everybody needs to do it" to "users in certain scenarios that are becoming rarer by the day need to do it". Given that backwards compatibility is a concern maybe failure to reseed on fork should only drop an error on the child process's error queue though? That behavior could potentially be a separate flag that OpenSSL uses by default (OPENSSL_TRY_TO_INIT_ATFORK), and then OPENSSL_INIT_ATFORK can be more strict about reseed failures if desired. -Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: From pwalten at au1.ibm.com Wed Aug 23 22:07:54 2017 From: pwalten at au1.ibm.com (Peter Waltenberg) Date: Thu, 24 Aug 2017 08:07:54 +1000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: References: <97E7DB38-D94A-4CED-BF11-F8AAF2511D81@akamai.com> <1c213baf-07b5-173c-7355-bd424ff0e8f6@python.org> <1502973289.8546.10.camel@redhat.com> <20170817201108.fnwg2cf3cy7mqdr3@roeckx.be> <1503040968.22810.2.camel@redhat.com> <20170818174256.22dskehi7qs32ezg@roeckx.be> Message-ID: The bad case I'm aware of is the fork() one as it's critical that the RNG state diverge on fork(). Without that you can get some very nasty behaviour in things like TLS servers. Some of which have a thread pool + fork() model to handle increasing load. While ideally you'd do a complete reseed, just different state in each RNG is a LOT better than nothing, and even PID + whatever else you can scrounge up will help a lot. Even the high res counters available on most current CPU's would help there because forking multiple processes isn't quite synchronous. I don't think 'telling the user to fix it' is a particularly good option in these cases as in general the user will be even less capable of dealing with this than OpenSSL. By all means warn the users that behaviour may not be ideal, but do your best first. Peter From: Paul Kehrer To: "openssl-dev at openssl.org" Date: 24/08/2017 07:13 Subject: Re: [openssl-dev] Work on a new RNG for OpenSSL Sent by: "openssl-dev" On August 19, 2017 at 2:48:19 AM, Salz, Rich via openssl-dev ( openssl-dev at openssl.org) wrote: I think the safest thing is for us to not change the default. Programs that know they are going to fork can do the right/safe thing. It would be nicer if we could automatically always do the right thing, but I don?t think it?s possible. It appears the current position is that since there will be edge cases where a reseed would fail (thus either halting the RNG or silently not reseeding it) that we should not attempt to reseed? I would argue it is better to attempt to reseed and document that edge cases may need to reseed themselves. This dramatically narrows the window from "everybody needs to do it" to "users in certain scenarios that are becoming rarer by the day need to do it". Given that backwards compatibility is a concern maybe failure to reseed on fork should only drop an error on the child process's error queue though? That behavior could potentially be a separate flag that OpenSSL uses by default (OPENSSL_TRY_TO_INIT_ATFORK), and then OPENSSL_INIT_ATFORK can be more strict about reseed failures if desired. -Paul-- openssl-dev mailing list To unsubscribe: https://urldefense.proofpoint.com/v2/url?u=https-3A__mta.openssl.org_mailman_listinfo_openssl-2Ddev&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=K53ZTnW2gq2IjM1tbpz7kYoHgvTfJ_aR8s4bK_o2xzY&m=fF3EsvxEsVa_Gqvi75hhE7fVtPzma348fsGdbb_GMLc&s=OhHmJAPRZokV7vWQaJy75wCYUxEWJLfAf79YJH6Mhao&e= -------------- next part -------------- An HTML attachment was scrubbed... URL: From rsalz at akamai.com Wed Aug 23 23:13:50 2017 From: rsalz at akamai.com (Salz, Rich) Date: Wed, 23 Aug 2017 23:13:50 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: References: <97E7DB38-D94A-4CED-BF11-F8AAF2511D81@akamai.com> <1c213baf-07b5-173c-7355-bd424ff0e8f6@python.org> <1502973289.8546.10.camel@redhat.com> <20170817201108.fnwg2cf3cy7mqdr3@roeckx.be> <1503040968.22810.2.camel@redhat.com> <20170818174256.22dskehi7qs32ezg@roeckx.be> Message-ID: <6AFBBCC2-A5D0-4AFE-8D01-95F1B9B395C7@akamai.com> Okay: https://github.com/openssl/openssl/pull/4239 -------------- next part -------------- An HTML attachment was scrubbed... URL: From steffen at sdaoden.eu Thu Aug 24 13:47:01 2017 From: steffen at sdaoden.eu (Steffen Nurpmeso) Date: Thu, 24 Aug 2017 15:47:01 +0200 Subject: [openssl-dev] Work on a new RNG for OpenSSL References: <1502973289.8546.10.camel@redhat.com> <20170817201108.fnwg2cf3cy7mqdr3@roeckx.be> <1503040968.22810.2.camel@redhat.com> <20170818174256.22dskehi7qs32ezg@roeckx.be> <17AA9277-52E0-4CBE-80D3-AF1128D5B15C@ll.mit.edu> <01BCDAFE-3A62-49A0-9005-AC5D6FCD3A82@akamai.com> <365FF22A-8D78-4874-BEF4-EF9E56A80208@ll.mit.edu> <34101F20-8DD8-4C2E-9150-8FE6FC76D922@akamai.com> <20170821161216.c3lj6emzrddchsvs@roeckx.be> <95605C33-9954-4726-B398-BAE394EA468C@ll.mit.edu> Message-ID: <20170824134701.XX0XN%steffen@sdaoden.eu> "Blumenthal, Uri - 0553 - MITLL" wrote: |> So I guess you want an interface that can both add things to the |> "entropy" pool, and to the "additional data" pool? It shouldn't |> be that hard, I'll try to come up with some proposal soon. | |I?d say the interface that Rich Salz proposed would be good enough: | |> ? But I think a new API, RAND_add_ex() that took a flag that had \ |> values like |> RAND_ADD_GLOBAL, RAND_ADD_LOCAL, RAND_ADD_PRIVATE, |> RAND_LOCAL_PRIVATE indicating which to seed. Thoughts? | |It exposes what?s necessary, but nothing more. Another benefit ? malicious \ |input would not compromise the entropy pool. | |> We should think carefully about what API?s we are exposing, and might \ |> want to wait for 1.1.2 | |Nothing wrong with thinking about what API to expose, and how. Since \ |1.1.1 is what?s currently being shaped ? there?s no reason to postpone \ |that thinking. Especially since the RNG/DRBG work is being done on \ |1.1.1 now, and this is a part of it. If you say it and continue like that then let me wonder why the objects as such cannot be accessed like RAND_object_get(SPECIAL_CONSTANT_OR_NULL), or RAND_object_create(SPECIAL_CONSTANT_OR_NULL), plus RAND_object_bytes() and RAND_object_add(). And all the current could be macros or inline functions to the corresponding new object based functions. But i am not writing a multithreaded server with the need for performance comparisons, RAND_bytes() is good enough for us. (And since OpenSSL is optional we still have to do some searching around to find a PRNG anyway, with a builtin arc4 fallback.) --steffen | |Der Kragenbaer, The moon bear, |der holt sich munter he cheerfully and one by one |einen nach dem anderen runter wa.ks himself off |(By Robert Gernhardt) From brett.r.nicholas.th at dartmouth.edu Thu Aug 24 15:10:26 2017 From: brett.r.nicholas.th at dartmouth.edu (Brett Nicholas) Date: Thu, 24 Aug 2017 15:10:26 +0000 Subject: [openssl-dev] confusion with rsa_meth_st in a custom RSA engine In-Reply-To: References: Message-ID: An HTML attachment was scrubbed... URL: From kurt at roeckx.be Thu Aug 24 16:48:32 2017 From: kurt at roeckx.be (Kurt Roeckx) Date: Thu, 24 Aug 2017 18:48:32 +0200 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: References: <97E7DB38-D94A-4CED-BF11-F8AAF2511D81@akamai.com> <1c213baf-07b5-173c-7355-bd424ff0e8f6@python.org> <1502973289.8546.10.camel@redhat.com> <20170817201108.fnwg2cf3cy7mqdr3@roeckx.be> <1503040968.22810.2.camel@redhat.com> <20170818174256.22dskehi7qs32ezg@roeckx.be> Message-ID: <20170824164831.w73sqz24muljckr7@roeckx.be> On Wed, Aug 23, 2017 at 05:12:56PM -0400, Paul Kehrer wrote: > On August 19, 2017 at 2:48:19 AM, Salz, Rich via openssl-dev ( > openssl-dev at openssl.org) wrote: > > > I think the safest thing is for us to not change the default. Programs that > know they are going to fork can do the right/safe thing. It would be nicer > if we could automatically always do the right thing, but I don?t think it?s > possible. > > > It appears the current position is that since there will be edge cases > where a reseed would fail (thus either halting the RNG or silently not > reseeding it) that we should not attempt to reseed? I would argue it is > better to attempt to reseed and document that edge cases may need to reseed > themselves. This dramatically narrows the window from "everybody needs to > do it" to "users in certain scenarios that are becoming rarer by the day > need to do it". Given that backwards compatibility is a concern maybe > failure to reseed on fork should only drop an error on the child process's > error queue though? That behavior could potentially be a separate flag that > OpenSSL uses by default (OPENSSL_TRY_TO_INIT_ATFORK), and then > OPENSSL_INIT_ATFORK can be more strict about reseed failures if desired. An idea that I had was to default to reseed on fork if we know we have a working syscall. Kurt From kurt at roeckx.be Thu Aug 24 16:50:06 2017 From: kurt at roeckx.be (Kurt Roeckx) Date: Thu, 24 Aug 2017 18:50:06 +0200 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: References: <1c213baf-07b5-173c-7355-bd424ff0e8f6@python.org> <1502973289.8546.10.camel@redhat.com> <20170817201108.fnwg2cf3cy7mqdr3@roeckx.be> <1503040968.22810.2.camel@redhat.com> <20170818174256.22dskehi7qs32ezg@roeckx.be> Message-ID: <20170824165005.dlyfzozebds7tban@roeckx.be> On Thu, Aug 24, 2017 at 08:07:54AM +1000, Peter Waltenberg wrote: > The bad case I'm aware of is the fork() one as it's critical that the RNG > state diverge on fork(). Without that you can get some very nasty > behaviour in things like TLS servers. Some of which have a thread pool + > fork() model to handle increasing load. > > While ideally you'd do a complete reseed, just different state in each RNG > is a LOT better than nothing, and even PID + whatever else you can > scrounge up will help a lot. Even the high res counters available on most > current CPU's would help there because forking multiple processes isn't > quite synchronous. This is why I want to add things like that by default in the additional data. Kurt From uri at ll.mit.edu Thu Aug 24 17:24:32 2017 From: uri at ll.mit.edu (Blumenthal, Uri - 0553 - MITLL) Date: Thu, 24 Aug 2017 17:24:32 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <20170824134701.XX0XN%steffen@sdaoden.eu> References: <1502973289.8546.10.camel@redhat.com> <20170817201108.fnwg2cf3cy7mqdr3@roeckx.be> <1503040968.22810.2.camel@redhat.com> <20170818174256.22dskehi7qs32ezg@roeckx.be> <17AA9277-52E0-4CBE-80D3-AF1128D5B15C@ll.mit.edu> <01BCDAFE-3A62-49A0-9005-AC5D6FCD3A82@akamai.com> <365FF22A-8D78-4874-BEF4-EF9E56A80208@ll.mit.edu> <34101F20-8DD8-4C2E-9150-8FE6FC76D922@akamai.com> <20170821161216.c3lj6emzrddchsvs@roeckx.be> <95605C33-9954-4726-B398-BAE394EA468C@ll.mit.edu> <20170824134701.XX0XN%steffen@sdaoden.eu> Message-ID: On 8/24/17, 09:45, "openssl-dev on behalf of Steffen Nurpmeso" wrote: >> ? But I think a new API, >> RAND_add_ex() that took a flag >> RAND_ADD_GLOBAL, RAND_ADD_LOCAL, RAND_ADD_PRIVATE, >> RAND_LOCAL_PRIVATE indicating which to seed. > > If you say it and continue like that then let me wonder why the > objects as such cannot be accessed like > > RAND_object_get(SPECIAL_CONSTANT_OR_NULL), or > RAND_object_create(SPECIAL_CONSTANT_OR_NULL), plus > RAND_object_bytes() and RAND_object_add(). And all the current > could be macros or inline functions to the corresponding new > object based functions. My take and motivation is this: - OpenSSL is designed to run on all kinds of architectures, CPU, and OS. Some have better entropy sources then others. - The current blog essentially says that regardless of how many RNGs you defined ? if the first one provides ?enough? entropy, other RNGs won?t be asked to contribute. Some applications may not want to rely on one RNG for tasks they consider critical. - Applications that need better entropy guarantees may want to contribute to the entropy pool by mixing in entropy from sources other than the default RNG. - NIST SP 800-90A Rev 1 understood this need and defined interface ?Additional input? that can be applied to both reseeding and generation. Since not every application has such need, NIST made this interface optional. - OpenSSL is not developed with one specific application in mind, therefore it make sense to implement and provide that optional interface, allowing so applications to mix in randomness from whatever other sources they trust and have available. I personally see no harm if these RNG objects are made available to applications that need/use them. But my first priority is being able to add to the RNG entropy pool more randomness from a source I trust. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5211 bytes Desc: not available URL: From rsalz at akamai.com Thu Aug 24 17:32:15 2017 From: rsalz at akamai.com (Salz, Rich) Date: Thu, 24 Aug 2017 17:32:15 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <20170824164831.w73sqz24muljckr7@roeckx.be> References: <97E7DB38-D94A-4CED-BF11-F8AAF2511D81@akamai.com> <1c213baf-07b5-173c-7355-bd424ff0e8f6@python.org> <1502973289.8546.10.camel@redhat.com> <20170817201108.fnwg2cf3cy7mqdr3@roeckx.be> <1503040968.22810.2.camel@redhat.com> <20170818174256.22dskehi7qs32ezg@roeckx.be> <20170824164831.w73sqz24muljckr7@roeckx.be> Message-ID: <03512938-A182-4BB8-84A1-5B2D9EDA9A5E@akamai.com> > An idea that I had was to default to reseed on fork if we know we > have a working syscall. Or /dev device too? From rsalz at akamai.com Thu Aug 24 17:41:38 2017 From: rsalz at akamai.com (Salz, Rich) Date: Thu, 24 Aug 2017 17:41:38 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: References: <1502973289.8546.10.camel@redhat.com> <20170817201108.fnwg2cf3cy7mqdr3@roeckx.be> <1503040968.22810.2.camel@redhat.com> <20170818174256.22dskehi7qs32ezg@roeckx.be> <17AA9277-52E0-4CBE-80D3-AF1128D5B15C@ll.mit.edu> <01BCDAFE-3A62-49A0-9005-AC5D6FCD3A82@akamai.com> <365FF22A-8D78-4874-BEF4-EF9E56A80208@ll.mit.edu> <34101F20-8DD8-4C2E-9150-8FE6FC76D922@akamai.com> <20170821161216.c3lj6emzrddchsvs@roeckx.be> <95605C33-9954-4726-B398-BAE394EA468C@ll.mit.edu> <20170824134701.XX0XN%steffen@sdaoden.eu> Message-ID: <9B44B53A-0018-4EC3-B69D-3D4D9E19DCBB@akamai.com> > I personally see no harm if these RNG objects are made available to applications that need/use them But decisions like this live forever. It is therefore VERY important to spend time thinking about what becomes part of the public API and what remains hidden so that we can change things later when we have a better understanding. This concept is new to OpenSSL :) But we just put out a 1.1.0 release that made things opaque, so it?s more fresh in the minds of at least some of the dev team. Personally, since DRBG is new in 1.1.1 I would be quite happy if we didn?t expose anything public until the release after that. From rsalz at akamai.com Thu Aug 24 17:47:55 2017 From: rsalz at akamai.com (Salz, Rich) Date: Thu, 24 Aug 2017 17:47:55 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <20170824165005.dlyfzozebds7tban@roeckx.be> References: <1c213baf-07b5-173c-7355-bd424ff0e8f6@python.org> <1502973289.8546.10.camel@redhat.com> <20170817201108.fnwg2cf3cy7mqdr3@roeckx.be> <1503040968.22810.2.camel@redhat.com> <20170818174256.22dskehi7qs32ezg@roeckx.be> <20170824165005.dlyfzozebds7tban@roeckx.be> Message-ID: <1DCD7373-878B-424A-A7AD-F47618CF0EF8@akamai.com> > This is why I want to add things like that by default in the > additional data. On reseed or generate? From uri at ll.mit.edu Thu Aug 24 18:03:45 2017 From: uri at ll.mit.edu (Blumenthal, Uri - 0553 - MITLL) Date: Thu, 24 Aug 2017 18:03:45 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <9B44B53A-0018-4EC3-B69D-3D4D9E19DCBB@akamai.com> References: <1502973289.8546.10.camel@redhat.com> <20170817201108.fnwg2cf3cy7mqdr3@roeckx.be> <1503040968.22810.2.camel@redhat.com> <20170818174256.22dskehi7qs32ezg@roeckx.be> <17AA9277-52E0-4CBE-80D3-AF1128D5B15C@ll.mit.edu> <01BCDAFE-3A62-49A0-9005-AC5D6FCD3A82@akamai.com> <365FF22A-8D78-4874-BEF4-EF9E56A80208@ll.mit.edu> <34101F20-8DD8-4C2E-9150-8FE6FC76D922@akamai.com> <20170821161216.c3lj6emzrddchsvs@roeckx.be> <95605C33-9954-4726-B398-BAE394EA468C@ll.mit.edu> <20170824134701.XX0XN%steffen@sdaoden.eu> <9B44B53A-0018-4EC3-B69D-3D4D9E19DCBB@akamai.com> Message-ID: <3F138A1A-79A5-4568-859E-085D029FB4DF@ll.mit.edu> >> I personally see no harm if these RNG objects are made available to applications that need/use them > > But decisions like this live forever. It is therefore VERY important to spend time thinking about what > becomes part of the public API and what remains hidden so that we can change things later when we > have a better understanding. No argument here, especially since I can?t claim to understand all the implications of exposing RNG (yet ;). But it is clear that at least some applications need to work with an RNG more closely than others, and would want more control over its behavior. > This concept is new to OpenSSL :) :-) :-) > But we just put out a 1.1.0 release that made things opaque, so it?s more fresh in the minds of at > least some of the dev team. In general, making things opaque is a good idea, RNG included. But: > Personally, since DRBG is new in 1.1.1 I would be quite happy if we didn?t expose anything public > until the release after that. Even opaque objects usually have some public interface. I think exposing RAND_add_ex() would be a good idea for 1.1..1, and it?s likely to serve as an acceptable ?live forever? API. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5211 bytes Desc: not available URL: From kurt at roeckx.be Thu Aug 24 18:12:33 2017 From: kurt at roeckx.be (Kurt Roeckx) Date: Thu, 24 Aug 2017 20:12:33 +0200 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <03512938-A182-4BB8-84A1-5B2D9EDA9A5E@akamai.com> References: <1c213baf-07b5-173c-7355-bd424ff0e8f6@python.org> <1502973289.8546.10.camel@redhat.com> <20170817201108.fnwg2cf3cy7mqdr3@roeckx.be> <1503040968.22810.2.camel@redhat.com> <20170818174256.22dskehi7qs32ezg@roeckx.be> <20170824164831.w73sqz24muljckr7@roeckx.be> <03512938-A182-4BB8-84A1-5B2D9EDA9A5E@akamai.com> Message-ID: <20170824181232.nv3p75obywhzk7xs@roeckx.be> On Thu, Aug 24, 2017 at 05:32:15PM +0000, Salz, Rich via openssl-dev wrote: > > > An idea that I had was to default to reseed on fork if we know we > > have a working syscall. ... to get entropy > > Or /dev device too? The point is that you can't be sure that /dev is still going to be available, we need to be sure we still have a source available. Kurt From kurt at roeckx.be Thu Aug 24 18:14:23 2017 From: kurt at roeckx.be (Kurt Roeckx) Date: Thu, 24 Aug 2017 20:14:23 +0200 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <1DCD7373-878B-424A-A7AD-F47618CF0EF8@akamai.com> References: <1502973289.8546.10.camel@redhat.com> <20170817201108.fnwg2cf3cy7mqdr3@roeckx.be> <1503040968.22810.2.camel@redhat.com> <20170818174256.22dskehi7qs32ezg@roeckx.be> <20170824165005.dlyfzozebds7tban@roeckx.be> <1DCD7373-878B-424A-A7AD-F47618CF0EF8@akamai.com> Message-ID: <20170824181423.qjeemfajqzf74zkj@roeckx.be> On Thu, Aug 24, 2017 at 05:47:55PM +0000, Salz, Rich via openssl-dev wrote: > > > This is why I want to add things like that by default in the > > additional data. > > On reseed or generate? Mostly on generate, but I see little point in not doing it on reseed. Kurt From rsalz at akamai.com Thu Aug 24 18:22:38 2017 From: rsalz at akamai.com (Salz, Rich) Date: Thu, 24 Aug 2017 18:22:38 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: <3F138A1A-79A5-4568-859E-085D029FB4DF@ll.mit.edu> References: <1502973289.8546.10.camel@redhat.com> <20170817201108.fnwg2cf3cy7mqdr3@roeckx.be> <1503040968.22810.2.camel@redhat.com> <20170818174256.22dskehi7qs32ezg@roeckx.be> <17AA9277-52E0-4CBE-80D3-AF1128D5B15C@ll.mit.edu> <01BCDAFE-3A62-49A0-9005-AC5D6FCD3A82@akamai.com> <365FF22A-8D78-4874-BEF4-EF9E56A80208@ll.mit.edu> <34101F20-8DD8-4C2E-9150-8FE6FC76D922@akamai.com> <20170821161216.c3lj6emzrddchsvs@roeckx.be> <95605C33-9954-4726-B398-BAE394EA468C@ll.mit.edu> <20170824134701.XX0XN%steffen@sdaoden.eu> <9B44B53A-0018-4EC3-B69D-3D4D9E19DCBB@akamai.com> <3F138A1A-79A5-4568-859E-085D029FB4DF@ll.mit.edu> Message-ID: ? Even opaque objects usually have some public interface. I think exposing RAND_add_ex() would be a good idea for 1.1..1, and it?s likely to serve as an acceptable ?live forever? API. That?s my point. API decisions live forever. Suppose we move around the DRBG?s so that they are per-thread, or per-SSL_CTX or per-SSL object? Will that API still work? Or will we need a A ?RAND_ex_ex? function? We don?t have even consensus on when and how to reseed. From uri at ll.mit.edu Thu Aug 24 20:33:11 2017 From: uri at ll.mit.edu (Blumenthal, Uri - 0553 - MITLL) Date: Thu, 24 Aug 2017 20:33:11 +0000 Subject: [openssl-dev] Work on a new RNG for OpenSSL In-Reply-To: References: <1502973289.8546.10.camel@redhat.com> <20170817201108.fnwg2cf3cy7mqdr3@roeckx.be> <1503040968.22810.2.camel@redhat.com> <20170818174256.22dskehi7qs32ezg@roeckx.be> <17AA9277-52E0-4CBE-80D3-AF1128D5B15C@ll.mit.edu> <01BCDAFE-3A62-49A0-9005-AC5D6FCD3A82@akamai.com> <365FF22A-8D78-4874-BEF4-EF9E56A80208@ll.mit.edu> <34101F20-8DD8-4C2E-9150-8FE6FC76D922@akamai.com> <20170821161216.c3lj6emzrddchsvs@roeckx.be> <95605C33-9954-4726-B398-BAE394EA468C@ll.mit.edu> <20170824134701.XX0XN%steffen@sdaoden.eu> <9B44B53A-0018-4EC3-B69D-3D4D9E19DCBB@akamai.com> <3F138A1A-79A5-4568-859E-085D029FB4DF@ll.mit.edu> Message-ID: <3711646C-6859-4221-8EA8-74527E39734A@ll.mit.edu> >> Even opaque objects usually have some public interface. I think exposing RAND_add_ex() >> would be a good idea for 1.1..1, and it?s likely to serve as an acceptable ?live forever? API. > > That?s my point. API decisions live forever. Point well taken. Nonetheless? > Suppose we move around the DRBG?s so that they are per-thread, or > per-SSL_CTX or per-SSL object? Will that API still work? Or will we > need a A ?RAND_ex_ex? function? The *API* probably still would work. The *implementation* of it (which is supposed to stay under the hood) may need to change. Offhand, I don?t see why the user-level API call would have to be different depending on whether the relevant DRBG is thread-local, or SSL-object-local. The only difficulty would be if you want to have *both*? But my point is ? this (?reseeding assistant?) is a needed capability. So it should be available to the users rather sooner than later? > We don?t have even consensus on when and how to reseed. Luckily, for the interface(s) I?m asking for it?s simple ? reseed upon explicit request. ;-) I understand the concerns for the reseeding in general. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5211 bytes Desc: not available URL: From levitte at openssl.org Wed Aug 23 04:15:27 2017 From: levitte at openssl.org (Richard Levitte) Date: Wed, 23 Aug 2017 06:15:27 +0200 (CEST) Subject: [openssl-dev] confusion with rsa_meth_st in a custom RSA engine In-Reply-To: References: Message-ID: <20170823.061527.529748126471539023.levitte@openssl.org> In message on Wed, 23 Aug 2017 01:50:04 +0000, "Brett R. Nicholas" said: > I am trying to develop a engine for a custom RSA hardware accelerator, and have a few questions > about the RSA_METHOD stucture implementation. ... > I'm confused as to which RSA_METHOD function pointers that my engine needs to implement. I > show the structure below for reference: > > struct rsa_meth_st { > char *name; > int (*rsa_pub_enc) (int flen, const unsigned char *from, > unsigned char *to, RSA *rsa, int padding); > int (*rsa_pub_dec) (int flen, const unsigned char *from, > unsigned char *to, RSA *rsa, int padding); > int (*rsa_priv_enc) (int flen, const unsigned char *from, > unsigned char *to, RSA *rsa, int padding); > int (*rsa_priv_dec) (int flen, const unsigned char *from, > unsigned char *to, RSA *rsa, int padding); > > int (*rsa_mod_exp) (BIGNUM *r0, const BIGNUM *I, RSA *rsa, BN_CTX *ctx); > > int (*bn_mod_exp) (BIGNUM *r, const BIGNUM *a, const BIGNUM *p, > const BIGNUM *m, BN_CTX *ctx, BN_MONT_CTX *m_ctx); > /* ....stuff.... */ > int flags; > /* .... stuff ... */ > }; // TYPEDEF'ED TO RSA_METHOD in include/ossl_typ.h > > So, three questions: > > 1 Is it possible for the standard OpenSSL RSA implementation to use my engine's > "modular exponentiation" function, without having to rewrite the RSA_ > [public|private]_[encrypt|decrypt] family of functions from /include/openssl/rsa.h? Yes. > 2 If so, does it suffice to only implement the rsa_mod_exp function? Or must I implement > both public_enc/dec and private_enc/dec functions as well? I ask, because the source code for > the old Intel RSAX engine > (https://gist.github.com/bigbrett/91903f773f9d150b7329c7d462cd220a) does this, but I can't > figure out how and when in the "RSA flow" the engine's function gets invoked. I'd like to point out this part of the code, which is relevant: #ifndef OPENSSL_NO_RSA meth1 = RSA_PKCS1_SSLeay(); e_rsax_rsa.rsa_pub_enc = meth1->rsa_pub_enc; e_rsax_rsa.rsa_pub_dec = meth1->rsa_pub_dec; e_rsax_rsa.rsa_priv_enc = meth1->rsa_priv_enc; e_rsax_rsa.rsa_priv_dec = meth1->rsa_priv_dec; e_rsax_rsa.bn_mod_exp = meth1->bn_mod_exp; e_rsax_rsa.finish = meth1->finish; #endif You you see, you don't need to actually *reimplement* the public/private encrypt/decrypt functions, but you need to make sure there's *some* implementation available through the method you build up. Incidently, in more modern OpenSSLs, you should use RSA_PKCS1_OpenSSL() rather than RSA_PKCS1_SSLeay(). Anyhow, the standard public/private encrypt/decrypt functions will call the mod_exp functions that you implement > 3 In /include/openssl/rsa.h, I saw the following macro for the RSA_METHOD flag field (line 55): > > /* > * This flag means the private key operations will be handled by rsa_mod_exp > * and that they do not depend on the private key components being present: > * for example a key stored in external hardware. Without this flag > * bn_mod_exp gets called when private key components are absent. > */ > # define RSA_FLAG_EXT_PKEY 0x0020 > > Does this mean that if I use this flag in the "flags" field of RSA_METHOD, that I DO NOT > need to implement rsa_pub_enc/dec and friends? I guess I'm just confused as to at what > point in the RSA encryption/decryption process my engine should be invoked at. That flag means that the standard public/private encrypt/decrypt won't try to access the p, q, dmp1 and iqmp components of the RSA structure, i.e. the components that make up the private part. Instead, that's left entirely to the mod_exp function (i.e. what you actually do implement). If you want to see for yourself what's happening, I suggest a study of crypto/rsa/rsa_ossl.c (OpenSSL 1.1.0 and up) or crypro/rsa/rsa_eay.c (OpenSSL 1.0.2). Cheers, Richard -- Richard Levitte levitte at openssl.org OpenSSL Project http://www.openssl.org/~levitte/ From rsalz at akamai.com Sat Aug 26 15:07:37 2017 From: rsalz at akamai.com (Salz, Rich) Date: Sat, 26 Aug 2017 15:07:37 +0000 Subject: [openssl-dev] Reseeding at fork time Message-ID: For those interested, there is a lot of discussion going on over at https://github.com/openssl/openssl/pull/4239 I consider it bad that the conversation is happening there, not here, but so it goes. If you don?t have a GitHub account and wish to post, forward to me and I will post it for you. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Brett.R.Nicholas.TH at dartmouth.edu Sun Aug 27 15:34:05 2017 From: Brett.R.Nicholas.TH at dartmouth.edu (Brett R. Nicholas) Date: Sun, 27 Aug 2017 15:34:05 +0000 Subject: [openssl-dev] confusion with rsa_meth_st in a custom RSA engine Message-ID: I just realized that my reply to Richard and Douglas didn't go through....I had the emails bundled to a digest, so not sure how to properly reply to an individual thread. My prior email should have said "Thanks, I"ll give it a try and let you know how it goes". Disabled the digest now, though. @Richard, I've looked at the source like you suggested. One more question......quoting your reply below: > That flag means that the standard public/private encrypt/decrypt won't > try to access the p, q, dmp1 and iqmp components of the RSA structure, > i.e. the components that make up the private part. Instead, that's > left entirely to the mod_exp function (i.e. what you actually do implement). This makes sense to me, and it seems that is the desired behavior. However, if I *only* reimplement the rsa_mod_exp() function, and leave the encrypt/decrypt functions to the default openSSL implementations, how can my engine know which of the four of those functions called its rsa_mod_exp function()? To put it another way: my accelerator will need to know whether it is meant to perform public key or private key operations, since the inputs will be written to different memory addresses. From within rsa_mod_exp(), the only way I can think of determining whether the function has been called by private_encrypt/decrypt() or public_encrypt/decrypt would be to check if the p and q fields of the RSA *rsa struct passed into the function are NULL? But I can't tell from the source code if this will be guaranteed just by having RSA_FLAG_EXT_PKEY set in the "flags" field of my engine's RSA_METHOD. Thanks for the help so far! Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at openssl.org Mon Aug 28 01:45:17 2017 From: steve at openssl.org (Dr. Stephen Henson) Date: Mon, 28 Aug 2017 01:45:17 +0000 Subject: [openssl-dev] confusion with rsa_meth_st in a custom RSA engine In-Reply-To: References: Message-ID: <20170828014517.GA27863@openssl.org> On Sun, Aug 27, 2017, Brett R. Nicholas wrote: > > This makes sense to me, and it seems that is the desired behavior. However, > if I *only* reimplement the rsa_mod_exp() function, and leave the > encrypt/decrypt functions to the default openSSL implementations, how can my > engine know which of the four of those functions called its rsa_mod_exp > function()? To put it another way: my accelerator will need to know whether > it is meant to perform public key or private key operations, since the > inputs will be written to different memory addresses. From within > rsa_mod_exp(), the only way I can think of determining whether the function > has been called by private_encrypt/decrypt() or public_encrypt/decrypt would > be to check if the p and q fields of the RSA *rsa struct passed into the > function are NULL? But I can't tell from the source code if this will be > guaranteed just by having RSA_FLAG_EXT_PKEY set in the "flags" field of my > engine's RSA_METHOD. > The rsa_mod_exp function is only called for private key operations. You can't tell if it is a private encrypt or a private decrypt though but that shouldn't matter because the operation performed at that level is the same for both. Steve. -- Dr Stephen N. Henson. OpenSSL project core developer. Commercial tech support now available see: http://www.openssl.org From leonb at parsec.co.za Mon Aug 28 06:08:17 2017 From: leonb at parsec.co.za (Leon Brits) Date: Mon, 28 Aug 2017 06:08:17 +0000 Subject: [openssl-dev] Upgrading OpenSSL In-Reply-To: <13e455db80e548888c9cd15424bb146a@EXCHANGESRV.PARSEC.local> References: <13e455db80e548888c9cd15424bb146a@EXCHANGESRV.PARSEC.local> Message-ID: Thanks for the help. I've come to learn that my problem is the HSM. It removes the RSA values p,q and d from the EVP key before returning it. This is normal since it is protecting the key by keeping it in the HSM - duh. Anyway so, I cannot use it as a normal key. "Live and learn" So this bring me to the next question: Is there any changes I need to make in the OpenSSL Engine for my upgrade (0.9.8 -> 1.0.2) to be complete? Regards, Leon Brits System Engineer Mobile: +27 84 250 2855 [cid:image001.png at 01D31FD4.CD5D06B0] 76 Regency Drive Route 21 Corporate Park Irene 0157 Tel +27 12 678 9740 (ext. 9767) | Fax +27 12 345 2561 www.parsec.co.za [cid:image002.png at 01D31FD4.CD5D06B0] From: openssl-dev [mailto:openssl-dev-bounces at openssl.org] On Behalf Of Leon Brits Sent: 23 August 2017 11:52 AM To: openssl-dev at openssl.org Subject: [openssl-dev] Upgrading OpenSSL Hi all, I am task to update two machines from v0.9.8z to v1.0.2 (since it is LTS). With the minimal changes, I've been able to get the application on the machines to compile with the newer version and generate RSA 4096 key pairs. The applications are able to successfully use their respective private keys and certificates to establish TLS connection between them. However, when I used the CLI to check a dumped privatekey i got the following output: % openssl rsa -check -in privkey.pem unable to load Private Key 1995859152:error:0D078079:asn1 encoding routines:ASN1_ITEM_EX_D2I:field missing:tasn_dec.c:489:Field=d, Type=RSA 1995859152:error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121: 1995859152:error:0606F091:digital envelope routines:EVP_PKCS82PKEY:private key decode error:evp_pkey.c:92: 1995859152:error:0907B00D:PEM routines:PEM_READ_BIO_PRIVATEKEY:ASN1 lib:pem_pkey.c:141: Any suggestions at what is wrong with the key? Note that an ID is stored in the RSA extended data since the private key may be stored in HSM. Thanks for your time LJB -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 14099 bytes Desc: image001.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.png Type: image/png Size: 9367 bytes Desc: image002.png URL: From David.von.Oheimb at siemens.com Mon Aug 28 11:46:51 2017 From: David.von.Oheimb at siemens.com (David von Oheimb) Date: Mon, 28 Aug 2017 13:46:51 +0200 Subject: [openssl-dev] How to use BIO_do_connect(), blocking and non-blocking with timeout, coping with errors Message-ID: <69d6c8c3-c3ba-44f5-fad0-4b2294e210d3@siemens.com> Hi all, I'm currently enhancing HTTP(S) clients based on OpenSSL in several flavors, in particular a CMP client, which in turn uses simple HTTP clients for contacting CRL distribution points or OCSP responders. Getting the client connect right appears surprisingly messy when one needs to cope with all kinds of network error situations including domain name resolution issues and temporarily unreachable servers. Both indefinitely blocking and non-blocking behavior (i.e., connection attempts with and without a timeout) should be supported. Since these are pretty general problems I wonder why there there is rather limited support via generic higher-level OpenSSL or C library functions, or at least I was unable to find it. Instead, the OpenSSL apps contain code that calls BIO_do_connect directly (or the equivalent BIO_do_handshake), in particular query_responder() in apps/ocsp.c. (The situation is similar for the subsequent exchange of data via the BIO, optionally with a timeout). So I constructed my own abstraction, called bio_connect, which took quite some effort testing network error situations. Please see below its code including comments on some strange behavior I experienced and my workarounds for that. Does this code make sense, or do I miss anything? How about adding such a function for instance to crypto/bio/bio_lib.c? BTW, my code uses a handy generic helper function, socket_wait, for waiting for read/write form/to a socket, with a given timeout. Since several instances of that pretty common code pattern using select() are spread over the OpenSSL apps (and crypto lib), I suggest adding this function to the library. Where would be a good place to put it? Thanks, David > /* returns -1 on error, 0 on timeout, 1 on success */ > int bio_connect(BIO *bio, int timeout) { > int blocking; > time_t max_time; > int rv; > > blocking = timeout == 0; > max_time = timeout != 0 ? time(NULL) + timeout : 0; > > if (!blocking) > BIO_set_nbio(bio, 1); > retry: > rv = BIO_do_connect(bio); > if (rv <= 0 && (errno == ETIMEDOUT /* in blocking case, > despite blocking BIO, BIO_do_connect() timed out */ || > ERR_GET_REASON(ERR_peek_error()) == ETIMEDOUT/* when non-blocking, > BIO_do_connect() timed out early with rv == -1 and errno == 0 */)) { > ERR_clear_error(); > (void)BIO_reset(bio); /* otherwise, blocking next connect() may crash > and non-blocking next BIO_do_connect() will fail */ > goto retry; > } > if (!blocking && rv <= 0 && BIO_should_retry(bio)) { > int fd; > if (BIO_get_fd(bio, &fd) <= 0) > return -1; > rv = socket_wait(fd, 1, max_time - time(NULL)); > if (rv > 0) > /* for some reason, select() may wrongly have returned success */ > goto retry; > } > return rv; > } > /* returns < 0 on error, 0 on timeout, >0 on success */ > int socket_wait(int fd, int for_read, int timeout) > { > fd_set confds; > struct timeval tv; > > if (timeout <= 0) > return 0; > > FD_ZERO(&confds); > openssl_fdset(fd, &confds); > tv.tv_usec = 0; > tv.tv_sec = timeout; > return select(fd + 1, for_read ? &confds : NULL, > for_read ? NULL : &confds, NULL, &tv); > } From leonb at parsec.co.za Mon Aug 28 14:09:19 2017 From: leonb at parsec.co.za (Leon Brits) Date: Mon, 28 Aug 2017 14:09:19 +0000 Subject: [openssl-dev] Upgrading OpenSSL In-Reply-To: References: <13e455db80e548888c9cd15424bb146a@EXCHANGESRV.PARSEC.local> Message-ID: The upgrade is now working fine in one of the applications which make TLS connections. I can see the engine functions being called when some action (sign/verify) which require the privatekey. However, this engine is also used in a patched version of Racoon2. In one of the files (crypto_openssl.c) a function is called by the IKE daemon (iked) during setup with: : EVP_SignInit(&ctx, md); EVP_SignUpdate(&ctx, octets->v, octets->l); EVP_SignFinal(&ctx, (unsigned char*)sig->v, &siglen, pkey); : With the upgraded OpenSSL v1.0.2, the last function now fails with the error: 2017-08-28 15:44:14 [INTERNAL_ERR]: crypto_openssl.c:1238:eay_rsassa_pkcs1_v1_5_sign(): RSA_sign failed: 30972:error:0606B06E:digital envelope routines:EVP_SignFinal:wrong public key type:p_sign.c:123: Q: I assume it is looking for one of the missing parameters (p) in the RSA structure - correct? Q: If so, how did it work in v0.9.8? If I change the first command to: EVP_SignInit_ex(&ctx, md, engine); Then it segfaults in the "SignFinal" command :( The engine is dynamically loaded the same way in both my TLS connection application and in the Racoon2 application. Thanks for your time - any help is appreciated! Leon Brits System Engineer Mobile: +27 84 250 2855 [cid:image001.png at 01D32016.91CF07F0] 76 Regency Drive Route 21 Corporate Park Irene 0157 Tel +27 12 678 9740 (ext. 9767) | Fax +27 12 345 2561 www.parsec.co.za [cid:image002.png at 01D32016.91CF07F0] From: openssl-dev [mailto:openssl-dev-bounces at openssl.org] On Behalf Of Leon Brits Sent: 28 August 2017 08:08 AM To: openssl-dev at openssl.org Subject: Re: [openssl-dev] Upgrading OpenSSL Thanks for the help. I've come to learn that my problem is the HSM. It removes the RSA values p,q and d from the EVP key before returning it. This is normal since it is protecting the key by keeping it in the HSM - duh. Anyway so, I cannot use it as a normal key. "Live and learn" So this bring me to the next question: Is there any changes I need to make in the OpenSSL Engine for my upgrade (0.9.8 -> 1.0.2) to be complete? Regards, Leon Brits System Engineer Mobile: +27 84 250 2855 [cid:image001.png at 01D32016.91CF07F0] 76 Regency Drive Route 21 Corporate Park Irene 0157 Tel +27 12 678 9740 (ext. 9767) | Fax +27 12 345 2561 www.parsec.co.za [cid:image002.png at 01D32016.91CF07F0] From: openssl-dev [mailto:openssl-dev-bounces at openssl.org] On Behalf Of Leon Brits Sent: 23 August 2017 11:52 AM To: openssl-dev at openssl.org Subject: [openssl-dev] Upgrading OpenSSL Hi all, I am task to update two machines from v0.9.8z to v1.0.2 (since it is LTS). With the minimal changes, I've been able to get the application on the machines to compile with the newer version and generate RSA 4096 key pairs. The applications are able to successfully use their respective private keys and certificates to establish TLS connection between them. However, when I used the CLI to check a dumped privatekey i got the following output: % openssl rsa -check -in privkey.pem unable to load Private Key 1995859152:error:0D078079:asn1 encoding routines:ASN1_ITEM_EX_D2I:field missing:tasn_dec.c:489:Field=d, Type=RSA 1995859152:error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121: 1995859152:error:0606F091:digital envelope routines:EVP_PKCS82PKEY:private key decode error:evp_pkey.c:92: 1995859152:error:0907B00D:PEM routines:PEM_READ_BIO_PRIVATEKEY:ASN1 lib:pem_pkey.c:141: Any suggestions at what is wrong with the key? Note that an ID is stored in the RSA extended data since the private key may be stored in HSM. Thanks for your time LJB -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 14099 bytes Desc: image001.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.png Type: image/png Size: 9367 bytes Desc: image002.png URL: From Brett.R.Nicholas.TH at dartmouth.edu Mon Aug 28 15:56:14 2017 From: Brett.R.Nicholas.TH at dartmouth.edu (Brett R. Nicholas) Date: Mon, 28 Aug 2017 15:56:14 +0000 Subject: [openssl-dev] confusion with rsa_meth_st in a custom RSA engine In-Reply-To: <20170828014517.GA27863@openssl.org> References: , <20170828014517.GA27863@openssl.org> Message-ID: > The rsa_mod_exp function is only called for private key operations. You can't > tell if it is a private encrypt or a private decrypt though but that > shouldn't matter because the operation performed at that level is the same for > both. Ah, I see. So to clarify (pls correct me if I'm wrong): * rsa_mod_exp() is the modular exponentiation function that openSSL will attempt to use for all private key operations (if RSA_FLAG_EXT_PKEY is set, or the private parameters of that method are non-null), before defaulting to bn_mod_exp(). * And bn_mod_exp() is the modular exponentiation function used by all public key operations So in my case, I should set RSA_FLAG_EX_PKEY in my engine's RSA_METHOD struct, and then implement my own versions of rsa_mod_exp (for the private key encryption/decryption), and bn_mod_exp (for the public key encryption/decryption). Brett ________________________________ From: openssl-dev on behalf of Dr. Stephen Henson Sent: Sunday, August 27, 2017 6:45 PM To: openssl-dev at openssl.org Subject: Re: [openssl-dev] confusion with rsa_meth_st in a custom RSA engine On Sun, Aug 27, 2017, Brett R. Nicholas wrote: > > This makes sense to me, and it seems that is the desired behavior. However, > if I *only* reimplement the rsa_mod_exp() function, and leave the > encrypt/decrypt functions to the default openSSL implementations, how can my > engine know which of the four of those functions called its rsa_mod_exp > function()? To put it another way: my accelerator will need to know whether > it is meant to perform public key or private key operations, since the > inputs will be written to different memory addresses. From within > rsa_mod_exp(), the only way I can think of determining whether the function > has been called by private_encrypt/decrypt() or public_encrypt/decrypt would > be to check if the p and q fields of the RSA *rsa struct passed into the > function are NULL? But I can't tell from the source code if this will be > guaranteed just by having RSA_FLAG_EXT_PKEY set in the "flags" field of my > engine's RSA_METHOD. > The rsa_mod_exp function is only called for private key operations. You can't tell if it is a private encrypt or a private decrypt though but that shouldn't matter because the operation performed at that level is the same for both. Steve. -- Dr Stephen N. Henson. OpenSSL project core developer. Commercial tech support now available see: https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.openssl.org&data=02%7C01%7Cbrett.r.nicholas.th%40dartmouth.edu%7C5374bb23132c4377080c08d4edb671de%7C995b093648d640e5a31ebf689ec9446f%7C0%7C0%7C636394815232517002&sdata=AdSj9Ys%2FG%2FQ1VJIIaD5kZhAwbki5mcF0AL8b2UBxic4%3D&reserved=0 -- openssl-dev mailing list To unsubscribe: https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmta.openssl.org%2Fmailman%2Flistinfo%2Fopenssl-dev&data=02%7C01%7Cbrett.r.nicholas.th%40dartmouth.edu%7C5374bb23132c4377080c08d4edb671de%7C995b093648d640e5a31ebf689ec9446f%7C0%7C0%7C636394815232517002&sdata=zzQB9AlLiORZC2btJ6zi1MZQxVzoB8pgWHTtAC8LHtM%3D&reserved=0 -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at openssl.org Mon Aug 28 17:32:30 2017 From: steve at openssl.org (Dr. Stephen Henson) Date: Mon, 28 Aug 2017 17:32:30 +0000 Subject: [openssl-dev] confusion with rsa_meth_st in a custom RSA engine In-Reply-To: References: <20170828014517.GA27863@openssl.org> Message-ID: <20170828173230.GA10656@openssl.org> On Mon, Aug 28, 2017, Brett R. Nicholas wrote: > > The rsa_mod_exp function is only called for private key operations. You can't > > tell if it is a private encrypt or a private decrypt though but that > > shouldn't matter because the operation performed at that level is the same for > > both. > > Ah, I see. So to clarify (pls correct me if I'm wrong): > > * rsa_mod_exp() is the modular exponentiation function that openSSL will attempt to use for all private key operations (if RSA_FLAG_EXT_PKEY is set, or the private parameters of that method are non-null), before defaulting to bn_mod_exp(). > * And bn_mod_exp() is the modular exponentiation function used by all public key operations > > > So in my case, I should set RSA_FLAG_EX_PKEY in my engine's RSA_METHOD struct, and then implement my own versions of rsa_mod_exp (for the private key encryption/decryption), and bn_mod_exp (for the public key encryption/decryption). > > Yes. Note also that if you set the public key components (n, e) you don't need to perform the public key operations in your ENGINE if you keep the original bn_mod_exp(): OpenSSL will do them for you. If possible you should set the public key components anyway: some operations such as generating certificate requests require them to be present. Steve. -- Dr Stephen N. Henson. OpenSSL project core developer. Commercial tech support now available see: http://www.openssl.org From steve at openssl.org Mon Aug 28 17:36:02 2017 From: steve at openssl.org (Dr. Stephen Henson) Date: Mon, 28 Aug 2017 17:36:02 +0000 Subject: [openssl-dev] Upgrading OpenSSL In-Reply-To: References: <13e455db80e548888c9cd15424bb146a@EXCHANGESRV.PARSEC.local> Message-ID: <20170828173602.GB10656@openssl.org> On Mon, Aug 28, 2017, Leon Brits wrote: > The upgrade is now working fine in one of the applications which make TLS connections. I can see the engine functions being called when some action (sign/verify) which require the privatekey. > > However, this engine is also used in a patched version of Racoon2. > In one of the files (crypto_openssl.c) a function is called by the IKE daemon (iked) during setup with: > : > EVP_SignInit(&ctx, md); > EVP_SignUpdate(&ctx, octets->v, octets->l); > EVP_SignFinal(&ctx, (unsigned char*)sig->v, &siglen, pkey); > : > With the upgraded OpenSSL v1.0.2, the last function now fails with the error: > 2017-08-28 15:44:14 [INTERNAL_ERR]: crypto_openssl.c:1238:eay_rsassa_pkcs1_v1_5_sign(): RSA_sign failed: 30972:error:0606B06E:digital envelope routines:EVP_SignFinal:wrong public key type:p_sign.c:123: > > Q: I assume it is looking for one of the missing parameters (p) in the RSA structure - correct? > Q: If so, how did it work in v0.9.8? > > If I change the first command to: > EVP_SignInit_ex(&ctx, md, engine); > Then it segfaults in the "SignFinal" command :( > > The engine is dynamically loaded the same way in both my TLS connection application and in the Racoon2 application. > > Thanks for your time - any help is appreciated! > Have you recompiled the application? Some structures have changed between OpenSSL 0.9.8 and 1.0.2 so you'll get problems with applications (or an ENGINE) compiled against the wrong headers. If that isn't the problem then what is "md" set to? Steve. -- Dr Stephen N. Henson. OpenSSL project core developer. Commercial tech support now available see: http://www.openssl.org From Brett.R.Nicholas.TH at dartmouth.edu Mon Aug 28 18:05:44 2017 From: Brett.R.Nicholas.TH at dartmouth.edu (Brett R. Nicholas) Date: Mon, 28 Aug 2017 18:05:44 +0000 Subject: [openssl-dev] confusion with rsa_meth_st in a custom RSA engine In-Reply-To: <20170828173230.GA10656@openssl.org> References: <20170828014517.GA27863@openssl.org> , <20170828173230.GA10656@openssl.org> Message-ID: > Note also that if you set the public key components (n, e) you don't need > to perform the public key operations in your ENGINE if you keep the original > bn_mod_exp(): OpenSSL will do them for you This is good to know. However, I think I will be implementing modular exponentiation for both public and private keys in hardware for proof of concept. One more follow up question: > If possible you should set the public key components anyway: some operations > such as generating certificate requests require them to be present I'm confused what you mean here, since my engine doesn't "own" any instances of an RSA struct, it just has a static instance of RSA_METHOD struct defined. So therefore my engine never "sets" public or private key components. It just uses the modexp functions to write the public/private data (contained in the RSA struct passed as an argument from the higher level encrypt/decrypt functions) out to the hardware accelerator, and then return the result. So I could never "set the public key components anyway", as they would be set by whichever program calls RSA_public/private_encrypt/decrypt(). Is my implementing it in this way different than how you thought I was implementing it? It made sense to me to do it this way, however please let me know if you think I'm going about it wrong, or if there are issues with this particular strategy. I want to make sure I'm using the engine API in the most intuitive and efficient way! Brett ________________________________ From: openssl-dev on behalf of Dr. Stephen Henson Sent: Monday, August 28, 2017 10:32:30 AM To: openssl-dev at openssl.org Subject: Re: [openssl-dev] confusion with rsa_meth_st in a custom RSA engine On Mon, Aug 28, 2017, Brett R. Nicholas wrote: > > The rsa_mod_exp function is only called for private key operations. You can't > > tell if it is a private encrypt or a private decrypt though but that > > shouldn't matter because the operation performed at that level is the same for > > both. > > Ah, I see. So to clarify (pls correct me if I'm wrong): > > * rsa_mod_exp() is the modular exponentiation function that openSSL will attempt to use for all private key operations (if RSA_FLAG_EXT_PKEY is set, or the private parameters of that method are non-null), before defaulting to bn_mod_exp(). > * And bn_mod_exp() is the modular exponentiation function used by all public key operations > > > So in my case, I should set RSA_FLAG_EX_PKEY in my engine's RSA_METHOD struct, and then implement my own versions of rsa_mod_exp (for the private key encryption/decryption), and bn_mod_exp (for the public key encryption/decryption). > > Yes. Note also that if you set the public key components (n, e) you don't need to perform the public key operations in your ENGINE if you keep the original bn_mod_exp(): OpenSSL will do them for you. If possible you should set the public key components anyway: some operations such as generating certificate requests require them to be present. Steve. -- Dr Stephen N. Henson. OpenSSL project core developer. Commercial tech support now available see: https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.openssl.org&data=02%7C01%7Cbrett.r.nicholas.th%40dartmouth.edu%7Cb1b7883dd5584c8488d708d4ee3ac4e4%7C995b093648d640e5a31ebf689ec9446f%7C0%7C0%7C636395383568793933&sdata=wDesZm0QTSpMHyOtQyAaWQRzEVYFmp9ewgZaZDn6unI%3D&reserved=0 -- openssl-dev mailing list To unsubscribe: https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmta.openssl.org%2Fmailman%2Flistinfo%2Fopenssl-dev&data=02%7C01%7Cbrett.r.nicholas.th%40dartmouth.edu%7Cb1b7883dd5584c8488d708d4ee3ac4e4%7C995b093648d640e5a31ebf689ec9446f%7C0%7C0%7C636395383568793933&sdata=g8dPj%2Fx9GwR6ldjezTCNKWkvcIOzy1nK0wSLK30ZjjY%3D&reserved=0 -------------- next part -------------- An HTML attachment was scrubbed... URL: From rsalz at akamai.com Tue Aug 29 00:11:02 2017 From: rsalz at akamai.com (Salz, Rich) Date: Tue, 29 Aug 2017 00:11:02 +0000 Subject: [openssl-dev] CVE 2017-3735 OOB read Message-ID: <802EA6B2-0BE2-41D3-81BC-216BE0FEF478@akamai.com> From https://www.openssl.org/news/secadv/20170828.txt OpenSSL Security Advisory [28 Aug 2017] ======================================== Malformed X.509 IPAdressFamily could cause OOB read (CVE-2017-3735) =================================================================== Severity: Low If an X.509 certificate has a malformed IPAddressFamily extension, OpenSSL could do a one-byte buffer overread. The most likely result would be an erroneous display of the certificate in text format. As this is a low severity fix, no release is being made. The fix can be found in the source repository (1.0.2, 1.1.0, and master branches); see https://github.com/openssl/openssl/pull/4276. This bug has been present since 2006. This issue was found by Google's OSS-Fuzz project on August 22. The fix was developed by Rich Salz of the OpenSSL development team. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at openssl.org Tue Aug 29 00:18:08 2017 From: steve at openssl.org (Dr. Stephen Henson) Date: Tue, 29 Aug 2017 00:18:08 +0000 Subject: [openssl-dev] confusion with rsa_meth_st in a custom RSA engine In-Reply-To: References: <20170828014517.GA27863@openssl.org> <20170828173230.GA10656@openssl.org> Message-ID: <20170829001808.GA15690@openssl.org> On Mon, Aug 28, 2017, Brett R. Nicholas wrote: > > > One more follow up question: > > > > If possible you should set the public key components anyway: some operations > > > such as generating certificate requests require them to be present > > I'm confused what you mean here, since my engine doesn't "own" any instances of an RSA struct, it just has a static instance of RSA_METHOD struct defined. So therefore my engine never "sets" public or private key components. It just uses the modexp functions to write the public/private data (contained in the RSA struct passed as an argument from the higher level encrypt/decrypt functions) out to the hardware accelerator, and then return the result. So I could never "set the public key components anyway", as they would be set by whichever program calls RSA_public/private_encrypt/decrypt(). > > > Is my implementing it in this way different than how you thought I was implementing it? It made sense to me to do it this way, however please let me know if you think I'm going about it wrong, or if there are issues with this particular strategy. I want to make sure I'm using the engine API in the most intuitive and efficient way! > > Ah if you're performing crypto acceleration of already existing keys then that's fine. In some cases an ENGINE can load a private key (typically from an HSM) and returns the EVP_PKEY structure: in that case it would initialise the RSA structure for RSA keys. It's that case where (n, e) should be initialised if possible. Steve. -- Dr Stephen N. Henson. OpenSSL project core developer. Commercial tech support now available see: http://www.openssl.org From leonb at parsec.co.za Tue Aug 29 04:18:38 2017 From: leonb at parsec.co.za (Leon Brits) Date: Tue, 29 Aug 2017 04:18:38 +0000 Subject: [openssl-dev] Upgrading OpenSSL In-Reply-To: <20170828173602.GB10656@openssl.org> References: <13e455db80e548888c9cd15424bb146a@EXCHANGESRV.PARSEC.local> <20170828173602.GB10656@openssl.org> Message-ID: Hi Steve, > Have you recompiled the application? Some structures have changed between > OpenSSL 0.9.8 and 1.0.2 so you'll get problems with applications (or an > ENGINE) compiled against the wrong headers. In a build VM, both the TLS application and Racoon2 is compiled against freshly compiled OpenSSL 1.0.2L > If that isn't the problem then what is "md" set to? In the TLS application I only use SHA256, but in Racoon2 I can see it requesting SHA1 (in gdb). Thanks for your time LJB From leonb at parsec.co.za Tue Aug 29 07:17:59 2017 From: leonb at parsec.co.za (Leon Brits) Date: Tue, 29 Aug 2017 07:17:59 +0000 Subject: [openssl-dev] Upgrading OpenSSL In-Reply-To: References: <13e455db80e548888c9cd15424bb146a@EXCHANGESRV.PARSEC.local> <20170828173602.GB10656@openssl.org> Message-ID: <0c8075d5c8e54064a6faa28fcb623b06@EXCHANGESRV.PARSEC.local> Hi all, I've was able to get the private key from the HSM (added below). Testing it from the commandline shows: % openssl rsa -noout -check -in /etc/racoon2/Local/refB.pem RSA key ok Next I started from the default Racoon2 source code (20100526a) with NO patches. It now reads the private key from file. It however still fails with same error at "p_sign.c:123"!? Since I know this will be a RSA key I added code after where the file is read as EVP_PKEY (within ifdef): evp = PEM_read_PrivateKey(fp, NULL, NULL, NULL); #ifdef TEST RSA *rsa = EVP_PKEY_get1_RSA(evp); printf("\nRSA modulus: %d\n\n", RSA_size(rsa)); #endif The output is: "RSA modulus: 512" (WTF!) Output from CLI says modulus is: $openssl rsa -text -in /etc/racoon2/Local/refB.pem Private-Key: (4096 bit) : : I also read the file directly with PEM_read_RSAPrivateKey() and also got the modulus as 512. I also tested that the private key and public certificate matches by singing and verifying random data. So the private key must be correct - right? I've added a line to the Racoon2 init function to print the OpenSSL version and the output is: "OpenSSL version: 100020cf". OpenSSL and Racoon2 are both configured with specific parameters to point to "/usr/local/ssl" as the path to OpenSSL. Based on search results, I've added "OpenSSL_add_all_algorithms()" to the init code, but this did not make a difference. The other "add" and "load" stuff was there already. Thanks for all the help so far LJB -----BEGIN RSA PRIVATE KEY----- MIIJKAIBAAKCAgEA5B36X3g6AtbnELtkoKUfAmwAXfFsk8kH0U+vRcuRLQ0VIVHI FTZBIP4Ao68JPBh+fqS3eCtF5Ra1AxSTHH5+QkkN+c5j+iWbvlT55/gtrGz25dyW A+xJ1LyvBuLxgczgCTIdvegMQNlo8r/fm492tXI/X3pMJufdMD8ddpBPc+rD1MbX HcOYuWRdeDic0BepH1cvMnb9nCLz7lHv0j18NFYNYPeB0htu+jzP3EZndVtP3Zt3 C1gdjZf8j4SCI7gGtP/1kQ4lmemLCQhQG7sOgmqrNcX4Xk35sTEfn3ICE6p5vbsB 2hCCKpG8BaLS6jyoXecoXhtsbK4qLtexJdtJmDbKUm1dvFxr5qpes/pcQQR3KxIf 4woZmp24vT08WZHwCsmJd1WZffODBxCcEPn8ZdAABYUTsGlvjTEWnJCr+fJyUz6M K/J85+sCilvOKVWvTldyleFfe7RFNLRPxxx9KeoSszb4lWe3Ic33vn5W8PthEvkY ZwK36K4ripjksb0ROzTvFrhl4hgopnpMN5lu8eQExzktC+wyNbDHN3msy4UsQpou mZq9WfRbvfqOCs/z3hh/bO5efxXhJCTt/5y4fh0qBRjXVCLkZcnaJ0cwaCfF+v/B AHtJ8/Un3Aon7kFvVaaw7eMz6kC4y4mYMN6Zc+oYom9So+3wU1us5ezCr2UCAwEA AQKCAgAWXR4DqAy/5IOor0ZxPV7p5N6wVw/W+Tlr+szrIcbszMCKkLL16Wi/LEex xGI6YwhbsBhQjqV+jRhX4fd+LqSAKRtWpzZD+tXm9fu+EyWfJtuZs2N9pPekBI+A NFWK7SP1otUpngs9fFL+oXLxlEIGDdpKqPm4Qrj7luRzkDhJ2/Uw8gF66Icbfcxw EdTFzdwFt41n/CRm30Jc1emWCfMYSmtrWzh9+bSKkdWq+bgA2F/fEPO9x/i1vfXD uDDuAYeezQ0tGF71VOENlKQN4+vLO1vnnK35yNk21uMLxaiQdwESabgHweCQ/dKa Ftlon2O+OQBLIZMioHyANSMn2/S96RT6qiiJrf8leIsRRVmvNnSGC9ScPfGqZHHc /Rh7zifRIDaoRi1/AB9GLoF01musmvULmv0AW2JOomm7Rxp71kED9pKJN/nrZkLT mftAPBm/4pmrVt28bLQE0sn8JgtSaeP4Z5Y1VV3UHKit8rgY2AdAOCVeHYDeCZwy n4OGeed6GxJIK0NHa9qTlhznPVvK5WB+vzNIqpd4eQPZ3ETX3fXp0CgEymyGQZIY +rXkkR7XHPLcey7iEEdhmUdxXLMffuzvj5s1oOL7JY0rRLG6QoLhwNQX/kmmncZw J24zrGz3KOfQcw1KKTuu/g8iwna3b/PYT15gIhlx1YvdvH34QQKCAQEA9kkoaQGR fC56Uy9X+HrafDBj+JZc5+qM+9NVx9yigc9AS+Hw31eM9g2+ZotairL++X3ExSTP 0zJZQG5Af/oueX7PO7JpYJywMErymKTEjRnSa31RWgNy0SENWztGnshVjoBkC18U fiTKrp/mEC3LEnjWG47NP7XnP6kwlZJf5vXyN0ystJXx4CUUHdKiFZ/+Ji8918jP MADLHIqx6C8wF1nw3rsw7xbfj8hdL2uXc+ccTHaITEM49zf1NY6Y5FlEADmK8PHl nW+KLz4p1FePP3+7j0vgmPPRAzH+26c+2SkBVLG9tZ4nRQFFoXu3DUZ/yR/kcj6d 6iF7WLg4IZ0M1QKCAQEA7R1dNihL7OrY2V/6F9YkWGThTaChfAlY2qoEzjU+tSN9 Oz4T4ErmNRwZne1bZS+KrDVAT2NyuHqP/d8UCH7+4dPUrCtUa2uocR2ZkCB/8YE0 MBwUGLVvEg9p5UzN8oLi5HYgplXuQyKjNKsOXRsFT+asiIaufGeB6SQH3YsO9d1J cnFQ+I7syXIpgNVU4Hwva/A6OMn3F5ctHqSPcefoDrjJvZvuxNv2Q+OkfygOwIVl fG0+cM33TRoHBkgRFsiluxB+HmHvnPruyacnVJr4iMMHoP55o2WFLzLyviCdxWfG esI2UcYp//yGRomJfhH4ryo352AlZK+6XiGxFhsgUQKCAQEAoU+HfUd+r9SNYGC5 ANIOuprDT/XEQk55KxPfsnlLozRIy6rgJpjkheC4ndeWZRQaNvVuZSXYTV9D4BSW gHwv5nntaA9SP+pA6FCeluYMqtKH5Ml++DMcB+fbJK8tcSCMETr8zAnplTmp2zh1 6pDj2mR08DXXL2yIW0UIwl7OG6tpi1jYIau4cfQ3OdVVPk69musEWYp4UdujLT2W ixolKJZLUsdOpLrZkQZOKpoQ5+Quv9I/MZwx+pKXNA4DDEV/aZpG68u0diqxWoIf yanT9GZIRfIP2H6RnsMnR11Cp9/YtE16kTNWSzcrETrDyZksZ2JRMZbbvZdSf+ih Mzr3RQKCAQAoDEabCBFS0ZOQm1pFPmDjIR7AmrnLllSQkhi/r1SJCZZ0NBbtUhkx rS5Scy11mKMGVPQotHZC3EiHq27ZxIFOKRYJgkP+5K8Nc99k7WhOpPDok0V9qt84 oKvVE7TRGfQyfBUlouvcIPSJx75kUVUDWsWXRZAg6OaTNwexDnSXaCmoj6UKZjFu EL8byGxOggsMYwWiY9I9BMaVH1wu8+lI20MPqE+apkAg1UkKRPzA3Yb3jgf2y4LS XewDzpY8T+VWBAIZJZdv3x4HpxjIfGgvySj3syNeSp5FC4jePeoH1nA6eaTtCNg9 KSyQq1gyl9x10V6h0KZgLCIBFhWv0yMRAoIBAE9i9Llefme5ZHDcXiJa7x7d11Et KpHkMfrYwmCCPsZdijwAyN4oBfcfDbfASzSTtcK13qnzwq34xLSe7JgefbR0prws PsFEx1kSZHxPuCl3/L7cZSTJ3ZrNcdHtcqiGHX4zR3AJ8Vy8dB/w9ip3IDE6+/8L dujQ2kWHOYAKoQ7+pV8qzfBSVs07COFm73bz7omasZaHPsGH9cmTbgGZSJiUAo36 jAK27hnXSQzqD2Jtmr95wAVJk2PYxtgsCbGFrE1STj9ZHp1OEoZqAdnlOKQEz1S0 ZfzdAMnD2ntpROVrvEEuLT68oePoXgwlG+1C4njPlpwCb0WlHfCM60RcllU= -----END RSA PRIVATE KEY----- From leonb at parsec.co.za Tue Aug 29 08:02:23 2017 From: leonb at parsec.co.za (Leon Brits) Date: Tue, 29 Aug 2017 08:02:23 +0000 Subject: [openssl-dev] Upgrading OpenSSL In-Reply-To: <0c8075d5c8e54064a6faa28fcb623b06@EXCHANGESRV.PARSEC.local> References: <13e455db80e548888c9cd15424bb146a@EXCHANGESRV.PARSEC.local> <20170828173602.GB10656@openssl.org> <0c8075d5c8e54064a6faa28fcb623b06@EXCHANGESRV.PARSEC.local> Message-ID: <8440d5398c9249cf9469d90385a2d716@EXCHANGESRV.PARSEC.local> Hi all, I need to correct my WTF comment - RTFM !!!! RSA_size return bytes. Sorry LJB > evp = PEM_read_PrivateKey(fp, NULL, NULL, NULL); #ifdef TEST RSA *rsa = > EVP_PKEY_get1_RSA(evp); printf("\nRSA modulus: %d\n\n", RSA_size(rsa)); > #endif > > The output is: "RSA modulus: 512" (WTF!) From Matthias.St.Pierre at ncp-e.com Tue Aug 29 09:45:26 2017 From: Matthias.St.Pierre at ncp-e.com (Dr. Matthias St. Pierre) Date: Tue, 29 Aug 2017 09:45:26 +0000 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API Message-ID: Hi everybody, on the [openssl-dev] mailing list, there has been a long ongoing discussion about the new RAND_DRBG API and comparing it with the old RAND_METHOD API (see "[openssl-dev] Work on a new RNG for OpenSSL"). Two of the most controversal questions were: - Do we really need a new RNG API? Should the RAND_DRBG API be made public or kept private? (Currently, it's exported from libcrypto but only used internally by libssl.) - How much control should the user (programmer) be given over the reseeding process and/or should he be allowed to add his own additional randomness? Many developers seem to be realizing the interesting possibilities of the DRBG API and are asking for public access to this new and promising API. One of the driving forces behind it is the question about how to do seeding and reseeding right. Among others, Uri Blumenthal asked for making the DRBG API public. Currently, the OpenSSL core members seem to be reluctant to make the API public, at least at this early stage. I understand Rich Salz's viewpoint that this requires a thorough discussion, because a public interface can't be easily changed and wrong decisions in the early phase can become a heavy burdon. Nevertheless, I agree with Uri Blumenthal that the DRBG API should be made public. So here comes my ====================================== Plea for a new public OpenSSL RNG API: ====================================== The new RAND_DRBG is the superior API. It shouldn't be kept private and hidden behind the ancient RAND_METHOD API. The philosophy of the two APIs is not very well compatible, in particular when it comes to reseeding and adding additional unpredictable input. Hiding the RAND_DRBG behind the RAND_METHOD API only causes problems. Also, it will force people to patch their OpenSSL copy if they want to use the superior API. The RAND_DRBG API should become the new public OpenSSL RNG API and the old RAND_METHOD API should be deprecated in the long run. This transition does not need to be rushed, but it would be good if there would be early consent on the road map. I am thinking of a smooth transition with a phase of coexistence and a compatibility layer mapping the default RAND_METHOD to the default public RAND_DRBG instance. (This compatibility layer already exists, it's the 'RAND_OpenSSL()' method.) Historical Background ===================== As Rich already mentioned in his blog post, the RAND_DRBG isn't new. It's been a part of OpenSSL for a long time, hidden in the FIPS 2.0 Object Module. I have been working with the FIPS DRBG for quite a while now, using a FIPS-capable OpenSSL 1.0.2x crypto library. The reason why our company switched to the FIPS DRBG is that one of our products runs on a small hardware device which does not have a reliable entropy source, but the product has to meet high security standards, in particular w.r.t. its RNG. So we decided to use the SmartCard RNG as primary entropy source for a deterministic AES-CTR based RNG and use /dev/urandom as additional input. Reseeding should occur on every generate request. Using the FIPS DRBG, these requirements were easily met, because the API gives such a fine grained control over reseeding and adding additional entropy. The DRBG was well documented, its design in NIST SP800-90A (now: NIST SP800-90Ar1) and its API in the OpenSSL FIPS 2.0 User Guide. The implementation was thoroughly tested and assessed during the FIPS certification process. So the only minor obstacle was that we had to patch the crypto library (not the FIPS object module) in order to get public access to the FIPS_drbg_*() methods. I always considered the DRBG API more mature than the good old RAND_METHOD API and I wondered, why the DRBG code lay forgotten for so many years in the FIPS 2.0 object module sources and was never ported to master. When in June of this year the thread "[openssl-dev] Work on a new RNG for OpenSSL" popped up (https://mta.openssl.org/pipermail/openssl-dev/2017-June), I closely watched the discussion, and when John Denker suggested having a look at NIST SP800-90A, I was electrified: > Constructive suggestion: If you want to see what a RNG looks > like when designed by cryptographers, take a look at: > Elaine Barker and John Kelsey, > "Recommendation for Random Number Generation Using Deterministic Random Bit Generators" > http://csrc.nist.gov/publications/nistpubs/800-90A/SP800-90A.pdf > > That design may look complicated, but if you think you can > leave out some of the blocks in their diagram, proceed with > caution. Every one of those blocks is there for a reason. >From his mail and the reaction to it, I had the impression that nobody seemed to remember the fact that the DRBG code was already present in the FIPS object module. The hidden treasure seemed forgotten! When I seized the opportunity and proposed to port the FIPS DRBG code into master, Rich Salz liked my idea and immediately started working on PR #3789: I am very grateful to Rich for picking up the idea so fast and and giving it so much speed and momentum. He has done a lot of work to convince others and defend the idea against initial scepticism by voices objecting to the seemingly new and unknown API. In the middle of August, when the first bulk of work was finished, Rich wrote a detailed blog post to advertise the new DRBG and explain his work and the discussion restarted: "[openssl-dev] Work on a new RNG for OpenSSL> (see https://mta.openssl.org/pipermail/openssl-dev/2017-August) It became quickly evident that users that the DRBG API had promising features, so they started asking for public access to this new API. The driving force was the question about how to do seeding and reseeding right. Among others, Uri Blumenthal was a dedicated advocate to make the DRBG API public (https://mta.openssl.org/pipermail/openssl-dev/2017-August/009594.html). But the OpenSSL members currently seem to be reluctant to make the API public right away. I understand Rich's viewpoint that this decision requires a thorough discussion, because a public interface can't be easily changed and wrong decisions in the early phase can become a heavy burdon. Nevertheless, I agree with Uri Blumenthal that the DRBG API should be made public. Here are some of my arguments for it. The DRBG API supports multiple instances and chaining ===================================================== The NIST DRBG standard had chaining of multiple DRBG instances in the mind from the very beginnning, see for example footnote [4] on page 25 of NIST SP800-90Ar1: > Entropy input may be obtained from an entropy source or an NRBG, both of which provide fresh entropy. > Entropy input could also be obtained from a DRBG that has access to an entropy source or NRBG. The original OpenSSL FIPS DRBG implementation did not support chaining, but this support has been added by Rich during the DRBG port. The DRBG API has a highly flexible concept for seeding and reseeding ==================================================================== As mentioned previously, the DRBG has a callback mechanism with callbacks like get_entropy() and get_adin() callbacks, which make it easy to fine tune the default instantiation process by either adding additional randomness input, or changing the entropy source entirely. The callbacks are primarily intended for obtaining randomness instantiation and reseeing. There is a clear concept for reseeding, which can be adjusted by changing the reseed_interval: Normally, the DRBG reseeds itself automatically, whenever the reseed_interval has been reached. In addition, there is a way for the DRBG consumer to add his own unpredictability when requesting random bytes: by adding additional input 'adin' to the RAND_DRBG_generate() call: int RAND_DRBG_generate(RAND_DRBG *drbg, unsigned char *out, size_t outlen, int prediction_resistance, const unsigned char *adin, size_t adinlen) So why are there so many ways to add randomness and additional input? And what is the difference? * The get_entropy() and get_adin() callbacks are used by the DRBG itself to _pull_ unpredictable data from some backend entropy source (which can also be a chained DRBG which is connected to some entropy source) during instantiation or reseeding. * The 'adin' argument can be used by the DRBG consumer to _push_ unpredictable input into the DRBG when generating random output. Note that the DRBG consumer has no possibility to push out-of-band randomness into the DRBG. Adding randomness is always coupled with a generate or (re-)seed operation. This is a very import difference between the RAND_DRBG and the RAND_METHOD and one of the reasons why it's so hard to do reseeding right in both the RAND and RAND_DRBG API simultaneously. In fact, currently The 'RAND_add()/RAND_bytes()' pattern is broken =============================================== In OpenSSL, the classical way for the RNG consumer to add his own randomness is to call 'RAND_add()' before calling 'RAND_bytes()'. If the new 'RAND_OpenSSL()' method (the "compatibility layer" hiding the public RAND_DRBG instance) is the default, then this does not work as expected anymore: The reason is that a call to 'RAND_add()' adds the provided randomness only to a global buffer ('rand_bytes'), from which it will be pulled during the next reseed. But no reseed is triggered. So the next RAND_bytes() call will be unaffected from the RAND_add(), which is not what the consumer expected. (The same holds for 'RAND_seed()', since 'drbg_seed()' only calls into 'drbg_add()') Reseeding of DRBGs occurs only at the following occasions: * immediately after a 'fork()' (new) * if the 'reseed_counter' exceeds the 'reseed_interval' * if 'RAND_DRBG_generate()' is called requesting 'prediction_resistance' * 'RAND_DRBG_reseed()' is called explicitely *Note:* Currently it looks like the situation is even worse: if 'RAND_add()' is called multiple times before a reseed occurs, then the result of the previous call is overwritten. Reseeding the 'DRBG' whenever the user calls 'RAND_add()' does not seem a good solution. It would be too expensive, in particular if system entropy is pulled for reseeding. Of course it is possible to fix this issue, but the DRBG provides for a much simpler solution: it lets the consumer contribute to the entropy of the internal state by providing additional input. If the user input contains entropy, that's fine, if it's "snake oil", no harm. The additional input is mixed into the internal state in just the same way as the entropy buffer using the 'ctr_df()' derivation function. One might think of the 'entropy' input as trusted randomness and 'adin' as untrusted randomness. For this reason, I would like to see the 'RAND_add()/RAND_bytes()' pattern deprecated and the 'RAND_DRBG_generate() with additional input' pattern advertised instead. The DRBG API supports different implementations =============================================== Well, it _supported_ them, until recently. But that's not irreversible. The DRBG concept, as layed out in the NIST standard, provides a generic framework for deterministic RNGs (the acronym DRBG stands for Deterministic Random Bit Generator). It deals with general questions like how to instantiate and reseed the RNG, where does it get the entropy from, etc. The standard proposes three concrete implementations, Hash_DRBG, HMAC_DRBG, and CTR_DRBG. In the FIPS code, all three were implemented, and the genericity was achieved using a data union and a set of five function pointers, reminiscent of a vtable in C++: struct drbg_ctx_st { ... /* Implementation specific structures */ union { DRBG_HASH_CTX hash; DRBG_HMAC_CTX hmac; DRBG_CTR_CTX ctr; } d; /* Initialiase PRNG and setup callbacks below */ int (*init)(DRBG_CTX *ctx, int nid, int security, unsigned int flags); /* Intantiate PRNG */ int (*instantiate)(DRBG_CTX *ctx, const unsigned char *ent, size_t entlen, const unsigned char *nonce, size_t noncelen, const unsigned char *pers, size_t perslen); /* reseed */ int (*reseed)(DRBG_CTX *ctx, const unsigned char *ent, size_t entlen, const unsigned char *adin, size_t adinlen); /* generat output */ int (*generate)(DRBG_CTX *ctx, unsigned char *out, size_t outlen, const unsigned char *adin, size_t adinlen); /* uninstantiate */ int (*uninstantiate)(DRBG_CTX *ctx); ... }; This part of the code was removed during the DRBG port, because currently CTR_DRBG is the only impementation. I would like to suggest to restore this 'polymorphic' implementation, to ease adding new implementations CHACHA20_DRBG in the future The DRBG API is well documented and tested ========================================== The entire DRBG API is part of the OpenSSL FIPS 2.0 Module and as such is well tested and well documented. Most of the documentation for the manual pages still-to-be-written can be taken from the FIPS User Guide and converted into manual pages, starting with textual modifications like FIPS_drbg_xxx(...) -> RAND_DRBG_xxx(...) DRBG_CTX *dctx -> RAND_DRBG *dctx and taking the new typedefs into consideration. Here is for example a comparison of the API function to install the callbacks: FIPS DRBG: int FIPS_drbg_set_callbacks(DRBG_CTX *dctx, size_t (*get_entropy)(DRBG_CTX *ctx, ), void (*cleanup_entropy)(DRBG_CTX *ctx, ), size_t entropy_blocklen, size_t (*get_nonce)(DRBG_CTX *ctx, ), void (*cleanup_nonce)(DRBG_CTX *ctx, ) ); RAND_DRBG: typedef size_t (*RAND_DRBG_get_entropy_fn)(RAND_DRBG *ctx, ); typedef void (*RAND_DRBG_cleanup_entropy_fn)(RAND_DRBG *ctx, ); typedef size_t (*RAND_DRBG_get_nonce_fn)(RAND_DRBG *ctx, ); typedef void (*RAND_DRBG_cleanup_nonce_fn)(RAND_DRBG *ctx, ); int RAND_DRBG_set_callbacks(RAND_DRBG *dctx, RAND_DRBG_get_entropy_fn get_entropy, RAND_DRBG_cleanup_entropy_fn cleanup_entropy, RAND_DRBG_get_nonce_fn get_nonce, RAND_DRBG_cleanup_nonce_fn cleanup_nonce); Conclusion ========== I see no reason why the RAND_DRBG shouldn't be made public as soon as possible, keeping the API as close as possible to the original FIPS DRBG API (FIPS 3.0 is upcoming!). In a second step, the current compatibility binding from RAND_METHOD to RAND_DRBG could be deprecated and faded out smoothly. Looking forward to receiving your comments. (But please be patient with me, I'm currently on physical rehab after a surgery.) Matthias St. Pierre -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 4328 bytes Desc: not available URL: From matt at openssl.org Tue Aug 29 10:17:08 2017 From: matt at openssl.org (Matt Caswell) Date: Tue, 29 Aug 2017 11:17:08 +0100 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: References: Message-ID: <2f79c2b6-560f-b245-b39d-dc4147d5ab44@openssl.org> On 29/08/17 10:45, Dr. Matthias St. Pierre wrote > Currently, the OpenSSL core members seem to be reluctant to make the > API public, at least at this early stage. I understand Rich Salz's > viewpoint that this requires a thorough discussion, because a public > interface can't be easily changed and wrong decisions in the early > phase can become a heavy burdon. FWIW, I am not against making the API public although I too understand Rich's reluctance. However it seems inevitable to me that sooner or later this API will have to be made accessible and I see little benefit in delaying that and plenty of reasons for doing it now (it only frustrates those that know the functionality is there but can't get at it :-)). There is always a risk with every API added that you get it wrong, and then you are stuck with it for ever more. The only real answers to this are: 1) Make sure you "design" your API to start with and get it reviewed 2) Encourage people to try the API while it is still in dev i.e. so we can still change it before the final release. I'm not really convinced that keeping the API hidden now will mean that it will be better in some future release. It just delays the point at which we have the realisation of "if only we had designed it like *this* it would have been soooo much better". Better is to make it public in dev asap. If there are problems with it we can always revert it before it makes it into a release. > The 'RAND_add()/RAND_bytes()' pattern is broken > =============================================== > > In OpenSSL, the classical way for the RNG consumer to add his own > randomness is to call 'RAND_add()' before calling 'RAND_bytes()'. If > the new 'RAND_OpenSSL()' method (the "compatibility layer" hiding the > public RAND_DRBG instance) is the default, then this does not work > as expected anymore: > > The reason is that a call to 'RAND_add()' adds the provided > randomness only to a global buffer ('rand_bytes'), from which it will > be pulled during the next reseed. But no reseed is triggered. So the > next RAND_bytes() call will be unaffected from the RAND_add(), which > is not what the consumer expected. (The same holds for 'RAND_seed()', > since 'drbg_seed()' only calls into 'drbg_add()') > > Reseeding of DRBGs occurs only at the following occasions: > > * immediately after a 'fork()' (new) * if the 'reseed_counter' > exceeds the 'reseed_interval' * if 'RAND_DRBG_generate()' is called > requesting 'prediction_resistance' * 'RAND_DRBG_reseed()' is called > explicitely > > *Note:* Currently it looks like the situation is even worse: if > 'RAND_add()' is called multiple times before a reseed occurs, then > the result of the previous call is overwritten. > > Reseeding the 'DRBG' whenever the user calls 'RAND_add()' does not > seem a good solution. It would be too expensive, in particular if > system entropy is pulled for reseeding. Of course it is possible to > fix this issue, but the DRBG provides for a much simpler solution: it > lets the consumer contribute to the entropy of the internal state by > providing additional input. If the user input contains entropy, > that's fine, if it's "snake oil", no harm. The additional input is > mixed into the internal state in just the same way as the entropy > buffer using the 'ctr_df()' derivation function. One might think of > the 'entropy' input as trusted randomness and 'adin' as untrusted > randomness. > > For this reason, I would like to see the 'RAND_add()/RAND_bytes()' > pattern deprecated and the 'RAND_DRBG_generate() with additional > input' pattern advertised instead. Deprecation itself would probably have to wait for a 1.2.0 release. We have a binary/source compatibility guarantee for 1.1.0 which would not be met for builds using "--api" or "no-deprecated". Is there a potential security vulnerability here? Applications using the "old" APIs expect RAND_add() to behave in a particular way. If we have silently changed this behaviour in 1.1.1 are they exposed? Matt From Matthias.St.Pierre at ncp-e.com Tue Aug 29 11:31:03 2017 From: Matthias.St.Pierre at ncp-e.com (Dr. Matthias St. Pierre) Date: Tue, 29 Aug 2017 11:31:03 +0000 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: <2f79c2b6-560f-b245-b39d-dc4147d5ab44@openssl.org> References: <2f79c2b6-560f-b245-b39d-dc4147d5ab44@openssl.org> Message-ID: <7b6ea5fcc69c4a65b3e98bf8620906c8@Ex13.ncp.local> > -----Urspr?ngliche Nachricht----- > Von: openssl-dev [mailto:openssl-dev-bounces at openssl.org] Im Auftrag von Matt Caswell > Gesendet: Dienstag, 29. August 2017 12:17 > An: openssl-dev at openssl.org > Betreff: Re: [openssl-dev] Plea for a new public OpenSSL RNG API > > > On 29/08/17 10:45, Dr. Matthias St. Pierre wrote > > ... > > The 'RAND_add()/RAND_bytes()' pattern is broken > > =============================================== > > > > In OpenSSL, the classical way for the RNG consumer to add his own > > randomness is to call 'RAND_add()' before calling 'RAND_bytes()'. If > > the new 'RAND_OpenSSL()' method (the "compatibility layer" hiding the > > public RAND_DRBG instance) is the default, then this does not work > > as expected anymore: > > ... > > Is there a potential security vulnerability here? Applications using the > "old" APIs expect RAND_add() to behave in a particular way. If we have > silently changed this behaviour in 1.1.1 are they exposed? Don't worry, this issue is new, the global 'rand_bytes' buffer has only been introduced by the DRBG port to master in August. I don't think it's a big deal to fix it. The reason I mentioned it here was to emphasize, that it is really hard to get the different philosophies (push vs. pull) of the two APIs working together correctly. The code was reviewed by several people and nobody noticed it. By the way: the approach using the fixed size global 'rand_bytes' buffer has another issue, which I will try to write down on GitHub within the next days. Matthias -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 4328 bytes Desc: not available URL: From levitte at openssl.org Tue Aug 29 11:42:29 2017 From: levitte at openssl.org (Richard Levitte) Date: Tue, 29 Aug 2017 13:42:29 +0200 (CEST) Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: References: Message-ID: <20170829.134229.1463977663045816733.levitte@openssl.org> I'm late in the game, having only followed the development very superficially... If I understand correctly, the RAND_DRBG API is really a completely separate API that has nothing to do with the RAND_METHOD API pers se, i.e. any association between the two is more or less "accidental"? Frankly, I cannot see anything wrong with making that a public API. I wonder, though, if this is going to be an implementation that's kinda sorta built-in only, or if other implementations of the basic stuff will be possible... or in other words, will we have a "method" structure like we have with so many other APIs? In saying this, I'm counting in the possibility that some implementations could come in the form of engines, is this something that's been thought about? I do notive the DRGB_CTX structure, but that doesn't quite follow the usual "method" pattern we have... Cheers, Richard In message on Tue, 29 Aug 2017 09:45:26 +0000, "Dr. Matthias St. Pierre" said: Matthias.St.Pierre> Hi everybody, Matthias.St.Pierre> Matthias.St.Pierre> on the [openssl-dev] mailing list, there has been a long ongoing discussion about the new RAND_DRBG API and comparing it with the old RAND_METHOD API (see "[openssl-dev] Work on a new RNG for OpenSSL"). Two of the most controversal questions were: Matthias.St.Pierre> Matthias.St.Pierre> - Do we really need a new RNG API? Should the RAND_DRBG API be made public or kept private? (Currently, it's exported from libcrypto but only used internally by libssl.) Matthias.St.Pierre> - How much control should the user (programmer) be given over the reseeding process and/or should he be allowed to add his own additional randomness? Matthias.St.Pierre> Matthias.St.Pierre> Many developers seem to be realizing the interesting possibilities of the DRBG API and are asking for public access to this new and promising API. One of the driving forces behind it is the question about how to do seeding and reseeding right. Among others, Uri Blumenthal asked for making the DRBG API public. Matthias.St.Pierre> Matthias.St.Pierre> Currently, the OpenSSL core members seem to be reluctant to make the API public, at least at this early stage. I understand Rich Salz's viewpoint that this requires a thorough discussion, because a public interface can't be easily changed and wrong decisions in the early phase can become a heavy burdon. Matthias.St.Pierre> Matthias.St.Pierre> Nevertheless, I agree with Uri Blumenthal that the DRBG API should be made public. So here comes my Matthias.St.Pierre> Matthias.St.Pierre> ====================================== Matthias.St.Pierre> Plea for a new public OpenSSL RNG API: Matthias.St.Pierre> ====================================== Matthias.St.Pierre> Matthias.St.Pierre> The new RAND_DRBG is the superior API. It shouldn't be kept private and hidden behind the ancient RAND_METHOD API. Matthias.St.Pierre> The philosophy of the two APIs is not very well compatible, in particular when it comes to reseeding and adding Matthias.St.Pierre> additional unpredictable input. Hiding the RAND_DRBG behind the RAND_METHOD API only causes problems. Matthias.St.Pierre> Also, it will force people to patch their OpenSSL copy if they want to use the superior API. Matthias.St.Pierre> Matthias.St.Pierre> The RAND_DRBG API should become the new public OpenSSL RNG API and the old RAND_METHOD API should be deprecated Matthias.St.Pierre> in the long run. This transition does not need to be rushed, but it would be good if there would be early consent Matthias.St.Pierre> on the road map. I am thinking of a smooth transition with a phase of coexistence and a compatibility layer Matthias.St.Pierre> mapping the default RAND_METHOD to the default public RAND_DRBG instance. (This compatibility layer already exists, Matthias.St.Pierre> it's the 'RAND_OpenSSL()' method.) Matthias.St.Pierre> Matthias.St.Pierre> Matthias.St.Pierre> Matthias.St.Pierre> Historical Background Matthias.St.Pierre> ===================== Matthias.St.Pierre> Matthias.St.Pierre> As Rich already mentioned in his blog post, the RAND_DRBG isn't new. It's been a part of OpenSSL for a long time, hidden in the FIPS 2.0 Object Module. Matthias.St.Pierre> Matthias.St.Pierre> I have been working with the FIPS DRBG for quite a while now, using a FIPS-capable OpenSSL 1.0.2x crypto library. The reason why our company switched to the FIPS DRBG is that one of our products runs on a small hardware device which does not have a reliable entropy source, but the product has to meet high security standards, in particular w.r.t. its RNG. So we decided to use the SmartCard RNG as primary entropy source for a deterministic AES-CTR based RNG and use /dev/urandom as additional input. Reseeding should occur on every generate request. Using the FIPS DRBG, these requirements were easily met, because the API gives such a fine grained control over reseeding and adding additional entropy. Matthias.St.Pierre> Matthias.St.Pierre> The DRBG was well documented, its design in NIST SP800-90A (now: NIST SP800-90Ar1) and its API in the OpenSSL FIPS 2.0 User Guide. The implementation was thoroughly tested and assessed during the FIPS certification process. So the only minor obstacle was that we had to patch the crypto library (not the FIPS object module) in order to get public access to the FIPS_drbg_*() methods. Matthias.St.Pierre> Matthias.St.Pierre> I always considered the DRBG API more mature than the good old RAND_METHOD API and I wondered, why the DRBG code lay forgotten for so many years in the FIPS 2.0 object module sources and was never ported to master. Matthias.St.Pierre> Matthias.St.Pierre> When in June of this year the thread "[openssl-dev] Work on a new RNG for OpenSSL" popped up (https://mta.openssl.org/pipermail/openssl-dev/2017-June), I closely watched the discussion, and when John Denker suggested having a look at NIST SP800-90A, I was electrified: Matthias.St.Pierre> Matthias.St.Pierre> > Constructive suggestion: If you want to see what a RNG looks Matthias.St.Pierre> > like when designed by cryptographers, take a look at: Matthias.St.Pierre> > Elaine Barker and John Kelsey, > "Recommendation for Random Number Generation Using Deterministic Random Bit Generators" Matthias.St.Pierre> > http://csrc.nist.gov/publications/nistpubs/800-90A/SP800-90A.pdf Matthias.St.Pierre> > Matthias.St.Pierre> > That design may look complicated, but if you think you can Matthias.St.Pierre> > leave out some of the blocks in their diagram, proceed with Matthias.St.Pierre> > caution. Every one of those blocks is there for a reason. Matthias.St.Pierre> Matthias.St.Pierre> Matthias.St.Pierre> Matthias.St.Pierre> From his mail and the reaction to it, I had the impression that nobody seemed to remember the fact that the DRBG code was already present in the FIPS object module. The hidden treasure seemed forgotten! When I seized the opportunity and proposed to port the FIPS DRBG code into master, Rich Salz liked my idea and immediately started working on PR #3789: Matthias.St.Pierre> Matthias.St.Pierre> Matthias.St.Pierre> Matthias.St.Pierre> Matthias.St.Pierre> Matthias.St.Pierre> I am very grateful to Rich for picking up the idea so fast and and giving it so much speed and momentum. He has done a lot of work to convince others and defend the idea against initial scepticism by voices objecting to the seemingly new and unknown API. In the middle of August, when the first bulk of work was finished, Rich wrote a detailed blog post to advertise the new DRBG and explain his work and the discussion restarted: Matthias.St.Pierre> Matthias.St.Pierre> "[openssl-dev] Work on a new RNG for OpenSSL> (see https://mta.openssl.org/pipermail/openssl-dev/2017-August) Matthias.St.Pierre> Matthias.St.Pierre> It became quickly evident that users that the DRBG API had promising features, so they started asking for public access to this new API. The driving force was the question about how to do seeding and reseeding right. Among others, Uri Blumenthal was a dedicated advocate to make the DRBG API public (https://mta.openssl.org/pipermail/openssl-dev/2017-August/009594.html). Matthias.St.Pierre> Matthias.St.Pierre> But the OpenSSL members currently seem to be reluctant to make the API public right away. I understand Rich's viewpoint that this decision requires a thorough discussion, because a public interface can't be easily changed and wrong decisions in the early phase can become a heavy burdon. Matthias.St.Pierre> Matthias.St.Pierre> Matthias.St.Pierre> Nevertheless, I agree with Uri Blumenthal that the DRBG API should be made public. Here are some of my arguments for it. Matthias.St.Pierre> Matthias.St.Pierre> Matthias.St.Pierre> The DRBG API supports multiple instances and chaining Matthias.St.Pierre> ===================================================== Matthias.St.Pierre> Matthias.St.Pierre> The NIST DRBG standard had chaining of multiple DRBG instances in the mind from the very beginnning, see for example footnote [4] on page 25 of NIST SP800-90Ar1: Matthias.St.Pierre> Matthias.St.Pierre> > Entropy input may be obtained from an entropy source or an NRBG, both of which provide fresh entropy. Matthias.St.Pierre> > Entropy input could also be obtained from a DRBG that has access to an entropy source or NRBG. Matthias.St.Pierre> Matthias.St.Pierre> The original OpenSSL FIPS DRBG implementation did not support chaining, but this support has been added by Rich during the DRBG port. Matthias.St.Pierre> Matthias.St.Pierre> Matthias.St.Pierre> Matthias.St.Pierre> The DRBG API has a highly flexible concept for seeding and reseeding Matthias.St.Pierre> ==================================================================== Matthias.St.Pierre> Matthias.St.Pierre> As mentioned previously, the DRBG has a callback mechanism with callbacks like get_entropy() and get_adin() callbacks, which make it easy to fine tune the default instantiation process by either adding additional randomness input, or changing the entropy source entirely. The callbacks are primarily intended for obtaining randomness instantiation and reseeing. There is a clear concept for reseeding, which can be adjusted by changing the reseed_interval: Normally, the DRBG reseeds itself automatically, whenever the reseed_interval has been reached. Matthias.St.Pierre> Matthias.St.Pierre> In addition, there is a way for the DRBG consumer to add his own unpredictability when requesting random bytes: by adding additional input 'adin' to the RAND_DRBG_generate() call: Matthias.St.Pierre> Matthias.St.Pierre> int RAND_DRBG_generate(RAND_DRBG *drbg, unsigned char *out, size_t outlen, Matthias.St.Pierre> int prediction_resistance, Matthias.St.Pierre> const unsigned char *adin, size_t adinlen) Matthias.St.Pierre> Matthias.St.Pierre> So why are there so many ways to add randomness and additional input? And what is the difference? Matthias.St.Pierre> Matthias.St.Pierre> * The get_entropy() and get_adin() callbacks are used by the DRBG itself to _pull_ unpredictable data from some backend entropy source (which can also be a chained DRBG which is connected to some entropy source) during instantiation or reseeding. Matthias.St.Pierre> * The 'adin' argument can be used by the DRBG consumer to _push_ unpredictable input into the DRBG when generating random output. Matthias.St.Pierre> Matthias.St.Pierre> Matthias.St.Pierre> Note that the DRBG consumer has no possibility to push out-of-band randomness into the DRBG. Adding randomness is always coupled with a generate or (re-)seed operation. This is a very import difference between the RAND_DRBG and the RAND_METHOD and one of the reasons why it's so hard to do reseeding right in both the RAND and RAND_DRBG API simultaneously. In fact, currently Matthias.St.Pierre> Matthias.St.Pierre> Matthias.St.Pierre> The 'RAND_add()/RAND_bytes()' pattern is broken Matthias.St.Pierre> =============================================== Matthias.St.Pierre> Matthias.St.Pierre> In OpenSSL, the classical way for the RNG consumer to add his own randomness is to call 'RAND_add()' before calling 'RAND_bytes()'. If the new 'RAND_OpenSSL()' method (the "compatibility layer" hiding the public RAND_DRBG instance) is the default, then this does not work as expected anymore: Matthias.St.Pierre> Matthias.St.Pierre> The reason is that a call to 'RAND_add()' adds the provided randomness only to a global buffer ('rand_bytes'), from which it will be pulled during the next reseed. But no reseed is triggered. So the next RAND_bytes() call will be unaffected from the RAND_add(), which is not what the consumer expected. (The same holds for 'RAND_seed()', since 'drbg_seed()' only calls into 'drbg_add()') Matthias.St.Pierre> Matthias.St.Pierre> Reseeding of DRBGs occurs only at the following occasions: Matthias.St.Pierre> Matthias.St.Pierre> * immediately after a 'fork()' (new) Matthias.St.Pierre> * if the 'reseed_counter' exceeds the 'reseed_interval' Matthias.St.Pierre> * if 'RAND_DRBG_generate()' is called requesting 'prediction_resistance' Matthias.St.Pierre> * 'RAND_DRBG_reseed()' is called explicitely Matthias.St.Pierre> Matthias.St.Pierre> *Note:* Currently it looks like the situation is even worse: if 'RAND_add()' is called multiple times before a reseed occurs, then the result of the previous call is overwritten. Matthias.St.Pierre> Matthias.St.Pierre> Reseeding the 'DRBG' whenever the user calls 'RAND_add()' does not seem a good solution. It would be too expensive, in particular if system entropy is pulled for reseeding. Of course it is possible to fix this issue, but the DRBG provides for a much simpler solution: it lets the consumer contribute to the entropy of the internal state by providing additional input. If the user input contains entropy, that's fine, if it's "snake oil", no harm. The additional input is mixed into the internal state in just the same way as the entropy buffer using the 'ctr_df()' derivation function. One might think of the 'entropy' input as trusted randomness and 'adin' as untrusted randomness. Matthias.St.Pierre> Matthias.St.Pierre> For this reason, I would like to see the 'RAND_add()/RAND_bytes()' pattern deprecated and the 'RAND_DRBG_generate() with additional input' pattern advertised instead. Matthias.St.Pierre> Matthias.St.Pierre> Matthias.St.Pierre> The DRBG API supports different implementations Matthias.St.Pierre> =============================================== Matthias.St.Pierre> Matthias.St.Pierre> Well, it _supported_ them, until recently. But that's not irreversible. Matthias.St.Pierre> Matthias.St.Pierre> The DRBG concept, as layed out in the NIST standard, provides a generic framework for deterministic RNGs (the acronym DRBG stands for Deterministic Random Bit Generator). It deals with general questions like how to instantiate and reseed the RNG, where does it get the entropy from, etc. Matthias.St.Pierre> Matthias.St.Pierre> The standard proposes three concrete implementations, Hash_DRBG, HMAC_DRBG, and CTR_DRBG. In the FIPS code, all three were implemented, and the genericity was achieved using a data union and a set of five function pointers, reminiscent of a vtable in C++: Matthias.St.Pierre> Matthias.St.Pierre> struct drbg_ctx_st Matthias.St.Pierre> { Matthias.St.Pierre> ... Matthias.St.Pierre> Matthias.St.Pierre> /* Implementation specific structures */ Matthias.St.Pierre> union Matthias.St.Pierre> { Matthias.St.Pierre> DRBG_HASH_CTX hash; Matthias.St.Pierre> DRBG_HMAC_CTX hmac; Matthias.St.Pierre> DRBG_CTR_CTX ctr; Matthias.St.Pierre> } d; Matthias.St.Pierre> /* Initialiase PRNG and setup callbacks below */ Matthias.St.Pierre> int (*init)(DRBG_CTX *ctx, int nid, int security, unsigned int flags); Matthias.St.Pierre> /* Intantiate PRNG */ Matthias.St.Pierre> int (*instantiate)(DRBG_CTX *ctx, Matthias.St.Pierre> const unsigned char *ent, size_t entlen, Matthias.St.Pierre> const unsigned char *nonce, size_t noncelen, Matthias.St.Pierre> const unsigned char *pers, size_t perslen); Matthias.St.Pierre> /* reseed */ Matthias.St.Pierre> int (*reseed)(DRBG_CTX *ctx, Matthias.St.Pierre> const unsigned char *ent, size_t entlen, Matthias.St.Pierre> const unsigned char *adin, size_t adinlen); Matthias.St.Pierre> /* generat output */ Matthias.St.Pierre> int (*generate)(DRBG_CTX *ctx, Matthias.St.Pierre> unsigned char *out, size_t outlen, Matthias.St.Pierre> const unsigned char *adin, size_t adinlen); Matthias.St.Pierre> /* uninstantiate */ Matthias.St.Pierre> int (*uninstantiate)(DRBG_CTX *ctx); Matthias.St.Pierre> Matthias.St.Pierre> ... Matthias.St.Pierre> }; Matthias.St.Pierre> Matthias.St.Pierre> Matthias.St.Pierre> This part of the code was removed during the DRBG port, because currently CTR_DRBG is the only impementation. I would like to suggest to restore this 'polymorphic' implementation, to ease adding new implementations CHACHA20_DRBG in the future Matthias.St.Pierre> Matthias.St.Pierre> Matthias.St.Pierre> The DRBG API is well documented and tested Matthias.St.Pierre> ========================================== Matthias.St.Pierre> Matthias.St.Pierre> Matthias.St.Pierre> The entire DRBG API is part of the OpenSSL FIPS 2.0 Module and as such is well tested and well documented. Most of the documentation for the manual pages still-to-be-written can be taken from the FIPS User Guide and converted into manual pages, starting with textual modifications like Matthias.St.Pierre> Matthias.St.Pierre> Matthias.St.Pierre> FIPS_drbg_xxx(...) -> RAND_DRBG_xxx(...) Matthias.St.Pierre> DRBG_CTX *dctx -> RAND_DRBG *dctx Matthias.St.Pierre> Matthias.St.Pierre> and taking the new typedefs into consideration. Here is for example a comparison of the API function to install the callbacks: Matthias.St.Pierre> Matthias.St.Pierre> FIPS DRBG: Matthias.St.Pierre> Matthias.St.Pierre> int FIPS_drbg_set_callbacks(DRBG_CTX *dctx, Matthias.St.Pierre> size_t (*get_entropy)(DRBG_CTX *ctx, ), Matthias.St.Pierre> void (*cleanup_entropy)(DRBG_CTX *ctx, ), Matthias.St.Pierre> size_t entropy_blocklen, Matthias.St.Pierre> size_t (*get_nonce)(DRBG_CTX *ctx, ), Matthias.St.Pierre> void (*cleanup_nonce)(DRBG_CTX *ctx, ) Matthias.St.Pierre> ); Matthias.St.Pierre> Matthias.St.Pierre> Matthias.St.Pierre> RAND_DRBG: Matthias.St.Pierre> Matthias.St.Pierre> typedef size_t (*RAND_DRBG_get_entropy_fn)(RAND_DRBG *ctx, ); Matthias.St.Pierre> typedef void (*RAND_DRBG_cleanup_entropy_fn)(RAND_DRBG *ctx, ); Matthias.St.Pierre> typedef size_t (*RAND_DRBG_get_nonce_fn)(RAND_DRBG *ctx, ); Matthias.St.Pierre> typedef void (*RAND_DRBG_cleanup_nonce_fn)(RAND_DRBG *ctx, ); Matthias.St.Pierre> Matthias.St.Pierre> int RAND_DRBG_set_callbacks(RAND_DRBG *dctx, Matthias.St.Pierre> RAND_DRBG_get_entropy_fn get_entropy, Matthias.St.Pierre> RAND_DRBG_cleanup_entropy_fn cleanup_entropy, Matthias.St.Pierre> RAND_DRBG_get_nonce_fn get_nonce, Matthias.St.Pierre> RAND_DRBG_cleanup_nonce_fn cleanup_nonce); Matthias.St.Pierre> Matthias.St.Pierre> Matthias.St.Pierre> Matthias.St.Pierre> Matthias.St.Pierre> Matthias.St.Pierre> Conclusion Matthias.St.Pierre> ========== Matthias.St.Pierre> Matthias.St.Pierre> I see no reason why the RAND_DRBG shouldn't be made public as soon as possible, keeping the API as close as possible to the original FIPS DRBG API (FIPS 3.0 is upcoming!). In a second step, the current compatibility binding from RAND_METHOD to RAND_DRBG could be deprecated and faded out smoothly. Matthias.St.Pierre> Matthias.St.Pierre> Matthias.St.Pierre> Looking forward to receiving your comments. (But please be patient with me, I'm currently on physical rehab after a surgery.) Matthias.St.Pierre> Matthias.St.Pierre> Matthias.St.Pierre> Matthias St. Pierre Matthias.St.Pierre> Matthias.St.Pierre> From Matthias.St.Pierre at ncp-e.com Tue Aug 29 13:05:21 2017 From: Matthias.St.Pierre at ncp-e.com (Dr. Matthias St. Pierre) Date: Tue, 29 Aug 2017 13:05:21 +0000 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: <20170829.134229.1463977663045816733.levitte@openssl.org> References: <20170829.134229.1463977663045816733.levitte@openssl.org> Message-ID: <0e062727611c44bb8039170bbec11a57@Ex13.ncp.local> > If I understand correctly, the RAND_DRBG API is really a completely > separate API that has nothing to do with the RAND_METHOD API pers se, > i.e. any association between the two is more or less "accidental"? Yes, you're right. The original DRBG API was implemented for the FIPS object module (only) by Stephen Henson. A FIPS capable libcrypto 1.0.2y implements a binding from RAND_METHOD to the FIPS DRBG, FIPS_drbg_method(), which becomes the default RAND_METHOD when FIPS mode is enabled. So this was how the FIPS DRBG was 'plugged in' at runtime. *Note*: In those days, the FIPS DRBG pulled its entropy using the (non-fips) RAND_SSLeay() method, which was seeded using RAND_poll()/RAND_add(). Alltogether a really weird construction. > Frankly, I cannot see anything wrong with making that a public API. I > wonder, though, if this is going to be an implementation that's kinda > sorta built-in only, or if other implementations of the basic stuff > will be possible... or in other words, will we have a "method" > structure like we have with so many other APIs? In saying this, I'm > counting in the possibility that some implementations could come in > the form of engines, is this something that's been thought about? I > do notive the DRGB_CTX structure, but that doesn't quite follow the > usual "method" pattern we have... Currently it's only possible to customize the callbacks but not the deterministic algorithm. IMHO this is sufficient for the needs of a standard OpenSSL user who only wants control over the entropy source. A true new algorithm (like e.g. CHACHA2_DRBG) should be implemented by experts and added mainstream. So I don't see any advantage of having an engine over using the 'vtable' approach from the FIPS DRBG, which has been removed. Regards, Matthias -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 4328 bytes Desc: not available URL: From levitte at openssl.org Tue Aug 29 13:16:05 2017 From: levitte at openssl.org (Richard Levitte) Date: Tue, 29 Aug 2017 15:16:05 +0200 (CEST) Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: <0e062727611c44bb8039170bbec11a57@Ex13.ncp.local> References: <20170829.134229.1463977663045816733.levitte@openssl.org> <0e062727611c44bb8039170bbec11a57@Ex13.ncp.local> Message-ID: <20170829.151605.794709116568547408.levitte@openssl.org> In message <0e062727611c44bb8039170bbec11a57 at Ex13.ncp.local> on Tue, 29 Aug 2017 13:05:21 +0000, "Dr. Matthias St. Pierre" said: Matthias.St.Pierre> > Frankly, I cannot see anything wrong with making that a public API. I Matthias.St.Pierre> > wonder, though, if this is going to be an implementation that's kinda Matthias.St.Pierre> > sorta built-in only, or if other implementations of the basic stuff Matthias.St.Pierre> > will be possible... or in other words, will we have a "method" Matthias.St.Pierre> > structure like we have with so many other APIs? In saying this, I'm Matthias.St.Pierre> > counting in the possibility that some implementations could come in Matthias.St.Pierre> > the form of engines, is this something that's been thought about? I Matthias.St.Pierre> > do notive the DRGB_CTX structure, but that doesn't quite follow the Matthias.St.Pierre> > usual "method" pattern we have... Matthias.St.Pierre> Matthias.St.Pierre> Currently it's only possible to customize the callbacks but not the deterministic algorithm. IMHO this is sufficient for the needs of a standard OpenSSL user who only wants control over the entropy source. A true new algorithm (like e.g. CHACHA2_DRBG) should be implemented by experts and added mainstream. So I don't see any advantage of having an engine over using the 'vtable' approach from the FIPS DRBG, which has been removed. Essentially, the argument for your last remark is in-structure vtable vs refered to vtable. I tend to prefer the latter (and that's the usual OpenSSL pattern too, even though there are exceptions). -- Richard Levitte levitte at openssl.org OpenSSL Project http://www.openssl.org/~levitte/ From Matthias.St.Pierre at ncp-e.com Tue Aug 29 13:27:20 2017 From: Matthias.St.Pierre at ncp-e.com (Dr. Matthias St. Pierre) Date: Tue, 29 Aug 2017 13:27:20 +0000 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: <20170829.151605.794709116568547408.levitte@openssl.org> References: <20170829.134229.1463977663045816733.levitte@openssl.org> <0e062727611c44bb8039170bbec11a57@Ex13.ncp.local> <20170829.151605.794709116568547408.levitte@openssl.org> Message-ID: > Essentially, the argument for your last remark is in-structure vtable > vs refered to vtable. I tend to prefer the latter (and that's the > usual OpenSSL pattern too, even though there are exceptions). You are the experts and much more familiar with the code then I am. My role was only to give the starting shot, the rest is up to you. Regards, Matthias -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 4328 bytes Desc: not available URL: From levitte at openssl.org Tue Aug 29 13:28:35 2017 From: levitte at openssl.org (Richard Levitte) Date: Tue, 29 Aug 2017 15:28:35 +0200 (CEST) Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: References: <0e062727611c44bb8039170bbec11a57@Ex13.ncp.local> <20170829.151605.794709116568547408.levitte@openssl.org> Message-ID: <20170829.152835.149197151251372241.levitte@openssl.org> In message on Tue, 29 Aug 2017 13:27:20 +0000, "Dr. Matthias St. Pierre" said: Matthias.St.Pierre> > Essentially, the argument for your last remark is in-structure vtable Matthias.St.Pierre> > vs refered to vtable. I tend to prefer the latter (and that's the Matthias.St.Pierre> > usual OpenSSL pattern too, even though there are exceptions). Matthias.St.Pierre> Matthias.St.Pierre> You are the experts and much more familiar with Matthias.St.Pierre> the code then I am. My role was only to give the Matthias.St.Pierre> starting shot, the rest is up to you. Fair enough! :-) -- Richard Levitte levitte at openssl.org OpenSSL Project http://www.openssl.org/~levitte/ From steve at openssl.org Tue Aug 29 13:29:03 2017 From: steve at openssl.org (Dr. Stephen Henson) Date: Tue, 29 Aug 2017 13:29:03 +0000 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: <20170829.134229.1463977663045816733.levitte@openssl.org> References: <20170829.134229.1463977663045816733.levitte@openssl.org> Message-ID: <20170829132903.GA25562@openssl.org> On Tue, Aug 29, 2017, Richard Levitte wrote: > I'm late in the game, having only followed the development very > superficially... > > If I understand correctly, the RAND_DRBG API is really a completely > separate API that has nothing to do with the RAND_METHOD API pers se, > i.e. any association between the two is more or less "accidental"? > > Frankly, I cannot see anything wrong with making that a public API. I > wonder, though, if this is going to be an implementation that's kinda > sorta built-in only, or if other implementations of the basic stuff > will be possible... or in other words, will we have a "method" > structure like we have with so many other APIs? In saying this, I'm > counting in the possibility that some implementations could come in > the form of engines, is this something that's been thought about? I > do notive the DRGB_CTX structure, but that doesn't quite follow the > usual "method" pattern we have... > In terms of future FIPS 140-2 compliance having a way to redirect all PRNG operations to a FIPS compliant version is essential. This would require some kind of method. We can redirect via RAND_METHOD but that currently means that any ENGINE supplied PRNG is severely restricted (one global PRNG only) compared to the built in DRBG. We could resolve that via a DRBG_METHOD or some extended RAND_METHOD. That's for the future though: there wont be a new FIPS module for some time that will need this functionality. In terms of the API itself. We could make RAND_DRBG public and I can see reasons both for and against that. The main objection I can see is that it commits us to DRBG like PRNGs: that is if we (or an ENGINE) in future wants to use a RNG that isn't a good fit for the DRBG API it's a problem. We're partially hitting that now because the DRBG API is not a good fit for the older RAND API. One alternative is to expand RAND_METHOD so it treats a DRBG as a special case of the expanded API and it is flexible enough to handle any future needs. That would (for example) have extended RAND APIs which can be passed an "instance" of a PRNG (something like RAND_CTX which is analagous to the DRBG instance structure we currently have). That is tricky because RAND_METHOD is alas not an opaque structure and can't be made so before OpenSSL 1.2.0. Tricky but not impossible. Steve. -- Dr Stephen N. Henson. OpenSSL project core developer. Commercial tech support now available see: http://www.openssl.org From Matthias.St.Pierre at ncp-e.com Tue Aug 29 13:39:02 2017 From: Matthias.St.Pierre at ncp-e.com (Dr. Matthias St. Pierre) Date: Tue, 29 Aug 2017 13:39:02 +0000 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: <20170829.152835.149197151251372241.levitte@openssl.org> References: <0e062727611c44bb8039170bbec11a57@Ex13.ncp.local> <20170829.151605.794709116568547408.levitte@openssl.org> <20170829.152835.149197151251372241.levitte@openssl.org> Message-ID: <3a54fcefd17d4735920426d21b3b868e@Ex13.ncp.local> > -----Urspr?ngliche Nachricht----- > Von: openssl-dev [mailto:openssl-dev-bounces at openssl.org] Im Auftrag von Richard Levitte > Gesendet: Dienstag, 29. August 2017 15:29 > An: openssl-dev at openssl.org > Betreff: Re: [openssl-dev] Plea for a new public OpenSSL RNG API > > In message on Tue, 29 Aug 2017 13:27:20 +0000, "Dr. > Matthias St. Pierre" said: > > Matthias.St.Pierre> > Essentially, the argument for your last remark is in-structure vtable > Matthias.St.Pierre> > vs refered to vtable. I tend to prefer the latter (and that's the > Matthias.St.Pierre> > usual OpenSSL pattern too, even though there are exceptions). > Matthias.St.Pierre> > Matthias.St.Pierre> You are the experts and much more familiar with > Matthias.St.Pierre> the code then I am. My role was only to give the > Matthias.St.Pierre> starting shot, the rest is up to you. > > Fair enough! :-) Just a sudden inspiration: If the RAND_DRBG becomes a truly independent API it might be better to strip the RAND_ prefix and redesign the API such that one has - a DRBG_CTX structure for the data members - a DRBG_METHOD structure for its methods Would this look more OpenSSL-like to you? Matthias -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 4328 bytes Desc: not available URL: From levitte at openssl.org Tue Aug 29 13:52:29 2017 From: levitte at openssl.org (Richard Levitte) Date: Tue, 29 Aug 2017 15:52:29 +0200 (CEST) Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: <3a54fcefd17d4735920426d21b3b868e@Ex13.ncp.local> References: <20170829.152835.149197151251372241.levitte@openssl.org> <3a54fcefd17d4735920426d21b3b868e@Ex13.ncp.local> Message-ID: <20170829.155229.1483789374609601848.levitte@openssl.org> In message <3a54fcefd17d4735920426d21b3b868e at Ex13.ncp.local> on Tue, 29 Aug 2017 13:39:02 +0000, "Dr. Matthias St. Pierre" said: Matthias.St.Pierre> Just a sudden inspiration: If the RAND_DRBG becomes a truly independent API it might be better to strip the RAND_ prefix and redesign the API such that one has Matthias.St.Pierre> Matthias.St.Pierre> - a DRBG_CTX structure for the data members Matthias.St.Pierre> - a DRBG_METHOD structure for its methods Matthias.St.Pierre> Matthias.St.Pierre> Would this look more OpenSSL-like to you? Yes. And per fairly recent recommendations to avoid cluttering the name space, that would be OSSL_DRGB_CTX and OSSL_DRGB_METHOD, btw. -- Richard Levitte levitte at openssl.org OpenSSL Project http://www.openssl.org/~levitte/ From rsalz at akamai.com Tue Aug 29 14:02:57 2017 From: rsalz at akamai.com (Salz, Rich) Date: Tue, 29 Aug 2017 14:02:57 +0000 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: <2f79c2b6-560f-b245-b39d-dc4147d5ab44@openssl.org> References: <2f79c2b6-560f-b245-b39d-dc4147d5ab44@openssl.org> Message-ID: <63C3E608-B02D-435F-9F50-CB93FE288DE5@akamai.com> dev asap. If there are problems with it we can always revert it before it makes it into a release. Someone on the OMC called that ?flip-flopping? and seemed to be against this kind of thing. If we are going to have an additional API, then we should at least see a draft of the header file first. Keep in mind that the current DRBG organization might change, and we don?t want to lose that freedom. From rsalz at akamai.com Tue Aug 29 14:10:51 2017 From: rsalz at akamai.com (Salz, Rich) Date: Tue, 29 Aug 2017 14:10:51 +0000 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: <20170829.155229.1483789374609601848.levitte@openssl.org> References: <20170829.152835.149197151251372241.levitte@openssl.org> <3a54fcefd17d4735920426d21b3b868e@Ex13.ncp.local> <20170829.155229.1483789374609601848.levitte@openssl.org> Message-ID: ? Yes. And per fairly recent recommendations to avoid cluttering the ? name space, that would be OSSL_DRGB_CTX and OSSL_DRGB_METHOD, btw. DRBG is used for RAND stuff, so I don?t see a good reason to not using RAND_DRBG as the prefix From rsalz at akamai.com Tue Aug 29 14:15:58 2017 From: rsalz at akamai.com (Salz, Rich) Date: Tue, 29 Aug 2017 14:15:58 +0000 Subject: [openssl-dev] How to use BIO_do_connect(), blocking and non-blocking with timeout, coping with errors In-Reply-To: <69d6c8c3-c3ba-44f5-fad0-4b2294e210d3@siemens.com> References: <69d6c8c3-c3ba-44f5-fad0-4b2294e210d3@siemens.com> Message-ID: Getting the client connect right appears surprisingly messy when one needs to cope with all kinds of network error situations including domain name resolution issues and temporarily unreachable servers. Both indefinitely blocking and non-blocking behavior (i.e., connection attempts with and without a timeout) should be supported. It is a complicated issue and hard to get right for all definitions of right for all applications ? A set of API?s that set up all the TLS ?metadata?, and took a connected socket might be a way through the maze. For example: SSL *SSL_connection(int socket, const char *servername, ?whatever?) From matt at openssl.org Tue Aug 29 14:36:22 2017 From: matt at openssl.org (Matt Caswell) Date: Tue, 29 Aug 2017 15:36:22 +0100 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: <63C3E608-B02D-435F-9F50-CB93FE288DE5@akamai.com> References: <2f79c2b6-560f-b245-b39d-dc4147d5ab44@openssl.org> <63C3E608-B02D-435F-9F50-CB93FE288DE5@akamai.com> Message-ID: <1bcf6fcb-fff7-0cdb-562e-1651dfea2ded@openssl.org> On 29/08/17 15:02, Salz, Rich via openssl-dev wrote: > > dev asap. If there are problems with it we can always revert it before > it makes it into a release. > > Someone on the OMC called that ?flip-flopping? and seemed to be against this kind of thing. > > If we are going to have an additional API, then we should at least see a draft of the header file first. Matthias, would you be willing to create such a draft as a github PR (marked as WIP for now)? Matt From uri at ll.mit.edu Tue Aug 29 15:06:12 2017 From: uri at ll.mit.edu (Blumenthal, Uri - 0553 - MITLL) Date: Tue, 29 Aug 2017 15:06:12 +0000 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: <63C3E608-B02D-435F-9F50-CB93FE288DE5@akamai.com> References: <2f79c2b6-560f-b245-b39d-dc4147d5ab44@openssl.org> <63C3E608-B02D-435F-9F50-CB93FE288DE5@akamai.com> Message-ID: <3AF1D54F-3A54-400F-8898-AED62D9AF67B@ll.mit.edu> Could you please be more specific wrt. DRBG organization that in your opinion could impact the UI? NIST 90Ar1 seems specific enough on what functionality DRBG can provide to users, and it doesn't seem like a very elaborate or rich interface. Why would it change, regardless of what you may need to do with it internally? Regards, Uri Sent from my iPhone > On Aug 29, 2017, at 10:03, Salz, Rich via openssl-dev wrote: > > > dev asap. If there are problems with it we can always revert it before > it makes it into a release. > > Someone on the OMC called that ?flip-flopping? and seemed to be against this kind of thing. > > If we are going to have an additional API, then we should at least see a draft of the header file first. > > Keep in mind that the current DRBG organization might change, and we don?t want to lose that freedom. > > > > -- > openssl-dev mailing list > To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 4223 bytes Desc: not available URL: From bkaduk at akamai.com Tue Aug 29 15:17:52 2017 From: bkaduk at akamai.com (Benjamin Kaduk) Date: Tue, 29 Aug 2017 10:17:52 -0500 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: <2f79c2b6-560f-b245-b39d-dc4147d5ab44@openssl.org> References: <2f79c2b6-560f-b245-b39d-dc4147d5ab44@openssl.org> Message-ID: <3af301f3-e02c-00ea-e66e-1fe1092d3699@akamai.com> On 08/29/2017 05:17 AM, Matt Caswell wrote: > On 29/08/17 10:45, Dr. Matthias St. Pierre wrote >> Currently, the OpenSSL core members seem to be reluctant to make the >> API public, at least at this early stage. I understand Rich Salz's >> viewpoint that this requires a thorough discussion, because a public >> interface can't be easily changed and wrong decisions in the early >> phase can become a heavy burdon. > FWIW, I am not against making the API public although I too understand > Rich's reluctance. However it seems inevitable to me that sooner or > later this API will have to be made accessible and I see little benefit > in delaying that and plenty of reasons for doing it now (it only > frustrates those that know the functionality is there but can't get at > it :-)). There is always a risk with every API added that you get it > wrong, and then you are stuck with it for ever more. The only real > answers to this are: > > 1) Make sure you "design" your API to start with and get it reviewed Definitely. It might be nice to see a high-level overview of what parts of the design changed while integrating the FIPS DRBG into the current form in master -- the global "rand_bytes" buffer and the removal of non-CTR implementation support are just a couple things that have been mentioned already in this thread. > 2) Encourage people to try the API while it is still in dev i.e. so we > can still change it before the final release. That also seems reasonable to me. I might go further and also propose that we seek a more stringent review standard than usual for this work, since it is such a critical subsystem, and the current level of review let a few things slip through unnoticed. > I'm not really convinced that keeping the API hidden now will mean that > it will be better in some future release. It just delays the point at > which we have the realisation of "if only we had designed it like *this* > it would have been soooo much better". Better is to make it public in > dev asap. If there are problems with it we can always revert it before > it makes it into a release. > >> The 'RAND_add()/RAND_bytes()' pattern is broken >> =============================================== >> >> In OpenSSL, the classical way for the RNG consumer to add his own >> randomness is to call 'RAND_add()' before calling 'RAND_bytes()'. If >> the new 'RAND_OpenSSL()' method (the "compatibility layer" hiding the >> public RAND_DRBG instance) is the default, then this does not work >> as expected anymore: >> >> The reason is that a call to 'RAND_add()' adds the provided >> randomness only to a global buffer ('rand_bytes'), from which it will >> be pulled during the next reseed. But no reseed is triggered. So the >> next RAND_bytes() call will be unaffected from the RAND_add(), which >> is not what the consumer expected. (The same holds for 'RAND_seed()', >> since 'drbg_seed()' only calls into 'drbg_add()') >> >> Reseeding of DRBGs occurs only at the following occasions: >> >> * immediately after a 'fork()' (new) * if the 'reseed_counter' >> exceeds the 'reseed_interval' * if 'RAND_DRBG_generate()' is called >> requesting 'prediction_resistance' * 'RAND_DRBG_reseed()' is called >> explicitely >> >> *Note:* Currently it looks like the situation is even worse: if >> 'RAND_add()' is called multiple times before a reseed occurs, then >> the result of the previous call is overwritten. >> >> Reseeding the 'DRBG' whenever the user calls 'RAND_add()' does not >> seem a good solution. It would be too expensive, in particular if >> system entropy is pulled for reseeding. Of course it is possible to >> fix this issue, but the DRBG provides for a much simpler solution: it >> lets the consumer contribute to the entropy of the internal state by >> providing additional input. If the user input contains entropy, >> that's fine, if it's "snake oil", no harm. The additional input is >> mixed into the internal state in just the same way as the entropy >> buffer using the 'ctr_df()' derivation function. One might think of >> the 'entropy' input as trusted randomness and 'adin' as untrusted >> randomness. >> >> For this reason, I would like to see the 'RAND_add()/RAND_bytes()' >> pattern deprecated and the 'RAND_DRBG_generate() with additional >> input' pattern advertised instead. > Deprecation itself would probably have to wait for a 1.2.0 release. We > have a binary/source compatibility guarantee for 1.1.0 which would not > be met for builds using "--api" or "no-deprecated". > > Is there a potential security vulnerability here? Applications using the > "old" APIs expect RAND_add() to behave in a particular way. If we have > silently changed this behaviour in 1.1.1 are they exposed? > Is there potential for vulnerability here? Yes. Whether there is actual vulnerability would require some careful analysis, and potentially input from people like Uri who have requirements for including TRBG input. -Ben -------------- next part -------------- An HTML attachment was scrubbed... URL: From rsalz at akamai.com Tue Aug 29 15:33:10 2017 From: rsalz at akamai.com (Salz, Rich) Date: Tue, 29 Aug 2017 15:33:10 +0000 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: <3AF1D54F-3A54-400F-8898-AED62D9AF67B@ll.mit.edu> References: <2f79c2b6-560f-b245-b39d-dc4147d5ab44@openssl.org> <63C3E608-B02D-435F-9F50-CB93FE288DE5@akamai.com> <3AF1D54F-3A54-400F-8898-AED62D9AF67B@ll.mit.edu> Message-ID: <2D67CCC2-32E2-48B7-9E7A-D42C3D925AAC@akamai.com> Could you please be more specific wrt. DRBG organization that in your opinion could impact the UI? From your use-case: you want to add entropy into a specific DRBG. You want to push it, as opposed to the DRBG ?pull when needed? model. That?s an additional API. Also from your use-case: you want to specify which DRBG instance gets that entropy. If we move to a pair per thread, as opposed to one per SSL and two in the global space, how do we make sure that API still works and does the right thing. Does that makes sense, and does it answer your question? From kurt at roeckx.be Tue Aug 29 16:27:16 2017 From: kurt at roeckx.be (Kurt Roeckx) Date: Tue, 29 Aug 2017 18:27:16 +0200 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: References: Message-ID: <20170829162716.ovfcl5p7j6ewuti4@roeckx.be> I actually plan to work on most of not all the things you mention. This will probably result in the API changing before it's made public. I'm probably slow, so please be patient. Kurt From kurt at roeckx.be Tue Aug 29 16:37:53 2017 From: kurt at roeckx.be (Kurt Roeckx) Date: Tue, 29 Aug 2017 18:37:53 +0200 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: <7b6ea5fcc69c4a65b3e98bf8620906c8@Ex13.ncp.local> References: <2f79c2b6-560f-b245-b39d-dc4147d5ab44@openssl.org> <7b6ea5fcc69c4a65b3e98bf8620906c8@Ex13.ncp.local> Message-ID: <20170829163752.3qceqgckzybewmky@roeckx.be> On Tue, Aug 29, 2017 at 11:31:03AM +0000, Dr. Matthias St. Pierre wrote: > > -----Urspr?ngliche Nachricht----- > > Von: openssl-dev [mailto:openssl-dev-bounces at openssl.org] Im Auftrag von Matt Caswell > > Gesendet: Dienstag, 29. August 2017 12:17 > > An: openssl-dev at openssl.org > > Betreff: Re: [openssl-dev] Plea for a new public OpenSSL RNG API > > > > > > On 29/08/17 10:45, Dr. Matthias St. Pierre wrote > > > ... > > > The 'RAND_add()/RAND_bytes()' pattern is broken > > > =============================================== > > > > > > In OpenSSL, the classical way for the RNG consumer to add his own > > > randomness is to call 'RAND_add()' before calling 'RAND_bytes()'. If > > > the new 'RAND_OpenSSL()' method (the "compatibility layer" hiding the > > > public RAND_DRBG instance) is the default, then this does not work > > > as expected anymore: > > > ... > > > > Is there a potential security vulnerability here? Applications using the > > "old" APIs expect RAND_add() to behave in a particular way. If we have > > silently changed this behaviour in 1.1.1 are they exposed? > > Don't worry, this issue is new, the global 'rand_bytes' buffer has only been introduced by the DRBG port to master in August. I don't think it's a big deal to fix it. The reason I mentioned it here was to emphasize, that it is really hard to get the different philosophies (push vs. pull) of the two APIs working together correctly. The code was reviewed by several people and nobody noticed it. By the way: the approach using the fixed size global 'rand_bytes' buffer has another issue, which I will try to write down on GitHub within the next days. I've actually noticed how this works and I have already partially rewritten it, but I'm still not very happy about it. I think by RAND_add() should not be called internally. But the question then is what to do when an application calls RAND_add(), we should be doing something with the buffer that's given. I think the best way to deal with it is with the DRBG API is used, RAND_add() is used for additional data. We now have 2 global DRBGs, and I think I want to have 1 of them chain to the other. RAND_add() could then be used for the master. An other problem with the current implemenation is that the randomness parameter that's now given to RAND_add() is just ignored, it assumes it's the same as the length. Kurt From kurt at roeckx.be Tue Aug 29 16:41:26 2017 From: kurt at roeckx.be (Kurt Roeckx) Date: Tue, 29 Aug 2017 18:41:26 +0200 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: <0e062727611c44bb8039170bbec11a57@Ex13.ncp.local> References: <20170829.134229.1463977663045816733.levitte@openssl.org> <0e062727611c44bb8039170bbec11a57@Ex13.ncp.local> Message-ID: <20170829164125.uvcdinbdjzpk3uca@roeckx.be> On Tue, Aug 29, 2017 at 01:05:21PM +0000, Dr. Matthias St. Pierre wrote: > > Currently it's only possible to customize the callbacks but not the deterministic algorithm. IMHO this is sufficient for the needs of a standard OpenSSL user who only wants control over the entropy source. A true new algorithm (like e.g. CHACHA2_DRBG) should be implemented by experts and added mainstream. So I don't see any advantage of having an engine over using the 'vtable' approach from the FIPS DRBG, which has been removed. I've been looking at implementing a chacha20 based DRBG, which isn't that hard. But there are various choices you can make, and every chacha20 RNG that I've seen seems to take a random set of those choices. I really wish there was some standardized version. Kurt From rsalz at akamai.com Tue Aug 29 16:44:36 2017 From: rsalz at akamai.com (Salz, Rich) Date: Tue, 29 Aug 2017 16:44:36 +0000 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: <20170829163752.3qceqgckzybewmky@roeckx.be> References: <2f79c2b6-560f-b245-b39d-dc4147d5ab44@openssl.org> <7b6ea5fcc69c4a65b3e98bf8620906c8@Ex13.ncp.local> <20170829163752.3qceqgckzybewmky@roeckx.be> Message-ID: ? An other problem with the current implemenation is that the ? randomness parameter that's now given to RAND_add() is just ? ignored, it assumes it's the same as the length. For what it?s worth, this was done deliberately, make RAND_add and RAND_seed equivalent. I am skeptical of the ability to get that estimate correct. Someone on GH there is a conversation thread about turning that into a percentage, which seems like the best thing to do for any new API. From uri at ll.mit.edu Tue Aug 29 18:50:37 2017 From: uri at ll.mit.edu (Blumenthal, Uri - 0553 - MITLL) Date: Tue, 29 Aug 2017 18:50:37 +0000 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: References: <2f79c2b6-560f-b245-b39d-dc4147d5ab44@openssl.org> <7b6ea5fcc69c4a65b3e98bf8620906c8@Ex13.ncp.local> <20170829163752.3qceqgckzybewmky@roeckx.be> Message-ID: <9BEAA98D-EC21-46CA-88CB-829C609B82F7@ll.mit.edu> On 8/29/17, 12:45, "openssl-dev on behalf of Salz, Rich via openssl-dev" wrote: ? An other problem with the current implemenation is that the ? randomness parameter that's now given to RAND_add() is just ? ignored, it assumes it's the same as the length. For what it?s worth, this was done deliberately, make RAND_add and RAND_seed equivalent. I am skeptical of the ability to get that estimate correct. Someone on GH there is a conversation thread about turning that into a percentage, which seems like the best thing to do for any new API. What?s the point of having this potentially harmful parameter? If it weren?t ignored ? how would OpenSSL use it? If, based on its value, OpenSSL may decide that it now got ?enough? entropy and doesn?t need to pull more from other sources before serving randomness to requestors ? then it is harmful. ?Over-confidence? in this value by the caller can negatively impact the quality of the produced random numbers. If this value is not used to guide OpenSSL when to stop pulling entropy sources and start serving randomness ? then it causes no harm, but what?s its purpose? IMHO this interface is a way for the user to improve the quality of the randomness it would get from the given RNG, *not* to replace (or diminish) its other sources. My proposal is to abolish this parameter, especially since now it is simply ignored (and IMHO ? for a good reason). -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5211 bytes Desc: not available URL: From bkaduk at akamai.com Tue Aug 29 18:52:18 2017 From: bkaduk at akamai.com (Benjamin Kaduk) Date: Tue, 29 Aug 2017 13:52:18 -0500 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: <9BEAA98D-EC21-46CA-88CB-829C609B82F7@ll.mit.edu> References: <2f79c2b6-560f-b245-b39d-dc4147d5ab44@openssl.org> <7b6ea5fcc69c4a65b3e98bf8620906c8@Ex13.ncp.local> <20170829163752.3qceqgckzybewmky@roeckx.be> <9BEAA98D-EC21-46CA-88CB-829C609B82F7@ll.mit.edu> Message-ID: <324d1dbf-da0c-db3e-2fe2-cc5d393bf353@akamai.com> On 08/29/2017 01:50 PM, Blumenthal, Uri - 0553 - MITLL wrote: > IMHO this interface is a way for the user to improve the quality of the randomness it would get from the given RNG, *not* to replace (or diminish) its other sources. My proposal is to abolish this parameter, especially since now it is simply ignored (and IMHO ? for a good reason). That's a fine proposal ... it just can't be implemented until a major release boundary, when our ABI stability policy permits such breaking changes. -Ben -------------- next part -------------- An HTML attachment was scrubbed... URL: From levitte at openssl.org Tue Aug 29 19:02:34 2017 From: levitte at openssl.org (Richard Levitte) Date: Tue, 29 Aug 2017 21:02:34 +0200 (CEST) Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: <3AF1D54F-3A54-400F-8898-AED62D9AF67B@ll.mit.edu> References: <2f79c2b6-560f-b245-b39d-dc4147d5ab44@openssl.org> <63C3E608-B02D-435F-9F50-CB93FE288DE5@akamai.com> <3AF1D54F-3A54-400F-8898-AED62D9AF67B@ll.mit.edu> Message-ID: <20170829.210234.2290485732319802683.levitte@openssl.org> In message <3AF1D54F-3A54-400F-8898-AED62D9AF67B at ll.mit.edu> on Tue, 29 Aug 2017 15:06:12 +0000, "Blumenthal, Uri - 0553 - MITLL" said: uri> Could you please be more specific wrt. DRBG organization that in your opinion could impact the UI? Are you talking about the UI API or something else? -- Richard Levitte levitte at openssl.org OpenSSL Project http://www.openssl.org/~levitte/ From rsalz at akamai.com Tue Aug 29 19:04:11 2017 From: rsalz at akamai.com (Salz, Rich) Date: Tue, 29 Aug 2017 19:04:11 +0000 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: <9BEAA98D-EC21-46CA-88CB-829C609B82F7@ll.mit.edu> References: <2f79c2b6-560f-b245-b39d-dc4147d5ab44@openssl.org> <7b6ea5fcc69c4a65b3e98bf8620906c8@Ex13.ncp.local> <20170829163752.3qceqgckzybewmky@roeckx.be> <9BEAA98D-EC21-46CA-88CB-829C609B82F7@ll.mit.edu> Message-ID: ? IMHO this interface is a way for the user to improve the quality of the randomness it would get from the given RNG, *not* to replace (or diminish) its other sources. My proposal is to abolish this parameter, especially since now it is simply ignored (and IMHO ? for a good reason). We cannot do this in a minor release; we have to wait until 1.2 From uri at ll.mit.edu Tue Aug 29 19:04:48 2017 From: uri at ll.mit.edu (Blumenthal, Uri - 0553 - MITLL) Date: Tue, 29 Aug 2017 19:04:48 +0000 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: <20170829.210234.2290485732319802683.levitte@openssl.org> References: <2f79c2b6-560f-b245-b39d-dc4147d5ab44@openssl.org> <63C3E608-B02D-435F-9F50-CB93FE288DE5@akamai.com> <3AF1D54F-3A54-400F-8898-AED62D9AF67B@ll.mit.edu> <20170829.210234.2290485732319802683.levitte@openssl.org> Message-ID: <168CEF3C-D655-47BF-9874-048308154AFD@ll.mit.edu> uri> Could you please be more specific wrt. DRBG organization that in your opinion could impact the UI? Are you talking about the UI API or something else? No-no-no, just the UI API. I used the term ?UI? to emphasize that this is the ?public? part of the API, exposed to the user, expected to be called by the user, and (probably) designed to be stable for some (hopefully considerable) time. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5211 bytes Desc: not available URL: From uri at ll.mit.edu Tue Aug 29 19:04:53 2017 From: uri at ll.mit.edu (Blumenthal, Uri - 0553 - MITLL) Date: Tue, 29 Aug 2017 19:04:53 +0000 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: <324d1dbf-da0c-db3e-2fe2-cc5d393bf353@akamai.com> References: <2f79c2b6-560f-b245-b39d-dc4147d5ab44@openssl.org> <7b6ea5fcc69c4a65b3e98bf8620906c8@Ex13.ncp.local> <20170829163752.3qceqgckzybewmky@roeckx.be> <9BEAA98D-EC21-46CA-88CB-829C609B82F7@ll.mit.edu> <324d1dbf-da0c-db3e-2fe2-cc5d393bf353@akamai.com> Message-ID: IMHO this interface is a way for the user to improve the quality of the randomness it would get from the given RNG, *not* to replace (or diminish) its other sources. My proposal is to abolish this parameter, especially since now it is simply ignored (and IMHO ? for a good reason). That's a fine proposal ... it just can't be implemented until a major release boundary, when our ABI stability policy permits such breaking changes. And that is fine. The sooner the better, but ABI stability makes sense too. My main point is: ?RAND_add() and whatever similar in purpose interface calls we may define in the future should exhibit the following behavior: Mix the provided randomness into the RNG state *immediately*, and Keep pulling other sources and mixing them into the state ? don?t subtract from the ?needed entropy? count the amount you presumably got from the user. Frankly, the need to provide double entropy argument doesn?t bother me all that much ? especially if the value 0 is accepted there. ;-) -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5211 bytes Desc: not available URL: From uri at ll.mit.edu Tue Aug 29 19:10:24 2017 From: uri at ll.mit.edu (Blumenthal, Uri - 0553 - MITLL) Date: Tue, 29 Aug 2017 19:10:24 +0000 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: <2D67CCC2-32E2-48B7-9E7A-D42C3D925AAC@akamai.com> References: <2f79c2b6-560f-b245-b39d-dc4147d5ab44@openssl.org> <63C3E608-B02D-435F-9F50-CB93FE288DE5@akamai.com> <3AF1D54F-3A54-400F-8898-AED62D9AF67B@ll.mit.edu> <2D67CCC2-32E2-48B7-9E7A-D42C3D925AAC@akamai.com> Message-ID: <8B137ECD-3077-48E6-B46B-16E0CF972134@ll.mit.edu> On 8/29/17, 11:33, "openssl-dev on behalf of Salz, Rich via openssl-dev" wrote: Could you please be more specific wrt. DRBG organization that in your opinion could impact the UI? > From your use-case: you want to add entropy into a specific DRBG. Yes. > You want to push it, as opposed to the DRBG ?pull when needed? model. I?d like to suggest that any approach other than ?immediately mix the received randomness into the RNG state? is bad. If a user does RAND_add() now, as opposed to 100 source code lines before, there may be a reason for that. Would you agree? Besides possible but probably negligible performance difference ? why would you ever want to delay mixing the received ?additional input? in? > That?s an additional API. I wouldn?t call it ?additional?. It may be a change from the current behavior ? but I think everybody would welcome such a change. IMHO it can only help, never hurt. > Also from your use-case: you want to specify which DRBG instance gets that entropy. Yeah, but that probably isn?t a part of the API that DRBG ?object? exposes. One ?API? is how to get a reference/pointer/instance/whatever to the DRBG object responsible for the operation I?m now concerned with, and that I want to influence/improve. This would probably differ between per-SSL DRBG and per-thread DRBG, etc. I haven?t even thought about this part of the API (and I?m sure others on the team have more experience and understanding of the OpenSSL code to do it better than I would anyway). Another ?API? is how to tell this specific DRBG instance what exactly I want from it now. E.g., mix more randomness that I provide into its state, give me some random bits, whatever. This part probably can stay close to what it currently is. I think 90A would be satisfied with 3-4 interface calls here? > If we move to a pair per thread, as opposed to one per SSL and two in the global space, how do we make sure that API still works and does the right thing. Yeah. That?s the ?other? part of the API. I think the two API ?groups? are pretty (completely?) orthogonal ? independent from each other. > Does that makes sense, and does it answer your question? Yeah? What do you think of my reasoning above? -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5211 bytes Desc: not available URL: From rsalz at akamai.com Tue Aug 29 19:21:57 2017 From: rsalz at akamai.com (Salz, Rich) Date: Tue, 29 Aug 2017 19:21:57 +0000 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: <8B137ECD-3077-48E6-B46B-16E0CF972134@ll.mit.edu> References: <2f79c2b6-560f-b245-b39d-dc4147d5ab44@openssl.org> <63C3E608-B02D-435F-9F50-CB93FE288DE5@akamai.com> <3AF1D54F-3A54-400F-8898-AED62D9AF67B@ll.mit.edu> <2D67CCC2-32E2-48B7-9E7A-D42C3D925AAC@akamai.com> <8B137ECD-3077-48E6-B46B-16E0CF972134@ll.mit.edu> Message-ID: ? I?d like to suggest that any approach other than ?immediately mix the received randomness into the RNG state? is bad. If a user does RAND_add() now, as opposed to 100 source code lines before, there may be a reason for that. I think the only way to do that in the DRBG model is to treat it as ?additional input? and do a generate call. I think that would achieve the same effect, but it wouldn?t delay any possible seeding of randomness that the DRBG itself needs at some point. ? One ?API? is how to get a reference/pointer/instance/whatever to the DRBG object responsible for the operation I?m now concerned with, and that I want to influence/improve. This would probably differ between per-SSL DRBG and per-thread DRBG, etc. I haven?t even thought about this part of the API (and I?m sure others on the team have more experience and understanding of the OpenSSL code to do it better than I would anyway). Yes, it is separate. But I want to make sure that if we are doing a comprehensive RAND overhaul that this is included in the requirements. From levitte at openssl.org Tue Aug 29 19:23:31 2017 From: levitte at openssl.org (Richard Levitte) Date: Tue, 29 Aug 2017 21:23:31 +0200 (CEST) Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: <168CEF3C-D655-47BF-9874-048308154AFD@ll.mit.edu> References: <3AF1D54F-3A54-400F-8898-AED62D9AF67B@ll.mit.edu> <20170829.210234.2290485732319802683.levitte@openssl.org> <168CEF3C-D655-47BF-9874-048308154AFD@ll.mit.edu> Message-ID: <20170829.212331.749934136233111186.levitte@openssl.org> In message <168CEF3C-D655-47BF-9874-048308154AFD at ll.mit.edu> on Tue, 29 Aug 2017 19:04:48 +0000, "Blumenthal, Uri - 0553 - MITLL" said: uri> uri> Could you please be more specific wrt. DRBG organization that in your opinion could impact the UI? uri> uri> Are you talking about the UI API or something else? uri> uri> No-no-no, just the UI API. uri> uri> I used the term ?UI? to emphasize that this is the ?public? part of the API, exposed to the user, expected to be called by the user, and (probably) designed to be stable for some (hopefully considerable) time. Ah, ok. In an OpenSSL context, this gets a bit confusing since there is an API called UI ( ). -- Richard Levitte levitte at openssl.org OpenSSL Project http://www.openssl.org/~levitte/ From uri at ll.mit.edu Tue Aug 29 19:55:14 2017 From: uri at ll.mit.edu (Blumenthal, Uri - 0553 - MITLL) Date: Tue, 29 Aug 2017 19:55:14 +0000 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: References: <2f79c2b6-560f-b245-b39d-dc4147d5ab44@openssl.org> <63C3E608-B02D-435F-9F50-CB93FE288DE5@akamai.com> <3AF1D54F-3A54-400F-8898-AED62D9AF67B@ll.mit.edu> <2D67CCC2-32E2-48B7-9E7A-D42C3D925AAC@akamai.com> <8B137ECD-3077-48E6-B46B-16E0CF972134@ll.mit.edu> Message-ID: <2545E83E-01DC-41A3-8B38-CCD300AABAE0@ll.mit.edu> On 8/29/17, 15:22, "openssl-dev on behalf of Salz, Rich via openssl-dev" wrote: ? I?d like to suggest that any approach other than ?immediately mix the received randomness into the RNG state? is bad. If a user does RAND_add() now, as opposed to 100 source code lines before, there may be a reason for that. > I think the only way to do that in the DRBG model is to treat it as ?additional input? and > do a generate call. I think that would achieve the same effect, but it wouldn?t delay any > possible seeding of randomness that the DRBG itself needs at some point. That is OK if the application manually invokes generate(). If it?s the RNG responsible for feeding key generators (who would call generate() transparently to user), then the only way I see is DRBG reseed(). Even though it implies a complete state replacement. AFAIK, there are three cases when randomness source(s) is pulled: a. At the instantiation time (page 29 of NIST SP 800-90A Rev1 draft). b. Upon reseed call (pages 30-31). c. During generation call(pages 32-36). ?Additional input? (i.e., what I?m after :) is present in two of the above: reseeding, and generation (pages 19 and 23). The draft is very clear that this ?additional input? shall not be counted towards the ?official? entropy (aka calculations when to pull the ?official? entropy sources). Which suits my purposes just fine. ;-) I realize that reseed() not only mixes my ?additional input? but also replaces the entire state. NIST does not specify interface to ?just? mix the ?additional input? into the state without replacing the whole state with some fresh entropy by calling Get_entropy_input(). Maybe we can provide such a function call (that?s what I think RAND_add() is supposed to do), but I?m not certain here? Wonder what others think? Do you believe that if you want to mix some more randomness into the RNG state to improve upon less-than-perfect entropy sources, you must also replace the entire state with the fresh pull from those sources? Or is it OK to just ?spice? the current state with ?additional input?? Note that regardless of the answer to the above, IMHO reseed() should be able to take ?additional input?. (NIST considers it optional, but sensitive/critical applications don?t. ;) ? One ?API? is how to get a reference/pointer/instance/whatever to the DRBG object responsible for the operation I?m now concerned with, ? and that I want to influence/improve. This would probably differ between per-SSL DRBG and per-thread DRBG, etc. I haven?t even ? thought about this part of the API (and I?m sure others on the team have more experience and understanding of the OpenSSL code ? to do it better than I would anyway). > Yes, it is separate. But I want to make sure that if we are doing a comprehensive RAND overhaul that this is included in the requirements. Yes, it should be included. But don?t let it stop us from defining/reviewing the ?clearer? part of the API. ;-) -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5211 bytes Desc: not available URL: From kurt at roeckx.be Tue Aug 29 20:32:33 2017 From: kurt at roeckx.be (Kurt Roeckx) Date: Tue, 29 Aug 2017 22:32:33 +0200 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: <9BEAA98D-EC21-46CA-88CB-829C609B82F7@ll.mit.edu> References: <2f79c2b6-560f-b245-b39d-dc4147d5ab44@openssl.org> <7b6ea5fcc69c4a65b3e98bf8620906c8@Ex13.ncp.local> <20170829163752.3qceqgckzybewmky@roeckx.be> <9BEAA98D-EC21-46CA-88CB-829C609B82F7@ll.mit.edu> Message-ID: <20170829203232.lpvfn5a3yr5tpxt5@roeckx.be> On Tue, Aug 29, 2017 at 06:50:37PM +0000, Blumenthal, Uri - 0553 - MITLL wrote: > On 8/29/17, 12:45, "openssl-dev on behalf of Salz, Rich via openssl-dev" wrote: > > ? An other problem with the current implemenation is that the > ? randomness parameter that's now given to RAND_add() is just > ? ignored, it assumes it's the same as the length. > > For what it?s worth, this was done deliberately, make RAND_add and RAND_seed equivalent. > > I am skeptical of the ability to get that estimate correct. > > Someone on GH there is a conversation thread about turning that into a percentage, which seems like the best thing to do for any new API. > > > What?s the point of having this potentially harmful parameter? If it weren?t ignored ? how would OpenSSL use it? > > If, based on its value, OpenSSL may decide that it now got ?enough? entropy and doesn?t need to pull more from other sources before serving randomness to requestors ? then it is harmful. ?Over-confidence? in this value by the caller can negatively impact the quality of the produced random numbers. As long as you have sources that don't provide 1 bit of randomness per bit that you provide you need to have an estimate of how much randomness it really contains. And you should probably seriously underestimate it so that you're sure that you collect enough. The problem with ignoring an existing parameter is that people could be calling it with for instance the value of 0, knowing it contains as good as none entropy. Or they could feed the unwithened output of an TRNG in that with an estimate of randomness it provides. And OpenSSL used to do the right thing with that. But now we just ignore it and assume every bit with get contains 1 bit of randomness and we're sundenly seriously overestimating the amount of randomness we're getting. This is a documented public API, you can't just go and ignore this parameter. Kurt From uri at ll.mit.edu Tue Aug 29 20:38:09 2017 From: uri at ll.mit.edu (Blumenthal, Uri - 0553 - MITLL) Date: Tue, 29 Aug 2017 20:38:09 +0000 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: <20170829203232.lpvfn5a3yr5tpxt5@roeckx.be> References: <2f79c2b6-560f-b245-b39d-dc4147d5ab44@openssl.org> <7b6ea5fcc69c4a65b3e98bf8620906c8@Ex13.ncp.local> <20170829163752.3qceqgckzybewmky@roeckx.be> <9BEAA98D-EC21-46CA-88CB-829C609B82F7@ll.mit.edu> <20170829203232.lpvfn5a3yr5tpxt5@roeckx.be> Message-ID: > If, based on its value, OpenSSL may decide that it now got ?enough? entropy and doesn?t need to > pull more from other sources before serving randomness to requestors ? then it is harmful. > ?Over-confidence? in this value by the caller can negatively impact the quality of the produced > random numbers. As long as you have sources that don't provide 1 bit of randomness per bit that you provide you need to have an estimate of how much randomness it really contains. And you should probably seriously underestimate it so that you're sure that you collect enough. So let me underestimate it to 0. ;-) The problem with ignoring an existing parameter is that people could be calling it with for instance the value of 0, knowing it contains as good as none entropy. Or they could feed the unwithened output of an TRNG in that with an estimate of randomness it provides. And OpenSSL used to do the right thing with that. I *don?t want* OpenSSL to make *any* estimation of the amount of provided entropy. All I want it to do is to mix these bits into the RNG state. It?s *my* business how much entropy I?m providing ? but I don?t want OpenSSL to make any decision regarding pull from other entropy sources based on my input. Does it sound reasonable? (He, it does to me ;) But now we just ignore it and assume every bit with get contains 1 bit of randomness and we're sundenly seriously overestimating the amount of randomness we're getting. If I had my way, you?d assume that every bit contains 0 bits of entropy, but mix it in regardless because that?s what the user is asking you to do. This is a documented public API, you can't just go and ignore this parameter. :-) -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5211 bytes Desc: not available URL: From rsalz at akamai.com Tue Aug 29 20:44:11 2017 From: rsalz at akamai.com (Salz, Rich) Date: Tue, 29 Aug 2017 20:44:11 +0000 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: <20170829203232.lpvfn5a3yr5tpxt5@roeckx.be> References: <2f79c2b6-560f-b245-b39d-dc4147d5ab44@openssl.org> <7b6ea5fcc69c4a65b3e98bf8620906c8@Ex13.ncp.local> <20170829163752.3qceqgckzybewmky@roeckx.be> <9BEAA98D-EC21-46CA-88CB-829C609B82F7@ll.mit.edu> <20170829203232.lpvfn5a3yr5tpxt5@roeckx.be> Message-ID: ? But now we just ignore it and assume every bit with get contains 1 ? bit of randomness and we're sundenly seriously overestimating the ? amount of randomness we're getting. This is a documented public API, ? you can't just go and ignore this parameter. Sure I can. Because the DRBG seeds from the system anyway From kurt at roeckx.be Tue Aug 29 21:39:58 2017 From: kurt at roeckx.be (Kurt Roeckx) Date: Tue, 29 Aug 2017 23:39:58 +0200 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: References: <2f79c2b6-560f-b245-b39d-dc4147d5ab44@openssl.org> <7b6ea5fcc69c4a65b3e98bf8620906c8@Ex13.ncp.local> <20170829163752.3qceqgckzybewmky@roeckx.be> <9BEAA98D-EC21-46CA-88CB-829C609B82F7@ll.mit.edu> <20170829203232.lpvfn5a3yr5tpxt5@roeckx.be> Message-ID: <20170829213958.pvsq3ymqm5zwajmh@roeckx.be> On Tue, Aug 29, 2017 at 08:38:09PM +0000, Blumenthal, Uri - 0553 - MITLL wrote: > > If, based on its value, OpenSSL may decide that it now got ?enough? entropy and doesn?t need to > > pull more from other sources before serving randomness to requestors ? then it is harmful. > > ?Over-confidence? in this value by the caller can negatively impact the quality of the produced > > random numbers. > > As long as you have sources that don't provide 1 bit of randomness > per bit that you provide you need to have an estimate of how much > randomness it really contains. And you should probably seriously > underestimate it so that you're sure that you collect enough. > > So let me underestimate it to 0. ;-) > > The problem with ignoring an existing parameter is that people > could be calling it with for instance the value of 0, knowing it > contains as good as none entropy. Or they could feed the unwithened > output of an TRNG in that with an estimate of randomness it provides. > And OpenSSL used to do the right thing with that. > > I *don?t want* OpenSSL to make *any* estimation of the amount of provided entropy. All I want it to do is to mix these bits into the RNG state. It?s *my* business how much entropy I?m providing ? but I don?t want OpenSSL to make any decision regarding pull from other entropy sources based on my input. > > Does it sound reasonable? (He, it does to me ;) But that is not the API that RAND_add() provides. It's a push not a pull API. With the DRBG you can do this, assuming using it's an extraction / derivative function. One of the suggestions I did before is to have RAND_poll_ex() take a parameter about how much randomness is needed, but I think it's also a wrong API and I'm thinking about removing it. > But now we just ignore it and assume every bit with get contains 1 > bit of randomness and we're sundenly seriously overestimating the > amount of randomness we're getting. > > If I had my way, you?d assume that every bit contains 0 bits of entropy, but mix it in regardless because that?s what the user is asking you to do. Which is why I suggested we use this for the additional data. But I think that as long as we have both APIs we might actually need it for the entropy input. If there is no other way to add randomness, RAND_add() is our current documented way to add it, and it will need to keep working. Kurt From kurt at roeckx.be Tue Aug 29 21:40:54 2017 From: kurt at roeckx.be (Kurt Roeckx) Date: Tue, 29 Aug 2017 23:40:54 +0200 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: References: <2f79c2b6-560f-b245-b39d-dc4147d5ab44@openssl.org> <7b6ea5fcc69c4a65b3e98bf8620906c8@Ex13.ncp.local> <20170829163752.3qceqgckzybewmky@roeckx.be> <9BEAA98D-EC21-46CA-88CB-829C609B82F7@ll.mit.edu> <20170829203232.lpvfn5a3yr5tpxt5@roeckx.be> Message-ID: <20170829214054.woxlt2vho77ngqql@roeckx.be> On Tue, Aug 29, 2017 at 08:44:11PM +0000, Salz, Rich via openssl-dev wrote: > Sure I can. Because the DRBG seeds from the system anyway You can't assume that will work for all users. Kurt From rsalz at akamai.com Tue Aug 29 23:02:59 2017 From: rsalz at akamai.com (Salz, Rich) Date: Tue, 29 Aug 2017 23:02:59 +0000 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: <20170829214054.woxlt2vho77ngqql@roeckx.be> References: <2f79c2b6-560f-b245-b39d-dc4147d5ab44@openssl.org> <7b6ea5fcc69c4a65b3e98bf8620906c8@Ex13.ncp.local> <20170829163752.3qceqgckzybewmky@roeckx.be> <9BEAA98D-EC21-46CA-88CB-829C609B82F7@ll.mit.edu> <20170829203232.lpvfn5a3yr5tpxt5@roeckx.be> <20170829214054.woxlt2vho77ngqql@roeckx.be> Message-ID: ? > Sure I can. Because the DRBG seeds from the system anyway ? You can't assume that will work for all users. And for places where the systesm doen?t have enough randomness, there is nothing we can do. From uri at ll.mit.edu Wed Aug 30 01:45:24 2017 From: uri at ll.mit.edu (Blumenthal, Uri - 0553 - MITLL) Date: Tue, 29 Aug 2017 21:45:24 -0400 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: <20170829213958.pvsq3ymqm5zwajmh@roeckx.be> References: <2f79c2b6-560f-b245-b39d-dc4147d5ab44@openssl.org> <7b6ea5fcc69c4a65b3e98bf8620906c8@Ex13.ncp.local> <20170829163752.3qceqgckzybewmky@roeckx.be> <9BEAA98D-EC21-46CA-88CB-829C609B82F7@ll.mit.edu> <20170829203232.lpvfn5a3yr5tpxt5@roeckx.be> <20170829213958.pvsq3ymqm5zwajmh@roeckx.be> Message-ID: <5654D35B-7102-49B3-AF13-1E754E4E2D33@ll.mit.edu> >> I *don?t want* OpenSSL to make *any* estimation of the amount of provided entropy. All I want it to do is to mix these bits into the RNG state. It?s *my* business how much entropy I?m providing ? but I don?t want OpenSSL to make any decision regarding pull from other entropy sources based on my input. >> >> Does it sound reasonable? (He, it does to me ;) > > But that is not the API that RAND_add() provides. Yes, I realize that, sort of. But nonetheless, that?s the interface/behavior that I think an RNG should offer. > It's a push not > a pull API. With the DRBG you can do this, assuming using it's an > extraction / derivative function. Not sure I understand. But ??you can do this?? sounds encouraging. ;-) > One of the suggestions I did before is to have RAND_poll_ex() take > a parameter about how much randomness is needed, but I think it's > also a wrong API and I'm thinking about removing it. There are two aspects: The RNG itself keeping track of how much entropy it has in its pool/state, and thus when to pull its (standard) entropy sources for more (and specifically for how much more). I don?t want to interfere with this part. The user-visible/accessible interface to RNG that allows: Forcing the ?reseed?, making the RNG to to its sources for ?fresh? entropy regardless of what?s up with its internal state; Telling the RNG to immediately mix the given bits into its state, without adjusting its count of available/present entropy. >> If I had my way, you?d assume that every bit contains 0 bits of entropy, but mix it in regardless because that?s what the user is asking you to do. > > Which is why I suggested we use this for the additional data. Yes, precisely. That?s exactly what I plan to use this interface for - mixing in additional data. NIST 800-90A Rev1 has ?additional input? only as part of Reseed() or Generate(). I can live with that (i.e., with the fact that right before my randomness is mixed in, the entire state s replaced). Though ideally we?d have ?RAND_add_ex()? or such that does not require replacing the entire state with fresh data from entropy source (because if that capability is needed, it is easy to emulate by issuing two calls: ?Reseed(); Add_ex();" > But > I think that as long as we have both APIs we might actually need > it for the entropy input. If there is no other way to add > randomness, RAND_add() is our current documented way to add it, > and it will need to keep working. It?s perfectly OK for me if RAND_add() keeps working. What I?d like to change about it is exactly when the mixing occurs. I?d like the state updated immediately upon RAND_add(), not when the RNG in question decides that it needs to replenish its entropy supply. I think this change would only improve security of applications that use it. TNX! -------------- next part -------------- An HTML attachment was scrubbed... URL: From leonb at parsec.co.za Wed Aug 30 04:55:43 2017 From: leonb at parsec.co.za (Leon Brits) Date: Wed, 30 Aug 2017 04:55:43 +0000 Subject: [openssl-dev] Upgrading OpenSSL In-Reply-To: <20170828173602.GB10656@openssl.org> References: <13e455db80e548888c9cd15424bb146a@EXCHANGESRV.PARSEC.local> <20170828173602.GB10656@openssl.org> Message-ID: I now have Racoon2 working. Steve's comment made me think about the digests used in Racoon2 and I went searching for any commands using SHA1. I found two hardcoded as string "SHA1". I changed it to SHA256 and bobs-your-uncle. I guess this is due to the phasing-out of the SHA1 hash which was not true in the days of v0.9.8? Sorry for using you guys as a soundboard, but I have no other option. Thanks for all your help Regards, LJB From paul.dale at oracle.com Wed Aug 30 04:59:57 2017 From: paul.dale at oracle.com (Paul Dale) Date: Tue, 29 Aug 2017 21:59:57 -0700 (PDT) Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: References: Message-ID: <97ae072a-e9c4-4803-8daf-4c9822a41947@default> My thoughts are that the new RNG API should be made public once it has been properly designed. We've a chance to get this right, let's take the time and make an effort to do so. There is no rush. I also feel that having an engine interface to the new RNG API is worthwhile. It allows hardware sources to be used via the same API. Again, this will require design work to get right. Likewise, supporting multiple DRBGs seems reasonable from a defence in depth perspective. I would like to see an entropy argument to the equivalent to RAND_add. Anyone is welcome to always pass zero in but there should be the option to not. Consider an hardware source with _provable_ output quality, why shouldn't it be allowed to meaningfully contribute? Each RNG should decide if it uses the argument or not. Thus, a DRBG that uses RAND_add to provide additional data needn't count it. I like the idea of two independent global RNGs. Keeping the generation of long lived key material segregated from other uses of randomness seems sensible -- there is no possibility of cross compromise. This does increase seeding requirements however. Pauli -- Oracle Dr Paul Dale | Cryptographer | Network Security & Encryption Phone +61 7 3031 7217 Oracle Australia -----Original Message----- From: Dr. Matthias St. Pierre [mailto:Matthias.St.Pierre at ncp-e.com] Sent: Tuesday, 29 August 2017 7:45 PM To: openssl-dev at openssl.org Subject: [openssl-dev] Plea for a new public OpenSSL RNG API Hi everybody, on the [openssl-dev] mailing list, there has been a long ongoing discussion about the new RAND_DRBG API and comparing it with the old RAND_METHOD API (see "[openssl-dev] Work on a new RNG for OpenSSL"). Two of the most controversal questions were: - Do we really need a new RNG API? Should the RAND_DRBG API be made public or kept private? (Currently, it's exported from libcrypto but only used internally by libssl.) - How much control should the user (programmer) be given over the reseeding process and/or should he be allowed to add his own additional randomness? Many developers seem to be realizing the interesting possibilities of the DRBG API and are asking for public access to this new and promising API. One of the driving forces behind it is the question about how to do seeding and reseeding right. Among others, Uri Blumenthal asked for making the DRBG API public. Currently, the OpenSSL core members seem to be reluctant to make the API public, at least at this early stage. I understand Rich Salz's viewpoint that this requires a thorough discussion, because a public interface can't be easily changed and wrong decisions in the early phase can become a heavy burdon. Nevertheless, I agree with Uri Blumenthal that the DRBG API should be made public. So here comes my ====================================== Plea for a new public OpenSSL RNG API: ====================================== The new RAND_DRBG is the superior API. It shouldn't be kept private and hidden behind the ancient RAND_METHOD API. The philosophy of the two APIs is not very well compatible, in particular when it comes to reseeding and adding additional unpredictable input. Hiding the RAND_DRBG behind the RAND_METHOD API only causes problems. Also, it will force people to patch their OpenSSL copy if they want to use the superior API. The RAND_DRBG API should become the new public OpenSSL RNG API and the old RAND_METHOD API should be deprecated in the long run. This transition does not need to be rushed, but it would be good if there would be early consent on the road map. I am thinking of a smooth transition with a phase of coexistence and a compatibility layer mapping the default RAND_METHOD to the default public RAND_DRBG instance. (This compatibility layer already exists, it's the 'RAND_OpenSSL()' method.) Historical Background ===================== As Rich already mentioned in his blog post, the RAND_DRBG isn't new. It's been a part of OpenSSL for a long time, hidden in the FIPS 2.0 Object Module. I have been working with the FIPS DRBG for quite a while now, using a FIPS-capable OpenSSL 1.0.2x crypto library. The reason why our company switched to the FIPS DRBG is that one of our products runs on a small hardware device which does not have a reliable entropy source, but the product has to meet high security standards, in particular w.r.t. its RNG. So we decided to use the SmartCard RNG as primary entropy source for a deterministic AES-CTR based RNG and use /dev/urandom as additional input. Reseeding should occur on every generate request. Using the FIPS DRBG, these requirements were easily met, because the API gives such a fine grained control over reseeding and adding additional entropy. The DRBG was well documented, its design in NIST SP800-90A (now: NIST SP800-90Ar1) and its API in the OpenSSL FIPS 2.0 User Guide. The implementation was thoroughly tested and assessed during the FIPS certification process. So the only minor obstacle was that we had to patch the crypto library (not the FIPS object module) in order to get public access to the FIPS_drbg_*() methods. I always considered the DRBG API more mature than the good old RAND_METHOD API and I wondered, why the DRBG code lay forgotten for so many years in the FIPS 2.0 object module sources and was never ported to master. When in June of this year the thread "[openssl-dev] Work on a new RNG for OpenSSL" popped up (https://mta.openssl.org/pipermail/openssl-dev/2017-June), I closely watched the discussion, and when John Denker suggested having a look at NIST SP800-90A, I was electrified: > Constructive suggestion: If you want to see what a RNG looks > like when designed by cryptographers, take a look at: > Elaine Barker and John Kelsey, > "Recommendation for Random Number Generation Using Deterministic Random Bit Generators" > http://csrc.nist.gov/publications/nistpubs/800-90A/SP800-90A.pdf > > That design may look complicated, but if you think you can > leave out some of the blocks in their diagram, proceed with > caution. Every one of those blocks is there for a reason. >From his mail and the reaction to it, I had the impression that nobody seemed to remember the fact that the DRBG code was already present in the FIPS object module. The hidden treasure seemed forgotten! When I seized the opportunity and proposed to port the FIPS DRBG code into master, Rich Salz liked my idea and immediately started working on PR #3789: I am very grateful to Rich for picking up the idea so fast and and giving it so much speed and momentum. He has done a lot of work to convince others and defend the idea against initial scepticism by voices objecting to the seemingly new and unknown API. In the middle of August, when the first bulk of work was finished, Rich wrote a detailed blog post to advertise the new DRBG and explain his work and the discussion restarted: "[openssl-dev] Work on a new RNG for OpenSSL> (see https://mta.openssl.org/pipermail/openssl-dev/2017-August) It became quickly evident that users that the DRBG API had promising features, so they started asking for public access to this new API. The driving force was the question about how to do seeding and reseeding right. Among others, Uri Blumenthal was a dedicated advocate to make the DRBG API public (https://mta.openssl.org/pipermail/openssl-dev/2017-August/009594.html). But the OpenSSL members currently seem to be reluctant to make the API public right away. I understand Rich's viewpoint that this decision requires a thorough discussion, because a public interface can't be easily changed and wrong decisions in the early phase can become a heavy burdon. Nevertheless, I agree with Uri Blumenthal that the DRBG API should be made public. Here are some of my arguments for it. The DRBG API supports multiple instances and chaining ===================================================== The NIST DRBG standard had chaining of multiple DRBG instances in the mind from the very beginnning, see for example footnote [4] on page 25 of NIST SP800-90Ar1: > Entropy input may be obtained from an entropy source or an NRBG, both of which provide fresh entropy. > Entropy input could also be obtained from a DRBG that has access to an entropy source or NRBG. The original OpenSSL FIPS DRBG implementation did not support chaining, but this support has been added by Rich during the DRBG port. The DRBG API has a highly flexible concept for seeding and reseeding ==================================================================== As mentioned previously, the DRBG has a callback mechanism with callbacks like get_entropy() and get_adin() callbacks, which make it easy to fine tune the default instantiation process by either adding additional randomness input, or changing the entropy source entirely. The callbacks are primarily intended for obtaining randomness instantiation and reseeing. There is a clear concept for reseeding, which can be adjusted by changing the reseed_interval: Normally, the DRBG reseeds itself automatically, whenever the reseed_interval has been reached. In addition, there is a way for the DRBG consumer to add his own unpredictability when requesting random bytes: by adding additional input 'adin' to the RAND_DRBG_generate() call: int RAND_DRBG_generate(RAND_DRBG *drbg, unsigned char *out, size_t outlen, int prediction_resistance, const unsigned char *adin, size_t adinlen) So why are there so many ways to add randomness and additional input? And what is the difference? * The get_entropy() and get_adin() callbacks are used by the DRBG itself to _pull_ unpredictable data from some backend entropy source (which can also be a chained DRBG which is connected to some entropy source) during instantiation or reseeding. * The 'adin' argument can be used by the DRBG consumer to _push_ unpredictable input into the DRBG when generating random output. Note that the DRBG consumer has no possibility to push out-of-band randomness into the DRBG. Adding randomness is always coupled with a generate or (re-)seed operation. This is a very import difference between the RAND_DRBG and the RAND_METHOD and one of the reasons why it's so hard to do reseeding right in both the RAND and RAND_DRBG API simultaneously. In fact, currently The 'RAND_add()/RAND_bytes()' pattern is broken =============================================== In OpenSSL, the classical way for the RNG consumer to add his own randomness is to call 'RAND_add()' before calling 'RAND_bytes()'. If the new 'RAND_OpenSSL()' method (the "compatibility layer" hiding the public RAND_DRBG instance) is the default, then this does not work as expected anymore: The reason is that a call to 'RAND_add()' adds the provided randomness only to a global buffer ('rand_bytes'), from which it will be pulled during the next reseed. But no reseed is triggered. So the next RAND_bytes() call will be unaffected from the RAND_add(), which is not what the consumer expected. (The same holds for 'RAND_seed()', since 'drbg_seed()' only calls into 'drbg_add()') Reseeding of DRBGs occurs only at the following occasions: * immediately after a 'fork()' (new) * if the 'reseed_counter' exceeds the 'reseed_interval' * if 'RAND_DRBG_generate()' is called requesting 'prediction_resistance' * 'RAND_DRBG_reseed()' is called explicitely *Note:* Currently it looks like the situation is even worse: if 'RAND_add()' is called multiple times before a reseed occurs, then the result of the previous call is overwritten. Reseeding the 'DRBG' whenever the user calls 'RAND_add()' does not seem a good solution. It would be too expensive, in particular if system entropy is pulled for reseeding. Of course it is possible to fix this issue, but the DRBG provides for a much simpler solution: it lets the consumer contribute to the entropy of the internal state by providing additional input. If the user input contains entropy, that's fine, if it's "snake oil", no harm. The additional input is mixed into the internal state in just the same way as the entropy buffer using the 'ctr_df()' derivation function. One might think of the 'entropy' input as trusted randomness and 'adin' as untrusted randomness. For this reason, I would like to see the 'RAND_add()/RAND_bytes()' pattern deprecated and the 'RAND_DRBG_generate() with additional input' pattern advertised instead. The DRBG API supports different implementations =============================================== Well, it _supported_ them, until recently. But that's not irreversible. The DRBG concept, as layed out in the NIST standard, provides a generic framework for deterministic RNGs (the acronym DRBG stands for Deterministic Random Bit Generator). It deals with general questions like how to instantiate and reseed the RNG, where does it get the entropy from, etc. The standard proposes three concrete implementations, Hash_DRBG, HMAC_DRBG, and CTR_DRBG. In the FIPS code, all three were implemented, and the genericity was achieved using a data union and a set of five function pointers, reminiscent of a vtable in C++: struct drbg_ctx_st { ... /* Implementation specific structures */ union { DRBG_HASH_CTX hash; DRBG_HMAC_CTX hmac; DRBG_CTR_CTX ctr; } d; /* Initialiase PRNG and setup callbacks below */ int (*init)(DRBG_CTX *ctx, int nid, int security, unsigned int flags); /* Intantiate PRNG */ int (*instantiate)(DRBG_CTX *ctx, const unsigned char *ent, size_t entlen, const unsigned char *nonce, size_t noncelen, const unsigned char *pers, size_t perslen); /* reseed */ int (*reseed)(DRBG_CTX *ctx, const unsigned char *ent, size_t entlen, const unsigned char *adin, size_t adinlen); /* generat output */ int (*generate)(DRBG_CTX *ctx, unsigned char *out, size_t outlen, const unsigned char *adin, size_t adinlen); /* uninstantiate */ int (*uninstantiate)(DRBG_CTX *ctx); ... }; This part of the code was removed during the DRBG port, because currently CTR_DRBG is the only impementation. I would like to suggest to restore this 'polymorphic' implementation, to ease adding new implementations CHACHA20_DRBG in the future The DRBG API is well documented and tested ========================================== The entire DRBG API is part of the OpenSSL FIPS 2.0 Module and as such is well tested and well documented. Most of the documentation for the manual pages still-to-be-written can be taken from the FIPS User Guide and converted into manual pages, starting with textual modifications like FIPS_drbg_xxx(...) -> RAND_DRBG_xxx(...) DRBG_CTX *dctx -> RAND_DRBG *dctx and taking the new typedefs into consideration. Here is for example a comparison of the API function to install the callbacks: FIPS DRBG: int FIPS_drbg_set_callbacks(DRBG_CTX *dctx, size_t (*get_entropy)(DRBG_CTX *ctx, ), void (*cleanup_entropy)(DRBG_CTX *ctx, ), size_t entropy_blocklen, size_t (*get_nonce)(DRBG_CTX *ctx, ), void (*cleanup_nonce)(DRBG_CTX *ctx, ) ); RAND_DRBG: typedef size_t (*RAND_DRBG_get_entropy_fn)(RAND_DRBG *ctx, ); typedef void (*RAND_DRBG_cleanup_entropy_fn)(RAND_DRBG *ctx, ); typedef size_t (*RAND_DRBG_get_nonce_fn)(RAND_DRBG *ctx, ); typedef void (*RAND_DRBG_cleanup_nonce_fn)(RAND_DRBG *ctx, ); int RAND_DRBG_set_callbacks(RAND_DRBG *dctx, RAND_DRBG_get_entropy_fn get_entropy, RAND_DRBG_cleanup_entropy_fn cleanup_entropy, RAND_DRBG_get_nonce_fn get_nonce, RAND_DRBG_cleanup_nonce_fn cleanup_nonce); Conclusion ========== I see no reason why the RAND_DRBG shouldn't be made public as soon as possible, keeping the API as close as possible to the original FIPS DRBG API (FIPS 3.0 is upcoming!). In a second step, the current compatibility binding from RAND_METHOD to RAND_DRBG could be deprecated and faded out smoothly. Looking forward to receiving your comments. (But please be patient with me, I'm currently on physical rehab after a surgery.) Matthias St. Pierre From Matthias.St.Pierre at ncp-e.com Wed Aug 30 05:54:45 2017 From: Matthias.St.Pierre at ncp-e.com (Dr. Matthias St. Pierre) Date: Wed, 30 Aug 2017 05:54:45 +0000 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: References: <2f79c2b6-560f-b245-b39d-dc4147d5ab44@openssl.org> <63C3E608-B02D-435F-9F50-CB93FE288DE5@akamai.com> <3AF1D54F-3A54-400F-8898-AED62D9AF67B@ll.mit.edu> <2D67CCC2-32E2-48B7-9E7A-D42C3D925AAC@akamai.com> <8B137ECD-3077-48E6-B46B-16E0CF972134@ll.mit.edu> Message-ID: <728dae1ae650419fb63257d688b29608@Ex13.ncp.local> Hi Rich, > I think the only way to do that in the DRBG model is to treat it as ?additional input? and do a generate > call. I think that would achieve the same effect, but it wouldn?t delay any possible seeding of randomness > that the DRBG itself needs at some point. You're right, well, almost. The implementation of drbg_add() (which implements RAND_add()) could be actually as simple as this little function: static int drbg_add(const void *buf, int num, double randomness) { int ret = 0; CRYPTO_THREAD_write_lock(rand_drbg.lock); ret = RAND_DRBG_reseed(&rand_drbg, buf, num); err: CRYPTO_THREAD_unlock(rand_drbg.lock); return ret; } The problem is that currently this would create a circular dependency, because now RAND_add() is involved both in pushing and in pulling entropy. RAND_add() -> drbg_add() -> RAND_DRBG_reseed() -> get_entropy() -> get_entropy_from_system() -> RAND_poll_ex() -> RAND_add() Using RAND_DRBG_generate() instead of RAND_DRBG_reseed() does not solve the problem, because the former can occasionally trigger a reseed. I already have the solution in my mind how to break up this vicious circle, but I need some time to write it down. Here is a sketch: 1) Use RAND_add() only for pushing to break the circular dependency =================================================================== Add a new function rand_drbg_add() which is identical to RAND_DRBG_reseed(), except that it does not pull fresh entropy (and consequently does not reset the reseed_counter). Then the left hand side of the becomes RAND_add() -> drbg_add() -> rand_drbg_add() The code of drbg_add() and rand_drbg_add() is at the end of this mail. (This is the easy part :-) ) 2) Reorganize the backend code for pulling entropy ================================================== Reorganize get_entropy_from_system() in a way that it collects the entropy in the way it was originally intended, without calling RAND_add() and without using this evil fixed size 'rand_bytes' buffer, from which I tried to dissuade you ever since, Rich (no offence ;-) ) This part is more complicated, because one has to be careful not to break RAND_poll() for applications which don't use the default RAND_OpenSSL() method. So please give me some time, I will try to turn my ideas into a PR. Matthias ___ drbg_lib.c: /* * Mix in the specified data without reseeding |drbg|. * * Used by drbg_add() only. * * Similar to RAND_DRBG_reseed(), but there is no call to * get_entropy() and the reseed_counter is not reset */ int rand_drbg_add(RAND_DRBG *drbg, const unsigned char *adin, size_t adinlen) { if (drbg->state == DRBG_ERROR) { RANDerr(RAND_F_RAND_DRBG_RESEED, RAND_R_IN_ERROR_STATE); return 0; } if (drbg->state == DRBG_UNINITIALISED) { RANDerr(RAND_F_RAND_DRBG_RESEED, RAND_R_NOT_INSTANTIATED); return 0; } if (adin == NULL) adinlen = 0; else if (adinlen > drbg->max_adinlen) { RANDerr(RAND_F_RAND_DRBG_RESEED, RAND_R_ADDITIONAL_INPUT_TOO_LONG); return 0; } drbg->state = DRBG_ERROR; if (!ctr_reseed(drbg, NULL, 0, adin, adinlen)) goto end; drbg->state = DRBG_READY; end: if (drbg->state == DRBG_READY) return 1; return 0; } ... static int drbg_add(const void *buf, int num, double randomness) { int ret = 0; CRYPTO_THREAD_write_lock(rand_drbg.lock); ret = rand_drbg_add(&rand_drbg, buf, num); err: CRYPTO_THREAD_unlock(rand_drbg.lock); return ret; } -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 4328 bytes Desc: not available URL: From Matthias.St.Pierre at ncp-e.com Wed Aug 30 06:09:09 2017 From: Matthias.St.Pierre at ncp-e.com (Dr. Matthias St. Pierre) Date: Wed, 30 Aug 2017 06:09:09 +0000 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: <2545E83E-01DC-41A3-8B38-CCD300AABAE0@ll.mit.edu> References: <2f79c2b6-560f-b245-b39d-dc4147d5ab44@openssl.org> <63C3E608-B02D-435F-9F50-CB93FE288DE5@akamai.com> <3AF1D54F-3A54-400F-8898-AED62D9AF67B@ll.mit.edu> <2D67CCC2-32E2-48B7-9E7A-D42C3D925AAC@akamai.com> <8B137ECD-3077-48E6-B46B-16E0CF972134@ll.mit.edu> <2545E83E-01DC-41A3-8B38-CCD300AABAE0@ll.mit.edu> Message-ID: > I realize that reseed() not only mixes my ?additional input? but also replaces the entire state. NIST does > not specify interface to ?just? mix the ?additional input? into the state without replacing the whole state > with some fresh entropy by calling Get_entropy_input(). Maybe we can provide such a function call (that?s > what I think RAND_add() is supposed to do), but I?m not certain here? The reseed() operation pulls fresh entropy, but the old state is not discarded. Instead, the fresh entropy is mixed in, just like the additional input (and the reseed_counter is reset). Have a look at ctr_update() and ctr_df() and you will see that the 'entropy' and the 'adin' input are treated the same way to change the internal state. The derivation function treats 'entropy' and 'adin' equally, it's just a difference in interpretation (trusted vs. untrusted randomness). The NIST document has also a lot of nice graphics to explain the inner workings. This helped me a lot to understand the details. Matthias -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 4328 bytes Desc: not available URL: From Matthias.St.Pierre at ncp-e.com Wed Aug 30 07:08:12 2017 From: Matthias.St.Pierre at ncp-e.com (Dr. Matthias St. Pierre) Date: Wed, 30 Aug 2017 07:08:12 +0000 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: <1bcf6fcb-fff7-0cdb-562e-1651dfea2ded@openssl.org> References: <2f79c2b6-560f-b245-b39d-dc4147d5ab44@openssl.org> <63C3E608-B02D-435F-9F50-CB93FE288DE5@akamai.com> <1bcf6fcb-fff7-0cdb-562e-1651dfea2ded@openssl.org> Message-ID: <57c58e9f8ef144a79bd763bc0ae90a6d@Ex13.ncp.local> > -----Urspr?ngliche Nachricht----- > Von: openssl-dev [mailto:openssl-dev-bounces at openssl.org] Im Auftrag von Matt Caswell > Gesendet: Dienstag, 29. August 2017 16:36 > An: openssl-dev at openssl.org > Betreff: Re: [openssl-dev] Plea for a new public OpenSSL RNG API > > > > On 29/08/17 15:02, Salz, Rich via openssl-dev wrote: > > > > dev asap. If there are problems with it we can always revert it before > > it makes it into a release. > > > > Someone on the OMC called that ?flip-flopping? and seemed to be against this kind of thing. > > > > If we are going to have an additional API, then we should at least see a draft of the header file first. > > Matthias, would you be willing to create such a draft as a github PR > (marked as WIP for now)? > > Matt Oh thanks, that's a great honor to be asked to do this. I will give it a thought. Currently, the RAND_add()/RAND_bytes() problem needs to fixed first, so it will take a little while. Matthias -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 4328 bytes Desc: not available URL: From uri at ll.mit.edu Wed Aug 30 14:26:44 2017 From: uri at ll.mit.edu (Blumenthal, Uri - 0553 - MITLL) Date: Wed, 30 Aug 2017 14:26:44 +0000 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: <97ae072a-e9c4-4803-8daf-4c9822a41947@default> References: <97ae072a-e9c4-4803-8daf-4c9822a41947@default> Message-ID: <39B4D013-3F63-421C-820C-40F21E15C7EC@ll.mit.edu> On 8/30/17, 00:59, "openssl-dev on behalf of Paul Dale" wrote: > My thoughts are that the new RNG API should be made public once it has > been properly designed. We've a chance to get this right, let's take the time > and make an effort to do so. There is no rush. Not quite. If there is an RNG involved in generating long-term keys now ? users better be able to control/affect it now. > I also feel that having an engine interface to the new RNG API is worthwhile. +1 > It allows hardware sources to be used via the same API. I rather doubt this. For example, my smartcard (accessible via PKCS#11) is a hardware source, which I occasionally use. How do you see it used with the same API? > I would like to see an entropy argument to the equivalent to RAND_add. > Anyone is welcome to always pass zero in but there should be the option to not. > Consider an hardware source with _provable_ output quality, why shouldn't it be > allowed to meaningfully contribute? What?s the purpose of passing the entropy argument? How is the callee (the RNG) going to use it? Why should the OpenSSL code in general trust the received value (how can OpenSSL tell that the received randomness is indeed from a hardware source with provable output quality)? Finally, what does it matter? And they all ?meaningfully contribute?. The only question is whether this contribution should prevent RNG from acquiring more entropy from other sources. My opinion is resounding no. > I like the idea of two independent global RNGs. +1 ;-) > This does increase seeding requirements however. If you can seed one, you can seed two. -----Original Message----- From: Dr. Matthias St. Pierre [mailto:Matthias.St.Pierre at ncp-e.com] Sent: Tuesday, 29 August 2017 7:45 PM To: openssl-dev at openssl.org Subject: [openssl-dev] Plea for a new public OpenSSL RNG API Hi everybody, on the [openssl-dev] mailing list, there has been a long ongoing discussion about the new RAND_DRBG API and comparing it with the old RAND_METHOD API (see "[openssl-dev] Work on a new RNG for OpenSSL"). Two of the most controversal questions were: - Do we really need a new RNG API? Should the RAND_DRBG API be made public or kept private? (Currently, it's exported from libcrypto but only used internally by libssl.) - How much control should the user (programmer) be given over the reseeding process and/or should he be allowed to add his own additional randomness? Many developers seem to be realizing the interesting possibilities of the DRBG API and are asking for public access to this new and promising API. One of the driving forces behind it is the question about how to do seeding and reseeding right. Among others, Uri Blumenthal asked for making the DRBG API public. Currently, the OpenSSL core members seem to be reluctant to make the API public, at least at this early stage. I understand Rich Salz's viewpoint that this requires a thorough discussion, because a public interface can't be easily changed and wrong decisions in the early phase can become a heavy burdon. Nevertheless, I agree with Uri Blumenthal that the DRBG API should be made public. So here comes my ====================================== Plea for a new public OpenSSL RNG API: ====================================== The new RAND_DRBG is the superior API. It shouldn't be kept private and hidden behind the ancient RAND_METHOD API. The philosophy of the two APIs is not very well compatible, in particular when it comes to reseeding and adding additional unpredictable input. Hiding the RAND_DRBG behind the RAND_METHOD API only causes problems. Also, it will force people to patch their OpenSSL copy if they want to use the superior API. The RAND_DRBG API should become the new public OpenSSL RNG API and the old RAND_METHOD API should be deprecated in the long run. This transition does not need to be rushed, but it would be good if there would be early consent on the road map. I am thinking of a smooth transition with a phase of coexistence and a compatibility layer mapping the default RAND_METHOD to the default public RAND_DRBG instance. (This compatibility layer already exists, it's the 'RAND_OpenSSL()' method.) Historical Background ===================== As Rich already mentioned in his blog post, the RAND_DRBG isn't new. It's been a part of OpenSSL for a long time, hidden in the FIPS 2.0 Object Module. I have been working with the FIPS DRBG for quite a while now, using a FIPS-capable OpenSSL 1.0.2x crypto library. The reason why our company switched to the FIPS DRBG is that one of our products runs on a small hardware device which does not have a reliable entropy source, but the product has to meet high security standards, in particular w.r.t. its RNG. So we decided to use the SmartCard RNG as primary entropy source for a deterministic AES-CTR based RNG and use /dev/urandom as additional input. Reseeding should occur on every generate request. Using the FIPS DRBG, these requirements were easily met, because the API gives such a fine grained control over reseeding and adding additional entropy. The DRBG was well documented, its design in NIST SP800-90A (now: NIST SP800-90Ar1) and its API in the OpenSSL FIPS 2.0 User Guide. The implementation was thoroughly tested and assessed during the FIPS certification process. So the only minor obstacle was that we had to patch the crypto library (not the FIPS object module) in order to get public access to the FIPS_drbg_*() methods. I always considered the DRBG API more mature than the good old RAND_METHOD API and I wondered, why the DRBG code lay forgotten for so many years in the FIPS 2.0 object module sources and was never ported to master. When in June of this year the thread "[openssl-dev] Work on a new RNG for OpenSSL" popped up (https://mta.openssl.org/pipermail/openssl-dev/2017-June), I closely watched the discussion, and when John Denker suggested having a look at NIST SP800-90A, I was electrified: > Constructive suggestion: If you want to see what a RNG looks > like when designed by cryptographers, take a look at: > Elaine Barker and John Kelsey, > "Recommendation for Random Number Generation Using Deterministic Random Bit Generators" > http://csrc.nist.gov/publications/nistpubs/800-90A/SP800-90A.pdf > > That design may look complicated, but if you think you can > leave out some of the blocks in their diagram, proceed with > caution. Every one of those blocks is there for a reason. From his mail and the reaction to it, I had the impression that nobody seemed to remember the fact that the DRBG code was already present in the FIPS object module. The hidden treasure seemed forgotten! When I seized the opportunity and proposed to port the FIPS DRBG code into master, Rich Salz liked my idea and immediately started working on PR #3789: I am very grateful to Rich for picking up the idea so fast and and giving it so much speed and momentum. He has done a lot of work to convince others and defend the idea against initial scepticism by voices objecting to the seemingly new and unknown API. In the middle of August, when the first bulk of work was finished, Rich wrote a detailed blog post to advertise the new DRBG and explain his work and the discussion restarted: "[openssl-dev] Work on a new RNG for OpenSSL> (see https://mta.openssl.org/pipermail/openssl-dev/2017-August) It became quickly evident that users that the DRBG API had promising features, so they started asking for public access to this new API. The driving force was the question about how to do seeding and reseeding right. Among others, Uri Blumenthal was a dedicated advocate to make the DRBG API public (https://mta.openssl.org/pipermail/openssl-dev/2017-August/009594.html). But the OpenSSL members currently seem to be reluctant to make the API public right away. I understand Rich's viewpoint that this decision requires a thorough discussion, because a public interface can't be easily changed and wrong decisions in the early phase can become a heavy burdon. Nevertheless, I agree with Uri Blumenthal that the DRBG API should be made public. Here are some of my arguments for it. The DRBG API supports multiple instances and chaining ===================================================== The NIST DRBG standard had chaining of multiple DRBG instances in the mind from the very beginnning, see for example footnote [4] on page 25 of NIST SP800-90Ar1: > Entropy input may be obtained from an entropy source or an NRBG, both of which provide fresh entropy. > Entropy input could also be obtained from a DRBG that has access to an entropy source or NRBG. The original OpenSSL FIPS DRBG implementation did not support chaining, but this support has been added by Rich during the DRBG port. The DRBG API has a highly flexible concept for seeding and reseeding ==================================================================== As mentioned previously, the DRBG has a callback mechanism with callbacks like get_entropy() and get_adin() callbacks, which make it easy to fine tune the default instantiation process by either adding additional randomness input, or changing the entropy source entirely. The callbacks are primarily intended for obtaining randomness instantiation and reseeing. There is a clear concept for reseeding, which can be adjusted by changing the reseed_interval: Normally, the DRBG reseeds itself automatically, whenever the reseed_interval has been reached. In addition, there is a way for the DRBG consumer to add his own unpredictability when requesting random bytes: by adding additional input 'adin' to the RAND_DRBG_generate() call: int RAND_DRBG_generate(RAND_DRBG *drbg, unsigned char *out, size_t outlen, int prediction_resistance, const unsigned char *adin, size_t adinlen) So why are there so many ways to add randomness and additional input? And what is the difference? * The get_entropy() and get_adin() callbacks are used by the DRBG itself to _pull_ unpredictable data from some backend entropy source (which can also be a chained DRBG which is connected to some entropy source) during instantiation or reseeding. * The 'adin' argument can be used by the DRBG consumer to _push_ unpredictable input into the DRBG when generating random output. Note that the DRBG consumer has no possibility to push out-of-band randomness into the DRBG. Adding randomness is always coupled with a generate or (re-)seed operation. This is a very import difference between the RAND_DRBG and the RAND_METHOD and one of the reasons why it's so hard to do reseeding right in both the RAND and RAND_DRBG API simultaneously. In fact, currently The 'RAND_add()/RAND_bytes()' pattern is broken =============================================== In OpenSSL, the classical way for the RNG consumer to add his own randomness is to call 'RAND_add()' before calling 'RAND_bytes()'. If the new 'RAND_OpenSSL()' method (the "compatibility layer" hiding the public RAND_DRBG instance) is the default, then this does not work as expected anymore: The reason is that a call to 'RAND_add()' adds the provided randomness only to a global buffer ('rand_bytes'), from which it will be pulled during the next reseed. But no reseed is triggered. So the next RAND_bytes() call will be unaffected from the RAND_add(), which is not what the consumer expected. (The same holds for 'RAND_seed()', since 'drbg_seed()' only calls into 'drbg_add()') Reseeding of DRBGs occurs only at the following occasions: * immediately after a 'fork()' (new) * if the 'reseed_counter' exceeds the 'reseed_interval' * if 'RAND_DRBG_generate()' is called requesting 'prediction_resistance' * 'RAND_DRBG_reseed()' is called explicitely *Note:* Currently it looks like the situation is even worse: if 'RAND_add()' is called multiple times before a reseed occurs, then the result of the previous call is overwritten. Reseeding the 'DRBG' whenever the user calls 'RAND_add()' does not seem a good solution. It would be too expensive, in particular if system entropy is pulled for reseeding. Of course it is possible to fix this issue, but the DRBG provides for a much simpler solution: it lets the consumer contribute to the entropy of the internal state by providing additional input. If the user input contains entropy, that's fine, if it's "snake oil", no harm. The additional input is mixed into the internal state in just the same way as the entropy buffer using the 'ctr_df()' derivation function. One might think of the 'entropy' input as trusted randomness and 'adin' as untrusted randomness. For this reason, I would like to see the 'RAND_add()/RAND_bytes()' pattern deprecated and the 'RAND_DRBG_generate() with additional input' pattern advertised instead. The DRBG API supports different implementations =============================================== Well, it _supported_ them, until recently. But that's not irreversible. The DRBG concept, as layed out in the NIST standard, provides a generic framework for deterministic RNGs (the acronym DRBG stands for Deterministic Random Bit Generator). It deals with general questions like how to instantiate and reseed the RNG, where does it get the entropy from, etc. The standard proposes three concrete implementations, Hash_DRBG, HMAC_DRBG, and CTR_DRBG. In the FIPS code, all three were implemented, and the genericity was achieved using a data union and a set of five function pointers, reminiscent of a vtable in C++: struct drbg_ctx_st { ... /* Implementation specific structures */ union { DRBG_HASH_CTX hash; DRBG_HMAC_CTX hmac; DRBG_CTR_CTX ctr; } d; /* Initialiase PRNG and setup callbacks below */ int (*init)(DRBG_CTX *ctx, int nid, int security, unsigned int flags); /* Intantiate PRNG */ int (*instantiate)(DRBG_CTX *ctx, const unsigned char *ent, size_t entlen, const unsigned char *nonce, size_t noncelen, const unsigned char *pers, size_t perslen); /* reseed */ int (*reseed)(DRBG_CTX *ctx, const unsigned char *ent, size_t entlen, const unsigned char *adin, size_t adinlen); /* generat output */ int (*generate)(DRBG_CTX *ctx, unsigned char *out, size_t outlen, const unsigned char *adin, size_t adinlen); /* uninstantiate */ int (*uninstantiate)(DRBG_CTX *ctx); ... }; This part of the code was removed during the DRBG port, because currently CTR_DRBG is the only impementation. I would like to suggest to restore this 'polymorphic' implementation, to ease adding new implementations CHACHA20_DRBG in the future The DRBG API is well documented and tested ========================================== The entire DRBG API is part of the OpenSSL FIPS 2.0 Module and as such is well tested and well documented. Most of the documentation for the manual pages still-to-be-written can be taken from the FIPS User Guide and converted into manual pages, starting with textual modifications like FIPS_drbg_xxx(...) -> RAND_DRBG_xxx(...) DRBG_CTX *dctx -> RAND_DRBG *dctx and taking the new typedefs into consideration. Here is for example a comparison of the API function to install the callbacks: FIPS DRBG: int FIPS_drbg_set_callbacks(DRBG_CTX *dctx, size_t (*get_entropy)(DRBG_CTX *ctx, ), void (*cleanup_entropy)(DRBG_CTX *ctx, ), size_t entropy_blocklen, size_t (*get_nonce)(DRBG_CTX *ctx, ), void (*cleanup_nonce)(DRBG_CTX *ctx, ) ); RAND_DRBG: typedef size_t (*RAND_DRBG_get_entropy_fn)(RAND_DRBG *ctx, ); typedef void (*RAND_DRBG_cleanup_entropy_fn)(RAND_DRBG *ctx, ); typedef size_t (*RAND_DRBG_get_nonce_fn)(RAND_DRBG *ctx, ); typedef void (*RAND_DRBG_cleanup_nonce_fn)(RAND_DRBG *ctx, ); int RAND_DRBG_set_callbacks(RAND_DRBG *dctx, RAND_DRBG_get_entropy_fn get_entropy, RAND_DRBG_cleanup_entropy_fn cleanup_entropy, RAND_DRBG_get_nonce_fn get_nonce, RAND_DRBG_cleanup_nonce_fn cleanup_nonce); Conclusion ========== I see no reason why the RAND_DRBG shouldn't be made public as soon as possible, keeping the API as close as possible to the original FIPS DRBG API (FIPS 3.0 is upcoming!). In a second step, the current compatibility binding from RAND_METHOD to RAND_DRBG could be deprecated and faded out smoothly. Looking forward to receiving your comments. (But please be patient with me, I'm currently on physical rehab after a surgery.) Matthias St. Pierre -- openssl-dev mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5211 bytes Desc: not available URL: From Matthias.St.Pierre at ncp-e.com Wed Aug 30 14:41:36 2017 From: Matthias.St.Pierre at ncp-e.com (Dr. Matthias St. Pierre) Date: Wed, 30 Aug 2017 14:41:36 +0000 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: <39B4D013-3F63-421C-820C-40F21E15C7EC@ll.mit.edu> References: <97ae072a-e9c4-4803-8daf-4c9822a41947@default> <39B4D013-3F63-421C-820C-40F21E15C7EC@ll.mit.edu> Message-ID: > -----Urspr?ngliche Nachricht----- > Von: openssl-dev [mailto:openssl-dev-bounces at openssl.org] Im Auftrag von Blumenthal, Uri - 0553 - MITLL > Gesendet: Mittwoch, 30. August 2017 16:27 > An: openssl-dev at openssl.org > Betreff: Re: [openssl-dev] Plea for a new public OpenSSL RNG API > ... > > It allows hardware sources to be used via the same API. > > I rather doubt this. For example, my smartcard (accessible via PKCS#11) is a hardware source, which I > occasionally use. How do you see it used with the same API? We have a similar situation, on a small hardware device with little own entropy but with a smartcard reader. We implemented a get_entropy() call which fetches the entropy via PKCS#11, and modified the rand_method such that RAND_DRBG_generate() is always called with prediction_resistance=1. So every generate() triggers a reseed(), the entropy is fetched from the smartcard and it is immediately postprocessed by the AES-CTR DRBG. The /dev/urandom device was only used as additional input. So we didn't feel the need for an extra API call. Matthias -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 4328 bytes Desc: not available URL: From uri at ll.mit.edu Wed Aug 30 14:49:57 2017 From: uri at ll.mit.edu (Blumenthal, Uri - 0553 - MITLL) Date: Wed, 30 Aug 2017 14:49:57 +0000 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: References: <97ae072a-e9c4-4803-8daf-4c9822a41947@default> <39B4D013-3F63-421C-820C-40F21E15C7EC@ll.mit.edu> Message-ID: <19E16623-7884-40CB-A1E6-6958FC55A975@ll.mit.edu> >> > It allows hardware sources to be used via the same API. >> >> I rather doubt this. For example, my smartcard (accessible via PKCS#11) is a hardware source, which I >> occasionally use. How do you see it used with the same API? > > We have a similar situation, on a small hardware device with little own entropy > but with a smartcard reader. Yes, but in most cases you cannot count on the smartcard (or smartcard-like device) being in the reader. Which is why in my opinion this is an ideal case for ?additional input? source, but rather poor as the ?main? entropy source. > We implemented a get_entropy() call which fetches the entropy via PKCS#11, and > modified the rand_method such that RAND_DRBG_generate() is always called with > prediction_resistance=1. So every generate() triggers a reseed(), the entropy is fetched > from the smartcard and it is immediately postprocessed by the AES-CTR DRBG. > The /dev/urandom device was only used as additional input. > So we didn't feel the need for an extra API call. I would do exactly the opposite. ?Normal? entropy is fetched from the default sources (/dev/urandom). But when a sensitive (aka long-term) keys are generated, a (portable :) hardware RNG is plugged in and used with RAND_add() equivalent. Reason ? in my setup reliable trusted hardware RNG is not always on. If it were, I?d use it as the main entropy source and be done with it. -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5211 bytes Desc: not available URL: From Matthias.St.Pierre at ncp-e.com Wed Aug 30 15:06:53 2017 From: Matthias.St.Pierre at ncp-e.com (Dr. Matthias St. Pierre) Date: Wed, 30 Aug 2017 15:06:53 +0000 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: <19E16623-7884-40CB-A1E6-6958FC55A975@ll.mit.edu> References: <97ae072a-e9c4-4803-8daf-4c9822a41947@default> <39B4D013-3F63-421C-820C-40F21E15C7EC@ll.mit.edu> <19E16623-7884-40CB-A1E6-6958FC55A975@ll.mit.edu> Message-ID: > > We have a similar situation, on a small hardware device with little own entropy > > but with a smartcard reader. > > Yes, but in most cases you cannot count on the smartcard (or smartcard-like device) being in the reader. > Which is why in my opinion this is an ideal case for ?additional input? source, but rather poor as the > ?main? entropy source. > > ... > > > We implemented a get_entropy() call which fetches the entropy via PKCS#11, and > > modified the rand_method such that RAND_DRBG_generate() is always called with > > prediction_resistance=1. So every generate() triggers a reseed(), the entropy is fetched > > from the smartcard and it is immediately postprocessed by the AES-CTR DRBG. > > The /dev/urandom device was only used as additional input. > > So we didn't feel the need for an extra API call. > > I would do exactly the opposite. ?Normal? entropy is fetched from the default sources (/dev/urandom). But > when a sensitive (aka long-term) keys are generated, a (portable :) hardware RNG is plugged in and used with > RAND_add() equivalent. Reason ? in my setup reliable trusted hardware RNG is not always on. If it were, I?d > use it as the main entropy source and be done with it. In general, I would agree with you. In our case, it was a requirement of the government to trust only the SmartCard RNG. Since we use it for VPN connections with SmartCard authentication this was no problem, because the SmartCard must be present in order to initiate the IKEv2 exchange. The only tricky part was to deal with temporary failures of the entropy source. Matthias -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 4328 bytes Desc: not available URL: From uri at ll.mit.edu Wed Aug 30 15:22:58 2017 From: uri at ll.mit.edu (Blumenthal, Uri - 0553 - MITLL) Date: Wed, 30 Aug 2017 15:22:58 +0000 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: References: <97ae072a-e9c4-4803-8daf-4c9822a41947@default> <39B4D013-3F63-421C-820C-40F21E15C7EC@ll.mit.edu> <19E16623-7884-40CB-A1E6-6958FC55A975@ll.mit.edu> Message-ID: >> I would do exactly the opposite. ?Normal? entropy is fetched from the default sources (/dev/urandom). But >> when a sensitive (aka long-term) keys are generated, a (portable :) hardware RNG is plugged in and used with >> RAND_add() equivalent. Reason ? in my setup reliable trusted hardware RNG is not always on. If it were, I?d >> use it as the main entropy source and be done with it. > > In general, I would agree with you. In our case, it was a requirement of the > government to trust only the SmartCard RNG. Since we use it for VPN > connections with SmartCard authentication this was no problem, because > the SmartCard must be present in order to initiate the IKEv2 exchange. Ah, that makes a lot of difference. > The only tricky part was to deal with temporary failures of the entropy source. Did you experience that often? How did you deal with it? -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 5211 bytes Desc: not available URL: From Matthias.St.Pierre at ncp-e.com Wed Aug 30 15:46:38 2017 From: Matthias.St.Pierre at ncp-e.com (Dr. Matthias St. Pierre) Date: Wed, 30 Aug 2017 15:46:38 +0000 Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: References: <97ae072a-e9c4-4803-8daf-4c9822a41947@default> <39B4D013-3F63-421C-820C-40F21E15C7EC@ll.mit.edu> <19E16623-7884-40CB-A1E6-6958FC55A975@ll.mit.edu> Message-ID: > -----Urspr?ngliche Nachricht----- > Von: openssl-dev [mailto:openssl-dev-bounces at openssl.org] Im Auftrag von Blumenthal, Uri - 0553 - MITLL > Gesendet: Mittwoch, 30. August 2017 17:23 > An: openssl-dev at openssl.org > Betreff: Re: [openssl-dev] Plea for a new public OpenSSL RNG API > > ... > > The only tricky part was to deal with temporary failures of the entropy source. > > Did you experience that often? How did you deal with it? > 1) If the SmartCard is missing when our VPN service starts at boot time, the initial instantiation of the DRBG fails. This error is ignored, the DRBG remains uninstantiated. 2) If the SmartCard is pulled after the VPN connection is established and a rekey occurs, the DRBG fails, because it pulls entropy on every generate request and the entropy source fails. The NIST standard demands that this is a fatal error and the DRBG has to be uninstantiated. So we uninstantiate and our custom drbg_bytes() returns an error, which leads to a VPN disconnect. In order to account for 1) and 2), in our custom drbg_bytes() implementation 3) we do a just-in-time (re-)instantiation of the DRBG before calling RAND_DRBG_generate(). Note that everything said above applies to the legacy setup of libcrypto 1.0.2x with FIPS DRBG. In the future, we would use RAND_DRBG_generate() directly. Regards, Matthias -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 4328 bytes Desc: not available URL: From paul.dale at oracle.com Wed Aug 30 22:00:18 2017 From: paul.dale at oracle.com (Paul Dale) Date: Wed, 30 Aug 2017 15:00:18 -0700 (PDT) Subject: [openssl-dev] Plea for a new public OpenSSL RNG API In-Reply-To: <39B4D013-3F63-421C-820C-40F21E15C7EC@ll.mit.edu> References: <97ae072a-e9c4-4803-8daf-4c9822a41947@default> <39B4D013-3F63-421C-820C-40F21E15C7EC@ll.mit.edu> Message-ID: <6342c8c5-6d42-449b-adb2-9a0a48455844@default> To access a PKCS#11 randomness source, it would be necessary to have an engine that implemented whatever new RNG API is defined which in turn talks to the P11 device. Possibly not ideal but workable. As for the entropy argument to RAND_add et al, the callee will use it in a manner suitable to it. For a DRBG, the buffer will likely be added as additional data and the argument ignored. Other RNGs could use it differently. NIST 800-90B 3.1.5.1.2 specifies the minimum amount of entropy that must be input in order achieve a desired output. In this case the RNG would accumulate the entropy arguments until it achieves the requisite level. The trust issue is both harder and easier. I agree completely that working out if you trust the assessed entropy or not is an incredibly difficult (impossible) task. However, we're a library and the user is telling us what _they_ trust. It isn't OpenSSL's place to try to second guess this, issue a warning at best. We don't stop users choosing poor passwords, using zero as an IV or generating a sixteen bit RSA key. As for the new RNG engine API, I've been considering the benefits of having two calls: one to get random bytes, the other to request entropy. The first can be whitened or produced by a DRBG etc, the second also returns an estimate as to the quality. Essentially the difference between RDRAND and RDSEED. Pauli -- Oracle Dr Paul Dale | Cryptographer | Network Security & Encryption Phone +61 7 3031 7217 Oracle Australia -----Original Message----- From: Blumenthal, Uri - 0553 - MITLL [mailto:uri at ll.mit.edu] Sent: Thursday, 31 August 2017 12:27 AM To: openssl-dev at openssl.org Subject: Re: [openssl-dev] Plea for a new public OpenSSL RNG API On 8/30/17, 00:59, "openssl-dev on behalf of Paul Dale" wrote: > My thoughts are that the new RNG API should be made public once it has > been properly designed. We've a chance to get this right, let's take the time > and make an effort to do so. There is no rush. Not quite. If there is an RNG involved in generating long-term keys now ? users better be able to control/affect it now. > I also feel that having an engine interface to the new RNG API is worthwhile. +1 > It allows hardware sources to be used via the same API. I rather doubt this. For example, my smartcard (accessible via PKCS#11) is a hardware source, which I occasionally use. How do you see it used with the same API? > I would like to see an entropy argument to the equivalent to RAND_add. > Anyone is welcome to always pass zero in but there should be the option to not. > Consider an hardware source with _provable_ output quality, why shouldn't it be > allowed to meaningfully contribute? What?s the purpose of passing the entropy argument? How is the callee (the RNG) going to use it? Why should the OpenSSL code in general trust the received value (how can OpenSSL tell that the received randomness is indeed from a hardware source with provable output quality)? Finally, what does it matter? And they all ?meaningfully contribute?. The only question is whether this contribution should prevent RNG from acquiring more entropy from other sources. My opinion is resounding no. > I like the idea of two independent global RNGs. +1 ;-) > This does increase seeding requirements however. If you can seed one, you can seed two. -----Original Message----- From: Dr. Matthias St. Pierre [mailto:Matthias.St.Pierre at ncp-e.com] Sent: Tuesday, 29 August 2017 7:45 PM To: openssl-dev at openssl.org Subject: [openssl-dev] Plea for a new public OpenSSL RNG API Hi everybody, on the [openssl-dev] mailing list, there has been a long ongoing discussion about the new RAND_DRBG API and comparing it with the old RAND_METHOD API (see "[openssl-dev] Work on a new RNG for OpenSSL"). Two of the most controversal questions were: - Do we really need a new RNG API? Should the RAND_DRBG API be made public or kept private? (Currently, it's exported from libcrypto but only used internally by libssl.) - How much control should the user (programmer) be given over the reseeding process and/or should he be allowed to add his own additional randomness? Many developers seem to be realizing the interesting possibilities of the DRBG API and are asking for public access to this new and promising API. One of the driving forces behind it is the question about how to do seeding and reseeding right. Among others, Uri Blumenthal asked for making the DRBG API public. Currently, the OpenSSL core members seem to be reluctant to make the API public, at least at this early stage. I understand Rich Salz's viewpoint that this requires a thorough discussion, because a public interface can't be easily changed and wrong decisions in the early phase can become a heavy burdon. Nevertheless, I agree with Uri Blumenthal that the DRBG API should be made public. So here comes my ====================================== Plea for a new public OpenSSL RNG API: ====================================== The new RAND_DRBG is the superior API. It shouldn't be kept private and hidden behind the ancient RAND_METHOD API. The philosophy of the two APIs is not very well compatible, in particular when it comes to reseeding and adding additional unpredictable input. Hiding the RAND_DRBG behind the RAND_METHOD API only causes problems. Also, it will force people to patch their OpenSSL copy if they want to use the superior API. The RAND_DRBG API should become the new public OpenSSL RNG API and the old RAND_METHOD API should be deprecated in the long run. This transition does not need to be rushed, but it would be good if there would be early consent on the road map. I am thinking of a smooth transition with a phase of coexistence and a compatibility layer mapping the default RAND_METHOD to the default public RAND_DRBG instance. (This compatibility layer already exists, it's the 'RAND_OpenSSL()' method.) Historical Background ===================== As Rich already mentioned in his blog post, the RAND_DRBG isn't new. It's been a part of OpenSSL for a long time, hidden in the FIPS 2.0 Object Module. I have been working with the FIPS DRBG for quite a while now, using a FIPS-capable OpenSSL 1.0.2x crypto library. The reason why our company switched to the FIPS DRBG is that one of our products runs on a small hardware device which does not have a reliable entropy source, but the product has to meet high security standards, in particular w.r.t. its RNG. So we decided to use the SmartCard RNG as primary entropy source for a deterministic AES-CTR based RNG and use /dev/urandom as additional input. Reseeding should occur on every generate request. Using the FIPS DRBG, these requirements were easily met, because the API gives such a fine grained control over reseeding and adding additional entropy. The DRBG was well documented, its design in NIST SP800-90A (now: NIST SP800-90Ar1) and its API in the OpenSSL FIPS 2.0 User Guide. The implementation was thoroughly tested and assessed during the FIPS certification process. So the only minor obstacle was that we had to patch the crypto library (not the FIPS object module) in order to get public access to the FIPS_drbg_*() methods. I always considered the DRBG API more mature than the good old RAND_METHOD API and I wondered, why the DRBG code lay forgotten for so many years in the FIPS 2.0 object module sources and was never ported to master. When in June of this year the thread "[openssl-dev] Work on a new RNG for OpenSSL" popped up (https://mta.openssl.org/pipermail/openssl-dev/2017-June), I closely watched the discussion, and when John Denker suggested having a look at NIST SP800-90A, I was electrified: > Constructive suggestion: If you want to see what a RNG looks > like when designed by cryptographers, take a look at: > Elaine Barker and John Kelsey, > "Recommendation for Random Number Generation Using Deterministic Random Bit Generators" > http://csrc.nist.gov/publications/nistpubs/800-90A/SP800-90A.pdf > > That design may look complicated, but if you think you can > leave out some of the blocks in their diagram, proceed with > caution. Every one of those blocks is there for a reason. From his mail and the reaction to it, I had the impression that nobody seemed to remember the fact that the DRBG code was already present in the FIPS object module. The hidden treasure seemed forgotten! When I seized the opportunity and proposed to port the FIPS DRBG code into master, Rich Salz liked my idea and immediately started working on PR #3789: I am very grateful to Rich for picking up the idea so fast and and giving it so much speed and momentum. He has done a lot of work to convince others and defend the idea against initial scepticism by voices objecting to the seemingly new and unknown API. In the middle of August, when the first bulk of work was finished, Rich wrote a detailed blog post to advertise the new DRBG and explain his work and the discussion restarted: "[openssl-dev] Work on a new RNG for OpenSSL> (see https://mta.openssl.org/pipermail/openssl-dev/2017-August) It became quickly evident that users that the DRBG API had promising features, so they started asking for public access to this new API. The driving force was the question about how to do seeding and reseeding right. Among others, Uri Blumenthal was a dedicated advocate to make the DRBG API public (https://mta.openssl.org/pipermail/openssl-dev/2017-August/009594.html). But the OpenSSL members currently seem to be reluctant to make the API public right away. I understand Rich's viewpoint that this decision requires a thorough discussion, because a public interface can't be easily changed and wrong decisions in the early phase can become a heavy burdon. Nevertheless, I agree with Uri Blumenthal that the DRBG API should be made public. Here are some of my arguments for it. The DRBG API supports multiple instances and chaining ===================================================== The NIST DRBG standard had chaining of multiple DRBG instances in the mind from the very beginnning, see for example footnote [4] on page 25 of NIST SP800-90Ar1: > Entropy input may be obtained from an entropy source or an NRBG, both of which provide fresh entropy. > Entropy input could also be obtained from a DRBG that has access to an entropy source or NRBG. The original OpenSSL FIPS DRBG implementation did not support chaining, but this support has been added by Rich during the DRBG port. The DRBG API has a highly flexible concept for seeding and reseeding ==================================================================== As mentioned previously, the DRBG has a callback mechanism with callbacks like get_entropy() and get_adin() callbacks, which make it easy to fine tune the default instantiation process by either adding additional randomness input, or changing the entropy source entirely. The callbacks are primarily intended for obtaining randomness instantiation and reseeing. There is a clear concept for reseeding, which can be adjusted by changing the reseed_interval: Normally, the DRBG reseeds itself automatically, whenever the reseed_interval has been reached. In addition, there is a way for the DRBG consumer to add his own unpredictability when requesting random bytes: by adding additional input 'adin' to the RAND_DRBG_generate() call: int RAND_DRBG_generate(RAND_DRBG *drbg, unsigned char *out, size_t outlen, int prediction_resistance, const unsigned char *adin, size_t adinlen) So why are there so many ways to add randomness and additional input? And what is the difference? * The get_entropy() and get_adin() callbacks are used by the DRBG itself to _pull_ unpredictable data from some backend entropy source (which can also be a chained DRBG which is connected to some entropy source) during instantiation or reseeding. * The 'adin' argument can be used by the DRBG consumer to _push_ unpredictable input into the DRBG when generating random output. Note that the DRBG consumer has no possibility to push out-of-band randomness into the DRBG. Adding randomness is always coupled with a generate or (re-)seed operation. This is a very import difference between the RAND_DRBG and the RAND_METHOD and one of the reasons why it's so hard to do reseeding right in both the RAND and RAND_DRBG API simultaneously. In fact, currently The 'RAND_add()/RAND_bytes()' pattern is broken =============================================== In OpenSSL, the classical way for the RNG consumer to add his own randomness is to call 'RAND_add()' before calling 'RAND_bytes()'. If the new 'RAND_OpenSSL()' method (the "compatibility layer" hiding the public RAND_DRBG instance) is the default, then this does not work as expected anymore: The reason is that a call to 'RAND_add()' adds the provided randomness only to a global buffer ('rand_bytes'), from which it will be pulled during the next reseed. But no reseed is triggered. So the next RAND_bytes() call will be unaffected from the RAND_add(), which is not what the consumer expected. (The same holds for 'RAND_seed()', since 'drbg_seed()' only calls into 'drbg_add()') Reseeding of DRBGs occurs only at the following occasions: * immediately after a 'fork()' (new) * if the 'reseed_counter' exceeds the 'reseed_interval' * if 'RAND_DRBG_generate()' is called requesting 'prediction_resistance' * 'RAND_DRBG_reseed()' is called explicitely *Note:* Currently it looks like the situation is even worse: if 'RAND_add()' is called multiple times before a reseed occurs, then the result of the previous call is overwritten. Reseeding the 'DRBG' whenever the user calls 'RAND_add()' does not seem a good solution. It would be too expensive, in particular if system entropy is pulled for reseeding. Of course it is possible to fix this issue, but the DRBG provides for a much simpler solution: it lets the consumer contribute to the entropy of the internal state by providing additional input. If the user input contains entropy, that's fine, if it's "snake oil", no harm. The additional input is mixed into the internal state in just the same way as the entropy buffer using the 'ctr_df()' derivation function. One might think of the 'entropy' input as trusted randomness and 'adin' as untrusted randomness. For this reason, I would like to see the 'RAND_add()/RAND_bytes()' pattern deprecated and the 'RAND_DRBG_generate() with additional input' pattern advertised instead. The DRBG API supports different implementations =============================================== Well, it _supported_ them, until recently. But that's not irreversible. The DRBG concept, as layed out in the NIST standard, provides a generic framework for deterministic RNGs (the acronym DRBG stands for Deterministic Random Bit Generator). It deals with general questions like how to instantiate and reseed the RNG, where does it get the entropy from, etc. The standard proposes three concrete implementations, Hash_DRBG, HMAC_DRBG, and CTR_DRBG. In the FIPS code, all three were implemented, and the genericity was achieved using a data union and a set of five function pointers, reminiscent of a vtable in C++: struct drbg_ctx_st { ... /* Implementation specific structures */ union { DRBG_HASH_CTX hash; DRBG_HMAC_CTX hmac; DRBG_CTR_CTX ctr; } d; /* Initialiase PRNG and setup callbacks below */ int (*init)(DRBG_CTX *ctx, int nid, int security, unsigned int flags); /* Intantiate PRNG */ int (*instantiate)(DRBG_CTX *ctx, const unsigned char *ent, size_t entlen, const unsigned char *nonce, size_t noncelen, const unsigned char *pers, size_t perslen); /* reseed */ int (*reseed)(DRBG_CTX *ctx, const unsigned char *ent, size_t entlen, const unsigned char *adin, size_t adinlen); /* generat output */ int (*generate)(DRBG_CTX *ctx, unsigned char *out, size_t outlen, const unsigned char *adin, size_t adinlen); /* uninstantiate */ int (*uninstantiate)(DRBG_CTX *ctx); ... }; This part of the code was removed during the DRBG port, because currently CTR_DRBG is the only impementation. I would like to suggest to restore this 'polymorphic' implementation, to ease adding new implementations CHACHA20_DRBG in the future The DRBG API is well documented and tested ========================================== The entire DRBG API is part of the OpenSSL FIPS 2.0 Module and as such is well tested and well documented. Most of the documentation for the manual pages still-to-be-written can be taken from the FIPS User Guide and converted into manual pages, starting with textual modifications like FIPS_drbg_xxx(...) -> RAND_DRBG_xxx(...) DRBG_CTX *dctx -> RAND_DRBG *dctx and taking the new typedefs into consideration. Here is for example a comparison of the API function to install the callbacks: FIPS DRBG: int FIPS_drbg_set_callbacks(DRBG_CTX *dctx, size_t (*get_entropy)(DRBG_CTX *ctx, ), void (*cleanup_entropy)(DRBG_CTX *ctx, ), size_t entropy_blocklen, size_t (*get_nonce)(DRBG_CTX *ctx, ), void (*cleanup_nonce)(DRBG_CTX *ctx, ) ); RAND_DRBG: typedef size_t (*RAND_DRBG_get_entropy_fn)(RAND_DRBG *ctx, ); typedef void (*RAND_DRBG_cleanup_entropy_fn)(RAND_DRBG *ctx, ); typedef size_t (*RAND_DRBG_get_nonce_fn)(RAND_DRBG *ctx, ); typedef void (*RAND_DRBG_cleanup_nonce_fn)(RAND_DRBG *ctx, ); int RAND_DRBG_set_callbacks(RAND_DRBG *dctx, RAND_DRBG_get_entropy_fn get_entropy, RAND_DRBG_cleanup_entropy_fn cleanup_entropy, RAND_DRBG_get_nonce_fn get_nonce, RAND_DRBG_cleanup_nonce_fn cleanup_nonce); Conclusion ========== I see no reason why the RAND_DRBG shouldn't be made public as soon as possible, keeping the API as close as possible to the original FIPS DRBG API (FIPS 3.0 is upcoming!). In a second step, the current compatibility binding from RAND_METHOD to RAND_DRBG could be deprecated and faded out smoothly. Looking forward to receiving your comments. (But please be patient with me, I'm currently on physical rehab after a surgery.) Matthias St. Pierre -- openssl-dev mailing list To unsubscribe: https://mta.openssl.org/mailman/listinfo/openssl-dev From bhat.jayalakshmi at gmail.com Thu Aug 31 05:55:16 2017 From: bhat.jayalakshmi at gmail.com (Jayalakshmi bhat) Date: Thu, 31 Aug 2017 11:25:16 +0530 Subject: [openssl-dev] how to compile out selected ciphers Message-ID: Hi All, I am trying to build openssl. As part of that I want to remove some ciphers like md4, rc5 etc. I tried ./config no-md5, no-rc5 and ./Configure no-md5, no-rc5. In both the case MD4 and RC5 directories are still getting compiled. Please can you let me know what could be going wrong. Regards Jaya -------------- next part -------------- An HTML attachment was scrubbed... URL: From levitte at openssl.org Thu Aug 31 09:13:13 2017 From: levitte at openssl.org (Richard Levitte) Date: Thu, 31 Aug 2017 11:13:13 +0200 (CEST) Subject: [openssl-dev] how to compile out selected ciphers In-Reply-To: References: Message-ID: <20170831.111313.259508043850633407.levitte@openssl.org> In message on Thu, 31 Aug 2017 11:25:16 +0530, Jayalakshmi bhat said: bhat.jayalakshmi> Hi All, bhat.jayalakshmi> bhat.jayalakshmi> I am trying to build openssl. As part of that I want bhat.jayalakshmi> to remove some ciphers like md4, rc5 etc. bhat.jayalakshmi> bhat.jayalakshmi> I tried ./config no-md5, no-rc5 and ./Configure bhat.jayalakshmi> no-md5, no-rc5. In both the case MD4 and RC5 bhat.jayalakshmi> directories are still getting compiled. bhat.jayalakshmi> bhat.jayalakshmi> Please can you let me know what could be going wrong. Your configuration line says 'no-md5', which is an attempt to remove MD5, not MD4. Your config line should be this: ./config no-md4 no-rc5 It's possible, though, that you really meant to remove MD5... unfortunately, it's such an integral part of most SSL/TLS protocol versions that we cannot for the moment allow it to be disabled. That's the issue you're hitting. Cheers, Richard -- Richard Levitte levitte at openssl.org OpenSSL Project http://www.openssl.org/~levitte/ From rsalz at akamai.com Thu Aug 31 12:43:56 2017 From: rsalz at akamai.com (Salz, Rich) Date: Thu, 31 Aug 2017 12:43:56 +0000 Subject: [openssl-dev] [openssl-users] how to compile out selected ciphers In-Reply-To: References: Message-ID: <7919C15F-1D08-4537-A314-8412070EBBA0@akamai.com> What version of openssl are you building? -------------- next part -------------- An HTML attachment was scrubbed... URL: From hkario at redhat.com Thu Aug 31 13:52:16 2017 From: hkario at redhat.com (Hubert Kario) Date: Thu, 31 Aug 2017 15:52:16 +0200 Subject: [openssl-dev] how to compile out selected ciphers In-Reply-To: <20170831.111313.259508043850633407.levitte@openssl.org> References: <20170831.111313.259508043850633407.levitte@openssl.org> Message-ID: <5974271.8uaZBKUojM@pintsize.usersys.redhat.com> On Thursday, 31 August 2017 11:13:13 CEST Richard Levitte wrote: > In message > on > Thu, 31 Aug 2017 11:25:16 +0530, Jayalakshmi bhat > said: > > bhat.jayalakshmi> Hi All, > bhat.jayalakshmi> > bhat.jayalakshmi> I am trying to build openssl. As part of that I want > bhat.jayalakshmi> to remove some ciphers like md4, rc5 etc. > bhat.jayalakshmi> > bhat.jayalakshmi> I tried ./config no-md5, no-rc5 and ./Configure > bhat.jayalakshmi> no-md5, no-rc5. In both the case MD4 and RC5 > bhat.jayalakshmi> directories are still getting compiled. > bhat.jayalakshmi> > bhat.jayalakshmi> Please can you let me know what could be going wrong. > > Your configuration line says 'no-md5', which is an attempt to remove > MD5, not MD4. Your config line should be this: > > ./config no-md4 no-rc5 > > It's possible, though, that you really meant to remove MD5... > unfortunately, it's such an integral part of most SSL/TLS protocol > versions that we cannot for the moment allow it to be disabled. > That's the issue you're hitting. It's not integral part of TLS 1.2 though so allowing for disabling of MD5 when SSL, TLS1.0 and TLS 1.1 are disabled isn't unreasonable. At the same time, the problem of data-at-rest remains, because while disabling it for TLS is a good idea, disabling it for decryption of PKCS#12 or PKCS#8 (private keys), CMS or S/MIME at the same time could create issues that manifest only quite a bit later. -- Regards, Hubert Kario Senior Quality Engineer, QE BaseOS Security team Web: www.cz.redhat.com Red Hat Czech s.r.o., Purky?ova 115, 612 00 Brno, Czech Republic -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 819 bytes Desc: This is a digitally signed message part. URL: From matt at openssl.org Thu Aug 31 14:06:51 2017 From: matt at openssl.org (Matt Caswell) Date: Thu, 31 Aug 2017 15:06:51 +0100 Subject: [openssl-dev] how to compile out selected ciphers In-Reply-To: <5974271.8uaZBKUojM@pintsize.usersys.redhat.com> References: <20170831.111313.259508043850633407.levitte@openssl.org> <5974271.8uaZBKUojM@pintsize.usersys.redhat.com> Message-ID: On 31/08/17 14:52, Hubert Kario wrote: > On Thursday, 31 August 2017 11:13:13 CEST Richard Levitte wrote: >> In message >> on >> Thu, 31 Aug 2017 11:25:16 +0530, Jayalakshmi bhat >> said: >> >> bhat.jayalakshmi> Hi All, >> bhat.jayalakshmi> >> bhat.jayalakshmi> I am trying to build openssl. As part of that I want >> bhat.jayalakshmi> to remove some ciphers like md4, rc5 etc. >> bhat.jayalakshmi> >> bhat.jayalakshmi> I tried ./config no-md5, no-rc5 and ./Configure >> bhat.jayalakshmi> no-md5, no-rc5. In both the case MD4 and RC5 >> bhat.jayalakshmi> directories are still getting compiled. >> bhat.jayalakshmi> >> bhat.jayalakshmi> Please can you let me know what could be going wrong. >> >> Your configuration line says 'no-md5', which is an attempt to remove >> MD5, not MD4. Your config line should be this: >> >> ./config no-md4 no-rc5 >> >> It's possible, though, that you really meant to remove MD5... >> unfortunately, it's such an integral part of most SSL/TLS protocol >> versions that we cannot for the moment allow it to be disabled. >> That's the issue you're hitting. > > It's not integral part of TLS 1.2 though so allowing for disabling of MD5 when > SSL, TLS1.0 and TLS 1.1 are disabled isn't unreasonable. > > At the same time, the problem of data-at-rest remains, because while disabling > it for TLS is a good idea, disabling it for decryption of PKCS#12 or PKCS#8 > (private keys), CMS or S/MIME at the same time could create issues that > manifest only quite a bit later. > Note (as an aside) that no-md5 was removed as an option from OpenSSL 1.1.0 (and master). Matt -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 480 bytes Desc: OpenPGP digital signature URL: