endless loop in probable_prime

[Jakob Bohm]

On 2020-06-18 18:13, Salz, Rich via openssl-users wrote:
>>     BN_bin2bn assumes that the size of a BN_ULONG (the type of a bn->d) is
>      BN_BYTES. You have already told us that sizeof(*d) is 4. So BN_BYTES
>      should also be 4. If BN_BYTES is being incorrectly set to 8 on your
>      platform then that would explain the discrepancy. Can you check?
>
> This seems HIGHLY likely since Ronny said earlier that the same config/toolchain is used for 32bit userspace and 64bit kernel, right?
Maybe the internal headers should contain lines that abort compilation if
inconsistencies are found in the values provided by the (public or private)
headers.

For example, if BN_BYTES > sizeof(BN_ULONG), compilation should stop via
an abstraction over the presence/absence of the _Static_assert, 
static_assert
or macro emulation of same in any given compiler.

/* Something like this, but via a macro abstraction: */
#if (some C++ compilers)
   /* Works if  defined(__cplusplus) && __cplusplus >= 201103l */
   /* Works for clang++ if has_feature(cxx_static_assert) */
   /* Works for g++ >= 4.3.x if defined(__GXX_EXPERIMENTAL_CXX0X__) */
   /* Works for MSC++ >= 16.00 */
   /* Fails for g++ 4.7.x specifically */
   /* Fails for some versions of Apple XCode */
   static_assert(
     (BN_BYTES <= sizeof(BN_ULONG)),
     "Failed static assert: " "BN_BYTES <= sizeof(BN_ULONG)");
#elif (some C compilers)
   /* Works for clang with has_feature(c_static_assert) */
   /* Works for gcc >= 4.6.x */
   /* Fails for some versions of Apple XCode */
   _Static_assert(
     (BN_BYTES <= sizeof(BN_ULONG)),
     "Failed static assert: " "BN_BYTES <= sizeof(BN_ULONG)");
#else
   /* Portable fallback, but some fudging may be needed for compilers
    *    without __COUNTER__ */
   /* If assertion fails, compiler will complain about invalid array size */
   /* If assertion is not a const expression, compiler will complain 
about that */
   typedef char OSSL_const_assert_##fudge##__LINE__##_##__COUNTER__[
     (BN_BYTES <= sizeof(BN_ULONG)) ? 1 : -1];
#endif


Enjoy

Jakob
-- 
Jakob Bohm, CIO, Partner, WiseMo A/S.  https://www.wisemo.com
Transformervej 29, 2860 Søborg, Denmark.  Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded