opensssl 1.1.1g test failure(s)

[Jakob Bohm]

Summary: The OpenSSL 1.1.1g test suite contains at least two bugs:

TestBug#1: Test suite fails if local network has no IPv6, error message 
(non-
   verbose) doesn't say that's the issue.  [ Testing IPv6 makes sense, 
rejecting
   regression tests on inadequate machines is important to avoid silent 
failures,
   but not telling testers that a test precondition failed or which ones 
is bad.
   ]

TestBug#2: Test suite uses ambiguous wording to report the index of 
failed test.
   Should have said "Failed test indexes:  2" or "Failed test: #2" (latter
   needs to repeat "#" for each index listed)

On 21/04/2020 19:34, Claus Assmann wrote:
> On Tue, Apr 21, 2020, Benjamin Kaduk via openssl-users wrote:
>> On Tue, Apr 21, 2020 at 07:22:38PM +0200, Claus Assmann wrote:
>>> ../test/recipes/80-test_ssl_old.t ..................
>>> Dubious, test returned 1 (wstat 256, 0x100)
>> Please run again with `make V=1 TESTS=test_ssl_old test` and post the relevant parts of the output?
> Thanks for the reply, below is the output, It seems it only fails
> because the host doesn't support IPv6?
>
> make depend && make _tests
> ( cd test;  mkdir -p test-runs;  SRCTOP=../.  BLDTOP=../.  RESULT_D=test-runs  PERL="/usr/bin/perl"  EXE_EXT=  OPENSSL_ENGINES=`cd .././engines 2>/dev/null && pwd`  OPENSSL_DEBUG_MEMORY=on  /usr/bin/perl .././test/run_tests.pl test_ssl_old )
> ../test/recipes/80-test_ssl_old.t ..
> 1..6
> # Subtest: test_ss
>      1..17
...
> 0:error:0200E016:system library:setsockopt:Invalid argument:crypto/bio/b_sock2.c:255:
> 0:error:2008B088:BIO routines:BIO_listen:listen v6 only:crypto/bio/b_sock2.c:256:
> Doing handshakes=1 bytes=256
> TLSv1.3, cipher (NONE) (NONE)
> ../../util/shlib_wrap.sh ../ssltest_old -s_key keyU.ss -s_cert certU.ss -c_key keyU.ss -c_cert certU.ss -ipv6 => 1
>      not ok 13 - test TLS via IPv6
>      #   Failed test 'test TLS via IPv6'
>      #   at ../test/recipes/80-test_ssl_old.t line 390.
>      # Looks like you failed 1 test of 13.
> not ok 2 - standard SSL tests
> #   Failed test 'standard SSL tests'
> #   at /home/ca/pd/security/openssl-1.1.1g/test/../util/perl/OpenSSL/Test.pm line 1212.
...
> # Looks like you failed 1 test of 6.
> Dubious, test returned 1 (wstat 256, 0x100)
> Failed 1/6 subtests
>
> Test Summary Report
> -------------------
> ../test/recipes/80-test_ssl_old.t (Wstat: 256 Tests: 6 Failed: 1)
>    Failed test:  2
>    Non-zero exit status: 1
> Files=1, Tests=6, 12 wallclock secs ( 0.04 usr  0.06 sys +  1.77 cusr  9.78 csys = 11.65 CPU)
> Result: FAIL
> *** Error 1 in . (Makefile:217 '_tests')
> *** Error 1 in /home/ca/pd/security/openssl-1.1.1g (Makefile:205 'tests')


Enjoy

Jakob
-- 
Jakob Bohm, CIO, Partner, WiseMo A/S.  http://www.wisemo.com
Transformervej 29, 2860 Soborg, Denmark.  Direct +45 31 13 16 10
This public discussion message is non-binding and may contain errors.
WiseMo - Remote Service Management for PCs, Phones and Embedded