Update110 [FreeBSD WIKI]

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
Process to update %%OpenSSL%% in base\\ 
(for 1.1.1-pre8 WIP see [[OpenSSL/Base/Update111]])

====== Summary ======

For the impatient...

<code>
svn checkout https://svn.FreeBSD.org/base/head /usr/src
fetch -o ~/patch-D15791 'https://reviews.freebsd.org/D15791?download=true'
fetch -o ~/openssl-1.1.0h.tar.gz https://www.openssl.org/source/openssl-1.1.0h.tar.gz
cd /usr/src
patch -p0 < ~/patch-D15791
mv /usr/src/crypto/openssl /usr/src/crypto/openssl-1.0.2
tar xf ~/openssl-1.1.0h.tar.gz -C /usr/src/crypto
mv /usr/src/crypto/openssl-1.1.0h /usr/src/crypto/openssl
cd /usr/src/secure/lib/libcrypto
pkg install perl5
rm amd64/*
mv /usr/obj/usr/src/amd64.amd64/secure/lib/libcrypto/*.S amd64/
make man-makefile-update
rm man/*
make man-update
cd ../libssl
make man-makefile-update
rm man/*
make man-update
</code>

====== Preliminary steps ======

1. Move current version of %%OpenSSL%%

<code>
$ mv /usr/src/crypto/openssl /usr/src/crypto/openssl-1.0.2
</code>

2. Download, verify

<code>
$ fetch https://www.openssl.org/source/openssl-1.1.0h.tar.gz
openssl-1.1.0h.tar.gz                         100% of 5295 kB   28 MBps 00m00s
$ fetch https://www.openssl.org/source/openssl-1.1.0h.tar.gz.asc
openssl-1.1.0h.tar.gz.asc                     100% of  455  B 4490 kBps 00m00s
$ sha256 ~/openssl-1.1.0h.tar.gz
SHA256 (/root/openssl-1.1.0h.tar.gz) = 5835626cde9e99656585fc7aaa2302a73a7e1340bf8c14fd635a62c66802a517
$ grep 5835626cde9e99656585fc7aaa2302a73a7e1340bf8c14fd635a62c66802a517 /usr/ports/security/openssl-devel/*
/usr/ports/security/openssl-devel/distinfo:SHA256 (openssl-1.1.0h.tar.gz) = 5835626cde9e99656585fc7aaa2302a73a7e1340bf8c14fd635a62c66802a517
$ gpg --verify openssl-1.1.0h.tar.gz.asc openssl-1.1.0h.tar.gz
gpg: WARNING: using insecure memory!
gpg: please see http://www.gnupg.org/documentation/faqs.html for more information
gpg: Signature made Tue Mar 27 15:50:44 2018 CEST using RSA key ID 0E604491
gpg: Good signature from "Matt Caswell <matt@openssl.org>"
gpg:                 aka "Matt Caswell <frodo@baggins.org>"
Primary key fingerprint: 8657 ABB2 60F0 56B1 E519  0839 D9C4 D26D 0E60 4491
</code>

3. Unpack, remove version from directory

<code>
$ tar xf ~/openssl-1.1.0h.tar.gz -C /usr/src/crypto
$ mv /usr/src/crypto/openssl-1.1.0h /usr/src/crypto/openssl
</code>

4. Update ''%%/usr/src/secure/lib/libcrypto/Makefile.inc%%''

<code>
...
# OpenSSL version used for manual page generation
OPENSSL_VER=    1.1.0h
OPENSSL_DATE=   2018-03-27
...
</code>

5. Required tools: Perl
We need ''%%pod2man%%'' from perl and ''%%perl%%'' to generate assembly from the perl scripts

<code>
$ pkg install perl5
...
</code>

====== Generate .h files ======

Complication here, as there are different ''%%opensslconf.h%%'' files for various arch. For now I generated an opensslconf.h like so

<code>
$ uname -a
FreeBSD freebsd12 12.0-CURRENT FreeBSD 12.0-CURRENT #0 r334983: Tue Jun 12 01:42:40 UTC 2018     root@releng3.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64
$ ./config -v
Operating system: amd64-whatever-freebsd
Configuring for BSD-x86_64
/usr/local/bin/perl5 ./Configure BSD-x86_64
Configuring OpenSSL version 1.1.0h (0x1010008fL)
    no-asan         [default]  OPENSSL_NO_ASAN
    no-crypto-mdebug [default]  OPENSSL_NO_CRYPTO_MDEBUG
    no-crypto-mdebug-backtrace [default]  OPENSSL_NO_CRYPTO_MDEBUG_BACKTRACE
    no-ec_nistp_64_gcc_128 [default]  OPENSSL_NO_EC_NISTP_64_GCC_128
    no-egd          [default]  OPENSSL_NO_EGD
    no-fuzz-afl     [default]  OPENSSL_NO_FUZZ_AFL
    no-fuzz-libfuzzer [default]  OPENSSL_NO_FUZZ_LIBFUZZER
    no-heartbeats   [default]  OPENSSL_NO_HEARTBEATS
    no-md2          [default]  OPENSSL_NO_MD2 (skip dir)
    no-msan         [default]  OPENSSL_NO_MSAN
    no-rc5          [default]  OPENSSL_NO_RC5 (skip dir)
    no-sctp         [default]  OPENSSL_NO_SCTP
    no-ssl-trace    [default]  OPENSSL_NO_SSL_TRACE
    no-ssl3         [default]  OPENSSL_NO_SSL3
    no-ssl3-method  [default]  OPENSSL_NO_SSL3_METHOD
    no-ubsan        [default]  OPENSSL_NO_UBSAN
    no-unit-test    [default]  OPENSSL_NO_UNIT_TEST
    no-weak-ssl-ciphers [default]  OPENSSL_NO_WEAK_SSL_CIPHERS
    no-zlib         [default]
    no-zlib-dynamic [default]
Configuring for BSD-x86_64
CC            =cc
CFLAG         =-DL_ENDIAN -Wall -O3 -pthread -D_THREAD_SAFE -D_REENTRANT
SHARED_CFLAG  =-fPIC
DEFINES       =DSO_DLFCN HAVE_DLFCN_H NDEBUG OPENSSL_THREADS OPENSSL_NO_STATIC_ENGINE OPENSSL_PIC OPENSSL_IA32_SSE2 OPENSSL_BN_ASM_MONT OPENSSL_BN_ASM_MONT5 OPENSSL_BN_ASM_GF2m SHA1_ASM SHA256_ASM SHA512_ASM RC4_ASM MD5_ASM AES_ASM VPAES_ASM BSAES_ASM GHASH_ASM ECP_NISTZ256_ASM PADLOCK_ASM POLY1305_ASM
LFLAG         =
PLIB_LFLAG    =
EX_LIBS       =
APPS_OBJ      =
CPUID_OBJ     =x86_64cpuid.o
UPLINK_OBJ    =
BN_ASM        =asm/x86_64-gcc.o x86_64-mont.o x86_64-mont5.o x86_64-gf2m.o rsaz_exp.o rsaz-x86_64.o rsaz-avx2.o
EC_ASM        =ecp_nistz256.o ecp_nistz256-x86_64.o
DES_ENC       =des_enc.o fcrypt_b.o
AES_ENC       =aes-x86_64.o vpaes-x86_64.o bsaes-x86_64.o aesni-x86_64.o aesni-sha1-x86_64.o aesni-sha256-x86_64.o aesni-mb-x86_64.o
BF_ENC        =bf_enc.o
CAST_ENC      =c_enc.o
RC4_ENC       =rc4-x86_64.o rc4-md5-x86_64.o
RC5_ENC       =rc5_enc.o
MD5_OBJ_ASM   =md5-x86_64.o
SHA1_OBJ_ASM  =sha1-x86_64.o sha256-x86_64.o sha512-x86_64.o sha1-mb-x86_64.o sha256-mb-x86_64.o
RMD160_OBJ_ASM=
CMLL_ENC      =cmll-x86_64.o cmll_misc.o
MODES_OBJ     =ghash-x86_64.o aesni-gcm-x86_64.o
PADLOCK_OBJ   =e_padlock-x86_64.o
CHACHA_ENC    =chacha-x86_64.o
POLY1305_OBJ  =poly1305-x86_64.o
BLAKE2_OBJ    =
PROCESSOR     =
RANLIB        =ranlib
ARFLAGS       =
PERL          =/usr/local/bin/perl5

SIXTY_FOUR_BIT_LONG mode

Configured for BSD-x86_64.
</code>

**TODO: Generate proper opensslconf.h files for all archs**\\ 
Same applies to crypto/include/internal/bn_conf.h\\ 
This is a hint from the %%OpenSSL%% Makefile

<code>
GENERATED_MANDATORY=crypto/include/internal/bn_conf.h crypto/include/internal/dso_conf.h include/openssl/opensslconf.h
</code>

and from the make output

<code>
/usr/local/bin/perl5 "-I." -Mconfigdata "util/dofile.pl"  "-oMakefile" crypto/include/internal/bn_conf.h.in > crypto/include/internal/bn_conf.h
/usr/local/bin/perl5 "-I." -Mconfigdata "util/dofile.pl"  "-oMakefile" crypto/include/internal/dso_conf.h.in > crypto/include/internal/dso_conf.h
/usr/local/bin/perl5 "-I." -Mconfigdata "util/dofile.pl"  "-oMakefile" include/openssl/opensslconf.h.in > include/openssl/opensslconf.h
</code>

====== Generate assembly .S files ======
From ''%%secure/libcrypto/Makefile.asm%%'':

<code>
# $FreeBSD: head/secure/lib/libcrypto/Makefile.asm 307976 2016-10-26 20:02:22Z jkim $
# Use this to help generate the asm *.S files after an import.  It is not
# perfect by any means, but does what is needed.
# Do a 'make -f Makefile.asm all' and it will generate *.S.  Move them
# to the arch subdir, and correct any exposed paths and $ FreeBSD $ tags.
</code>

Let's do that!

<code>
make -f Makefile.asm all
[Creating objdir /usr/obj/usr/src/amd64.amd64/secure/lib/libcrypto...]
( echo '/* $'FreeBSD'$ */' ; echo '/* Do not modify. This file is auto-generated from aes-x86_64.pl. */' ; env CC=cc perl /usr/src/crypto/openssl/crypto/aes/asm/aes-x86_64.pl elf ) > aes-x86_64.S
( echo '/* $'FreeBSD'$ */' ; echo '/* Do not modify. This file is auto-generated from aesni-mb-x86_64.pl. */' ; env CC=cc perl /usr/src/crypto/openssl/crypto/aes/asm/aesni-mb-x86_64.pl elf ) > aesni-mb-x86_64.S
...
( echo '/* $'FreeBSD'$ */' ; echo '/* Do not modify. This file is auto-generated from sha512-x86_64.pl. */' ; cat sha256-x86_64.s ) > sha256-x86_64.S
env CC=cc perl /usr/src/crypto/openssl/crypto/sha/asm/sha512-x86_64.pl elf sha512-x86_64.s
( echo '/* $'FreeBSD'$ */' ; echo '/* Do not modify. This file is auto-generated from sha512-x86_64.pl. */' ; cat sha512-x86_64.s ) > sha512-x86_64.S
</code>

**TODO: Verify that we have all necessary ''%%.S%%'' files**\\ 
The files processed are defined in the Makefile, which is thus version-specific.

===== Update Makefile.asm .pl sources =====

%%OpenSSL%% 1.1 added some perl files that generate ASM.
From ''%%${WRKSRC}/Configurations/00-base-templates.conf%%'' we can find out what %%OpenSSL%%'s current list of assembly files is per architecture.
Quite some additional files to be added. Chacha20/Poly1305 for all archs, some other for ARM(64).

====== Manpage generation ======

===== TL;DR =====

After fixing Makefile.inc ''%%MKLINK%%'' output

<code>
$ make -C /usr/src/secure/lib/libcrypto man-makefile-update
$ make -C /usr/src/secure/lib/libcrypto man-update
$ make -C /usr/src/secure/lib/libssl man-makefile-update
$ make -C /usr/src/secure/lib/libssl man-update
$ make -C /usr/src/secure/usr.bin/openssl man-makefile-update
$ make -C /usr/src/secure/usr.bin/openssl man-update
</code>

Work log:

Two targets in Makefile.inc: ''%%man-update%%'' and ''%%man-makefile-update%%''.

===== man-makefile-update =====
''%%Makefile.man%%'' is included for installing the manpages, the first lines indicate this is generated by the ''%%man-makefile-update%%'' update. Let's run that!

<code>
$ make man-makefile-update
rm -f /usr/src/secure/lib/libcrypto/Makefile.man
echo '# $'FreeBSD'$' >> /usr/src/secure/lib/libcrypto/Makefile.man
echo '# DO NOT EDIT: generated from man-makefile-update target' >>  /usr/src/secure/lib/libcrypto/Makefile.man
for i in crypto; do  for j in /usr/src/crypto/openssl/doc/$i/*.pod; do  test -f $j || continue;  fn=`basename $j .pod`;  if ! echo 'SSLeay_version des_modes' | grep -qw "$fn"; then  echo "MAN+= $fn.3";  fi;  done;  done | env LANG=C s
ort >> /usr/src/secure/lib/libcrypto/Makefile.man
for i in crypto; do  for j in /usr/src/crypto/openssl/doc/$i/*.pod; do  test -f $j || continue;  fn=`basename $j .pod`;  if ! echo 'SSLeay_version des_modes' | grep -qw "$fn"; then  perl /usr/src/crypto/openssl/util/extract-names.pl <
 $j |  awk "/^$fn\$/ { next; }  { print \"MLINKS+= $fn.3 \" \$1 \".3\" }";  fi;  done;  done | env LANG=C sort >> /usr/src/secure/lib/libcrypto/Makefile.man
Can't open perl script "/usr/src/crypto/openssl/util/extract-names.pl": No such file or directory
(repeat many times)
</code>

Big OOPS! Something changed in upstream %%OpenSSL%% that breaks this process. ''%%extract-names.pl%%'' no longer exists, replaced by ''%%process_docs.pl%%''.\\ 
Looks like first loop succeeds (''%%MAN+=%%''), but second loop (extract manpage aliases/links) does not (''%%MLINKS+=%%'').

===== man-update =====

Let's try ''%%man-update%%'' target

<code>
$ make man-update
ASN1_INTEGER_get_int64.3
ASN1_OBJECT_new.3
...
o2i_SCT_LIST.3
x509.3
</code>

Converts .pod into manpages with ''%%pod2man%%''

===== Iterate over libcrypto, libssl and apps =====

Looks like this has to be repeated for ''%%secure/lib/libcrypto%%'', ''%%secure/lib/libssl%%'' and ''%%secure/lib/openssl%%''
Let's try this in secure/lib/libssl

<code>
$ cd secure/lib/libssl
$ make man-update
[Creating objdir /usr/obj/usr/src/amd64.amd64/secure/lib/libssl...]
SSL_CIPHER_get_name.3
SSL_COMP_add_compression_method.3
...
SSL_CTX_free.3
SSL_CTX_get0_param.3
Can't open SSL_CTX_get_ex_new_index.pod: No such file or directory at /usr/local/bin/pod2man line 71.
*** Error code 2

Stop.
make: stopped in /usr/src/secure/lib/libssl
</code>

Another OOPS! Let's try ''%%man-makefile-update%%'' and then ''%%man-update%%''

<code>
$ make man-makefile-update
rm -f /usr/src/secure/lib/libssl/Makefile.man
echo '# $'FreeBSD'$' >> /usr/src/secure/lib/libssl/Makefile.man
echo '# DO NOT EDIT: generated from man-makefile-update target' >>  /usr/src/secure/lib/libssl/Makefile.man
for i in ssl; do  for j in /usr/src/crypto/openssl/doc/$i/*.pod; do  test -f $j || continue;  fn=`basename $j .pod`;  if ! echo 'SSLeay_version des_modes' | grep -qw "$fn"; then  echo "MAN+= $fn.3";  fi;  done;  done | env LANG=C sort
 >> /usr/src/secure/lib/libssl/Makefile.man
for i in ssl; do  for j in /usr/src/crypto/openssl/doc/$i/*.pod; do  test -f $j || continue;  fn=`basename $j .pod`;  if ! echo 'SSLeay_version des_modes' | grep -qw "$fn"; then  perl /usr/src/crypto/openssl/util/extract-names.pl < $j
 |  awk "/^$fn\$/ { next; }  { print \"MLINKS+= $fn.3 \" \$1 \".3\" }";  fi;  done;  done | env LANG=C sort >> /usr/src/secure/lib/libssl/Makefile.man
Can't open perl script "/usr/src/crypto/openssl/util/extract-names.pl": No such file or directory
(repeat many times)
$ make man-update
DTLSv1_listen.3
OPENSSL_init_ssl.3
...
d2i_SSL_SESSION.3
ssl.3
</code>

Conclusion: Part of man-makefile-update that collects ''%%.pod%%'' files works, part that extracts aliases doesn't.\\ 
<del>**TODO: Fix ''%%MLINKS+=%%'' part of ''%%man-makefile-update%%''** (hint: ''%%process-docs.pl%%'')</del>

===== extract-names.pl and process_docs.pl =====

extract-names.pl is the old and process_docs is the new script

==== extract-names.pl ====

Looks like all it does is read the ''%%=head1 NAME%%'' section of the ''%%.pod%%'' files and extract the comma-separated entries contained.\\ 
Example from ''%%ASN1_ITEM_lookup.pod%%'':

<code>
=head1 NAME

ASN1_ITEM_lookup, ASN1_ITEM_get - lookup ASN.1 structures
</code>

Results in a Perl hash-table

<code>
ASN1_ITEM_lookup
ASN1_ITEM_get
</code>

**NOTE:** The manpage here is ASN1_ITEM_lookup.3 so somewhere the perl script filters out that link to itself.\\ 
This should lead to a single sym-link ''%%ASN1_ITEM_get.3 -> ASN1_ITEM_lookup.3%%''

==== process_docs.pl ====

Looks like the part we're interested in is [[https://github.com/openssl/openssl/blob/master/util/process_docs.pl#L169|line 169 onwards]].
This uses a feature from %%OpenSSL%%::Util::Pod which isn't directly accessible.

An alternative in ''%%sed%%'' is

<code>
sed '/=head1 NAME/,/=head1 /! d;/=head1.*/d;/^\s*$/d;s/ - .*$//;s/,//g' BIO_ADDR.pod || tr ' ' '\n'
</code>

==== Diff for secure/lib/libcrypto/Makefile.inc ====

Includes the changes for the version

<code>
Index: Makefile.inc
===================================================================
--- Makefile.inc	(revision 335034)
+++ Makefile.inc	(working copy)
@@ -3,7 +3,7 @@
 .include <bsd.own.mk>

 # OpenSSL version used for manual page generation
-OPENSSL_VER=   1.0.2o
+OPENSSL_VER=   1.1.0h
 OPENSSL_DATE=  2018-03-27

 LCRYPTO_SRC=   ${SRCTOP}/crypto/openssl
@@ -106,7 +106,8 @@
			test -f $$j || continue; \
			fn=`basename $$j .pod`; \
			if ! echo '${_skip}' | grep -qw "$$fn"; then \
-		       perl ${LCRYPTO_SRC}/util/extract-names.pl < $$j | \
+		       sed '/=head1 NAME/,/^=.*/! d;/^=.*/d;/^\s*$$/d' $$j | tr '\n' ' ' | \
+			 sed 's/ - .*$$//;s/,/ /g;s/  / /g' | tr ' ' '\n' | \
			  awk "/^$$fn\$$/ { next; } \
			  { print \"MLINKS+= $$fn.${_sec} \" \$$1 \".${_sec}\" }"; \
			fi; \
</code>

====== Update secure/libcrypto/Makefile ======

Sources to be checked is ''%%crypto/*/build.info%%'' which defines the sources to be used. \\ 
Helper to extract SRCS+= from a dir

<code>
cd /usr/src/crypto/openssl/crypto
for file in */build.info ; do
   echo -e "\n$file"
   sed -E '/^SOURCE/,/^($|INCLUDE.*)/! d;/^INCLUDE.*/d;s/^SOURCE.*=//;s/\\//g;s/\{.*\}//g;s/[[:space:]]+/ /g;s/^ //;s/ $//;/^$/d' $file \
| tr ' ' '\n' | sort | tr '\n' ' ' | fold -s -w70 | \                                                                                                                                                                                  sed 's/$/\\/;s/^/	/;1s/^/SRCS+=/;$s/ \\//'                                                                                                                                                                                   done
</code>

copy-paste.