Process to update %%OpenSSL%% in base\\ (for 1.1.1-pre8 WIP see [[OpenSSL/Base/Update111]]) ====== Summary ====== For the impatient... svn checkout https://svn.FreeBSD.org/base/head /usr/src fetch -o ~/patch-D15791 'https://reviews.freebsd.org/D15791?download=true' fetch -o ~/openssl-1.1.0h.tar.gz https://www.openssl.org/source/openssl-1.1.0h.tar.gz cd /usr/src patch -p0 < ~/patch-D15791 mv /usr/src/crypto/openssl /usr/src/crypto/openssl-1.0.2 tar xf ~/openssl-1.1.0h.tar.gz -C /usr/src/crypto mv /usr/src/crypto/openssl-1.1.0h /usr/src/crypto/openssl cd /usr/src/secure/lib/libcrypto pkg install perl5 rm amd64/* mv /usr/obj/usr/src/amd64.amd64/secure/lib/libcrypto/*.S amd64/ make man-makefile-update rm man/* make man-update cd ../libssl make man-makefile-update rm man/* make man-update ====== Preliminary steps ====== 1. Move current version of %%OpenSSL%% $ mv /usr/src/crypto/openssl /usr/src/crypto/openssl-1.0.2 2. Download, verify $ fetch https://www.openssl.org/source/openssl-1.1.0h.tar.gz openssl-1.1.0h.tar.gz 100% of 5295 kB 28 MBps 00m00s $ fetch https://www.openssl.org/source/openssl-1.1.0h.tar.gz.asc openssl-1.1.0h.tar.gz.asc 100% of 455 B 4490 kBps 00m00s $ sha256 ~/openssl-1.1.0h.tar.gz SHA256 (/root/openssl-1.1.0h.tar.gz) = 5835626cde9e99656585fc7aaa2302a73a7e1340bf8c14fd635a62c66802a517 $ grep 5835626cde9e99656585fc7aaa2302a73a7e1340bf8c14fd635a62c66802a517 /usr/ports/security/openssl-devel/* /usr/ports/security/openssl-devel/distinfo:SHA256 (openssl-1.1.0h.tar.gz) = 5835626cde9e99656585fc7aaa2302a73a7e1340bf8c14fd635a62c66802a517 $ gpg --verify openssl-1.1.0h.tar.gz.asc openssl-1.1.0h.tar.gz gpg: WARNING: using insecure memory! gpg: please see http://www.gnupg.org/documentation/faqs.html for more information gpg: Signature made Tue Mar 27 15:50:44 2018 CEST using RSA key ID 0E604491 gpg: Good signature from "Matt Caswell " gpg: aka "Matt Caswell " Primary key fingerprint: 8657 ABB2 60F0 56B1 E519 0839 D9C4 D26D 0E60 4491 3. Unpack, remove version from directory $ tar xf ~/openssl-1.1.0h.tar.gz -C /usr/src/crypto $ mv /usr/src/crypto/openssl-1.1.0h /usr/src/crypto/openssl 4. Update ''%%/usr/src/secure/lib/libcrypto/Makefile.inc%%'' ... # OpenSSL version used for manual page generation OPENSSL_VER= 1.1.0h OPENSSL_DATE= 2018-03-27 ... 5. Required tools: Perl We need ''%%pod2man%%'' from perl and ''%%perl%%'' to generate assembly from the perl scripts $ pkg install perl5 ... ====== Generate .h files ====== Complication here, as there are different ''%%opensslconf.h%%'' files for various arch. For now I generated an opensslconf.h like so $ uname -a FreeBSD freebsd12 12.0-CURRENT FreeBSD 12.0-CURRENT #0 r334983: Tue Jun 12 01:42:40 UTC 2018 root@releng3.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 $ ./config -v Operating system: amd64-whatever-freebsd Configuring for BSD-x86_64 /usr/local/bin/perl5 ./Configure BSD-x86_64 Configuring OpenSSL version 1.1.0h (0x1010008fL) no-asan [default] OPENSSL_NO_ASAN no-crypto-mdebug [default] OPENSSL_NO_CRYPTO_MDEBUG no-crypto-mdebug-backtrace [default] OPENSSL_NO_CRYPTO_MDEBUG_BACKTRACE no-ec_nistp_64_gcc_128 [default] OPENSSL_NO_EC_NISTP_64_GCC_128 no-egd [default] OPENSSL_NO_EGD no-fuzz-afl [default] OPENSSL_NO_FUZZ_AFL no-fuzz-libfuzzer [default] OPENSSL_NO_FUZZ_LIBFUZZER no-heartbeats [default] OPENSSL_NO_HEARTBEATS no-md2 [default] OPENSSL_NO_MD2 (skip dir) no-msan [default] OPENSSL_NO_MSAN no-rc5 [default] OPENSSL_NO_RC5 (skip dir) no-sctp [default] OPENSSL_NO_SCTP no-ssl-trace [default] OPENSSL_NO_SSL_TRACE no-ssl3 [default] OPENSSL_NO_SSL3 no-ssl3-method [default] OPENSSL_NO_SSL3_METHOD no-ubsan [default] OPENSSL_NO_UBSAN no-unit-test [default] OPENSSL_NO_UNIT_TEST no-weak-ssl-ciphers [default] OPENSSL_NO_WEAK_SSL_CIPHERS no-zlib [default] no-zlib-dynamic [default] Configuring for BSD-x86_64 CC =cc CFLAG =-DL_ENDIAN -Wall -O3 -pthread -D_THREAD_SAFE -D_REENTRANT SHARED_CFLAG =-fPIC DEFINES =DSO_DLFCN HAVE_DLFCN_H NDEBUG OPENSSL_THREADS OPENSSL_NO_STATIC_ENGINE OPENSSL_PIC OPENSSL_IA32_SSE2 OPENSSL_BN_ASM_MONT OPENSSL_BN_ASM_MONT5 OPENSSL_BN_ASM_GF2m SHA1_ASM SHA256_ASM SHA512_ASM RC4_ASM MD5_ASM AES_ASM VPAES_ASM BSAES_ASM GHASH_ASM ECP_NISTZ256_ASM PADLOCK_ASM POLY1305_ASM LFLAG = PLIB_LFLAG = EX_LIBS = APPS_OBJ = CPUID_OBJ =x86_64cpuid.o UPLINK_OBJ = BN_ASM =asm/x86_64-gcc.o x86_64-mont.o x86_64-mont5.o x86_64-gf2m.o rsaz_exp.o rsaz-x86_64.o rsaz-avx2.o EC_ASM =ecp_nistz256.o ecp_nistz256-x86_64.o DES_ENC =des_enc.o fcrypt_b.o AES_ENC =aes-x86_64.o vpaes-x86_64.o bsaes-x86_64.o aesni-x86_64.o aesni-sha1-x86_64.o aesni-sha256-x86_64.o aesni-mb-x86_64.o BF_ENC =bf_enc.o CAST_ENC =c_enc.o RC4_ENC =rc4-x86_64.o rc4-md5-x86_64.o RC5_ENC =rc5_enc.o MD5_OBJ_ASM =md5-x86_64.o SHA1_OBJ_ASM =sha1-x86_64.o sha256-x86_64.o sha512-x86_64.o sha1-mb-x86_64.o sha256-mb-x86_64.o RMD160_OBJ_ASM= CMLL_ENC =cmll-x86_64.o cmll_misc.o MODES_OBJ =ghash-x86_64.o aesni-gcm-x86_64.o PADLOCK_OBJ =e_padlock-x86_64.o CHACHA_ENC =chacha-x86_64.o POLY1305_OBJ =poly1305-x86_64.o BLAKE2_OBJ = PROCESSOR = RANLIB =ranlib ARFLAGS = PERL =/usr/local/bin/perl5 SIXTY_FOUR_BIT_LONG mode Configured for BSD-x86_64. **TODO: Generate proper opensslconf.h files for all archs**\\ Same applies to crypto/include/internal/bn_conf.h\\ This is a hint from the %%OpenSSL%% Makefile GENERATED_MANDATORY=crypto/include/internal/bn_conf.h crypto/include/internal/dso_conf.h include/openssl/opensslconf.h and from the make output /usr/local/bin/perl5 "-I." -Mconfigdata "util/dofile.pl" "-oMakefile" crypto/include/internal/bn_conf.h.in > crypto/include/internal/bn_conf.h /usr/local/bin/perl5 "-I." -Mconfigdata "util/dofile.pl" "-oMakefile" crypto/include/internal/dso_conf.h.in > crypto/include/internal/dso_conf.h /usr/local/bin/perl5 "-I." -Mconfigdata "util/dofile.pl" "-oMakefile" include/openssl/opensslconf.h.in > include/openssl/opensslconf.h ====== Generate assembly .S files ====== From ''%%secure/libcrypto/Makefile.asm%%'': # $FreeBSD: head/secure/lib/libcrypto/Makefile.asm 307976 2016-10-26 20:02:22Z jkim $ # Use this to help generate the asm *.S files after an import. It is not # perfect by any means, but does what is needed. # Do a 'make -f Makefile.asm all' and it will generate *.S. Move them # to the arch subdir, and correct any exposed paths and $ FreeBSD $ tags. Let's do that! make -f Makefile.asm all [Creating objdir /usr/obj/usr/src/amd64.amd64/secure/lib/libcrypto...] ( echo '/* $'FreeBSD'$ */' ; echo '/* Do not modify. This file is auto-generated from aes-x86_64.pl. */' ; env CC=cc perl /usr/src/crypto/openssl/crypto/aes/asm/aes-x86_64.pl elf ) > aes-x86_64.S ( echo '/* $'FreeBSD'$ */' ; echo '/* Do not modify. This file is auto-generated from aesni-mb-x86_64.pl. */' ; env CC=cc perl /usr/src/crypto/openssl/crypto/aes/asm/aesni-mb-x86_64.pl elf ) > aesni-mb-x86_64.S ... ( echo '/* $'FreeBSD'$ */' ; echo '/* Do not modify. This file is auto-generated from sha512-x86_64.pl. */' ; cat sha256-x86_64.s ) > sha256-x86_64.S env CC=cc perl /usr/src/crypto/openssl/crypto/sha/asm/sha512-x86_64.pl elf sha512-x86_64.s ( echo '/* $'FreeBSD'$ */' ; echo '/* Do not modify. This file is auto-generated from sha512-x86_64.pl. */' ; cat sha512-x86_64.s ) > sha512-x86_64.S **TODO: Verify that we have all necessary ''%%.S%%'' files**\\ The files processed are defined in the Makefile, which is thus version-specific. ===== Update Makefile.asm .pl sources ===== %%OpenSSL%% 1.1 added some perl files that generate ASM. From ''%%${WRKSRC}/Configurations/00-base-templates.conf%%'' we can find out what %%OpenSSL%%'s current list of assembly files is per architecture. Quite some additional files to be added. Chacha20/Poly1305 for all archs, some other for ARM(64). ====== Manpage generation ====== ===== TL;DR ===== After fixing Makefile.inc ''%%MKLINK%%'' output $ make -C /usr/src/secure/lib/libcrypto man-makefile-update $ make -C /usr/src/secure/lib/libcrypto man-update $ make -C /usr/src/secure/lib/libssl man-makefile-update $ make -C /usr/src/secure/lib/libssl man-update $ make -C /usr/src/secure/usr.bin/openssl man-makefile-update $ make -C /usr/src/secure/usr.bin/openssl man-update Work log: Two targets in Makefile.inc: ''%%man-update%%'' and ''%%man-makefile-update%%''. ===== man-makefile-update ===== ''%%Makefile.man%%'' is included for installing the manpages, the first lines indicate this is generated by the ''%%man-makefile-update%%'' update. Let's run that! $ make man-makefile-update rm -f /usr/src/secure/lib/libcrypto/Makefile.man echo '# $'FreeBSD'$' >> /usr/src/secure/lib/libcrypto/Makefile.man echo '# DO NOT EDIT: generated from man-makefile-update target' >> /usr/src/secure/lib/libcrypto/Makefile.man for i in crypto; do for j in /usr/src/crypto/openssl/doc/$i/*.pod; do test -f $j || continue; fn=`basename $j .pod`; if ! echo 'SSLeay_version des_modes' | grep -qw "$fn"; then echo "MAN+= $fn.3"; fi; done; done | env LANG=C s ort >> /usr/src/secure/lib/libcrypto/Makefile.man for i in crypto; do for j in /usr/src/crypto/openssl/doc/$i/*.pod; do test -f $j || continue; fn=`basename $j .pod`; if ! echo 'SSLeay_version des_modes' | grep -qw "$fn"; then perl /usr/src/crypto/openssl/util/extract-names.pl < $j | awk "/^$fn\$/ { next; } { print \"MLINKS+= $fn.3 \" \$1 \".3\" }"; fi; done; done | env LANG=C sort >> /usr/src/secure/lib/libcrypto/Makefile.man Can't open perl script "/usr/src/crypto/openssl/util/extract-names.pl": No such file or directory (repeat many times) Big OOPS! Something changed in upstream %%OpenSSL%% that breaks this process. ''%%extract-names.pl%%'' no longer exists, replaced by ''%%process_docs.pl%%''.\\ Looks like first loop succeeds (''%%MAN+=%%''), but second loop (extract manpage aliases/links) does not (''%%MLINKS+=%%''). ===== man-update ===== Let's try ''%%man-update%%'' target $ make man-update ASN1_INTEGER_get_int64.3 ASN1_OBJECT_new.3 ... o2i_SCT_LIST.3 x509.3 Converts .pod into manpages with ''%%pod2man%%'' ===== Iterate over libcrypto, libssl and apps ===== Looks like this has to be repeated for ''%%secure/lib/libcrypto%%'', ''%%secure/lib/libssl%%'' and ''%%secure/lib/openssl%%'' Let's try this in secure/lib/libssl $ cd secure/lib/libssl $ make man-update [Creating objdir /usr/obj/usr/src/amd64.amd64/secure/lib/libssl...] SSL_CIPHER_get_name.3 SSL_COMP_add_compression_method.3 ... SSL_CTX_free.3 SSL_CTX_get0_param.3 Can't open SSL_CTX_get_ex_new_index.pod: No such file or directory at /usr/local/bin/pod2man line 71. *** Error code 2 Stop. make: stopped in /usr/src/secure/lib/libssl Another OOPS! Let's try ''%%man-makefile-update%%'' and then ''%%man-update%%'' $ make man-makefile-update rm -f /usr/src/secure/lib/libssl/Makefile.man echo '# $'FreeBSD'$' >> /usr/src/secure/lib/libssl/Makefile.man echo '# DO NOT EDIT: generated from man-makefile-update target' >> /usr/src/secure/lib/libssl/Makefile.man for i in ssl; do for j in /usr/src/crypto/openssl/doc/$i/*.pod; do test -f $j || continue; fn=`basename $j .pod`; if ! echo 'SSLeay_version des_modes' | grep -qw "$fn"; then echo "MAN+= $fn.3"; fi; done; done | env LANG=C sort >> /usr/src/secure/lib/libssl/Makefile.man for i in ssl; do for j in /usr/src/crypto/openssl/doc/$i/*.pod; do test -f $j || continue; fn=`basename $j .pod`; if ! echo 'SSLeay_version des_modes' | grep -qw "$fn"; then perl /usr/src/crypto/openssl/util/extract-names.pl < $j | awk "/^$fn\$/ { next; } { print \"MLINKS+= $fn.3 \" \$1 \".3\" }"; fi; done; done | env LANG=C sort >> /usr/src/secure/lib/libssl/Makefile.man Can't open perl script "/usr/src/crypto/openssl/util/extract-names.pl": No such file or directory (repeat many times) $ make man-update DTLSv1_listen.3 OPENSSL_init_ssl.3 ... d2i_SSL_SESSION.3 ssl.3 Conclusion: Part of man-makefile-update that collects ''%%.pod%%'' files works, part that extracts aliases doesn't.\\ **TODO: Fix ''%%MLINKS+=%%'' part of ''%%man-makefile-update%%''** (hint: ''%%process-docs.pl%%'') ===== extract-names.pl and process_docs.pl ===== extract-names.pl is the old and process_docs is the new script ==== extract-names.pl ==== Looks like all it does is read the ''%%=head1 NAME%%'' section of the ''%%.pod%%'' files and extract the comma-separated entries contained.\\ Example from ''%%ASN1_ITEM_lookup.pod%%'': =head1 NAME ASN1_ITEM_lookup, ASN1_ITEM_get - lookup ASN.1 structures Results in a Perl hash-table ASN1_ITEM_lookup ASN1_ITEM_get **NOTE:** The manpage here is ASN1_ITEM_lookup.3 so somewhere the perl script filters out that link to itself.\\ This should lead to a single sym-link ''%%ASN1_ITEM_get.3 -> ASN1_ITEM_lookup.3%%'' ==== process_docs.pl ==== Looks like the part we're interested in is [[https://github.com/openssl/openssl/blob/master/util/process_docs.pl#L169|line 169 onwards]]. This uses a feature from %%OpenSSL%%::Util::Pod which isn't directly accessible. An alternative in ''%%sed%%'' is sed '/=head1 NAME/,/=head1 /! d;/=head1.*/d;/^\s*$/d;s/ - .*$//;s/,//g' BIO_ADDR.pod || tr ' ' '\n' ==== Diff for secure/lib/libcrypto/Makefile.inc ==== Includes the changes for the version Index: Makefile.inc =================================================================== --- Makefile.inc (revision 335034) +++ Makefile.inc (working copy) @@ -3,7 +3,7 @@ .include # OpenSSL version used for manual page generation -OPENSSL_VER= 1.0.2o +OPENSSL_VER= 1.1.0h OPENSSL_DATE= 2018-03-27 LCRYPTO_SRC= ${SRCTOP}/crypto/openssl @@ -106,7 +106,8 @@ test -f $$j || continue; \ fn=`basename $$j .pod`; \ if ! echo '${_skip}' | grep -qw "$$fn"; then \ - perl ${LCRYPTO_SRC}/util/extract-names.pl < $$j | \ + sed '/=head1 NAME/,/^=.*/! d;/^=.*/d;/^\s*$$/d' $$j | tr '\n' ' ' | \ + sed 's/ - .*$$//;s/,/ /g;s/ / /g' | tr ' ' '\n' | \ awk "/^$$fn\$$/ { next; } \ { print \"MLINKS+= $$fn.${_sec} \" \$$1 \".${_sec}\" }"; \ fi; \ ====== Update secure/libcrypto/Makefile ====== Sources to be checked is ''%%crypto/*/build.info%%'' which defines the sources to be used. \\ Helper to extract SRCS+= from a dir cd /usr/src/crypto/openssl/crypto for file in */build.info ; do echo -e "\n$file" sed -E '/^SOURCE/,/^($|INCLUDE.*)/! d;/^INCLUDE.*/d;s/^SOURCE.*=//;s/\\//g;s/\{.*\}//g;s/[[:space:]]+/ /g;s/^ //;s/ $//;/^$/d' $file \ | tr ' ' '\n' | sort | tr '\n' ' ' | fold -s -w70 | \ sed 's/$/\\/;s/^/ /;1s/^/SRCS+=/;$s/ \\//' done copy-paste.