Process to update %%OpenSSL%% in base\\ 
(for 1.1.0h WIP see [[OpenSSL/Base/Update110]])

====== Summary ======

For the impatient...

<code>
svn checkout https://svn.FreeBSD.org/base/head /usr/src
fetch -o ~/patch-D15791 'https://reviews.freebsd.org/D15791?download=true'
fetch -o ~/openssl-1.1.1-pre8.tar.gz https://www.openssl.org/source/openssl-1.1.1-pre8.tar.gz
cd /usr/src
patch -p0 < ~/patch-D15791
mv /usr/src/crypto/openssl /usr/src/crypto/openssl-1.0.2
tar xf ~/openssl-1.1.1-pre8.tar.gz -C /usr/src/crypto
mv /usr/src/crypto/openssl-1.1.1-pre8 /usr/src/crypto/openssl
cd /usr/src/secure/lib/libcrypto
pkg install perl5
rm amd64/*
mv /usr/obj/usr/src/amd64.amd64/secure/lib/libcrypto/*.S amd64/
make man-makefile-update
rm man/*
make man-update
</code>

====== Preliminary steps ======

1. Move current version of %%OpenSSL%%

<code>
$ mv /usr/src/crypto/openssl /usr/src/crypto/openssl-1.0.2o
</code>

2. Download, verify

<code>
$ fetch https://www.openssl.org/source/openssl-1.1.1-pre8.tar.gz
openssl-1.1.1-pre8.tar.gz                     100% of 8139 kB   41 MBps 00m00s
$ fetch https://www.openssl.org/source/openssl-1.1.1-pre8.tar.gz.asc
openssl-1.1.1-pre8.tar.gz.asc                 100% of  488  B 5178 kBps 00m00s
$ gpg --verify openssl-1.1.1-pre8.tar.gz.asc openssl-1.1.1-pre8.tar.gz
gpg: WARNING: using insecure memory!
gpg: please see http://www.gnupg.org/documentation/faqs.html for more information
gpg: Signature made Wed Jun 20 16:48:22 2018 CEST using RSA key ID 0E604491
gpg: Good signature from "Matt Caswell <matt@openssl.org>"
gpg:                 aka "Matt Caswell <frodo@baggins.org>"
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: 8657 ABB2 60F0 56B1 E519  0839 D9C4 D26D 0E60 4491
</code>

3. Unpack, remove version from directory

<code>
$ tar xf ~/openssl-1.1.1-pre8-tar.gz -C /usr/src/crypto
$ mv /usr/src/crypto/openssl-1.1.1-pre8 /usr/src/crypto/openssl
</code>

4. Update ''%%/usr/src/secure/lib/libcrypto/Makefile.inc%%''

<code>
...
# OpenSSL version used for manual page generation
OPENSSL_VER=    1.1.1.p8
OPENSSL_DATE=   2018-06-20
...
</code>

5. Required tools: Perl
We need ''%%pod2man%%'' from perl and ''%%perl%%'' to generate assembly from the perl scripts

<code>
$ pkg install perl5
...
</code>

====== Generate .h files ======

Complication here, as there are different ''%%opensslconf.h%%'' files for various arch. For now I generated an opensslconf.h like so

<code>
$ uname -a
FreeBSD freebsd12 12.0-CURRENT FreeBSD 12.0-CURRENT #0 r334983: Tue Jun 12 01:42:40 UTC 2018     root@releng3.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC  amd64
$ ./config -v --openssldir=/usr/local/openssl --prefix=/usr/local enable-ec_nistp_64_gcc_128 no-sm2 no-sm3
./config -v --openssldir=/usr/local/openssl --prefix=/usr/local enable-ec_nistp_64_gcc_128 no-sm2 no-sm3
Operating system: amd64-whatever-freebsd
/usr/bin/env __CNF_CPPDEFINES='' __CNF_CPPINCLUDES='' __CNF_CPPFLAGS='' __CNF_CFLAGS='' __CNF_CXXFLAGS='' __CNF_LDFLAGS='' __CNF_LDLIBS='' /usr/local/bin/perl5 ./Configure BSD-x86_64 '--openssldir=/usr/local/openssl' '--prefix=/usr/local' 'enable-ec_nistp_64_gcc_128' 'no-sm2' 'no-sm3'
Configuring OpenSSL version 1.1.1-pre8 (0x10101008L) for BSD-x86_64
Using os-specific seed configuration
Creating configdata.pm
Creating Makefile

**********************************************************************
***                                                                ***
***   If you want to report a building issue, please include the   ***
***   output from this command:                                    ***
***                                                                ***
***     perl configdata.pm --dump                                  ***
***                                                                ***
**********************************************************************
</code>

Now we can run ''%%make build_generated%%'' which will generate the files we want

<code>
$ make build_generated
/usr/local/bin/perl5 "-I." -Mconfigdata "util/dofile.pl"  "-oMakefile" crypto/include/internal/bn_conf.h.in > crypto/include/internal/bn_conf.h
/usr/local/bin/perl5 "-I." -Mconfigdata "util/dofile.pl"  "-oMakefile" crypto/include/internal/dso_conf.h.in > crypto/include/internal/dso_conf.h
/usr/local/bin/perl5 "-I." -Mconfigdata "util/dofile.pl"  "-oMakefile" include/openssl/opensslconf.h.in > include/openssl/opensslconf.h
</code>

Merge the diff into opensslconf-x86.h.in
**TODO: Generate proper opensslconf.h files for all archs**\\ 
Same applies to crypto/include/internal/bn_conf.h\\ 
This is a hint from the %%OpenSSL%% Makefile

<code>
GENERATED_MANDATORY=crypto/include/internal/bn_conf.h crypto/include/internal/dso_conf.h include/openssl/opensslconf.h
</code>

and from the make output

<code>
/usr/local/bin/perl5 "-I." -Mconfigdata "util/dofile.pl"  "-oMakefile" crypto/include/internal/bn_conf.h.in > crypto/include/internal/bn_conf.h
/usr/local/bin/perl5 "-I." -Mconfigdata "util/dofile.pl"  "-oMakefile" crypto/include/internal/dso_conf.h.in > crypto/include/internal/dso_conf.h
/usr/local/bin/perl5 "-I." -Mconfigdata "util/dofile.pl"  "-oMakefile" include/openssl/opensslconf.h.in > include/openssl/opensslconf.h
</code>

1.1.1 introduces yet another generate header, ''%%apps/progs.h%%''

====== Generate assembly .S files ======
From ''%%secure/libcrypto/Makefile.asm%%'':

<code>
# $FreeBSD: head/secure/lib/libcrypto/Makefile.asm 307976 2016-10-26 20:02:22Z jkim $
# Use this to help generate the asm *.S files after an import.  It is not
# perfect by any means, but does what is needed.
# Do a 'make -f Makefile.asm all' and it will generate *.S.  Move them
# to the arch subdir, and correct any exposed paths and $ FreeBSD $ tags.
</code>

Let's do that!

<code>
make -f Makefile.asm all
[Creating objdir /usr/obj/usr/src/amd64.amd64/secure/lib/libcrypto...]
( echo '/* $'FreeBSD'$ */' ; echo '/* Do not modify. This file is auto-generated from aes-x86_64.pl. */' ; env CC=cc perl /usr/src/crypto/openssl/crypto/aes/asm/aes-x86_64.pl elf ) > aes-x86_64.S
( echo '/* $'FreeBSD'$ */' ; echo '/* Do not modify. This file is auto-generated from aesni-mb-x86_64.pl. */' ; env CC=cc perl /usr/src/crypto/openssl/crypto/aes/asm/aesni-mb-x86_64.pl elf ) > aesni-mb-x86_64.S
...
( echo '/* $'FreeBSD'$ */' ; echo '/* Do not modify. This file is auto-generated from sha512-x86_64.pl. */' ; cat sha256-x86_64.s ) > sha256-x86_64.S
env CC=cc perl /usr/src/crypto/openssl/crypto/sha/asm/sha512-x86_64.pl elf sha512-x86_64.s
( echo '/* $'FreeBSD'$ */' ; echo '/* Do not modify. This file is auto-generated from sha512-x86_64.pl. */' ; cat sha512-x86_64.s ) > sha512-x86_64.S
</code>

**TODO: Verify that we have all necessary ''%%.S%%'' files**\\ 
The files processed are defined in the Makefile, which is thus version-specific.

===== Update Makefile.asm .pl sources =====

%%OpenSSL%% 1.1 added some perl files that generate ASM.
From ''%%${WRKSRC}/Configurations/00-base-templates.conf%%'' we can find out what %%OpenSSL%%'s current list of assembly files is per architecture.
Quite some additional files to be added. Chacha20/Poly1305 for all archs, some other for ARM(64).

====== Manpage generation ======

The structure in the source has changed, there's no distinction between the libs and/or apps. All we have is the ''%%docs%%'' directory containing sub-directories ''%%man1%%'' ''%%man3%%'' ''%%man5%%'' ''%%man7%%''. The ''%%man-update%%'' and ''%%man-makefile-update%%'' targets in ''%%secure/lib/libcrypto/Makefile.inc%%'' must be updated to reflect that.

Additionally, the scripts to convert the (Perl) ''%%.pod%%'' files to man-pages were changed.

All docs are now located in the ''%%secure/lib/libcrypto/man%%'' dir.

===== extract-names.pl and process_docs.pl =====

extract-names.pl is the old and process_docs is the new script

==== extract-names.pl ====

Looks like all it does is read the ''%%=head1 NAME%%'' section of the ''%%.pod%%'' files and extract the comma-separated entries contained.\\ 
Example from ''%%ASN1_ITEM_lookup.pod%%'':

<code>
=head1 NAME

ASN1_ITEM_lookup, ASN1_ITEM_get - lookup ASN.1 structures
</code>

Results in a Perl hash-table

<code>
ASN1_ITEM_lookup
ASN1_ITEM_get
</code>

**NOTE:** The manpage here is ASN1_ITEM_lookup.3 so somewhere the perl script filters out that link to itself.\\ 
This should lead to a single sym-link ''%%ASN1_ITEM_get.3 -> ASN1_ITEM_lookup.3%%''

==== process_docs.pl ====

Looks like the part we're interested in is ''%%https://github.com/openssl/openssl/blob/master/util/process_docs.pl#L169|line 169 onwards%%'' (This link is broken).

This uses a feature from %%OpenSSL%%::Util::Pod which isn't directly accessible.

An alternative in ''%%sed%%'' is

<code>
sed '/=head1 NAME/,/=head1 /! d;/=head1.*/d;/^\s*$/d;s/ - .*$//;s/,//g' BIO_ADDR.pod || tr ' ' '\n'
</code>

====== Update secure/lib/libcrypto/Makefile ======

Sources to be checked is ''%%crypto/*/build.info%%'' which defines the sources to be used. \\ 
Helper to extract SRCS+= from a dir

<code>
cd /usr/src/crypto/openssl/crypto
for file in */build.info ; do
   echo -e "\n# ${file%/build.info}"
   sed -E '/^SOURCE/,/^($|INCLUDE.*)/! d;/^INCLUDE.*/d;s/^SOURCE.*=//;s/\\//g;s/\{.*\}//g;s/[[:space:]]+/ /g;s/^ //;s/ $//;/^$/d' $file \
   | tr ' ' '\n' | sort | tr '\n' ' ' | fold -s -w70 \
   | sed 's/$/\\/;s/^/	/;1s/^/SRCS+=/;$s/ \\//'
   echo
done
</code>

copy-paste.