Differences

This shows you the differences between two versions of the page.

--- usersfetchingsource [2013/01/28 16:30] – IsaacLevy
+++ usersfetchingsource [2018/06/09 03:38] (current) – historical page MarkLinimon
@@ Line 1: / Line 1: @@
--01-27
+Page created: 2013-01-27
-==== The Upcoming cvsup src/base Problem ====
+This page is being linked to by various *BSD news sources and user groups, (Patrick %%McEvoy%% et. al.), we'll try to keep this page up to date with the current state of this issue.  Eventually, I hope this page exists merely to announce it's own deprecation- once there is a resolution accepted the %%FreeBSD%% community at large.
-For %%FreeBSD%% users, (particularly systems administrators and integrators), fetching source has always been easy - via either c[v]sup or CTM.  Until recently, they have also been used to fetch the ports collection.  CTM has not been heavily promoted and has not been as widely used as c[v]sup.
+==== UPDATE 2 : 2015-08-06 ====
-CVS has been deprecated in %%FreeBSD%% development for several years now, and all %%FreeBSD%% development has moved to the more modern [[http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/svn.html|SVN]].  While [[http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/svn.html|SVN]] is a great leap forward for the development process, c[v]sup has been heavily used by server administrators and system integrators for source management - and a replacement has fallen behind as the CVS infrastructure is being phased out.
+Most of the contents of this page are seriously out of date.
-=== At the time of this writing (2013-01-27): ===
+There is now an in-base version of svn, as /usr/bin/svnlite. This is a stripped down but fully functional client but has the bulky API / language bindings removed.  It has a relatively lightweight footprint and no external dependencies.  svnlite will make use of the ca_root_nss package in order to validate https certificates, if it is present.
-  * ports, cvsup access **end-of-lifed Feb 28, 2013**
+If you do not have svnlite, or prefer to use the official packaged version, it can be securely obtained:
-  * source, cvsup access deprecated - **end-of-life date not set**
-[[http://www.mavetju.org/mail/view_message.php?list=freebsd-current&id=209027|Deja-vu…  This reminds me of cvsup+modula-3.]] - [[http://lists.nycbug.org/pipermail/talk/2013-January/014613.html|Spork, NYC*BUG]]
+<code>
+# pkg install ca_root_nss
+# pkg install subversion
+</code>
-==== Possible Solutions ====
+If you wish to use secure https urls with svnlite, you should install the ca_root_nss package to obtain the current root certificate lists.
-The objective: get sources for everything using just the tools in base.
+<code>
+# pkg install ca_root_nss
+</code>
-  - [[http://svnweb.freebsd.org/base/user/des/svnsup/|svnsup]] or some downloader in base which can pull deltas from SVN servers. [[http://svnweb.freebsd.org/base/user/des/svnsup/|svnsup]] was started by [[DagErlingSmørgrav|DES]], but notcomplete. **UPDATE: [[http://lists.freebsd.org/pipermail/freebsd-stable/2013-January/071841.html|John Mehr is currently working on "a "lightweight, dependency-free, BSD licensed program to pull source using the svn protocol"]].** This feels like the obvious solution, to put a small program in base.  [[http://www.freebsd.org/cgi/man.cgi?query=portsnap&apropos=0&sektion=0&manpath=FreeBSD+9.1-RELEASE&arch=default&format=html|portsnap(8)]] is in base, for similar reason [[http://lists.freebsd.org/pipermail/freebsd-stable/2013-January/071909.html|Arrigo Marchiori discusses DES's svnup]]
+You should use https://svn.freebsd.org/ where practical and so long as ca_root_nss is installed and up to date, the certificate should be automatically checked and securely verified.
+svn.freebsd.org connects you to the closest project-operated mirror.  The old legacy mirror names should not be used.
+Official documentation is here: [[https://www.freebsd.org/doc/handbook/svn.html]]
+As of writing:
+  * cvs and cvsup is fully removed and should be ignored.
+  * ctm is obsolete and is thought to be dangerous. Its use should be discouraged.
+  * svnup can now use https to access svn.freebsd.org, but it needs the ca_root_nss package.
+  * Any remaining references to 'pkg_add -r' should be replaced with 'pkg add'.
+  * svnup is not expected to be added to base but will remain in ports/packages.
+  * ctm is expected to be removed from base and moved to ports/packages.
+  * The %%FreeBSD%% project encourages the use of svn or svnlite as the preferred source access method.
+Partially obsolete information follows:
+==== UPDATE ====
+There is now a working **svnup(1)** utility, which is tested functional!
+Fetch ports sources, then:
+<code>
+# pkg install svnup
+# svnup stable -h svn.freebsd.org
+</code>
+Currently it's a port- but it's such a small binary, it's absolutely a perfect candidate to replace csup(1) in base!  Please note, however, that this port does not yet support fetching over a secure channel (i.e. https), and therefore should only be used on trusted networks.
+=== Legacy cvsup Status, (updated 2015-05-26): ===
+  * source, ports, cvsup access fully removed.
+----
+==== The src/base Fetch Problem ====
+At the time of this writing, there is no //in-base// solution for fetching canonical RELEASE sources, (to build and maintain production grade systems).
+**UPDATE: no longer true; svnlite is in-base**
+For %%FreeBSD%% users, (particularly systems administrators and integrators), fetching source has always been easy - via either c[v]sup or CTM.  Until recently, these tools could also be used to fetch the ports collection.  CTM has not been heavily promoted and has not been as widely used as c[v]sup, but has a strong user base.
+[[CvsIsDeprecated|CVS has been deprecated in FreeBSD development for several years now]], and all %%FreeBSD%% development has moved to the more modern [[http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/svn.html|SVN]].  While [[http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/svn.html|SVN]] is a great leap forward for the development process, c[v]sup has been heavily used by server administrators and system integrators for source management - and a replacement has fallen behind as the CVS infrastructure is being phased out.
+==== Other methods of fetching source ====
+The objective: ability to fetch sources for everything using just the tools in base.
+  - svnup(1) a tiny utility which can pull deltas from SVN servers:
+**UPDATE: An early version is functional, but not in base. (2013 02 23)**
+The work has been added as a port, **net/svnup**.
+[[http://lists.freebsd.org/pipermail/freebsd-stable/2013-February/072026.html|John Mehr is currently working on "a "lightweight, dependency-free, BSD licensed program to pull source using the svn protocol]].  svnup is still being optimized, but is tested functional.  The original (very small) sources are available [[http://jcm.dsl.visi.com/freebsd/svnup/|on his website]].
+This feels like the obvious solution, to put a small program in base.  [[http://www.freebsd.org/cgi/man.cgi?query=portsnap&apropos=0&sektion=0&manpath=FreeBSD+9.1-RELEASE&arch=default&format=html|portsnap(8)]] is in base, for similar reason
+[[http://lists.freebsd.org/pipermail/freebsd-stable/2013-January/071909.html|Arrigo Marchiori discusses DES's svnup]]
 [[http://lists.freebsd.org/pipermail/freebsd-stable/2013-January/071836.html} I personally (maintainer of subversion port!) prefer csup over all other methods for "non-developers" systems too, and I'll be happy to see "svnup" when (if?) it will be created...]]
 - Black Lion AKA Lev Serebryakov %%<%%lev@%%FreeBSD%%.org>
+An early [[http://svnweb.freebsd.org/base/user/des/svnsup/|svnsup]] was started by [[DagErlingSmørgrav|DES]], but not completed.
   - [[http://www.freebsd.org/cgi/man.cgi?query=freebsd-update&apropos=0&sektion=0&manpath=FreeBSD+9.1-RELEASE&arch=default&format=html|freebsd-update(8)]] can be configured, via freebsd-update.conf(5), to download src only.  Updates are signed and automatically verified by freebsd-update.
@@ Line 29: / Line 94: @@
   - [[http://www.freebsd.org/cgi/man.cgi?query=ctm&sektion=1&apropos=0&manpath=FreeBSD+9.1-RELEASE|ctm(1)]] provides updates to all stable branches (4 through 9) and -current as well as the entire SVN repository.  It can function in either push mode (subscribe to the relevant mailing list and receive deltas) or pull mode (fetch deltas from a FTP mirror).
-    * Can only currently mirror -stable branches and -current.  Mirroring -release branches requires a different tool - freebsd-update(8) is the obvious choice
+    * Can only currently mirror -stable branches and -current.  Mirroring -release branches requires a different tool
-    * Pull mode is not secure (unless using HTTPS).  Verifying signatures in email requires installing one of the PGP ports,.
+    * Pull mode is not secured out of the box, (c[v]sup shares this problem, though it validates checksums).  Verifying signatures (in email, directly from shell, or by other means) requires installing PGP (one of the GPG ports, or source).
+  - subversion-static port, just added to the ports tree early Feb 2013.  This port aims to be a nearly dependency-free SVN build, built statically, as not to bring unwanted dependencies onto a system.  While this is an awesome leap foreword for developers, it still carries a lot of the problems inherent in merely fetching base/src.  The subversion-static port appears to be a lightweight and feature complete svn client for developers needs, as well.
-  - subversion-static port, just added to the ports tree this week, but not yet quite functional.  This aims to be a nearly dependency-free SVN build, built statically, as not to bring unwanted dependencies onto a system.  While this is an awesome leap foreword for developers, it still carries a lot of the problems inherent in merely fetching base/src.
+  - CTMup **UPDATE: stopped work, now that svnup(1) works** [[http://lists.freebsd.org/pipermail/ctm-users/2013-January/000399.html|Isaac (.ike) Levy was working on a c[v sup workalike in shell, which uses ctm(1)]].  Peculariaties of the ctm(1), particularly error/exit codes, made this a time consuming approach to implement.  This 'ctmup' utility was meant to get a c[v]sup workalike out there fast, but it is not at all meant to replace the svnup work.
-  - No in-base solution.  (possibly the worst solution).  [[http://www.freebsd.org/cgi/man.cgi?query=fetch&apropos=0&sektion=0&manpath=FreeBSD+9.1-RELEASE&arch=default&format=html|fetch(1)]] exists.  Leave users to their own devices- to download src.tbz tarballs via ftp/http infrastructure.  Sorting out REL, RELENG, and STABLE from these tarballs is *quite* the trick- and obviously could potentially soak more bandwidth from Project online resources.  Checksum/mtree userland tricks will pop up, hunting down signature files will become a new hassle.  Feels like it's back to the stone age with this one- (but perhaps that's good? back to basics?)
+  - No in-base solution - let users fetch/package/distribute sources themselves.  (possibly the worst solution).  [[http://www.freebsd.org/cgi/man.cgi?query=fetch&apropos=0&sektion=0&manpath=FreeBSD+9.1-RELEASE&arch=default&format=html|fetch(1)]] exists.  Leave users to their own devices- to download src.tbz tarballs via ftp/http infrastructure.  Sorting out REL, RELENG, and STABLE from these tarballs is *quite* the trick- and obviously could potentially soak more bandwidth from Project online resources.  Checksum/mtree userland tricks will pop up, hunting down signature files will become a new hassle.  Feels like it's back to the stone age with this.
 ==== FAQ (from lists) ====
 === Q. Why is cvsup/CVS deprecated? ===
-  - The %%FreeBSD%% project has been migrating development from SVN to CVS, since 2008.  Active development using the CVS repositories is over, and cvs service is currently becoming extra management overhead.  It needs to go, calmly.  The requirement to support the CVS exporter prevents use of some SVN functionality.
+  - The %%FreeBSD%% project has been migrating development from CVS to SVN, since 2008.  The CVS repository no longer exists.
 === Q. Why is installing SVN such a big deal for some users? ===
@@ Line 64: / Line 131: @@
     * %%FreeBSD%% users/admins typically **adore** C, UNIX.
-=== Q. Why would users/admins want to install anything other than REL? ===
+=== Q. Why would users/admins want to install anything other than RELEASE? ===
-  - Administrators run "production" environments using all of REL, RELENG, STABLE, CURRENT.
+  - Administrators run "production" environments using all of RELEASE, RELENG, STABLE, CURRENT.
-    * **REL** - most folks live here, it is the conservative/stable choice
+    * **RELEASE** - most folks live here, it is the conservative/stable choice
-    * **RELENG** - sometimes for a
+    * **RELENG** -
+      * sometimes for a security fix
       * sometimes for a bugfix
-      * sometimes for a feature
     * **STABLE**
       * prepping an environment for upcoming releases
-      * contain a necessary bugfix, security fix, or other necessary feature (often A/B testing style, running on selected/safe machines etc...)
+      * contain a necessary bugfix, new feature or other reason (often A/B testing style, running on selected/safe machines etc...)
     * **CURRENT** - yes, it happens in "prod" (typically if CURRENT hits production, some extreme bug/security fix is being "back-patched" for temporary resolution)
@@ Line 154: / Line 221: @@
 ----
-=== NEW canonical source fetching (work in progress behind SVN move): ===
+=== NEW canonical source fetching: ===
-  - Install subversion - at the time of this writing, [[http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/svn.html|choose your own adventure]]
+  - Ensure ca_root_nss is installed
-  - ''%% #noop# pkg_add -r subversion %%''
+<code>
-  - ''%% #noop# pkg install devel/subversion %%''} //(**noop note**: At the time of this writing, due to the security incident cleanup, no binaries available)//
+# pkg install ca_root_nss
+</code>
-  - fetch ports using [[http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/ports-using.html|portsnap]], and familiarize yourself with the new pkg(8) tool [[http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/pkgng-intro.html|pkgng or pkg(8)]], and don't forget there are still old tools necessary, like [[http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/ports-nextsteps.html|pkg_info]]
+  - If svnlite is not installed, or you prefer the official svn build
-  - Finally install subversion,
 <code>
-# cd /usr/ports/devel/subversion
+# pkg install subversion
-# make install clean
 </code>
-Now that we have subversion, [[http://lists.freebsd.org/pipermail/freebsd-stable/2013-January/071832.html|we just acquired this on our system too]]:
+  - Finally, use svnlite or svn to check out base, (typically to /usr/src)
-  * SQLite3 (not the SQLite in base
-  * APR
-  * Expat 2.x (not the Expat/libbsdxml now in base)
-  * Neon or Serf
-  * GNU gettext and libintl
-  * libiconv
-Any of these could be a problem/conflict/undesirable for system integrators.  APR for example could be a big problem if you're next steps are to load up a particular version of the Apache web server, (like a great number of %%FreeBSD%% users do).
+<code>
+# svnlite checkout https://svn.FreeBSD.org/base/head /usr/src
+</code>
-  - Finally, use svn to check out base, (typically to /usr/src)
+or:
 <code>
-# svn checkout https://svn0.us-east.FreeBSD.org/base/head /usr/src
+# svn checkout https://svn.FreeBSD.org/base/head /usr/src
-Error validating server certificate for 'https://svn0.us-east.freebsd.org:443':
-- The certificate is not issued by a trusted authority. Use the
-  fingerprint to validate the certificate manually!
-- The certificate hostname does not match.
-Certificate information:
-- Hostname: svnmir.nyi.FreeBSD.org
-- Valid: from Aug 12 23:01:31 2012 GMT until Aug 12 23:01:31 2013 GMT
-- Issuer: clusteradm, FreeBSD.org, (null), CA, US (clusteradm@FreeBSD.org)
-- Fingerprint: 06:D1:23:DE:5E:7A:F7:2B:7A:7E:74:95:5F:54:8D:5C:B0:D6:2E:8F
-(R)eject, accept (t)emporarily or accept (p)permanently?
 </code>
-Decide how to handle this SSL cert, ([[http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/svn-mirrors.html|compare the Fingerprint manually]] until the dust settles through the transition):
   - # move on to [[http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/makeworld.html|buildworld/buildkernel dance]]...
@@ Line 215: / Line 264: @@
 Kent S. Gordon, 1997
 http://www.mavetju.org/mail/view_message.php?list=freebsd-current&id=209027
+----
+{{tag>Historical}}