Differences

This shows you the differences between two versions of the page.

--- devsummit:201309:ports [2013/09/26 13:29] – MatthewSeaman
+++ devsummit:201309:ports [2021/04/25 10:00] (current) – Move to DevSummit namespace. JethroNederhof
@@ Line 26: / Line 26: @@
 Announcement: We now have INDEX-10 from portsnap
-===== PCBSD Package CDN  AllanJude =====
+===== PCBSD Package CDN - Allan Jude =====
 Packages sets would take mirrors weeks to upload.
@@ Line 32: / Line 32: @@
 pkgng: the master needs to be atomic.  If the pkg checksum doesn't match will fall back to alternative
-New high bandwidth master and using myCDN
+New high bandwidth master and using CDN
 rsync --delay-update  in two passes (files, then sym-links)
@@ Line 45: / Line 45: @@
 Servers sync last 15mins of traffic within seconds.  Staggered updates across servers.
-Automatic mirror selection.  GSLB from [[/ScaleEngine|ScaleEngine]].  LB checks mirrors for freshness on images and drops them from the real hosts.
+Automatic mirror selection.  GSLB from %%ScaleEngine%%.  LB checks mirrors for freshness on images and drops them from the real hosts.
 Conclusion: ZFS make replication easier and allows atomicity.
-PCBSD servers pull about 36TiB per day -- iso.cdn about 1 %%TiB%%/day, pkg.cdn ~85 %%GiB%%/day of that.
+PCBSD servers pull about 36TiB per month -- iso.cdn about 1 %%TiB%%/day, pkg.cdn ~85 %%GiB%%/day of that.
 bapt: want to look at this for %%FreeBSD%% pkg repo and steal any applicable ideas.
@@ Line 136: / Line 136: @@
 developer sub-package: should this be automatic or manual setup of sub-packages?  cf. PORTDOCS
 -- start by doing things manually, then provide more automation once it has matured.
+===== Package Sets -- Erwin =====
+Decouple src from package releases.
+Done:
+convert ports from CVS to SVN
+new distribution infrastructure
+new build cluster (not yet for old packages)
+pkgng
+Package/Ports releases:
+Wanted?
+How frequently?
+biweekly
++
+monthly
+Do we want a security branch?
+portssecteam@ mailing list -- but no formal ports sec team, no charter etc.  Needs to be formalised.
+Should they answer to portmgr or SO?  SO technically could just appoint a ports sec team (SO charter is surprisingly wide...)
+We rarely get advance notice of issues in ports.  So portssecteam does not need to be a small closed private team.
+Needs to expedite handling vulns that are already out in public.
+This is secteam responsibility, but t's already hard enough to stay on top of kernel+userland.
+secteam relies on knowing who has expertise for help on specific areas of kernel / userland.
+Doesn't apply with orts eg. gnome@ is an open mailing list and not a good place to discuss vulns.
+ports sec teams should have this knowledge.
+ports sec team has authority to override maintainer where necessary for security reasons. It's about
+quick response and authority.
+Need to find out who is on the team right now...  and are still available to work on stuff.
+The list isn't dead but ports sec team is not used properly at the moment.
+SO should answer to core.  Ports Sec Team should answer to SO.  Unfinished Charter is on website, needs
+review and ratification.  How can people join the ports security team?
+ports security team created by fission of secteam by Jacques around 2005.
+Member of PST at %%EuroBSDCon%% should meet up and formalize things.
+===== Package Set Policy - bapt =====
+Just an idea now: want to make it real starting next month.
+Want to create package sets quarterly, and maintain them properly.
+Q1
+Q2
+Q3
+Q4
+Weekly
+%%LatestQ%%
+branches in SVN.  Qx branches to be maintained for 1 year.  Only portmgr and PST
+can commit to Qx branches.
+Weekly is just a symlink to the latest package set pushed from build cluster each week.
+Security fixes only.
+packages which have security problems we can fix -- package will be removed from site
+and port will be marked restricted.  We need a mechanism to tell users they need to update
+to a new branch where no security update is available.
+pkg news to be displayed after pkg update.  Also add to Gnome etc. front-ends
+(Ref PC-BSD people for desktop support)
+To start on 1st Nov (Q4) and next year Q4 will be in Oct.  Need working Ports Sec Team,
+and working pkg build cluster.  Can start with currently available pkg-test cluster.
+Space usage on cluster with 5 package sets online at once.
+Will allow errata updates as well as security.  Will have weekly builds for testing, so
+should allow fixing errors before the big Qx releases.
+pkg-fallout: New hardware allows much more rigorous QA testing.  BD has finished setting up machines
+so expect to receive way more mails real soon.
+port maintainers' responsibility to contact portmgr or ports sec team if they have a security fix to push
+to Qx branches. (portmgr only for errata)
+Test update from Qx -> Qx+1, Qx -> Qx+2, Qx+1 -> Qx+2 (all combinations of supported branches)
+Huge need for base system people to inform portmgr of anything that will break ABI in enough time to
+build package sets for HEAD and test for regressions in the pkg build cluster.  Really important to know
+about breakages that will prevent running a newer jail for HEAD.
+Deadline for having working ports sec team?
+===== pkg_install removal - bapt =====
+We need to clean up pkg having dirty code to support pkg_install.  Supporting two versions is
+becoming harder and harder.  Can get rid of much dirty code in bsd.*.mk
+Once we have two or three Qx release sets working for users then we can say we don't need to support
+pkg_install any more.  portmaster works the same way with pkgng.  Binary packages can't be updated
+with pkg_install: so delete all and rebuild using pkgng.
+This is a big change late in the line of eg. 8.4-RELEASE.
+Need update to the pkg.txz package on pkg.freebsd.org so that it installs a more recent version
+Conversion is just
+/usr/sbin/pkg
+pkg2ng
+and then continue maintaining you system as previously if you don't want to switch to binary
+package management.
+We're blocked by pkg_install.
+We need a really strong PR campaign that this is coming.  6 months warning required.
+Bootstrapping pkgng when nothing else is installed still tells you to run pkg2ng
+Could just run a post install script.
+List of *shiny* new features that can be implmented in a post-pkg_install world to be included
+in the PR campaign.
+Could the ports tree support divergent feature sets for pkgng vs. pkg_install?
+Is very hard to maintain.  Involves lots of duplication.
+We don't have experience of a release which is pkgng-ified. (we should do by May
+when 10.?)
+Many large companies have already adopted pkgng.  Config management tools like
+puppet pretty much imply pkgng.
+Better to slip the pkg_install cut-off date if we discover a problem than delay pre-emptively.
+Do 3rd party tools support pkgng yet: puppet, chef, ansible, salt, webmin, packagekit: support
+is either already in the ports or ready to add to ports.
+Plans to fix huge securty hole which is pkgng bootstrapping?  This will be fixed by checking
+signatures on pkgs. and repo catalogues.  (Different certificates -- so you could bootstrap from
+an old CDRom, needs a cert less likely to be revoked)
+Automatic message warning when using pkg_install "84 days left until pkg_install goes away..."
+.3 doesn't provide pkg bootstrap.  Need to wait until April when 8.3-RELEASE expires.
+.3-RELEASE will be supported for it's lifetime.
+.4-RELEASE users will be told they have switch.
+===== Open tasks =====
+Package names: we're doing it badly for packages.  Renaming packages based on options based on options is really bad.
+If we want to be able to provide 3 versions of %%MySQL%% or php or whatever: different versions need different PKGNAMESUFFIX
+ports need distinct portnames.  We should not need LATEST_LINK.
+We should be able to get rid of all the 'portmaster -o ...' entries in UPDATING.
+(Don't use PREFIX -- that's used for different language versions eg p5- py37-)
+Distinct packages which happen to have the same name -- needs fixing.  Change in policy, as this used to be OK if the ports
+were in different categories.
+Most of the time the pkg name should reflect the directory name.  Need flexibility to override in some cases though.
+(This is the same issue as 'pkg install subversion' attempting to install 3 different versions of SVN).
+Do we really need four different versions of perl?
+Using metapackages eg. lang/python -- installs a mostly empty package which depends on the default version of python.
+options -- should be mostly converted into sub packages.  We shouldn't be aiming to support all possible different combinations
+of compilation options.  3 sorts of options.
+-- adds extra files (just use a sub package)
+-- depend on an alternative shlib (don't offer this
+in general, or provide flavours where sensible)
+-- enable or disable different compiled in modules (convert
+to using run-time modules)
+... perhaps also
+-- turn on or off compilation options or add an optional patch
+(use separate ports -- this is v. similar to type 2)
+We can't get rid of options entirely eg. bind
+We want to get rid of the origin as the package identifier and use the pkgname instead.
+Package names vary between linux distros.  Same thing with %%FreeBSD%%.  V. hard to coordinate even between
+just *BSD.
+DFly is switching to ports.  Will other BSDs follow suit?
+Convert build options into run-time modules (ref: nginx)
+----
+{{tag>Historical}}