• Check MASTER_SITE_BACKUP to see if the file is there → is it *completely* unfetchable or only unfetchable from all MASTER_SITES? ✔ implemented, not in cron yet, first check run going 200510132357

• Randomize recheck for known-good / known-bad urls, for a given site, so that we don't get synchronized and check millions every 14 days and few the rest of the time.

• Try GET after a failed HEAD, to see if a server (against the standard, but who cares about the standard?) replies differently?

• Try GET after a failed LIST, in case an ftp server doesn't permit LIST. See, e.g., http://www.freebsd.org/cgi/query-pr.cgi?q=90751

• Still some weirdness with timeouts vs. other errors

• ✔ Don't check any ports that:
  • set FETCH_ENV
  • set FETCH_CMD
  • have a do-fetch: target

• SecureIIS returns 406 error

• handle HTTP redirects to ftp: URL's

• make 421 There are too many anonymous users currently logged in. a permanently transient error.

• Chase down LWP saying that IO::Socket::INET says that the error is (Timeout) when connecting to a system that's not running an HTTP server.
  • Found it, but I don't like it. This is IO::Socket's connect() having an implementation of non-blocking connect for timeout that's incompatible with FreeBSD. Upgrading IO::Socket::INET at least got it to change to “connect: Invalid argument”, but it'd be nice to get to actually get “Connection refused”.

• Output DEPRECATED and EXPIRATION_DATE so people don't waste time on ports that will go away (or are motivated to fix before they go away)

• Say whether the file is on MASTER_SITE_BACKUP

• Levels of badness: right now it's just 100% unfetchable, what about levels at 25%, 50%, 75%? That would help point out impending problems, like the current ruby/contrib situation

• How many ports depend on this one, if it's unfetchable?

• Always output maintainer even if it's all fetchable?

• Suggestion from aDe: augmenting the ok/bad/skipped with a +/-/? character in the order they were checked (the default case for most people)

so you'd have something like: 3 ok, 2 bad, 1 skipped: +-?-++

• Identify groups of bad sites that are some MASTER_SITE_FOO and say “MASTER_SITE_FOO is all bad” instead of listing all sites.

• Want to have a lighter weight format for first viewing, then “drill down” if you're interested.

e.g., this is completely gratuitous:

File: arts-1.4.2.tar.bz2 has 106 possible URLs: 105 OK, 1 bad, 0 skipped 
Port maintainer: kde@FreeBSD.org 
ftp://ftp.scarlet.be/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://ftp.scarlet.be/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://mirrors.isc.org/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.gtlib.cc.gatech.edu/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://ftp.gtlib.cc.gatech.edu/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.oregonstate.edu/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.eu.uu.net/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.tiscali.nl/pub/mirrors/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://ftp.tiscali.nl/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.du.se/pub/mirrors/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://ftp.du.se/pub/mirrors/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.solnet.ch/mirror/KDE/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://sunsite.informatik.rwth-aachen.de/pub/Linux/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://ftp-stud.fht-esslingen.de/Mirrors/ftp.kde.org/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://ftp.gwdg.de/pub/x11/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://kde.mirrors.hoobly.com/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.informatik.uni-hamburg.de/pub/soft/X/contrib/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.rediris.es/mirror/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://ftp.rediris.es/mirror/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.mirrors.net.ar/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.man.szczecin.pl/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://gd.tuwien.ac.at/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://gd.tuwien.ac.at/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.informatik.hu-berlin.de/pub/Mirrors/ftp.kde.org/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://mirrors.dotsrc.org/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://mirrors.dotsrc.org/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://mirrors.ibiblio.org/pub/mirrors/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://ibiblio.org/pub/mirrors/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: 500 read timeout (Last OK result Sat Oct 8 01:07:02 2005 UTC) 
ftp://ftp.roedu.net/pub/mirrors/ftp.kde.org/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.fi.muni.cz/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.fh-heilbronn.de/pub/mirrors/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://ftp.fi.muni.cz/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.tuniv.szczecin.pl/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://ftp.tuniv.szczecin.pl/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://sunsite.icm.edu.pl/pub/unix/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://mirror.switch.ch/mirror/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://mirror.switch.ch/ftp/mirror/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.sunet.se/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://ftp.sunet.se/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.rz.uni-wuerzburg.de/pub/unix/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.dataplus.se/pub/linux/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.duth.gr/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://ftp.duth.gr/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.unina.it/pub/Linux/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://ftp.unina.it/pub/Linux/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.gwdg.de/pub/x11/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.kde.org.yu/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://ftp.kde.org.yu/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.tu-clausthal.de/pub/mirror/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.belnet.be/packages/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://ftp.belnet.be/packages/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.lip6.fr/pub/X11/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://www-ftp.lip6.fr/pub/X11/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.tu-chemnitz.de/pub/X11/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.uvsq.fr/pub/X11/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.rhnet.is/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://ftp.rhnet.is/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.esat.net/mirrors/ftp.kde.org/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://ftp.esat.net/mirrors/ftp.kde.org/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.mirror.ac.uk/sites/ftp.kde.org/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://download.mirror.ac.uk/sites/ftp.kde.org/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://kde.paralax.org/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.cronyx.ru/pub/mirror/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.fu-berlin.de/pub/unix/X11/gui/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://sunsite.sut.ac.jp/pub/archives/linux/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://sunsite.tus.ac.jp/pub/archives/linux/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://linux.cis.nctu.edu.tw/X/wm/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.na.kde.org/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.sun.ac.za/mirrorsites/ftp.kde.org/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://ftp.sun.ac.za/sites/ftp.kde.org/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.estpak.ee/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp-stud.fht-esslingen.de/pub/Mirrors/ftp.kde.org/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.kde.org/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.ntua.gr/pub/X11/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://ftp.ntua.gr/pub/X11/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.hol.gr/pub/mirror/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://ftp.hol.gr/mirror/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.pbone.net/mirror/ftp.kde.org/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.heanet.ie/mirrors/ftp.kde.org/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://ftp.heanet.ie/mirrors/ftp.kde.org/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://linux3.cc.ntu.edu.tw/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://linux3.cc.ntu.edu.tw/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://mi.mirror.garr.it/pub/mirrors/KDE/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://mi.mirror.garr.it/mirrors/KDE/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://mirror.xmission.com/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://mirror.xmission.com/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.iway.fr/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.kddlabs.co.jp/pub/X11/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://ftp.kddlabs.co.jp/pub/X11/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.no.kde.org/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.yz.yamagata-u.ac.jp/pub/X11/wm/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://ftp.yz.yamagata-u.ac.jp/pub/X11/wm/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.funet.fi/pub/mirrors/ftp.kde.org/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://ftp.funet.fi/pub/mirrors/ftp.kde.org/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.planetmirror.com/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://public.planetmirror.com/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://ftp.ussg.iu.edu/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://ftp.ussg.iu.edu/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://carroll.aset.psu.edu/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://chernabog.cc.vt.edu/pub/projects/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://chernabog.cc.vt.edu/pub/projects/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://mirror.cc.columbia.edu/pub/software/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://ftp.chg.ru/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://www.mirrorservice.org/sites/ftp.kde.org/pub/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
ftp://www.t.ring.gr.jp/pub/X/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok 
http://www.t.ring.gr.jp/archives/X/kde/stable/3.4.2/src/arts-1.4.2.tar.bz2: ok