The current pkg-tools uses an ineffective way of keeping track of installed packages and the goal is to make a selfcontained project replicating the original pkg_* programs with the use of the BerkeleyDB as a cache for making certain operations alot faster. Adding additional fields for the package install date and a more useful size information (human readable) could also be needed.

• Determine what information should be indexed in the database; Currently i'm indexing files installed by each package, and it's origin, by indexing each package the 'pkg_info -W' command executes much faster. By indexing it's origin, origin lookups work alot faster, but this is still a work in progress.
• Implement database support functions (lib); Functions are beeing continually added/removed as needed to make the queries easy and the database-layer abstract so that switching databases if needed, could be made easy for the future.
• Modify existing tools to make use of the database cache; So far pkg_info has been improved by using the cache, like the -W option, this is a work in progress and I'm planning to make better size queries, and perhaps adding more fields to the database (like install date, this would have to be discussed).
• Write and add a converter(pkg_convert) that caches /var/db/pkg into a database file so that you can make use of the cache
• Update the database indexes correctly in pkg_add/pkg_delete, so that you won't have to run pkg_convert after each add/delete.
• Change the output formatting of some of pkg_info's options, especially size.
• Identifying bottlenecks with profiling tools like gprof

If you have any suggestions or disagreements please contact me at andenore@FreeBSD.org

Last updated: 2008 07 15

Converter:

• Added convert program that indexes information to PKG_DBCACHE_FILE
• Added some features to lookup keys and print them (or print whole db)

Info:

• Improved performance for -W option using the dbcache
• Profiling showed that the default behavior for pkg_info was to read plist everytime even though not needed. Added check for this and speed improved significantly.
• Uses human readable output for -s (size option) I'm not sure if this breaks things, but it looks Ok. (The old output is available via the -b option)

Add:

• Indexes information to dbcache according to the add
• Added percentage progress for remote fetching ( -r option )
• Fixes dependencies when installing (scans installed ports, checks @pkgdep and adds matching dependency to the installing package's +REQUIRED_BY file)

Delete:

• Deindexes information according to the delete

Lib:

• Changed methods in match.c to use cache if available (matchallbyorigin, ispkginstalled)
• Added global variable 'database' of type DB* (the database object)
• Added database helper functions
• Added function to cache a Packagelist (cache_plist())
• The environment variable PKG_DBCAHE_FILE can be set to make the file location different than the default /var/db/pkg/pkgcache.db

Create:

• The -O option has been modified to cache information when installing ports from /usr/ports/

Packinglist:

• A @comment DATE:seconds-since-epoch has been added (affects add/create/info/lib)

If you would like to test the improved pkg_install tools here's what's to do:

1. Download the Source
2. Extract and make it with 'tar zxf pkg_improved.tar.gz ; cd pkg_install ; make'
3. Run pkg_convert with no arguments to build the cache ( './convert/pkg_convert' )
4. After this you can use the programs as you like, if you want to test it with ports, you could either copy the programs over the original in /usr/sbin/ or you could change the variables pointing to these programs in “/usr/ports/Mk/bsd.port.mk” (the variables that should be changed are PKG_CMD, PKG_ADD, PKG_DELETE, PKG_INFO and PKG_VERSION)

• Snapshot of the source (for those who want to test)
• Perforce depot: //depot/projects/soc2008/andenore_pkginstall/
• Blog (a little more info about the progress)