The current pkg-tools uses an ineffective way of keeping track of installed packages and the goal is to make a selfcontained project replicating the original pkg_* programs with the use of the BerkeleyDB as a cache for making certain operations alot faster. Adding additional fields for the package install date and a more useful size information (human readable) could also be needed.

• Determine what information should be indexed in the database; Currently i'm indexing files installed by each package, and it's origin, by indexing each package the 'pkg_info -W' command executes much faster. By indexing it's origin, origin lookups work alot faster, but this is still a work in progress.
• Implement database support functions (lib); Functions are beeing continually added/removed as needed to make the queries easy and the database-layer abstract so that switching databases if needed, could be made easy for the future.
• Modify existing tools to make use of the database cache; So far pkg_info has been improved by using the cache, like the -W option, this is a work in progress and I'm planning to make better size queries, and perhaps adding more fields to the database (like install date, this would have to be discussed).
• Write and add a converter(pkg_convert) that caches /var/db/pkg into a database file so that you can make use of the cache
• Update the database indexes correctly in pkg_add/pkg_delete, so that you won't have to run pkg_convert after each add/delete.
• Change the output formatting of some of pkg_info's options, especially size.
• Identifying bottlenecks with profiling tools like gprof

If you have any suggestions or disagreements please contact me at andenore@FreeBSD.org

Last updated: 2008 07 15

Converter:

• Added convert program that indexes information to PKG_DBCACHE_FILE
• Added some features to lookup keys and print them (or print whole db)

Info:

• Improved performance for -W option using the dbcache
• Profiling showed that the default behavior for pkg_info was to read plist everytime even though not needed. Added check for this and speed improved significantly.
• Uses human readable output for -s (size option) I'm not sure if this breaks things, but it looks Ok. (The old output is available via the -b option)

Add:

• Indexes information to dbcache according to the add
• Added percentage progress for remote fetching ( -r option )
• Fixes dependencies when installing (scans installed ports, checks @pkgdep and adds matching dependency to the installing package's +REQUIRED_BY file)

Delete:

• Deindexes information according to the delete

Lib:

• Changed methods in match.c to use cache if available (matchallbyorigin, ispkginstalled)
• Added global variable 'database' of type DB* (the database object)
• Added database helper functions
• Added function to cache a Packagelist (cache_plist())
• The environment variable PKG_DBCAHE_FILE can be set to make the file location different than the default /var/db/pkg/pkgcache.db

Create:

• The -O option has been modified to cache information when installing ports from /usr/ports/

Packinglist:

• A @comment DATE:seconds-since-epoch has been added (affects add/create/info/lib)

If you would like to test the improved pkg_install tools here's what's to do:

1. Download the Source
2. Extract and make it with 'tar zxf pkg_improved.tar.gz ; cd pkg_install ; make'
3. Run pkg_convert with no arguments to build the cache
4. After this you can use the programs as you like, if you want to test it with ports, you could either copy the programs over the original in /usr/sbin/ or you could change the variables pointing to these programs in “/usr/ports/Mk/bsd.port.mk” (the variables that should be changed are PKG_CMD, PKG_ADD, PKG_DELETE, PKG_INFO and PKG_VERSION)

• Snapshot of the source (for those who want to test)
• Perforce depot: //depot/projects/soc2008/andenore_pkginstall/
• Blog (a little more info about the progress)