====== PackageKit backend for pkgng ======
  * Student: [[/MattWindsor|MattWindsor]] (mattbw@%%FreeBSD%%.org)
  * Mentor: [[/JulienLaffaye|JulienLaffaye]] (jlaffaye@%%FreeBSD%%.org)

===== Project description =====

**This and following sections are written in the future tense because they were written before the project was carried out.**

I intend to develop, test and document a packagekit backend for pkgng, ideally with the view of being able to use an existing packagekit frontend such as Apper to install, remove and upgrade packages on a %%FreeBSD%% system.  This will be written in C against the packagekit and libpkg APIs and glib, and the deliverable will be a .so file that can be used by %%PackageKit%% as a backend as well as the source code to compile it.

This would make system administration using binary packages easier in these use cases:

  * Situations where multiple systems running different operating systems need to be updated by the same people; for example, University Radio York runs both Debian and %%FreeBSD%% systems which could theoretically both be served by packagekit if a pkgng backend is created.
  * Users who wish to install binary packages on %%FreeBSD%%, but would prefer to use a graphical user interface;
  * Users who wish to install binary packages on %%FreeBSD%% and already know packagekit from an existing supported package manager.

There is an existing packagekit backend for %%FreeBSD%% Ports; ideally I would like to implement as much of the functionality already present in that backend as is applicable to pkgng.  However, the core functionality (getdepends, getdetails, getupdates, installpackages, refreshcache, removepackages, searchgroup, searchname and updatepackages) would be the main focus of at least the first leg of the project.

===== Approach to solving the problem =====

I will be using the pkgng library, libpkg, to create a C-based packagekit backend (as opposed to, say, the ports backend, which uses Ruby).  This allows for the backend to be a relatively natural gluing of the two systems.

I intend to make full use of %%FreeBSD%%'s unique features (sbufs, stdlib extensions, etc) as well as GLib's featureset where applicable and appropriate to make the backend development easier.

Due to problems encountered with the latest version of packagekit, initial development efforts are being concentrated on the (obsolete) version of packagekit in %%FreeBSD%% ports (0.6.11) with the aim of porting the code upwards as far as possible after a decent feature set has been achieved.

===== Deliverables =====

==== Pre-midterm ====

  * A packagekit backend that has tested support for basic package installing, removing and updating (the feature matrix features getdepends, getupdates, installpackages, refreshcache, removepackages, updatepackages);
  * If possible, support or substantial progress towards support for searchgroup, searchname and getdetails (otherwise these will be implemented post-midterm);
  * Constant releases on an at least weekly basis of experimental source code and instructions for building for use with the version of packagekit available for %%FreeBSD%%.

==== Post-midterm ====

  * A fully tested packagekit backend that provides all functionality currently ascribed to the "ports" backend with the exception of any functionality not implemented or applicable for pkgng;
  * Documentation from both user and developer perspectives explaining what packagekit is, how to obtain it, how to use it with pkgng (including with GUIs) and how to further develop it;
  * A codebase (verified via static code analysis tools and peer review) that conforms to style(9) and best practices, and is ready for extension to support any features unimplemented by the end of the project.

===== Milestones =====

  * Week one (June 17): **Start of coding**, pk_backend boilerplate, mapping %%PackageKit%% package IDs to libpkg names, ideally some way of testing this (which probably entails implementing package description). By the end of week 1 the backend should be compiling and presenting a (very empty) feature set to packagekit, and this will be tested using pkcon.  **Done**
  * Week two (June 24): package description implemented and contingency (as week 1 may be the first foray into actually programming against the APIs, I imagine it'll take longer to "get right")  **Done (plus %%GetFiles%%, %%GetRepoList%%)**
  * Week three (July 1): local package installation  **Replaced with remote package installation, which was done instead**
  * Week four (July 8): package removal  **Done**
  * Week five (July 15): contacting external repositories, package database updating **Done**
  * Week six (July 22): package updating and remote package installation **Remote package installation done, updating appears to be done but was never fully tested.**
  * Week seven (July 29): Contingency and **mid-term evaluation**; I'll probably be going away on family holiday this week, but will try to do some off-Internet work (documentation and testing?)
  * Week eight (August 5): Remove query-based jobs; improve events system to provide correct errors on job failure; more in-depth checking for any issues remaining from the jobs change. **Removal done, error reporting attempted but without much progress**
  * Week nine (August 12): %%GetDepends%%, %%SearchDetails%% and %%GetRequires%%; at this stage with the exception of Cancel the backend should be at feature parity with ports.  In-depth attempts to fix the install crash bug. Try implementing Cancel, %%RepoEnable%% (text file hackery probably), and %%WhatProvides%%; this can roll over into other weeks if needs be.  **%%GetDepends%% and %%GetRequires%% done; %%SearchDetails%% not.**
  * Week ten (August 19): Contingency for the above, and also preparation of test environment.
  * Week eleven (August 26): Tests for each idempotent action.  **Unit tests for most major utility/glue functions.**
  * Week twelve (September 2): Tests, where possible, for non-idempotent actions, jail/VM if possible. **Basic install/get-files/removal workflow test written.**
  * Week thirteen (September 9): Create a port and upload to ports tree.  Also, final call for updating %%PackageKit%%: if possible, updating; if not, contingency and cleanup.  **Port submitted for CFT.  %%PackageKit%% not updated yet.**
  * September 16: End of coding (soft) 
  * September 23: End of coding (hard) 
  * Post-%%GSoC%%: I will ideally continue to maintain the backend and oversee the implementation of any functionality left out during %%GSoC%%.

===== Test Plan =====

==== Initial Ideas ====

**To be discussed with my mentor.  Test environment bring-up in week 10.**

Current testing efforts are concentrated on informal checking of the output of ''%%pkcon%%'' as well as some experimental automated acceptance testing via running both ''%%pkcon%%'' and ''%%pkg%%'''s equivalent processes on a set of packages and seeing if the output of one can be "mocked up" by the output of the other.

As well as this, static analysis (clang static analyser as well as simple compiler warnings at WARNS level 7) is used.

Possible other forms of testing include:
  * unit testing  **Unit tests available (using ATF)**
  * finding test users
  * checking on multiple architectures and systems (URY systems may be useful here)  **Appears to work on a 9.1 32-bit system and 9.2 64-bit VM.**
  * fuzz testing via specifically injecting invalid %%PackageIDs%%  **Not done yet**

==== Tests Available ====

At time of writing there are:

  * unit tests (using ATF, these work; use "make tests" in the backend directory, then "atf-run|atf-report" in the backend, backend/jobs and backend/query directories);
  * functional tests in shell script comparing pkcon output to pkg (mostly broken, will be fixed shortly).

==== Running the Tests ====

To be updated as I go along.  Ideally this should be reproducible elsewhere.

  * Make a new %%FreeBSD%% jail.
  * Bootstrap pkg.
  * Get a copy of the ports tree; install subversion and kuya either as ports or packages.
  * ''%%svn clone https://socsvn.freebsd.org/socsvn/soc2013/mattbw/%%''.
  * Either install %%PackageKit%% and ''%%make && make install%%'' in the "backend" directory, or test the port by ''%%cd%%''ing to "backend/port" and running "make install".  Note that the port will lag behind the svn repo.
  * For unit tests, run ''%%make tests%%'' then ''%%atf-run|atf-report%%'' in the "backend" and "query" directories.
  * For functional tests, check the "tests" subdirectory of "backend".  Some of these will need the pkgng repo changing to the test repository ([[http://www-student.cs.york.ac.uk/~mbw500/testrepo]]). 

===== The Code =====

The code (may not be in a compileable state at any given time) is at https://svnweb.freebsd.org/socsvn/soc2013/mattbw/.  For cloning, replace "svnweb" with "socsvn".

I usually try to prefix any broken commits' messages with warnings (eg "(BROKEN)").

The main backend development effort is in the directory //backend//; other elements (such as test code, documentation and suchlike) lie outside of this directory.

**All code is GPL2.** (Sorry!)  This is because %%PackageKit%% itself is GPL2.  Any code that does not link against %%PackageKit%% (should any code exist) can probably be re-licenced as %%FreeBSD%%, but I'd rather not do so without legal assurances.

==== Compiling, Installing and Using ====

**Remember to install ports-mgmt/packagekit and ports-mgmt/pkg before trying to compile.**

=== Compiling ===

Unlike the %%PackageKit%% backends that live in %%PackageKit%%'s tree, this backend uses a BSD-style Makefile; this means that you just need to run ''%%make%%'' and ''%%sudo make install%%''.  Compilation should work with both %%FreeBSD%% GCC and clang; any failure to build using the normal settings should be classed as a bug.  (The port should build with clang's -Weverything -Werror, as long as system includes are disabled.)

=== Installing ===

''%%make install%%'' installs to the normal %%PackageKit%% backends directory (''%%/usr/local/lib/packagekit-backend%%'').

You will need to edit the %%PackageKit%% configuration file (''%%/usr/local/etc/PackageKit/PackageKit.conf%%''), changing the default backend from ''%%ports%%'' to ''%%pkgng%%''.

=== Using ===

The %%PackageKit%% CLI is called ''%%pkcon%%''.  Here are a few sample commands to try:

  * ''%%pkcon get-details PACKAGE-NAME%%'' - Retrieves a summary of the package.
  * ''%%pkcon search name NAME%%'' - Searches for a package by name.
  * ''%%pkcon search group NAME%%'' - Searches for a package by group (these are basically ports categories, but with different names).
  * ''%%pkcon get-files PACKAGE-NAME%%'' - Retrieves a list of files for that package (must be installed)
  * ''%%pkcon install PACKAGE-NAME%%'' - Installs a package.
  * ''%%pkcon remove PACKAGE-NAME%%'' - Removes a package.
  * ''%%pkcon update PACKAGE-NAME%%'' - Updates a package.
  * ''%%pkcon refresh%%'' - Updates the repository catalogues.
  * ''%%pkcon resolve PACKAGE-NAME%%'' - Mainly an internal command, this shows the mapping from a package name to a package.

The search and resolve commands can take filters; currently these are ''%%--filter installed%%'' and ''%%--filter "~installed"%%'' to limit the search/resolution to installed or non-installed packages respectively.

To specify a version for a package, use a //package ID// instead of a name.  These are strings of the format ''%%name;version;arch;repository%%'' (where the last two can be omitted, the repository is ''%%installed%%'' or ''%%repo-REPOSITORYNAME%%'', and the arch is for example ''%%freebsd:9:x86:32%%''.

**Note that you may need to run pkcon as root.**  Technically this should not be the case, but %%ConsoleKit%%/%%PolKit%% wouldn't work for me.

In addition to ''%%pkcon%%'', you can use the GNOME %%PackageKit%% frontent for a graphical user interface...

=== Graphical user interface ===

By installing ''%%ports-mgmt/gnome-packagekit%%'', you can use the GTK user interface (''%%gpk-application%%'').  Again, you may need to run this as root (or solve the mysteries of %%PolKit%% and %%ConsoleKit%%...!).

The GTK interface is fairly self-explanatory, and is based around a menu of possible packages.  You can select a package group to look through on the left hand side, or search using the search bar in the top-left.  Other actions are available from the top-level menu.

There appear to be some minor glitches with the GTK interface, which may be due to problems in the backend.  Hopefully these will be fixed eventually!

===== Further Work =====

  * Updating %%PackageKit%% to 0.7.x or 0.8.x (will need some changes to the backend);
  * Implementing Cancel and any other features "left behind";
  * Implementing %%SearchDetails%%;
  * Implementing some switches for existing features;
  * Bug fixing.

===== Notes =====

  * Check the README in the root of the branch for information about how to compile, bugs, etc. etc.
  * I'm generally most accessible via IRC (irc.freenode.net #pkgng and #%%PackageKit%%, EFnet #freebsd-soc).  During the %%GSoC%% period I will be using the nick **mattbw** exclusively.  Afterwards, I will be **mattbw** or **hayashi**.

===== Useful links =====

  * [[http://www.packagekit.org|PackageKit homepage]].
  * [[http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/captainhayashi/1|Original proposal]].
  * [[http://www.google-melange.com/gsoc/project/google/gsoc2013/captainhayashi/19001|Project page on Melange]].
  * [[/JustinMuniz]]'s GSoC2013 project [[SummerOfCode2013/pkgQtGtk]] covers very similar ground, so there will likely be overlap and co-operation.
  * [[http://gitorious.org/packagekit/packagekit/blobs/master/backends/PORTING.txt|Porting guide for PackageKit 0.8.x]], for use when porting 0.6.x code over.

----

{{tag>Historical}}