Ideas Web App Design Document

A project to replace the static ideas.xml based system for describing desired or in-progress projects that could use additional developer volunteers.

Update: A prototype is running at http://apps.stokely.org/ideas/

• April 8, 2008 - v0.94 - Advanced search is fully implemented with results pages providing RSS feeds for complex queries.
• March 25, 2008 - Implemented keyword searching of idea summaries and bodies. Results page includes an RSS feed to the ideas that match your query so you can subscribe and get notified whenver a new idea that matches your query is added.
• March 25, 2008 - Moved to http://apps.stokely.org/ideas, as there were problems with the DNS for test.kuaitech.com.

There are many drawbacks to the current system:

• Strict XML schema requires careful text editing in a tedious file format to add new ideas to the list.
• Anyone can propose an idea, but there is no feedback mechanism to separate the good ideas from the very bad ideas on the list.
  • Many senior developers have very different opinions about technical ideas. The current system does not provide an avenue for them to list alternate points of view, add detailed technical comments of suggestions or criticism, etc..
  • It is not possible to see the ideas with the most recent “buzz” of users commenting/voting positively on them.
• Notifications are limited to emails sent to busy CVS mailing lists with all other web changes, so there is no notification mechanism for just the ideas list or the particular ideas one is interested in.

I propose that we replace, or at least augment, the current ideas.xml static web page based system with a dynamic web application. This application should have the following features :

• Allow anyone to propose a new idea.
• Allow anyone to comment and vote on previously proposed ideas.
• Provide an RSS feed of the newest ideas.
• Provide an RSS feed of the comments/votes for any specific idea.
• Export the entire ideas list to something closely approximating our existing ideas.xml file.
• Allow one to sort and search the ideas list by category, proposer, votes, summary title, or full text.
• Anonymous ideas/comments are hidden by default until cleared by a moderator.
• Moderator bits to be set for certain users so that they can moderate the above (can subscribe to an rss file for unmoderated ideas and comments needing their attention).
• Import functionality to import the current ideas.xml file.

I propose that we provide this functionality with a Python/Django based application running on a jailed server somewhere with Apache, mod_python, and MySQL or Postgres. This has many advantages including high level UI libraries, strong separation between data models and views, and very quick templating for the content. A quick mockup using the CSS from the main freebsd.org site was made in a few hours and is available in Perforce (//depot/user/murray/www/apps/django/) with the same look and feel of the main web site, but dynamically allowing anyone to add ideas, comment on ideas by others, and sort the existing ideas by category, score, or title (no searching or import/export yet as of Friday March 21).

In order to support commenting/voting on ideas, we need to have logins. Spam will also be a concern and so we'll need to possibly consider CAPTCHAs before allowing the user to enter a new idea or comment. It is strongly desired that we not require all freebsd developers and users to remember yet another password. For this reason, the application will use OpenID so that login credentials can be shared with the tens of thousands of sites (including Flickr, .name domain names, and more).

Update: There are many flaws with the OpenID model, as discussed on countless blogs such as here http://idcorner.org/2007/08/22/the-problems-with-openid/, however I still feel for this use case of an unimportant login for the purpose fo voting on an online ideas page this is completely sufficient. This is just a convenience so that users don't have to use yet another userid and password before voting and commenting.

Currently it is running on a jailed personal server, apps.stokely.org/ideas. I would like to move it to the FreeBSD cluster on sky.freebsd.org or another machine.

1. Install Apache (New: these steps are now covered in the FreeBSD Handbook)
2. Install MySQL 5.x
3. Install Python >= 2.4.
4. Install latest Django from subversion repository.
5. Install latest django-registration package from subversion repository.
6. Link to each of the above from your site-packages directory on the site.
7. Check out the code for the ideas web app from perforce.
8. For testing purposes, run local django runserver commands from django docs.
9. For production, edit apache httpd.conf.
10. Create the database where the ideas tables will be held. mysql -u <admin> create database freebsd_ideas;
11. Generate the tables that the webapp needs. cd ~/ideas-webapp-code; python manage.py syncdb

The body of an idea that is submitted through the web app may contain a limited number of HTML tags, including: <b>, <ul>, <li>, <p>, <br>. To ensure that only these tags are accepted, the entire input string is first encoded so that < becomes &lt; and > becomes &gt; and & becomes &amp; with python's cgi.escape method. Then, string replacement is done between the escaped and unescaped versions of just those tags listed above. This partially/possibly escaped idea body is then written to the sql database. If there are no html tags at all included, then the idea is prefixed with <pre> and appended with </pre> so that it recreates the text formatting input in the browser when displayed on the idea detail pages.

Cookies will be used, but for this limited domain cross-site scripting should not be an issue. As this solution utlizes a database backend, SQL injection attacks are theoretically possible. In general the Django library provides strong input validation and there will be no SQL commands in the highest level application itself as the data models thoroughly abstract those details in Django and provide escaping for all input strings. Also, we should run this on a jailed server with nothing else in the same database so that there is nothing important to be lost in any event. The import/export functionality of the webapp will make it trivial to recover from accidental data loss (much more likely to happen by an administrator error, especially if I'm anywhere near it.)

I'm still in the process of checking the code into Perforce, trying to decide the right location. I think that the root level in our RCS for this class of web app should be www/apps/django. This allows us to introduce a django-specific directory hierarchy beneath it for specific projects that use this famework and can share templates and python code.

The high level directories under www/apps/django would be :

• www/apps/django/templates/
• www/apps/django/${application name}
• www/apps/django/scripts/ (tools/scripts not specific to any one application that uses django).

For this specific application, I would propose:

• www/apps/django/templates/ideas
• www/apps/django/ideas
  • feeds.py (the implementations of the various RSS feeds.)
  • urls.py (the acceptable url patterns and mappings to functions in views.py)
  • models.py (the data structures for ideas, votes, etc..)
  • views.py (the basic functions for each page, to manipulate data then pass it to template system).
  • sortheaders.py (contributed code from the web which implements table header sorting)
  • templatetags/ (some shortcuts for the templates).

The basic code is checked into Perforce currently in //depot/user/murray/www/apps/django/

The main page, which I propose be visited from http://<something>.freebsd.org/ideas/. (I'm happy with something == www, testing, webapps, or whatever) :

The idea submission form :

The idea details form (shows comments and votes for/against this idea) :