IntellegentDownloadManager [FreeBSD WIKI]

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Intelligent download manager for the ports collection ======
  * Student: [[/Ambarisha|Ambarisha]] (<ambarisha@FreeBSD.org>)
  * Mentor: [[/XinLi|XinLi]]

===== Project description =====

Ports collection, as of now, simply uses fetch(1) to download distfiles. There are a few drawback to this approach:

  * Duplicate downloads
  * Picking a mirror to download is a problem
  * Cannot cap download speed or number of connections

The requirements are clearly put down on the ideas page at [[IdeasPage/IDMS]]

===== Implementation =====

Client:

The client will be a replacement of the current "fetch" program used by the ports collection. It will be used in the exact same way as fetch, but instead of actually fetching the files, it requests the Download Manager Service to download the files. It will be blocking on the service till the download is complete. It can frequently request the service for status updates.

Service:

The service assigns a "worker" thread to handle each request. If the request is for a file which is already being downloaded, the same download will be used to serve both requests. The service will also maintain a record of the download speeds from each mirror site. This record will be used to assign priority to the mirror site and the workers attempt to get a requested file from a site with higher priority. The service also updates any client with the status of the download upon request.

===== Deliverables =====

1. DMS client\\ 
2. DMS service (basic)\\ 
3. DMS service (improved - status updates, duplicate requests, checksum verification)\\ 
4. Site profiling\\ 
5. Job migration\\ 
6. Command line status dump tool\\ 

===== Schedule =====

(Jun 17 - Jun 23)   DMS Client\\ 
(Jun  24 - July 7)  DMS Service (Basic)\\ 
(July 8 - July  21) DMS Service (Improved)\\ 
(July 22 - July 28) Buffer\\ 
(July 29 - Aug 11)  Site profiling\\ 
(Aug 12 - Aug 18)   Buffer\\ 
(Aug 19 - Sep 1)    Job Migration\\ 
(Sep  2 - Sep 8)    Buffer\\ 
(Sep  9 - Sep 15)   Command line dump tool\\ 
(Sep 16 - Sep 22)   Buffer\\ 

===== The Code =====

[[https://socsvn.freebsd.org/socsvn/soc2013/ambarisha/|SVN repository]]