Show pageOld revisionsBacklinksBack to top This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong. ===== Ideas Page ===== ===== Introduction ===== Over the years, the %%FreeBSD%% Project has built up a list of ideas for implementation work that seems like it might be a good idea, but no hands are available to do it. If you would like to contribute to the %%FreeBSD%% Project, you might peruse this list to get a sense of the kinds of work that others have thought might be useful or available to do. Obviously, contributions are not limited to this list! Please contact us before starting on it though -- sometimes items remain on the list after they are completed, and sometimes they are just ideas, rather than a recipe for success. Searching our mailing list archives may turn up the discussions leading to ideas being put on the list. Frequently the initial goal would be to simply investigate the idea, rather than produce code. Many project ideas list contacts, who it is worth sending an e-mail. Otherwise (and perhaps as well), send e-mail to our hackers@ mailing list. **For Google Summer of Code Students** Ideas on this list are generally unsuitable for Summer of Code projects (too large, too small, no available mentor, etc) but there may still be ideas here that could form the basis of a good %%GSoC%% task. See: [[SummerOfCodeIdeas|FreeBSD Google Summer of Code Ideas]] <callout type="warning"> **Important**: When adding an idea, include a contact method, so interested parties can get in touch. </callout> ====== Wireless ====== Wireless ideas can be found on [[WiFi/IdeasPage]]. ====== Embedded ====== ====== File Systems ====== ===== Improve the performance of dump/restore ===== == Description == A performance evaluation of the split cache (as is) and an unified cache (like e.g. %%NetBSD%%) would be interesting. More details in [[http://lists.freebsd.org/pipermail/freebsd-hackers/2007-February/019666.html|this mail]] to the hackers mailing list. Additional improvements are welcome too. == Requirements == * Knowledge of C programming. * Basic understanding of backup/restore procedures. ===== OpenBFS ===== [[http://nobius.org/~dbg/practical-file-system-design.pdf|Practical Filesystem Design Book]] == Description == Before working at Apple, Dominic Giampaolo wrote a 64 bit journalling filesystem for %%BeOS%%. The objectives were to have a modern filesystem capable of streaming for multimedia applications. It also supports indexing and Extended Attributes for desktop use. [[https://www.haiku-os.org/|Haiku]] has [[https://git.haiku-os.org/haiku/tree/src/add-ons/kernel/file_systems/bfs|implemented]] a version of the filesystem based on the documentation which is used as their base filesystem and is available under an MIT license. Porting it, or rather using the existing implementation as the basis for a new one, would make an interesting case for an alternative filesystem. == Requirements == * Ability to read and understand foreign C code, familiarity with %%FreeBSD%%'s VFS is a plus. * Ability to interpret results from testsuites and find solutions. * Knowledge of filesystem internals. ===== Port Hammer2 ===== [[https://en.wikipedia.org/wiki/HAMMER2|HAMMER2 Filesystem]] == Description == The [[https://www.dragonflybsd.org/hammer/|Hammer]] is a recent copy-on-write filesystem developed by Matt Dillon as part of [[https://www.dragonflybsd.org/|DragonFlyBSD]]. Due to the differences in evolution in %%DragonFly%% it would be very difficult to port the original hammer but it should be possible to port Hammer2, which has most of the same advanced features and is specifically designed to support clustering. Both filesystems are available under a BSD license. The ports tree carries [[https://www.freshports.org/sysutils/hammer2/|sysutils/hammer2]] a read-only port of the kernel driver: figuring out how to finish the write support is being considered for %%NetBSD%%. == Requirements == * Strong knowledge of C. * Understanding of %%FreeBSD%%'s filesystem interfacing and VFS. * Willingness to dive into %%DragonFly%%BSD's locking semantics. ===== Porting HFS+ ===== [[HFS|HFS Wiki]] == Description == The Hierarchical File System was developed by Apple Inc. for use in %%MacOS%%. With the Release of %%MacOS%% X it received many new features, and the source code was made available as part of XNU. An initial %%FreeBSD%% 5.3 HFS port was made and although it was subsequently abandoned and support for locking has to be added, it would be excellent reference material for an updated port. A port would also be a good reference for bringing other interesting filesystems from Apple's Darwin. == Requirements == * Strong knowledge of C. * Understanding of %%FreeBSD%%'s filesystem interfacing and VFS. ====== Kernel ====== ===== DTrace ===== **Technical Contact** [[RobertWatson|rwatson@]] **Homepage**: [[http://people.freebsd.org/~jb/dtrace/index.html|DTrace for FreeBSD]] DTrace is a dynamic tracing facility designed by Sun Microsystems and released in Solaris 10. They have since released the major part of Solaris under the banner of %%OpenSolaris%% and the Common Development and Distribution License (CDDL) 1.0. * We need a clean CTF implementation for %%FreeBSD%% to avoid the license (CDDL) that Sun has on their code. A specification about the file format needs to be written, and someone who never looked at the Sun code (and doesn't while doing the work) would have to implement that and write tests for the implementation. * We can always use more[[https://wikis.oracle.com/display/DTrace/Providers|Providers]] . == Requirements == * Ability to read and understand foreign C code. * Ability to write C code. * A good understanding of the %%FreeBSD%% kernel. ===== Kernel Control Flow Integrity (kCFI) ===== Control Flow Integrity (CFI) is a security feature implemented mostly compilers and is fully supported by Clang/LLVM: the original prototype was indeed implemented for %%FreeBSD%%. CFI has greatly evolved and has been [[https://lwn.net/Articles/810077/|implemented in the Linux kernel]] and [[https://source.android.com/docs/security/test/kcfi|Android]]. CFI is relatively well [[https://clang.llvm.org/docs/ControlFlowIntegrityDesign.html|documented]] and should be easy to implement on %%FreeBSD%%. Using CFI we shouldn't need to use an additional stack protector at all. == Requirements == * Ability to read and understand foreign C code. * Ability to write C code and definitions. * Interest in compilers, linkers, and how they work. ===== IOCTL command checking tool ===== **Technical Contact**: [[BrooksDavis|brooks@]] The ioctl(2) system call takes integer commands which are typically defined using various ''%%_IO*()%%'' macros in ''%%sys/sys/ioccom.h%%''. One component of the command is the amount of data to be copied in and/or out of the kernel (usually specified by a C type to be copied). Any object which changes size between 32 and 64-bit variants of an architecture requires compatibility definitions and special handling (this commonly occurs if a pointer or ''%%long%%'' is included in the object directly.) Types may also change size due to differing padding rules. Additionally, all types containing pointers require special handling even if a complex definition allows the size not to change. It would be useful to have a tool to check all ioctl(2) commands for size stability and to check objects used as command arguments for pointers. == Requirements == * Ability to read and understand foreign C code. * Ability to write C code. * Tool may be written in any language. ===== Suspend to disk ===== Implement a suspend/resume from disk mechanism. Possibly use the dump functions to dump pages to disk, then use ACPI to put the system in S4 or power-off. Resume would require changes to the loader to load the memory image directly and then begin executing again. == Requirements == * Good knowledge of C. * Understanding of the hardware/software interface. * A laptop that works with ACPI. * Kernel awareness. ===== Solaris Doors IPC Implementation ===== * [[http://www.scn.rain.com/~neighorn/PDF/fast_sockets.pdf|Fast Sockets, An IPC Library to Boost Application Performance]] * [[http://www.rampant.org/doors/linux-doors.pdf|An Implementation of the Solaris Doors API for Linux]] Doors provide a mechanism for processes to issue remote procedure calls to functions in other processes running on the same system. The door APIs were developed by Sun Microsystems as a core part of the Spring operating system and were officially available in Solaris 2.6. They are extensively used in Illumos. In addition to the Solaris/Illumos port, which is well documented, there is also an outdated implementation for linux that can serve for comparison. The project would consist in understanding how the existing code works and designing a completely new, but compatible, implementation for %%FreeBSD%%. == Requirements == * Interest in Inter-process Communications * Ability to understand code and the existing implementations in Illumos and Linux. * Capacity to run your own tests and benchmarks. ===== Develop a netmap vnic for jails that allows connecting jails to vale ===== **Contact:** [[DanielEbdrup]] **Technical Contact:** <freebsd-virtualization@freebsd.org> Jail network connectivity has gotten a separate network stack through VNET/VIMAGE, but at present is limited to 10G between jails and the host. Utilizing a netmap based vnic for jails and the vale software switch to make IPC for jails <-> bhyve and the outside world much easier and faster. Additionally, it may very well be possible to adopt a lot of existing code from svn revision 293459. == Requirements == * Interest in IPC/VPC * Ability to write and/or adopt C code ====== Virtualization ====== ===== bhyve gdb-stub/dcons integration ===== %%FreeBSD%%'s 'bhyve' Hypervisor has a feature where it allows the kernel debugger to communicate with the outside world via a socket. Unfortunately it is quite slow, being polled one byte at a time. In addition it still uses the kernel on the VM to do all the work. Some avenues of improvement could include using the existing dcons memory buffer driver so that a whole buffer might be transferred at a time, or to make the current driver much faster. One might also intercept some of the commands (memory read for example) and perform them directly in the hypervisor so that cooperation of the virtualized system is not required to examine memory. It is even possible that by manipulating the processor flags appropriately, one could single step or 'break' the guest without its cooperation at all. == Requirements == * Willingness to dig into OS internals * Knowledge of i486/amd64 assembler and virtualization concepts * ability to decode intel/AMD processor specs ===== VirtualBox shared folder support for FreeBSD guests ===== **Technical Contact:** [[LiWenHsu|lwhsu@]], [[OleksandrTymoshenko|gonzo@]], [[BernhardFroehlich|decke@]] **Contact:** [[VirtualBox Porting Team|vbox@]] == Description == * https://github.com/lwhsu/freebsd-vboxfs * https://kernelnomicon.org/?p=650 * https://www.virtualbox.org/manual/ch04.html#sharedfolders Oracle VirtualBox does unofficially support %%FreeBSD%% as host and guest operating system. VirtualBox shared folder support allows to access folders of your host from within the guest system. This is similar how you would use network shares in Windows networks except that shared folders do not need require networking, only the VirtualBox Guest Additions. This task was part of %%GSoC%% 2013 where the main focus was on porting the code to %%FreeBSD%%. The result was almost working read only support but more bugs need to be fixed and read write support needs to be added and tested. == Requirements == * Ability to read and understand foreign C code * Ability to write C code * Knowledge of the VFS subsystem ====== Networking ====== ===== openPOWERLINK and other industrial ethernet protocols ===== The manufacturing industry and automation have special requirements for networking in industrial environments: while the cabling has to carry special isolation there is also the requirement to support real-time communications. In such environments, there is a tendency to move towards Ethernet to take advantage of the existing hardware. Several Ethernet types have been developed and standardized to support real-time, with Ethernet/IP and PROFINET being the most common. Ethernet %%PowerLink%% ([[https://www.ethernet-powerlink.org/|EPL]]) is particularly interesting as the protocol stack is available under a BSD license. References: [[http://openpowerlink.sourceforge.net/web/|openPowerLink]] ===== SCPS, Space Communication Protocol Standards ===== SCPS is a protocol suite designed to allow communication over challenging environments. Originally developed jointly by NASA and %%DoD%%'s USSPACECOM, these protocols are used for commercial, educational, and military environments. A student project in this area would involve implementing various network protocols according to specification (SCPS File Protocol, similar to FTP; SCPS-Transport Protocol, based on TCP; and others.) Note that European Space Agency has now an ESA [[http://sophia.estec.esa.int/socis2012/|Summer of Code in Space]] program and while %%FreeBSD%% is not a mentoring organization, interested students could motivate such a process. References: [[http://www.ccsds.org/|Consultative Committee for Space Data Systems]] == Requirements == * Good knowledge of C and TCP. * Able to understand the %%FreeBSD%% TCP/IP stack. * A testbed with at least two machines. ====== Porting ====== ===== Port FreeBSD to new platforms ===== Porting to new platforms is a good way to learn the internals of %%FreeBSD%% and serves to check the general portability of the base system. While there are important efforts to maintain an external toolchain working it would be ideal to start working on newer platforms that are already supported by the toolchain in base and where the hardware support is either easy to find or well supported through emulation. **Contact**: freebsd-arch@ == Requirements == * Good Knowledge of C and assembler of the target platform. * Familiarity with (cross)building %%FreeBSD%%. ===== Port FreeBSD on Tablet device ===== Porting %%FreeBSD%% to arm, intel tablets and next mobile device. **Contact**: freebsd-arch@ == Requirements == * Some kernel experience ====== Testing and Continuous Integration ====== ===== POSIX compliance testing framework ===== == Description == Standards compliance has always been one of the main objectives for the %%FreeBSD%% project. In the past we had [[https://people.freebsd.org/~schweikh/posix-utilities.html|some efforts]] to follow regular testing procedures but we haven't followed up the efforts with proper and sustainable infrastructure. The Open Group has made some [[http://www.opengroup.org/testing/downloads.html|testsuites]] freely available. In particular there used to be a [[https://www.freshports.org/misc/tet|FreeBSD port]] of the TET testing suite, but this was removed due to a lack of maintenance and other issues. The [[http://www.opengroup.org/testing/linux-test/lsb-vsx.html|lsb-vsx]] Linux testsuite should also be cleaned up so that together we are able to do regular testing on both linux-emulation and %%FreeBSD%%-native support. Any compliance issue should be reported in Bugzilla and an attempt to contact the respective group should be made to draw a plan towards compliance. ===== Travis Continuous Integration Support for FreeBSD ===== **Technical Contact:** [[CraigRodrigues|rodrigc@]], == Description == [[https://travis-ci.org|Travis Continuous Integration]] is a very popular Continuous Integration system used by projects hosted on GitHub. If a GitHub project has a .travis.yml config file in the root directory, the Travis system will build and test the project if new code is committed to the GitHub project. Currently, Travis only supports Linux and %%MacOS%% X. The Travis project closed [[https://github.com/travis-ci/travis-ci/issues/1818|issue 1818]] which was a request to add %%FreeBSD%% support to Travis, due to lack of resources. However, having this support would be very useful, and allow %%FreeBSD%% to test many third party projects on GitHub. === Requirements === * Knowledge of Shell scripting in /bin/sh, Python, Ruby * Basic knowledge of REST API's and the [[https://docs.github.com/en/free-pro-team@latest/rest|GitHub REST API]] * Knowledge of virtualization systems, such as [[bhyve]], QEMU, KVM ====== Userland / Installation Tools ====== ===== Profile Guided Optimization (PGO) ===== Both GCC and clang offer the option of generating code that is **optimized with CPU programming counter data** by tools like Linux [[https://perf.wiki.kernel.org/index.php/Main_Page|perf]] or %%FreeBSD%% [[https://man.freebsd.org/cgi/man.cgi?query=pmc&sektion=3&manpath=freebsd-release|pmc(3)]]. With Linux, the recommendation is to use the ''%%-b%%'' flag to use the Last Branch Record (LBR) to record call chains. This information is then converted with [[https://github.com/google/autofdo|autofdo]] to generate a profile. %%FreeBSD%% has no similar process. We should enhance tooling to enable generation of profiles in a compatible format. ==== Documentation ==== [[https://clang.llvm.org/docs/UsersManual.html#profile-guided-optimization|Profile Guided Optimization]] – Clang Compiler User's Manual LLVM SPGO profile generation tool: * llvm-profgen14, llvm-profgen(1) https://man.freebsd.org/cgi/man.cgi?query=llvm-profgen14&sektion=1&manpath=freebsd-ports * llvm-profgen15, llvm-profgen(1) https://man.freebsd.org/cgi/man.cgi?query=llvm-profgen15&sektion=1&manpath=freebsd-ports * llvm-profgen16, llvm-profgen(1) manual page installable with [[https://www.freshports.org/devel/llvm16/|devel/llvm16]], the online version is not yet available (2023-07-18). ===== NDMP data server ===== URL: [[https://www.snia.org/ndmp|Network Data Management Protocol (NDMP)]] The NDMP initiative was launched to create an open standard protocol for network-based backup for network-attached storage. Major commercial storage systems come with a compliant service. This allows major commercial backup systems to backup such NAS devices. Including a NDMP disk server into %%FreeBSD%% would allow to play nice out of the box (modulo some configuring) regarding backups in a corporate environment. * Evaluate the existing revisions of the NDMP standard. * Choose an appropriate revision (after checking of supported versions in commercial backup systems). * Implement at least a NDMP data server. * Bonus: implement a NDMP tape server (to allow attached tapes to be used). == Requirements == * Access to a commercial backup system with NDMP support (mostly for interoperability testing; since a NDMPcopy application seems to be available, this is not a hard requirement). * Good knowledge of a programming language which is included in the base system. * Knowledge about UFS snapshots. ===== Port prebind from OpenBSD ===== The %%OpenBSD%% prebind is a secure implementation of prelinking that is compatible with address space randomization. Prelinking allows to speed up application startup when a lot of libraries are involved. This should show a noticeable effect with e.g. GNOME/KDE. == Requirements == * Good C knowledge (reading and writing). ===== Proxy auto-config file support for libfetch ===== A proxy auto-config (PAC) file contains a %%JavaScript%% function "%%FindProxyForURL%%(url, host)" that determines which HTTP or SOCKS proxy, if any, to use to access a given URL. In most application the file may be specified manually or discovered using the Web Proxy Autodiscovery Protocol. Support for PAC files in libfetch would make fetch more versitle. Supporting PAC files nominally requires a fairly complete %%JavaScript%% implementation. Google's V8 %%JavaScript%% engine is BSD Licensed, however it compiles code to native machine code so platform support is an issue. However, the parser etc may provide a good starting point, and other engines may also exist and should be evaluated. A minimalist implementation of the language with commonly used constructs such as if/else, string comparison, and functions would be sufficient in many cases. References: * [[http://en.wikipedia.org/wiki/Proxy_auto-config|Wikipedia Article on Proxy auto-config]] * [[https://developer.mozilla.org/en-US/docs/Web/HTTP/Proxy_servers_and_tunneling/Proxy_Auto-Configuration_(PAC)_file|Proxy Auto-Configuration (PAC) file]] * [[http://wp.netscape.com/eng/mozilla/2.0/relnotes/demo/proxy-live.html|Proxy Auto-Config File Format]] == Requirements == * Strong knowledge of secure C programming. ===== PXE Installer ===== It would be great to have a bundled PXE installer. This would allow one to boot an install server from a %%FreeSBIE%% live CD-ROM on one box, set the BIOS on subsequent boxes to PXE boot, and then have the rest happen by magic. This would be very helpful for installing cluster nodes, etc. [[MarkusBoelter|m@]] is [[MarkusBoelter|working on]] a bundled PXE installer as part of his BSDInstaller project within the Google Summer of Code 2006. The PXE Installer is working but some non-PXE related issues have to be solved before it can enter the tree. == Requirements == * Good PXE knowledge. ===== Improve cron(8) and atrun(8) ===== Currently, cron(8) and atrun(8) are outdated in their implementation. Here are some directions for improvement: * Update cron(8) to ISC cron with security fixes from %%OpenBSD%%. * Integrate the atrun(8) functionality into cron(8), as it was done in %%NetBSD%%. == Requirements == * Strong knowledge of the C language and Unix API. ===== libpw ===== === Technical Contact === Create a library to be able to manage users/groups easily, it also should have a pam/nss-like plugin framework for different account systems. libutil has pw_* and gr_* undocumented functions which allow user/group manipulation writing pw_*() and gr_*() manpages known users for that library: pkgng, pw(8) == Requirements == * Knowledge of C * Knowledge of pam/nss ===== Safe crash dumps ===== **Technical Contact:** [[GavinAtkinson|gavin@]] Crash dumps are important for collecting debugging information but are also disabled by default because they can consume much space in /var if the user doesn't pay attention to them. What we miss is a safe way to enable crash dumps by default without having to worry about them filling up /var. == Requirements == * Knowledge of C * Knowledge of the %%FreeBSD%% system internals ===== Import syslogd improvements from NetBSD ===== **Technical Contacts:** [[EdMaste|emaste@]], [[MarkJohnston|markj@]] Note - there is work in progress on this project, please contact Ed or Mark for details. %%NetBSD%%'s syslogd has a number of improvements from a former Google Summer of Code project available for porting: * new syslog protocol api syslogp(3) that supports structured data and draft-rfc timestamps * reliable tcp connections with queuing * encrypted connections (TLS) The changes are in %%NetBSD%%'s repository in src/usr.sbin/syslogd, available for viewing on their cvsweb: * http://cvsweb.netbsd.org/bsdweb.cgi/src/usr.sbin/syslogd/?only_with_tag=MAIN == Requirements == * Good knowledge about the C programming language. ===== RAID and disk monitoring suite ===== **Technical Contact:** There have been several organizations that have independently developed RAID and disk failure monitoring tools. These should be gathered together into a unified group of daemons and monitoring scripts to provide a consistent view of disk status. * [[http://svnweb.freebsd.org/base/user/sbruno/ard/]] * [[http://svnweb.freebsd.org/base/user/sbruno/mfid/]] * [[http://svnweb.freebsd.org/base/user/sbruno/mptd/]] == Requirements == * Knowledge of /etc/rc.d scripts and ordering * Basic understanding of C and UNIX * Shell scripting in /bin/sh ====== Global Projects (may touch everything) ====== ===== EPUB Support in Documentation Build Infrastructure ===== //Suggested Summer of Code project idea// Enhance the [[http://www.freebsd.org/docproj/|FreeBSD Documentation Project]] build infrastructure to generate [[http://idpf.org/epub|EPUB]] format output suitable for eBook readers from such as iPads and Kindles. == Requirements == * Ability to work with Makefiles * Knowledge of SGML/XML transforms ===== PerfVisor (PERFormance adVISOR) ===== **Coordination:** [[AlexanderLeidinger|netchild@]] ====== Possible Old or No Interest ====== ===== Document all sysctls ===== **Technical Contact**: [[BradDavis|brd@]], [[EitanAdler|eadler@]] == Description == The sysctl(8) utility retrieves kernel states and allows processes with appropriate privilege to change kernel states. On request it is able to display description lines which document the kernel state. Unfortunately not every sysctl is documented. This task is possible to share with other volunteers. mat has done some development in Perforce, in the mat_sysctl_cleanup branch. * Find every undocumented sysctl in the kernel. * Try to determine what this sysctl is for and document it. == Requirements == * Ability to read and understand foreign C code. ===== BSD-licensed Text-Processing Tools ===== //Part of Summer of Code 2010//, //Part of Summer of Code 2008// **Technical Contact:** [[G'aborK"ovesd'an|gabor@]] diff/diff3/sdiff: Many command-line options are supported but some features are still missing. Maybe the three programs can be integrated into a single binary, this should be evaluated. A thorough performance benchmark should also be done. See [[SummerOfCode2012/JesseHagewood|SummerOfCode2012/JesseHagewood]] for last status. [[http://mdocml.bsd.lv/|mdocml]]: Some groff features are very hard to implement but they aren't strictly needed to render our man pages. Yet some manuals do not compile with mdocml. Investigate the reasons and create a migration plan. == Requirements == * Knowledge of C. The goal of this project is to get a tool which can analyze a system for performance bottlenecks and maybe even give some hints what to do next. This is not a %%GSoC%% project. Maybe small parts of it can be done during a %%GSoC%%. ==== Prerequisites you should know/read to understand the project steps: ==== * http://dtrace.org/blogs/brendan/2012/12/13/usenix-lisa-2012-performance-analysis-methodology/ * http://queue.acm.org/detail.cfm?id=2413037 or http://dtrace.org/blogs/brendan/2012/02/29/the-use-method/ * http://dtrace.org/blogs/brendan/2012/03/01/the-use-method-solaris-performance-checklist/ * http://dtrace.org/blogs/brendan/2012/03/07/the-use-method-linux-performance-checklist/ * http://www.slideshare.net/brendangregg/zfsperftools2012 or http://www.brendangregg.com/Slides/zfsperftools2012.pdf * http://www.dtracebook.com * http://queue.acm.org/detail.cfm?id=1809426 * http://dtrace.org/blogs/brendan/2012/12/10/usenix-lisa-2010-visualizations-for-performance-analysis/ * http://dtrace.org/blogs/brendan/2012/03/26/subsecond-offset-heat-maps/ * and http://dtrace.org/blogs/brendan/ in general... ==== Project phases ==== === Initial phase === * Identify/list possible resources in %%FreeBSD%%. See the performance checklist postings above for ideas. Those are just a start, this can iteratively be extended to include subitems (e.g. in a first iteration the "network stack" could be an item of a resource, in a second iteration each network interface is a resource, on a third iteration one network protocol could be a resource, on a fourth iteration just the TX queue of a NIC can be a resource, ...). Not everything needs to be identified initially, at one point the work on this resource list should be pushed back to a point where all or some of the following items are handled for the existing resources. When all OS resources are listed, further iterations could even include 3rd party applications (e.g. apache httpd requests, mysql table scans, ...). Interesting values for each resource can be bandwidth (total/percentage), operations per second, latency, usage duration, time until completion, ... Ideally the result is written down in the %%FreeBSD%% wiki. For each "high-level" resource (e.g. network stack) try to document some kind of "where to look next if saturated/on-error/..." resource (e.g. network interface, protocol checks, ...) * For each resource identify/document "utilization", "saturation" and "error" values (see the USE method in the prerequisites), and code points in the source which show them. If there's no code point in the source, mark the resource for later investigation. Hardware provided info, e.g. CPU counters, count as as "code points" in this sense too. * For each resource determine/document existing tools which provide the required information (saturation/error) and how to use them to get the required info. * For each resource without an existing way to get the required information, but with a code point in the source which provides the required information, write a dtrace/hwpmc/whatever script/program/whatever to query the required information. * For each resource without a code point in the source which provides the required information, find a "cheap" way to determine the resource value and add corresponding code readable by sysctl or dtrace (readout variable on existing probe or a specific SDT probe just for this) or whatever (and write a script/program/whatever as above). === Improvement phase === * For each existing code point with an existing script check if a new way of handling this info (sysctl, dtrace probe, ...) would make the sampling of this data cheaper (= less performance impact of the performance monitoring itself). === Visualization phase === * Write a tool (for the ports collection) which visualizes data (heat maps, flame graphs, ...) gathered from the above performance data collection scripts (either stored somewhere or by calling the scripts directly) and provides hints where to look next in case a resource is the bottleneck. The blog of Brendan Gregg gives a lot of ideas how to visualize with various types of graphs. Suggestion: make it system independent, foresee use on *BSD, Solaris and Linux. === Cloud phase (everything is cloudy nowadays...) === * Write an agent which is able to collect the data which the visualizing tool is able to handle and extend the visualizing tool to collect data from agents on other machines (via an encryted and authenticated connection if needed = possible but not mandatory, depending on the environment). === Datawarehouse phase (buzzword bingo!) === * Write a gateway/collector service which a) is able to "handle" agents in a datacenter (query and store data), b) is able to handle bulk-data-retrieval requests from remote locations (transfer data from remote datacenters to the company headquater), and c) is able to store/archive queried data for later use ("How did this look last week/month/year?"). ==== Discussion ==== The first item in the initial phase is a big item. It may be better to see the complete initial phase as an iterative or divide and conquer step. First determine some high level resources and process all steps of the initial phase on them. Then repeat the initial phase again by braking up the high level resources in more detailed items (e.g. first "all CPU in system" as one resource, second "each CPU in system" as a resrouce, third breaking up each CPU via hardware performance counters). ==== GSoC info ==== Each phase is a big project of its own. Do not expect to be able to do it during some weekends or during a %%GSoC%%. If you think you can, you did not get the full scope, think again. If you still insinst after rethinking, be our guest, but you better prepare a very good outline what you want to do when, how, at which level of detail (e.g. come with a list of resources for the first item in the initial phase), and also include what you will NOT do/cover/handle but could be related (e.g. CPU internal performance counters if you want to handle the CPU performance side of this). You already need to know %%FreeBSD%% (you use it already on your server or desktop since several months) and you shall not be afraid to ask questions or discuss on the mailinglists. This is not a project for 2h per day, if you are not motivated to spend time on this, you better chose a different topic. This can be made "big enough" to be a project suitable for finals at your university (depending on the requirements of your university, with or without extending it over the end of the %%GSoC%%). ====== Ports ====== ===== make db target in ports ===== **Technical Contact:** [[BaptisteDaroussin|bapt@]] A target in ports tree root that pulls all possible information out of the prots tree into a db, for example, sqlite. From which many things can be query fastly and historical data can be retrieved by querying the snapshots. References: * == Requirements == * Scripting * Basic knowledge in database ====== Other ====== If you are interested in working on a project not explicitly mentioned above, you may want to contact one of the potential //Technical Contact//s below: * File Systems: ScottLong AlfredPerlstein * [[GEOM]]: PawelJakubDawidek PoulHenningKamp AndreyElsukov * Networking: AlfredPerlstein RobertWatson AndreyElsukov * Wireless (802.11) Networking: AdrianChadd BernhardSchmidt * Release Engineering: ReleaseEngineering * [[Security|Security]] / [[TrustedBSD]]: RobertWatson Additionally, there are a lot of interesting mailing lists that can be used when searching information about specific subjects. - For porting ideas go to [[WantedPorts]] - For Beginner tasks check out [[JuniorJobs]] ideaspage.txt Last modified: 2023/10/04 15:41by CySchubert