| Both sides previous revision Previous revision Next revision | Previous revision |
| testclusteronepointers [2011/01/09 00:44] – Zoo's actually running 8 EdMaste | testclusteronepointers [2018/02/22 14:02] (current) – zoo runs 11-stable now EdMaste |
|---|
| ====== Netperf Cluster Pointers ====== | ====== Test Cluster One Pointers ====== |
| This page describes some useful hints on working with the [[http://www.FreeBSD.org/projects/netperf/cluster.html|FreeBSD netperf cluster]]. | This page describes some useful hints on working with the [[http://www.FreeBSD.org/projects/netperf/cluster.html|FreeBSD netperf cluster]]. |
| |
| ===== zoo.FreeBSD.org ===== | ===== zoo.FreeBSD.org ===== |
| zoo.%%FreeBSD%%.org is the central management host for the %%FreeBSD%% netperf cluster, and is the only link between the test boxes and the outside world. This host was donated by Sentex, along with its network connectivity. SSH into this box to access the netperf cluster. Contact netperf-admin to get set up with an account there; this typically requires that you already have a %%FreeBSD%%.org account, which will be cloned to the box; because the netperf cluster is geographically quite distinct from the main %%FreeBSD%%.org cluster, standard %%FreeBSD%%.org home directories are not available there. ''%%zoo.FreeBSD.org%%'' tends to run whatever the most reliable -STABLE branch is at a particular time. | zoo.%%FreeBSD%%.org is the central management host for the %%FreeBSD%% test cluster, and is the only link between the test boxes and the outside world. This host was donated by Sentex, along with its network connectivity. SSH into this box to access the test cluster. Contact netperf-admin to get set up with an account there; this typically requires that you already have a %%FreeBSD%%.org account, which will be cloned to the box; because the test cluster is geographically quite distinct from the main %%FreeBSD%%.org cluster, standard %%FreeBSD%%.org home directories are not available there. ''%%zoo.FreeBSD.org%%'' tends to run whatever the most reliable -STABLE branch is at a particular time. |
| |
| ===== Network, console, and power configuration ===== | ===== Network, console, and power configuration ===== |
| * They can be reached by a gigabit backbone from zoo, configured using DHCP | * They can be reached by a gigabit backbone from zoo, configured using DHCP |
| * They netboot from zoo using tftp, pxeboot, and NFS from /zoo/machinename | * They netboot from zoo using tftp, pxeboot, and NFS from /zoo/machinename |
| See NetperfClusterReservations for more specific information on how to reach the console and power control for your test box. In the common case you can type ''%%console system-name%%'' to connect to the console of the host reached via 'portnumber' as identified on the reservations page. Please contact the admins if you need help with this. | See TestClusterOneReservations for more specific information on how to reach the console and power control for your test box. In the common case you can type ''%%console system-name%%'' to connect to the console of the host reached via 'portnumber' as identified on the reservations page. Please contact the admins if you need help with this. |
| |
| ===== Machine and power console accounts ===== | ===== Machine and power console accounts ===== |
| The username for apc2 is 'apc'; the username for apc3 and apc4 is 'apc'. Passwords are likewise. | The username for apc2 is 'apc'; the username for apc3 and apc4 is 'apc'. Passwords are likewise. |
| |
| The username for apc5 is 'admin'. Password is likewise. | The username for apc5 is 'apc'. Password is likewise. |
| |
| ===== Source access ===== | ===== Source access ===== |
| Most netperf cluster users make use of Perforce to move locally edited source trees to the netperf cluster for testing; they may also do source code editing on zoo itself. A range of development models can be used, from netbooting kernels built on zoo or a test machine with an NFS root, to rebuilding and installing %%FreeBSD%% versions on test machine hard disks, to uploading kernels with embedded MFS file systems. | Most test cluster users make use of Perforce to move locally edited source trees to the test cluster for testing; they may also do source code editing on zoo itself. A range of development models can be used, from netbooting kernels built on zoo or a test machine with an NFS root, to rebuilding and installing %%FreeBSD%% versions on test machine hard disks, to uploading kernels with embedded MFS file systems. |
| |
| A copy of ''%%p4%%'' is installed in ''%%/usr/local/bin%%''. If you've not used p4 for work on multiple hosts at a time before, remember to create a new p4 client configuration (with a different name from your existing p4 client) when setting up the configuration. I.e., ''%%username_zoo%%''. Netperf cluster users are encouraged to back up their data, possibly using revision control, and especially not to rely on the persistence of data on test machine hard disks. | A copy of ''%%p4%%'' is installed in ''%%/usr/local/bin%%''. If you've not used p4 for work on multiple hosts at a time before, remember to create a new p4 client configuration (with a different name from your existing p4 client) when setting up the configuration. I.e., ''%%username_zoo%%''. Test cluster users are encouraged to back up their data, possibly using revision control, and especially not to rely on the persistence of data on test machine hard disks. |
| |
| A cvsup mirror of the %%FreeBSD%% CVS repository may be found in ''%%/zoo/cvsup/FreeBSD-CVS%%''. You'll need to use ''%%cvs -R%%'' as otherwise CVS will complain about not being able to write to the directory tree. The mirror is updated every couple of hours using cvsup. | A cvsup mirror of the %%FreeBSD%% CVS repository may be found in ''%%/zoo/cvsup/FreeBSD-CVS%%''. You'll need to use ''%%cvs -R%%'' as otherwise CVS will complain about not being able to write to the directory tree. The mirror is updated every couple of hours using cvsup. |
| |
| ===== Building ===== | ===== Building ===== |
| ''%%zoo.FreeBSD.org%%'' runs %%FreeBSD%% 8-STABLE, which may mean you have to build custom versions of config or build tools if you are building other versions of %%FreeBSD%%, especially kernels. Some users will keep a series of config versions in ''%%~/bin%%'' for this reason. | ''%%zoo.FreeBSD.org%%'' runs %%FreeBSD%% 11-STABLE, which may mean you have to build custom versions of config or build tools if you are building other versions of %%FreeBSD%%, especially kernels. Some users will keep a series of config versions in ''%%~/bin%%'' for this reason. |
| |
| The ''%%/zoo%%'' file system is a good place to keep your scratch files. Most users will have a ''%%/zoo/username%%''. You may find you wish to build on your test box, or you might want to build on zoo. If building on zoo, please run large builds under ''%%nice%%'' so as to avoid disrupting interactive sessions for other users. | The ''%%/zoo%%'' file system is a good place to keep your scratch files. Most users will have a ''%%/zoo/username%%''. You may find you wish to build on your test box, or you might want to build on zoo. If building on zoo, please run large builds under ''%%nice%%'' so as to avoid disrupting interactive sessions for other users. |
| ===== Network Configuration ===== | ===== Network Configuration ===== |
| Tests hosts and zoo are linked by a gigabit switch used for access to power consoles, test hosts via SSH, NFS mounting, DHCP, etc. Please do not do extensive high bandwidth testing on this network segment (192.168.5), as it will affect performance for other users, as well as produce unreliable results (as it is a shared network segment). Several test hosts are configured with direct gigabit links between them, most notably the hosts ''%%tiger-1%%'', ''%%tiger-2%%'', and ''%%tiger-3%%'', which each have four gigabit ports and are interconnected. ''%%cheetah%%'' and ''%%sloth%%'' are also connected to the tigers, and can be used for comparative measurements as well. These hosts should be used for any tests relating to IP forwarding, bridging, etc. ''%%leopard-1%%'', ''%%leopard-2%%'', ''%%leopard-3%%'', and ''%%hydra-2%%'' are linked via a dedicated 10gbps switch and are available for 10gbps testing. | Tests hosts and zoo are linked by a gigabit switch used for access to power consoles, test hosts via SSH, NFS mounting, DHCP, etc. Please do not do extensive high bandwidth testing on this network segment (192.168.5), as it will affect performance for other users, as well as produce unreliable results (as it is a shared network segment). Several test hosts are configured with direct gigabit links between them, most notably the hosts ''%%tiger-1%%'', ''%%tiger-2%%'', and ''%%tiger-3%%'', which each have four gigabit ports and are interconnected. ''%%cheetah%%'' and ''%%sloth%%'' are also connected to the tigers, and can be used for comparative measurements as well. These hosts should be used for any tests relating to IP forwarding, bridging, etc. ''%%leopard-1%%'', ''%%leopard-2%%'', ''%%leopard-3%%'', and ''%%hydra-2%%'' are linked via a dedicated 10gbps switch and are available for 10gbps testing. |
| | |
| | ===== Fetching external files ===== |
| | zoo is running //micro_proxy// from inetd to allow test machines to fetch individual files or packages. It is not supposed to help with any high speed downloading etc. Use ''%% setenv HTTP_PROXY http://192.168.5.1:8080/ %%'' or your shell equivalent (or zoo's IPv6 link-local in case you prefer IPv6 to IPv4) to use it. |
| | |
| | ===== Time Server ===== |
| | |
| | The cluster has its own time server which syncs to GPS. If you want to keep time on hosts in the test lab you can |
| | either run PTPv2 (/usr/ports/net/ptpd2) or use NTP to talk to the m600 (192.168.5.17). |
| |
| ==== some interconnects ==== | ==== some interconnects ==== |
| no/c --+ g8 (copper) | no/c --+ g8 (copper) |
| no/c --+ g9 (fiber) | no/c --+ g9 (fiber) |
| |
| no/c == no carrier | |
| ??? == active, other end / iface / ... unkown | |
| </code> | |
| |
| === switch2, netgear === | |
| |
| <code> | |
| mgmt .6 switch2 | |
| link g6 switch1 -------------------+ g2 | |
| | | |
| backup link g4 switch1 -------------------+ g3 | |
| | | |
| elephant 100 fxp0 00:00:e2:3c:70:59 mgmt. 104 | | |
| no/c fxp1 00:00:e2:3c:70:5a | | |
| 1000 em0 00:07:e9:1b:f2:d9 --------------------------------------------+ g4,vl2 | |
| | | |
| no/c ----+ g7 | |
| | | |
| tiger-1 1000 em0 00:07:e9:1a:6e:04 --------------------------------------------+ g6,vl2 | |
| 1000 em1 00:07:e9:1a:6e:05 ---------------+ | | |
| 1000 em2 00:0e:0c:09:b3:a8 mgmt .101 | | | |
| 1000 em3 00:0e:0c:09:b3:a9 ------------+ | | | |
| | | | | |
| tiger-2 1000 em0 00:07:e9:1a:69:f6 --------------------------------------------+ g5,vl2 | |
| 1000 em1 00:07:e9:1a:69:f7 ------------|-------------------------------|--------- orangutan port 2 (GE) | |
| 1000 em2 00:0e:0c:09:b2:12 mgmt .102 | | | | |
| 1000 em3 00:0e:0c:09:b2:13 ------------+ | | | |
| | | | |
| tiger-3 no/c em0 00:07:e9:1a:b4:20 | | | |
| 1000 em1 00:07:e9:1a:b4:21 --------------------------------------------+ g1,vl2 | |
| 1000 em2 00:0e:0c:09:ba:fe mgmt .103 | | | |
| 1000 em3 00:0e:0c:09:ba:ff -----------+ | | | |
| | | g8 + | |
| | | \ | |
| cheetah 100 fxp0 00:e0:81:28:66:cf mgmt .106 | | '---- mgmt .1 netgear GS724T | |
| 1000 bge0 00:e0:81:28:67:12 -----------+ | | |
| 1000 bge1 00:e0:81:28:67:13 ---------------+ | |
| down em0 00:04:23:b0:46:ba | |
| down em1 00:04:23:b0:46:bb | |
| |
| no/c == no carrier | no/c == no carrier |
| ===== Guidelines and Procedures for Use ===== | ===== Guidelines and Procedures for Use ===== |
| |
| Before being used, **systems must be reserved using the wiki**. This prevents accidental collisions in use and allows the cluster maintainers to decide when systems need recycling, may be redeployed for other uses, etc. NetperfClusterReservations may be used to reserve systems in the netperf cluster. For any long-term commitment of hardware (over a day or two), please contact the cluster administrators before making a reservation. | Before being used, **systems must be reserved using the wiki**. This prevents accidental collisions in use and allows the cluster maintainers to decide when systems need recycling, may be redeployed for other uses, etc. TestClusterOneReservations may be used to reserve systems in the test cluster. For any long-term commitment of hardware (over a day or two), please contact the cluster administrators before making a reservation. |
| |
| Please **do not reconfigure BIOS settings** on the hosts using the BIOS serial console without consulting the cluster admins. This can lead to difficult to diagnose and fix problems, inconsistent performance measurement results, etc. Under no circumstances disable BIOS serial redirection (if available) or disable PXE booting. | Please **do not reconfigure BIOS settings** on the hosts using the BIOS serial console without consulting the cluster admins. This can lead to difficult to diagnose and fix problems, inconsistent performance measurement results, etc. Under no circumstances disable BIOS serial redirection (if available) or disable PXE booting. |
| **Feel free to reinstall the operating systems or repartition the disks** on any test box you have a reservation for. However, when done with a testing session and reservation, **please restore a vanilla install of %%FreeBSD%%-CURRENT on the testbox** and its netboot tree so that other users don't have to clean up after you. Please coordinate any long-term use of the test cluster with the cluster administrators, and update the wiki to reflect your use. | **Feel free to reinstall the operating systems or repartition the disks** on any test box you have a reservation for. However, when done with a testing session and reservation, **please restore a vanilla install of %%FreeBSD%%-CURRENT on the testbox** and its netboot tree so that other users don't have to clean up after you. Please coordinate any long-term use of the test cluster with the cluster administrators, and update the wiki to reflect your use. |
| |
| Please **log use of the test cluster for any significant project** on the NetperfClusterReservations page, so that we can point at the interesting projects done there when soliciting further hardware donations. Please also mention the netperf cluster in commits and posts if you use it for significant testing and development as part of a project for the same reason. | Please **log use of the test cluster for any significant project** on the TestClusterOneReservations page, so that we can point at the interesting projects done there when soliciting further hardware donations. Please also mention the test cluster in commits and posts if you use it for significant testing and development as part of a project for the same reason. |
| |
| Do not power off the systems using APM/ACPI ie, **no 'shutdown -p now**', since this will mean (for at least some of the systems) that an on-site person has to push the power button instead of users just being able to power on systems remotely using the power controllers. | Do not power off the systems using APM/ACPI ie, **no 'shutdown -p now**', since this will mean (for at least some of the systems) that an on-site person has to push the power button instead of users just being able to power on systems remotely using the power controllers. |
| We would love it if someone added conserver support to kgdb, so that kgdb could attach to kernels via the existing conserver setup. Right now, the two appear to be mutually exclusive, requiring disconnecting conserver from a host before it can be debugged using kgdb. | We would love it if someone added conserver support to kgdb, so that kgdb could attach to kernels via the existing conserver setup. Right now, the two appear to be mutually exclusive, requiring disconnecting conserver from a host before it can be debugged using kgdb. |
| |
| We would love it if release disk images and memory file system images could be constructed without root access on the host where the build is done. Be it a UFS file system tool in userspace, tarfs and tar-based root file system images, or whatever, it would make life much easier if this support were integrated into our build and release environment. | We would love it if release disk images and memory file system images could be constructed without root access on the host where the build is done. Be it a UFS file system tool in userspace, tarfs and tar-based root file system images, or whatever, it would make life much easier if this support were integrated into our build and release environment. (See e.g. [[IdeasPage#Tar_output_mode_for_installworld|Tar output mode for installworld]].) |
| |
| ===== RMI MIPS eval board use ===== | ===== RMI MIPS eval board use ===== |