Differences

This shows you the differences between two versions of the page.

--- clusteradm:nyinetinstall [2012/06/06 23:33] – MarkLinimon
+++ clusteradm:nyinetinstall [2020/11/21 06:26] (current) – PhilipPaeps
@@ Line 1: / Line 1: @@
-==== installing a new machine at NYI ====
+====== installing a new machine at NYI ======
-([[/MarkLinimon|MarkLinimon]] note: WIP)
+This information is correct as of November 2020.
+===== Metadata =====
+Add the machine to https://rackmonkey.freebsd.org/.  Document the physical connections and any peculiarities of the new machine in the notes field.  Also note the MAC addresses so the machine can be found again if it inadvertently moves to a different switch port for whatever reason.
+  * Allocate IP addresses in the DNS.
+    * Remember to set up reverse DNS too.
+    * Also set up an IP address for IPMI.
+      * And reverse DNS for it.
+  * Poke holes in the firewall for IPMI
+  * Add IPMI to conserver
+After committing to admin-svn, wait for the update emails to show up and resolve any alerts.
+===== Power =====
+On admin.nyi, console to rpduXXXX/lpduXXXX and name the power outlets.
+===== Network =====
+Modern machines will have IPMI rather than serial ports.
+For a fresh installation, put both IPMI and network in the install vlan.  Also remember to name the port.
+On admin.nyi, ssh to rsw, and set things up.
+<code>
+set interfaces ge-N/0/XX description "hostname ipmi"
+set interfaces ge-N/0/YY description "hostname port 0"
+set interfaces ge-N/0/XX unit 0 family ethernet-switching vlan members install
+set interfaces ge-N/0/YY unit 0 family ethernet-switching vlan members install
+commit
+</code>
+Now grep ''%%/var/log/daemon%%'' for the IPMI MAC address on admin.nyi to find the DHCP lease in the install vlan.  Then you can get to the console with ''%%ipmitool -U ADMIN -P ADMIN -H 10.0.7.XX sol activate%%''.
+===== Initial install =====
+Follow the instructions in /etc/motd of the pxeroot environment.  The root password is in secrets/nyi.asc in the admin-svn repository.
+Before rebooting the machine, jail into the install environment and set up production network information in /etc/rc.conf.  You may also need to add something like ''%%comconsole_port="0x2f8"%%'' to /boot/loader.conf.
+When you're happy the machine has a chance of showing up on the network following a reboot, exit the jail and reboot the server.
+===== Move to production network =====
+While the machine is booting, move the network interface (not IPMI!) to the production vlan on the switch:
+<code>
+delete interfaces ge-N/0/YY unit 0 family ethernet-switching vlan members install
+set interfaces ge-N/0/YY unit 0 family ethernet-switching vlan members XXXXX
+commit
+</code>
+When the machine comes up, verify that the network works by pinging freebsd.org on the console.
+===== Finish installation =====
+Follow instructions in /etc/motd for the new install.
+You'll probably want to set up swap too.
+<code>
+gmirror label swap0 ada0p2 ada1p2
+gmirror label swap1 ada2p2 ada3p2
+</code>
+And adjust /etc/fstab accordingly.
+Do a ''%%sysrc local_unbound_enable="YES"%%''.  The clusteradm scripts will configure it.
+After /etc/motd is clear of warnings, reboot the machine to make sure everything still works.
+===== Move IPMI to production =====
+At this stage, the IPMI port is still in the install VLAN and you're talking to it with ipmitool on admin.nyi.
+ssh to the new machine from somewhere else and become root and disconnect your ipmitool on admin.nyi (with ~.).
+On the new machine, over ssh:
+<code>
+pkg install ipmitool
+ipmitool lan set 1 ipsrc static
+ipmitool lan set 1 ipaddr 10.0.0.XXX
+ipmitool lan set 1 netmask 255.255.255.0
+ipmitool lan set 1 defgw ipaddr 10.0.0.1
+</code>
+Now move the IPMI port to the netmgmt vlan on rsw:
+<code>
+delete interfaces ge-N/0/XX unit 0 family ethernet-switching vlan members install
+set interfaces ge-N/0/XX unit 0 family ethernet-switching vlan members netmgmt
+commit
+</code>
+If you've done everything right, after about a minute, you should be able to console to the new machine from admin.nyi.
+====== old information from before November 2020 ======
+([[/PhilipPaeps|PhilipPaeps]] note: this information is very old but may still have value when debugging)
+([[/MarkLinimon|MarkLinimon]] note: WIP; needs some information from [[clusteradm/YahooNetinstall]])
 === all machines ===
@@ Line 13: / Line 116: @@
   * make sure ''%%rsw02XX%%'' has the the interface's vlan set to **portsbuild** (e.g.)
-  * make sure the ''%%/usr/local/etc/dhcpd.conf%%'' entry for your machine does have entries for both ''%%fixed-address%%'' and ''%%option root-path%%''
+  * make sure the ''%%/usr/local/etc/dhcpd.conf%%'' entry for your machine has entries for both ''%%fixed-address%%'' and ''%%option root-path%%''
   * boot the machine
   * format the drive
@@ Line 25: / Line 128: @@
   * if you have not done so already:
-    * create a directory for pxebooting something like ''%%/home/linimon/pxeroot/amd64-install-9.0%%''
+    * create a kickstart directory for pxebooting, something like ''%%/home/linimon/pxeroot/amd64-install-9.0%%'' :!: if you are installing via ZFS, this directory will need to be exported **read-write**.
-    * copy over ''%%/home/linimon/pxeroot/amd64-install-9.0/doit%%'' to it and edit for that directory and release
+    * copy over the ''%%/home/linimon/pxeroot/amd64-install-9.0/getbootableiso%%'' script to it and edit for that directory and release
-    * run it
+    * ''%%sudo sh -x ./getbootableiso%%''
-  * make sure ''%%rsw02XX%%'' has the the interface's vlan set to **install**
+    * (see [[http://freebsd.so14k.com/freebsd9_pxe.shtml|BradDavis]]'s writeup for more detail of what ''%%getbootableiso%%'' does)
-  * make sure the ''%%/usr/local/etc/dhcpd.conf%%'' entry for your machine does not has entries for neither ''%%fixed-address%%'' nor ''%%option root-path%%'' so it will boot on the default **install** vlan
+  * make sure ''%%rsw02XX%%'' has the the proper interface's vlan set to **install**
-  * reboot the machine
+  * make sure the ''%%/usr/local/etc/dhcpd.conf%%'' entry for your machine has entries for neither ''%%fixed-address%%'' nor ''%%option root-path%%'' so it will boot from the default **install** vlan
-  * do the install
+  * reboot the machine.
+  * do the install.
+  * reboot the machine.  Halt the boot in the BIOS and set the disk to have a higher boot priority than the Ethernet adapter.
+  * if the machine comes up correctly, make sure the ''%%/usr/local/etc/dhcpd.conf%%'' entry for your machine has entries for both ''%%fixed-address%%'' and ''%%option root-path%%''.  The former should be an address that is in the range of the desired vlan.  (See [[Teams/nyi-admin/Networks]] for the list of vlans).
+  * reboot the machine again to get the proper new address you picked.
   * profit!