SystemAdministration/TheRiseAndFallOfNewNodeManagement
For those who opened this sacred tomb, take a moment to decide if you wish to truly proceed. There are better things to do with your life like walk around the world, or learn to play pinball with your feet. If you truly wish to proceed, remember that their be dragons here.
Initial Setup
(this guide assumes we're using Ubuntu 12.04 and are on Linode, most of this is still relevent in general, but ignore the linode bits)
Once a new node is created on Linode, you need to deploy Ubuntu 12.04, this can "Dashboard" tab easily. Make sure you give 512M of swap, power it up, write down the root password, then open a console. We've got work to do.
On helium, in the root home directory, there's a folder called deployment_kit which has all the files you need to copy in place.
Install All Updates
Linode's image is a bit out of date, so a quick upgrade is needed, first you need to update the package index
Last login: Sat Mar 22 22:02:47 2014 root@localhost:~# apt-get update Get:1 http://mirrors.linode.com precise Release.gpg [198 B] Get:2 http://mirrors.linode.com precise-updates Release.gpg [198 B] Get:3 http://mirrors.linode.com precise-backports Release.gpg [198 B] Get:4 http://mirrors.linode.com precise-security Release.gpg [198 B] -SNIP-
Then install updates
root@localhost:~# apt-get dist-upgrade Reading package lists... Done Building dependency tree Reading state information... Done Calculating upgrade... Done The following packages will be upgraded: accountsservice apport apt apt-transport-https apt-utils apt-xapian-index base-files bash-completion bc bind9-host curl dbus dmsetup dnsutils dosfstools dpkg file gnupg gpgv grub-common ifupdown initramfs-tools initramfs-tools-bin iproute isc-dhcp-client isc-dhcp-common landscape-common language-pack-en language-pack-en-base language-selector-common libaccountsservice0 libapt-inst1.4 libapt-pkg4.12 libasn1-8-heimdal libbind9-80 libc-bin libc6 libcurl3 libcurl3-gnutls libdbus-1-3 libdevmapper1.02.1 libdns81 libdrm-intel1 libdrm-nouveau1a libdrm-radeon1 libdrm2 libgcrypt11 libglib2.0-0 libgnutls26 libgssapi3-heimdal libhcrypto4-heimdal libheimbase1-heimdal libheimntlm0-heimdal libhx509-5-heimdal libisc83 libisccc80 libisccfg82 libkrb5-26-heimdal libldap-2.4-2 liblockfile-bin liblockfile1 liblwres80 libmagic1 libpci3 libplymouth2 libpolkit-gobject-1-0 libpython2.7 libroken18-heimdal libssl1.0.0 libudev0 libwind0-heimdal libxcb1 libxml2 lsb-base lsb-release multiarch-support openssl pciutils perl perl-base perl-modules plymouth plymouth-theme-ubuntu-text procps python python-apport python-apt python-apt-common python-httplib2 python-lazr.restfulclient python-minimal python-openssl python-problem-report python2.7 python2.7-minimal rsyslog sudo tzdata udev unzip update-manager-core w3m xkb-data 103 upgraded, 0 newly installed, 0 to remove and 0 not upgraded. Need to get 44.5 MB of archives. After this operation, 19.5 kB of additional disk space will be used. Do you want to continue [Y/n]?
This takes about 5-10 minutes. Drink a soda, and contemplate life ...
Set Hostname
Hostnames should be setup with the next item on the [HostnamePolicy|Hostname Policy]. On Ubuntu, you need to edit /etc/hostname, and /etc/hosts. You also need to add the LDAP server IP to the hosts file so it will function even if DNS is down
root@localhost:~# cat /etc/hostname boron root@localhost:~# cat /etc/hosts 127.0.0.1 boron.li684-22 boron localhost LDAP-IP ldap-server.li694-22 # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters
Load the new hostname with 'hostname -F'
root@localhost:~# hostname -F /etc/hostname root@localhost:~#
Note, prompt won't change until to log out/log back in.
Switch Over To Distro Kernels
Linode uses a customized kernel instead of stock Ubuntu kernels. While this works "well enough" for most people, it lacks AppArmor, and cause unexpected splats as it doesn't have a ramdisk.
Here's Linode's guide on how to fix it: https://library.linode.com/custom-instances/pv-grub-howto
When you're done, uname -a should say something like this
root@boron:~# uname -a Linux boron 3.2.0-60-virtual #91-Ubuntu SMP Wed Feb 19 04:13:28 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
Setup Networking
On the Linode panel, make sure the node has an internal IP address so that other nodes in the data centre can access it, then note it. You have to setup static IP address configure. Linode has a decent guide for this, but the quick and dirty version is you need to edit /etc/network/interfaces to look like this
# The loopback interface auto lo iface lo inet loopback # Configuration for eth0 and aliases # This line ensures that the interface will be brought up during boot. auto eth0 eth0:0 eth0:1 # eth0 - This is the main IP address that will be used for most outbound connections. # The address, netmask and gateway are all necessary. iface eth0 inet static address PUBLIC-IP-HERE netmask 255.255.255.0 gateway GATEWAY-HERE # eth0:0 # This is a the private IP address. iface eth0:0 inet static address INTERNAL-IP-HERE netmask 255.255.128.0
You can apply the new IP configuration with this
root@boron:~# ifdown eth0 && ifup eth0 eth0:0 resolvconf: Error: /etc/resolv.conf isn't a symlink, not doing anything. resolvconf: Error: /etc/resolv.conf isn't a symlink, not doing anything. ssh stop/waiting ssh start/running, process 1087 resolvconf: Error: /etc/resolv.conf isn't a symlink, not doing anything. ssh stop/waiting ssh start/running, process 1127 root@boron:~#
Then, kill the dhcp client by doing this:
root@boron:~# pidof dhclient 518 root@boron:~# kill 518 # use the value you got from pidof
You can then apply the new resolver config by setting /etc/resolv.conf to this:
domain li694-22 nameserver 192.168.174.17 nameserver 192.168.170.201 nameserver 72.14.188.5
Finally, you should install some useful packages to have which are missing on the stock linode config
root@boron:~# apt-get install command-not-found python-software-properties
Setting up LDAP
You need the reader password and the slapd_ca.pem, pam-configs_mkhomedir ssh_ldap.sh files from the deployment kit now
root@boron:~# apt-get install ldap-auth-client libpam-ldap ldap-utils
When asked configuration questions, here's what you enter:
- LDAP server identifer: ldap://ldap-server.li694-22/ ldap://ldap-slave01.li694-22/
- Distiquished Name: dc=li694-22
- LDAP version to use: 3
- Make local Root admin: No
- Does LDAP require login: Yes
- LDAP username: cn=ldapReader,dc=li694-22
- LDAP password is in the deployment kit
(if you make a mistake; type dpkg-reconfigure libpam-ldap to re-run the wizard)
Purge away nscd, we don't need it, and it causes issues (it gets auto-installed by libpam-ldap)
root@boron:~# apt-get purge nscd
Now, take the slapd_ca.pem, and stick it in /usr/share/ca-certificates/li694-22 (you have to make this folder)
Open up /etc/ca-certificates.conf in your favorite editor, add the following to the end
li694-22/slapd_ca.pem
You need to now install the certificate into the system. Just run update-ca-certificates
root@boron:/usr/share/ca-certificates/li694-22# update-ca-certificates Updating certificates in /etc/ssl/certs... 1 added, 0 removed; done.
Then open /etc/ldap.conf with your favorite editor
Find and uncomment:
#ssl start_tls
Open /etc/ldap/ldap.conf, and set the following:
BASE dc=li694-22 URI ldap://ldap-server.li694-22/ ldap://ldap-slave01.li694-22
ldap should be setup now, now you just need to enable it in PAM, and update NSS. First, you need to copy pam-configs_mkhomedir to its proper place
root@boron:~# cp pam-configs_mkhomedir /usr/share/pam-configs/mkhomedir
Then update PAM. PAM should list "Active mkhomedirs" as an option if the config file was properly setup.
root@boron:~# pam-auth-update root@boron:~# auth-client-config -t nss -p lac_ldap
You should be able to run id and get valid results at this point
root@boron:~# id mcasadevall uid=2500(mcasadevall) gid=2501(sysops) groups=2501(sysops),2500(firefighters),2502(db)
Setup Machine ACLs
Setup Sudoers
Now we need to setup the ability to sudo, and to limit who can access a box
Sudo configuration is dependent on the node, and its role, but every box should have the following line amended to sudoers
%sysops ALL=(ALL) NOPASSWD: ALL
No user should use a password at any time.
Setup ACL
Now that is done, the last step is to limit who can access the box based off POSIX groups. Once again, this varies on a box-by-box basis, but its controlled by a single line.
Open up /etc/security/access.conf
This file is a Debian-derivate ACL list controlling who can and can't login. The syntax is explained in the file, but here's an example
-:ALL EXCEPT root (firefighters) (global_services): ALL
In this example, root (local user) can login, as well as those in the firefighter group; all others are denied. You can add multiple groups or users, i.e.
-:ALL EXCEPT root (sysops) (db) (global_services) : ALL
Every machine should allow access to global_services so that service that need to pop into a shell can get in.
Setup SSH
Upgrading SSH
Unfortunately, the version of OpenSSH shipped in precise is too old to support LDAP key retrieval, so we need to upgrade it. I threw together an updated package and loaded it onto a PPA, available here: https://launchpad.net/~li69422-staff/+archive/backports-for-precise
Adding it to the system is quick and painless
root@boron:~# apt-add-repository ppa:li69422-staff/backports-for-precise You are about to add the following PPA to your system: More info: https://launchpad.net/~li69422-staff/+archive/backports-for-precise Press [ENTER] to continue or ctrl-c to cancel adding it gpg: keyring `/tmp/tmpsvGLrk/secring.gpg' created gpg: keyring `/tmp/tmpsvGLrk/pubring.gpg' created gpg: requesting key AEA37004 from hkp server keyserver.ubuntu.com gpg: /tmp/tmpsvGLrk/trustdb.gpg: trustdb created gpg: key AEA37004: public key "Launchpad PPA for Packages for li694-22" imported gpg: Total number processed: 1 gpg: imported: 1 (RSA: 1) OK
Now we just need to install upgrades, and grab it.
root@boron:~# apt-get update && apt-get dist-upgrade
APT will print the following before upgrading
The following packages will be upgraded: openssh-client openssh-server
Note: You can ignore the following error while installing it:
/var/lib/dpkg/info/openssh-server.postinst: 295: /var/lib/dpkg/info/openssh-server.postinst: deb-systemd-helper: not found
Setting Up SSH-LDAP Authetication
So the magic that makes SSH-LDAP authethication work is a command in sshd_config that allows it to dynamically pull authorized_keys from a script. This is a quick and easy two step process
First, copy ldap_ssh.sh to /etc/ssh
root@robot:~# cp ldap_ssh.sh /etc/ssh
Now open /etc/ssh/sshd_config and add the following lines at the bottom
AuthorizedKeysCommand /etc/ssh/ldap_ssh.sh AuthorizedKeysCommandUser nobody
Restart SSH
root@boron:~# service ssh restart ssh stop/waiting ssh start/running, process 5327
And test it by logging in via SSH directly
Locking Down SSH
Once you config that SSH is setup properly, lock down the configuration file. The following should be true
- SSH should only listen on internal IPs (we can proxy in from boron)
- Password authetication should be disabled
- Root login should be disabled
This requires three edits in /etc/ssh/sshd_config, first, the listen lines
ListenAddress INTERNAL-IP
Then Password Authetication
PasswordAuthentication no
And then disable root. You have to add this line as its not in the example config
PermitRootLogin no
Save your changes, and restart ssh
root@boron:~# service ssh restart ssh stop/waiting ssh start/running, process 5752
Setting up Kerberos (KRB5)
Setting up single-signon makes life really easy when working across a bunch of nodes, so we have kerberos available facility it. By running kinit, you can generate a kerberos ticket, and zoom around in style.
A quick APT gets us what we need:
root@lithium:~# apt-get install krb5-user
- Default kerberos realm: LI694-22
To test, switch to your user account, and run kinit:
mcasadevall@lithium:~$ kinit Password for mcasadevall@LI694-22: mcasadevall@lithium:~$ klist Ticket cache: FILE:/tmp/krb5cc_2500 Default principal: mcasadevall@LI694-22 Valid starting Expires Service principal 23/03/2014 08:22 23/03/2014 18:22 krbtgt/LI694-22@LI694-22 renew until 24/03/2014 08:22
If you got a ticket, you're done with user configuration.
Now you need to generate a ticket as an admin, and generate the necessary server site bits. Due to a bug, you need to edit /etc/krb.conf.
Under realms, add the following:
LI694-22 = {
admin_server = kdc-master.li694-22
}
sudo back to root, and add the principle
mcasadevall@lithium:~# kdestroy # (clean out old tickets) mcasadevall@lithium:~# kinit krb/admin # (password is in master_passwd on ldap-master) Password for krb/admin@LI694-22: mcasadevall@lithium:~$ kadmin Authenticating as principal krb/admin@LI694-22 with password. Password for krb/admin@LI694-22: kadmin:
From here, you have to generate a principle for the node, then export it into the keytab, That's done with two commands
kadmin: add_principal -randkey host/carbon.li694-22@LI694-22 WARNING: no policy specified for host/carbon.li694-22@LI694-22; defaulting to no policy add_principal: Principal or policy already exists while creating "host/carbon.li694-22@LI694-22". kadmin: ktadd -randkey host/carbon.li694-22@LI694-22 (lots of information) kadmin: quit
Now its time for round 2 with SSH's config, you need to edit both the client and server settings this time.
Set the following in /etc/ssh/sshd_config KerberosAuthentication yes GSSAPIAuthentication yes GSSAPICleanupCredentials yes
Now you need to edit the outbound config to send Kerberos tickets
Set the following in /etc/ssh/ssh_config (under Hosts *) GSSAPIDelegateCredentials yes
Confirm you can SSH freely between nodes, and you're done
Install Nagios Plugins
We use a fork of nagios for service monitoring. Setup is done on boron (sentinal.soylentnews.org), but the short version is you just need to install the plugins locally, then setup service monitoring via the icinga config files. This is a one liner.
$ sudo apt-get install nagios-plugins nagios-plugins-extra
Cleanup
Now install pwgen:
apt-get install pwgen
Generate a new root password with pwgen (i.e. pwgen 50 1), and set it
Update the master_password file, and take a breath, you're done!