SystemAdministration/TheRiseAndFallOfNewNodeManagement

For those who opened this sacred tomb, take a moment to decide if you wish to truly proceed. There are better things to do with your life like walk around the world, or learn to play pinball with your feet. If you truly wish to proceed, remember that their be dragons here.

Initial Setup

(this guide assumes we're using Ubuntu 12.04 and are on Linode, most of this is still relevent in general, but ignore the linode bits)

Once a new node is created on Linode, you need to deploy Ubuntu 12.04, this can "Dashboard" tab easily. Make sure you give 512M of swap, power it up, write down the root password, then open a console. We've got work to do.

On helium, in the root home directory, there's a folder called deployment_kit which has all the files you need to copy in place.

Install All Updates

Linode's image is a bit out of date, so a quick upgrade is needed, first you need to update the package index

Last login: Sat Mar 22 22:02:47 2014
root@localhost:~# apt-get update
Get:1 http://mirrors.linode.com precise Release.gpg [198 B]
Get:2 http://mirrors.linode.com precise-updates Release.gpg [198 B]
Get:3 http://mirrors.linode.com precise-backports Release.gpg [198 B]
Get:4 http://mirrors.linode.com precise-security Release.gpg [198 B]
-SNIP-

Then install updates

root@localhost:~# apt-get dist-upgrade
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  accountsservice apport apt apt-transport-https apt-utils apt-xapian-index base-files bash-completion bc bind9-host curl dbus dmsetup dnsutils dosfstools
  dpkg file gnupg gpgv grub-common ifupdown initramfs-tools initramfs-tools-bin iproute isc-dhcp-client isc-dhcp-common landscape-common language-pack-en
  language-pack-en-base language-selector-common libaccountsservice0 libapt-inst1.4 libapt-pkg4.12 libasn1-8-heimdal libbind9-80 libc-bin libc6 libcurl3
  libcurl3-gnutls libdbus-1-3 libdevmapper1.02.1 libdns81 libdrm-intel1 libdrm-nouveau1a libdrm-radeon1 libdrm2 libgcrypt11 libglib2.0-0 libgnutls26
  libgssapi3-heimdal libhcrypto4-heimdal libheimbase1-heimdal libheimntlm0-heimdal libhx509-5-heimdal libisc83 libisccc80 libisccfg82 libkrb5-26-heimdal
  libldap-2.4-2 liblockfile-bin liblockfile1 liblwres80 libmagic1 libpci3 libplymouth2 libpolkit-gobject-1-0 libpython2.7 libroken18-heimdal libssl1.0.0
  libudev0 libwind0-heimdal libxcb1 libxml2 lsb-base lsb-release multiarch-support openssl pciutils perl perl-base perl-modules plymouth
  plymouth-theme-ubuntu-text procps python python-apport python-apt python-apt-common python-httplib2 python-lazr.restfulclient python-minimal
  python-openssl python-problem-report python2.7 python2.7-minimal rsyslog sudo tzdata udev unzip update-manager-core w3m xkb-data
103 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 44.5 MB of archives.
After this operation, 19.5 kB of additional disk space will be used.
Do you want to continue [Y/n]? 

This takes about 5-10 minutes. Drink a soda, and compliate life ...

Set Hostname

Hostnames should be setup with the next item on the [HostnamePolicy|Hostname Policy]. On Ubuntu, you need to edit /etc/hostname, and /etc/hosts

root@localhost:~# cat /etc/hostname 
boron
root@localhost:~# cat /etc/hosts
127.0.0.1	localhost
127.0.1.1	boron

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

Load the new hostname with 'hostname -F'

root@localhost:~# hostname -F /etc/hostname 
root@localhost:~# 

Note, prompt won't change until to log out/log back in.

Switch Over To Distro Kernels

Linode uses a customized kernel instead of stock Ubuntu kernels. While this works "well enough" for most people, it lacks AppArmor, and cause unexpected splats as it doesn't have a ramdisk.

Here's Linode's guide on how to fix it: https://library.linode.com/custom-instances/pv-grub-howto

When you're done, uname -a should say something like this

root@boron:~# uname -a
Linux boron 3.2.0-60-virtual #91-Ubuntu SMP Wed Feb 19 04:13:28 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Setup Networking

On the Linode panel, make sure the node has an internal IP address so that other nodes in the data centre can access it, then note it. You have to setup static IP address configure. Linode has a decent guide for this, but the quick and dirty version is you need to edit /etc/network/interfaces to look like this

# The loopback interface
auto lo
iface lo inet loopback

# Configuration for eth0 and aliases

# This line ensures that the interface will be brought up during boot.
auto eth0 eth0:0 eth0:1

# eth0 - This is the main IP address that will be used for most outbound connections.
# The address, netmask and gateway are all necessary.
iface eth0 inet static
 address PUBLIC-IP-HERE
 netmask 255.255.255.0
 gateway   GATEWAY-HERE
# eth0:0
# This is a second public IP address.
iface eth0:0 inet static
 address INTERNAL-IP-HERE
 netmask 255.255.128.0

You can apply the new IP configuration with this

root@boron:~# ifdown eth0 && ifup eth0 eth0:0
resolvconf: Error: /etc/resolv.conf isn't a symlink, not doing anything.
resolvconf: Error: /etc/resolv.conf isn't a symlink, not doing anything.
ssh stop/waiting
ssh start/running, process 1087
resolvconf: Error: /etc/resolv.conf isn't a symlink, not doing anything.
ssh stop/waiting
ssh start/running, process 1127
root@boron:~#  

Finally, you should install some useful packages to have which are missing on the stock linode config

root@boron:~# apt-get install command-not-found python-software-properties

FIXME: RESOLVER

Setting up LDAP

You need the reader password and the slapd_ca.pem, pam-configs_mkhomedir ssh_ldap.sh files from the deployment kit now

root@boron:~# apt-get install ldap-auth-client libpam-ldap ldap-utils

When asked configuration questions, here's what you enter:

• LDAP server identifer: ldap://ldap-server.li694-22/
• Distiquished Name: dc=li694-22
• LDAP version to use: 3
• Make local Root admin: No
• Does LDAP require login: Yes
• LDAP username: cn=ldapReader,dc=li694-22
• LDAP password is in the deployment kit

(if you make a mistake; type dpkg-reconfigure libpam-ldap to re-run the wizard)

Purge away nscd, we don't need it, and it causes issues (it gets auto-installed by libpam-ldap)

root@boron:~# apt-get purge nscd

Now, take the slapd_ca.pem, and stick it in /usr/share/ca-certificates/li694-22 (you have to make this folder)

Open up /etc/ca-certifiates.conf in your favorite editor, add the following to the end

li694-22/slapd_ca.pem

You need to now install the certificate into the system. Just run update-ca-certificates

root@boron:/usr/share/ca-certificates/li694-22# update-ca-certificates 
Updating certificates in /etc/ssl/certs... 1 added, 0 removed; done.

Then open /etc/ldap.conf with your favorite editor

Find and uncomment:

#ssl start_tls

ldap should be setup now, now you just need to enable it in PAM, and update NSS. First, you need to copy pam-configs_mkhomedir to its proper place

root@boron:~# cp pam-configs_mkhomedir /usr/share/pam-configs/mkhomedir

Then update PAM. PAM should list "Active mkhomedirs" as an option if the config file was properly setup.

root@boron:~# pam-auth-update 
root@boron:~# auth-client-config -t nss -p lac_ldap

You should be able to run id and get valid results at this point

root@boron:~# id mcasadevall
uid=2500(mcasadevall) gid=2501(sysops) groups=2501(sysops),2500(firefighters),2502(db)

Setup SSH

Upgrading SSH

Unfortunately, the version of OpenSSH shipped in precise is too old to support LDAP key retrieval, so we need to upgrade it. I threw together an updated package and loaded it onto a PPA, available here: https://launchpad.net/~li69422-staff/+archive/backports-for-precise

Adding it to the system is quick and painless

root@boron:~# apt-add-repository ppa:li69422-staff/backports-for-precise
You are about to add the following PPA to your system:
 
 More info: https://launchpad.net/~li69422-staff/+archive/backports-for-precise
Press [ENTER] to continue or ctrl-c to cancel adding it

gpg: keyring `/tmp/tmpsvGLrk/secring.gpg' created
gpg: keyring `/tmp/tmpsvGLrk/pubring.gpg' created
gpg: requesting key AEA37004 from hkp server keyserver.ubuntu.com
gpg: /tmp/tmpsvGLrk/trustdb.gpg: trustdb created
gpg: key AEA37004: public key "Launchpad PPA for Packages for li694-22" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
OK

Now we just need to install upgrades, and grab it.

root@boron:~# apt-get update && apt-get dist-upgrade

APT will print the following before upgrading

The following packages will be upgraded:
  openssh-client openssh-server

Setting Up SSH-LDAP Authetication

So the magic that makes SSH-LDAP authethication work is a command in sshd_config that allows it to dynamically pull authorized_keys from a script. This is a quick and easy two step process

First, copy ldap_ssh.sh to /etc/ssh

root@robot:~# cp ldap_ssh.sh /etc/ssh

Now open /etc/ssh/sshd_config and add the following lines at the bottom

AuthorizedKeysCommand /etc/ssh/ldap_ssh.sh
AuthorizedKeysCommandUser nobody

Restart SSH

root@boron:~# service ssh restart
ssh stop/waiting
ssh start/running, process 5327

And test it by logging in via SSH directly