SystemAdministration

Welcome

This is a comphensive index dealing with aspects of system administration and management of our clusters, as well as some of the more archine bits of setup required to make it work.

Who we are

Sysop Team Main Page
nick	position	timezone
xlefay	Co-team leader	UTC+2 (CEST)
paulej72	Co-team leader	UTC-4 (EDT)
mechanicjay	Co-leader	UTC-4 (EST/EDT)
NCommander	Member	UTC-9 (AKDT)
robind	Member	?
Audioguy	Member	UTC-7 (PST/PDT)

Index of Development Pages and Resources

Servers

List of servers on linode: Category:SystemAdministration/Servers

soylent-www - Primary Apache and slash servers for main site.

Hydrogen, Fluorine, Boron

soylent-db -- mysql servers, holds the slash database.

Helium, Neon

dev -- Development server.

Lithium

staff-slash -- Staff only Slash server.

Nitrogen

irc -- IRC server and related services.

Carbon

backups -- Backup services.

Oxygen

directory services -- LDAP and Kerberos.

Helium, Boron

soylent-services - mail, wiki, other services as needed.

Beryllium

Known Problems

Need cron job to backup server
No init script for Apache.
Broken https configuration
- Mostly fixed, Slash is the problem child now
Gluster is occassionally misfiring, manifests as Apache or slashd crashing depending on the node, can be fixed with the following command cocktail

sudo umount -l /srv/soylentnews.org # tells linux to lazy unmount, required when glusterd took a dive
sudo service glusterfs-server restart
sudo mount -a # will remount gluster without an issue

Then restart Apache/Slashd as required

Stuff That Needs To Be Addressed

Hydrogen is off line due to performance problems
icinga/monitoring project needs to be picked up and completed
Should have some sort of SN password safe
Privilege Duplication - making sure that all services have multiple admins
DNS, Audioguy is investigating some goofiness
Systems Documentation needs to revised and brought up to date.
Work Coordination, not always good communication when fundamental things change.

Work Notes

DNS is completely run and managed by Linode's DNS Manager service. This was an expedient decision when trying to get off bluehost. We may want to investigate putting the master zone file on helium or boron and having external services handle serving out our dns.

Resources

Access Instructions - how to get on the nodes, get around, and kerberos for users primer
Group Permissions - understanding our LDAP groups, what machines they can access, and where you can sudo
LDAP Management for Dummies - how to do basic shit in that source of miserary known as LDAP
The Rise And Fall Of New Node Management - from bash to fully intergrated node, this doc has it
The Hitchhikerr's Guide to The li694-22 Domain - machine list, general information
Kerberos Administration Or Everything You Wanted To Know About Kerberos But Were Afraid To Ask
DnsRecords - copy of the zone file pasted from the machine that was accidentally turned off
Emergency Technical Procedures - in case of fire, break glass (Read before messing with servers)
Backup Information - information on where and what is being backed up.