Infrastructure

Select the monitoring framework and design the deployment of the selected framework, including gathering the necessary information for configuration.

Nagios seems like an excellent candidate for this, along with Cacti and Munin. Relevant email message from Nick:

I'm using a combination of Nagios, Munin, and Cacti to do monitoring and stats collecting. My Nagios setup is doing service and status checks and sending out alerts, while Cacti and Munin are doing graphing and stats.

I'm using Munin because it's dead simple to get running on a machine and has a good number of default plugins, while Cacti does the more advanced data collection. I have Cacti collecting data from several hundred switch ports through SNMP, and running some custom Bash and Python scripts to get data from some hardware without any good plugins or SNMP access.

While I don't have any experience collecting data with Nagios, I definitely recommend running Cacti along with it. Cacti is extremely customizable, and it's pretty easy to have a certain set of graphs display for certain users. It also has the ability to zoom in on graphs, and download data in csv format.

Here are some examples (the public username:password is visitor:NITscoc):

Cacti, with a limited set of graphs:
http://raptor.nceas.ucsb.edu/cacti/

Munin, with stats for the UCSB dataone server:
http://raptor.nceas.ucsb.edu/munin/nceas.ucsb.edu/dataone.nceas.ucsb.edu.html

Nagios alerts go out by email and IRC (check out #sysnotify our IRC server). I have the admin web interface running, but it's not currently set up for anyone except admin users.

I have all three installed from the Ubuntu-supported repositories on Ubuntu 8.04, so they're two years old at this point. The new versions probably look nicer.