NOC Tools Handout
Covered applications
Smokeping – Latency and service performance measurement
Big Sister – Monitoring services and hosts historical performance data logging
Rancid – Router and switch configuration diffing
Trac – ticket generation and tracking, wiki
Note, all of these applications are available from ports.
Smokeping
/usr/ports/net/smokeping/
make install
edit your /usr/local/etc/apachexx/httpd.conf
add:
#added for smokeping
ScriptAlias /smokeping.cgi /usr/local/smokeping/htdocs/smokeping.cgi
Alias /smokeimg/ /usr/local/smokeping/htdocs/img/
Depending on which version of apache you may have to override the default dirtory permissions which you can do for the entire webserver or just selected portions.
The configuration for smokeping is done in:
/usr/local/etc/smokeping/config...
The syntax is slightly daunting, at a minimum though you should change:
owner
contact
mailhost
at the top of the configuration, then towards the bottom, change the section starting with +World to reflect the hosts taht you want to monitor, for the local network I did something like the following:
+Local
menu = Local
title= local
host = router-inside.pacnog2.dnsdojo.net
+ World
menu = World
title = Worldwide Connectivity
++Teleglobe
host = if-4-0-0.bb2.LCN-LakeCowichan.Teleglobe.net
++Oregon
host = twin.uoregon.edu
++Blotto
host = blotto.ath.cx
Once the configuration file has been adequately dealt with you need to add it's directive into /etc/rc.conf and then you can start is as usual from /usr/local/etc/rc.d/smokeping.
once it is collecting data you can access that data by going to:
http://your_webserver/smokeping.cgi
Big Sister
Big sister is in /usr/ports/net-mgmt/bigsister
make install
bigsister is a cocmplicated package with a lot of files... so it installs itself in it's own tree at /usr/local/bigsister.
In /usr/local/bigsister among other things you'll see a directory called www which is where the bigsister generated webpages live. if you create symlink to this directory from inside /usr/local/www/apache22/data/ you can access the bigsister webpage by going to http://yourserver/name_of_symlink.
The adm directory is where the files which control what Big Sister monitors and what it does with that information.
If you look in the adm/uxmon-net that's where the tests are configured for each host. you can note that it already has some pre-configured values for monitoring the localsystem.
The documentation for Big Sister is here:
http://www.joerg.cc/html/bigsis/index.html
the monitoring we did in the classroom went into the uxmon-net looks something like this for each PC:
pc24.pacnog2.dnsdojo.net proto=icmp ping ssh smtp http
pc24.pacnog2.dnsdojo.net check=named service=domain tcp
Once the bigsister service /usr/local/etc/rc.d/bigsister is started it will begin monitoring conifgured hosts. if you make changes to the uxmon-net file while it's running, it will detect that and reload.
Rancid
rancid is /usr/ports/net-mgmt/rancid/
make install
/usr/local/etc/rancid/rancid.conf sets up some of rancid's environment. copy rancid.conf.sample to rancid.conf
At the bottom of the file there's item:
#LIST_OF_GROUPS="sl joebobisp"
Define at least one group give it a name that's reasonable.
for example routers, so:
#LIST_OF_GROUPS="routers"
edit /etc/mail/aliases
add two aliases
rancid-routers: email@address
rancid-admin-routers: email@address
run the newaliases command to regenerate your aliases
Pick a user or create a user called rancid. or use root, maybe but not very safe.
copy the file /usr/local/share/rancid/cloginrc.sample to ~username/.cloginrc
edit .cloginrc
while the cloginrc syntax is quite rich at it's simplest form for a cisco router it looks like:
add password routername vtypass enpass
if you want to use ssh first you would add:
add method routername ssh
normally RANCID will try telnet then ssh
Once you've saved that, attempt to log into the router you just configured by typing:
clogin routername
make sure that the .cloginrc has the permissions 600 (read only by owner). It contains your router passwords!
go to /usr/local/var/rancid.
type:
rancid-cvs (then the groupname ie routers in this case)
note that it will create the routers group and also setup the cvs directory
go into the routers directory:
edit the file router.db (it doesn't exist yet)
Put the route fqdn or ip address (something that would be matched by a password line in the cloginrc) the type of router and the state all on the same line, for example:
192.168.0.250:cisco:up Make sure that all the files and directories in /usr/local/var/rancid are writable or better yet owned by the user that's going to run rancid.
run the command rancid-run
if it succeeds without error it's just a question now of inserting a cronjob:
# run config differ hourly
1 * * * * /usr/local/bin/rancid-run
# clean out config differ logs
50 23 * * * /usr/bin/find /usr/local/var/rancid/logs -type f -mtime +2 -exec rm {};
If your router configuration is changed, you will get an email of the diffs. If the router reboots, you will also get a diffs email since some of the status information has changed. If the router becomes unreachable, you'll also be notified of that after four hours.
Note, Rancid isn't Just for Cisco Routers, out of the box it supports:
arancid Alteon WebOS switches
brancid Bay Networks (nortel)
cat5rancid Cisco catalyst switches
cssrancid Cisco content services switches
erancid ADC-kentrox EZ-T3 mux
f10rancid Force10
fnrancid Fortinet Firewalls
francid Foundry and HP procurve OEMs of Foundry
hrancid HP Procurve Switches
htranicd Hitachi Routers
jerancid Juniper Networks E-series
jrancid Juniper Networks
mrancid MRTd
prancid Procket Networks
rivrancid Riverstone
rrancid Redback
tntrancid Lucent TNT
nrancid Netscreen firewalls
nsrancid Netscaler
xrancid Extreme switches
zrancid Zebra routing software