NOC Tools Handout

Covered applications

Smokeping – Latency and service performance measurement

Big Sister – Monitoring services and hosts historical performance data logging

Rancid – Router and switch configuration diffing

Trac – ticket generation and tracking, wiki

Note, all of these applications are available from ports.

Smokeping

/usr/ports/net/smokeping/

make install

edit your /usr/local/etc/apachexx/httpd.conf

add:

#added for smokeping

ScriptAlias /smokeping.cgi /usr/local/smokeping/htdocs/smokeping.cgi

Alias /smokeimg/ /usr/local/smokeping/htdocs/img/

Depending on which version of apache you may have to override the default dirtory permissions which you can do for the entire webserver or just selected portions.

The configuration for smokeping is done in:

/usr/local/etc/smokeping/config...

The syntax is slightly daunting, at a minimum though you should change:

owner

contact

mailhost


at the top of the configuration, then towards the bottom, change the section starting with +World to reflect the hosts taht you want to monitor, for the local network I did something like the following:

+Local

menu = Local

title= local

host = router-inside.pacnog2.dnsdojo.net



+ World

menu = World

title = Worldwide Connectivity


++Teleglobe

host = if-4-0-0.bb2.LCN-LakeCowichan.Teleglobe.net


++Oregon

host = twin.uoregon.edu


++Blotto

host = blotto.ath.cx


Once the configuration file has been adequately dealt with you need to add it's directive into /etc/rc.conf and then you can start is as usual from /usr/local/etc/rc.d/smokeping.

once it is collecting data you can access that data by going to:

http://your_webserver/smokeping.cgi

Big Sister

Big sister is in /usr/ports/net-mgmt/bigsister


make install


bigsister is a cocmplicated package with a lot of files... so it installs itself in it's own tree at /usr/local/bigsister.


In /usr/local/bigsister among other things you'll see a directory called www which is where the bigsister generated webpages live. if you create symlink to this directory from inside /usr/local/www/apache22/data/ you can access the bigsister webpage by going to http://yourserver/name_of_symlink.


The adm directory is where the files which control what Big Sister monitors and what it does with that information.


If you look in the adm/uxmon-net that's where the tests are configured for each host. you can note that it already has some pre-configured values for monitoring the localsystem.


The documentation for Big Sister is here:


http://www.joerg.cc/html/bigsis/index.html


the monitoring we did in the classroom went into the uxmon-net looks something like this for each PC:


pc24.pacnog2.dnsdojo.net proto=icmp ping ssh smtp http

pc24.pacnog2.dnsdojo.net check=named service=domain tcp


Once the bigsister service /usr/local/etc/rc.d/bigsister is started it will begin monitoring conifgured hosts. if you make changes to the uxmon-net file while it's running, it will detect that and reload.


Rancid


rancid is /usr/ports/net-mgmt/rancid/


make install


/usr/local/etc/rancid/rancid.conf sets up some of rancid's environment. copy rancid.conf.sample to rancid.conf


At the bottom of the file there's item:


#LIST_OF_GROUPS="sl joebobisp"


Define at least one group give it a name that's reasonable.


for example routers, so:


#LIST_OF_GROUPS="routers"


edit /etc/mail/aliases


add two aliases


rancid-routers: email@address

rancid-admin-routers: email@address


run the newaliases command to regenerate your aliases


Pick a user or create a user called rancid. or use root, maybe but not very safe.


copy the file /usr/local/share/rancid/cloginrc.sample to ~username/.cloginrc


edit .cloginrc


while the cloginrc syntax is quite rich at it's simplest form for a cisco router it looks like:

add password routername  vtypass enpass


if you want to use ssh first you would add:


add method routername ssh


normally RANCID will try telnet then ssh


Once you've saved that, attempt to log into the router you just configured by typing:


clogin routername


make sure that the .cloginrc has the permissions 600 (read only by owner). It contains your router passwords!

go to /usr/local/var/rancid.


type:


rancid-cvs (then the groupname ie routers in this case)


note that it will create the routers group and also setup the cvs directory


go into the routers directory:


edit the file router.db (it doesn't exist yet)


Put the route fqdn or ip address (something that would be matched by a password line in the cloginrc) the type of router and the state all on the same line, for example:


 192.168.0.250:cisco:up

Make sure that  all the files and directories in /usr/local/var/rancid are writable or better yet owned by the user that's going to run rancid. 


run the command rancid-run


if it succeeds without error it's just a question now of inserting a cronjob:


# run config differ hourly
1 * * * * /usr/local/bin/rancid-run
        
# clean out config differ logs
50 23 * * * /usr/bin/find /usr/local/var/rancid/logs -type f -mtime +2 -exec rm {};


If your router configuration is changed, you will get an email of the diffs. If the router reboots, you will also get a diffs email since some of the status information has changed. If the router becomes unreachable, you'll also be notified of that after four hours.

Note, Rancid isn't Just for Cisco Routers, out of the box it supports:

       arancid        Alteon WebOS switches

       brancid        Bay Networks (nortel)

       cat5rancid     Cisco catalyst switches

       cssrancid      Cisco content services switches

       erancid        ADC-kentrox EZ-T3 mux

       f10rancid      Force10

       fnrancid       Fortinet Firewalls

       francid        Foundry and HP procurve OEMs of Foundry

       hrancid        HP Procurve Switches

       htranicd       Hitachi Routers

       jerancid       Juniper Networks E-series

       jrancid        Juniper Networks

       mrancid        MRTd

       prancid        Procket Networks

       rivrancid      Riverstone

       rrancid        Redback

       tntrancid      Lucent TNT

       nrancid        Netscreen firewalls

       nsrancid       Netscaler

       xrancid        Extreme switches

       zrancid        Zebra routing software