the second phase

In my industry, there’s alot of monitoring to be done. We have servers, services, processes, ports, log files, applications, you name it, we have to monitor it.

When I first started at DI, there was a current implementation of Nagios. Unfortunately, it was horribly done. Some agents used SSL, some didn’t. It was all done with separate configuration files, it was slow and if you had too many monitors, they didn’t go off in time.

In any case, we started investigating replacements for Nagios and we came upon Hyperic. Hyperic is an open source ( although they have an enterprise version ) monitoring system.  It is written in java and has individual agents that are installed on each of the systems you wish to monitor.  The agents have a very low overhead, although that is somewhat dependent on if they are running scripts and what those scripts are doing.

Read the rest of this entry »

I’ll have my review of Hyperic 3.2.2 posted in a bit and I’ll be frank to say that I’m not too happy with it.

This is a document I wrote a while back for work that I thought I would release in hopes that some people out there would find it useful.

Preferably, you have a backup of the file system that you can use. If not, the filesystem you are about to try to to recover a file on must meet these requirements:

• No new files have been created on the filesystem.
• No files have been extended.
• The filesystem is able to be unmounted.
• It is a JFS filesystem, not JFS2

If so, then please, drink a few more beers and continue, but before you do…

BACKUP THE CURRENT FILESYSTEM!
Read the rest of this entry »

It seems that lately all that I’ve written have been docs on how to do stuff at work. Mostly because I’ve been busy ( scuba, girlfriend, work, soccer, friends, etc. ) unlike some people I know ( hi pierre ). Anyways, recently while at work, we found that we wanted to switch the names of some of the network adapters on an AIX machine. However, this turns out to be a very complicated thing to do. You cannot just rename them. I also didn’t want to mess around with moving the cards around in the machine, rebooting, testing, etc. I just wanted to fix the damn names.

So I did. Here’s how.

First, get all the information about the adapters.

for i in ent0 ent1 ent2
do
odmget -q name=”$i” CuDv >> /tmp/$i
odmget -q name=”$i” CuAt >> /tmp/$i
odmget -q name=”$i” CuVPD >> /tmp/$i
done

Next, down the interfaces and detach them.

for i in en0 en1 en2 et0 et1 et2
do
ifconfig $i down
ifconfig $i detach
done

Now, remove all the references to the devices from the ODM

for i in ent0 ent1 ent2 en0 en1 en2 et0 et1 et2
do
odmdelete -q name=”$i” -o CuAt
odmdelete -q name=”$i” -o CuDv
odmdelete -q name=”$i” -o CuVPD
odmdelete -q value3=”$i” -o CuDvDr
done

We can verify that no adapters and no interfaces exist now by issuing the lsdev commands again. All we should see is the loopback interface.

lsdev -Cc adapter -l ent*
lsdev -Cc if
lo0 Available Loopback Network Interface

Edit the files we created the first step and replace every instance of the adapter name with the new adapter name. For instance, I would edit /tmp/ent0 and replace all instances of “ent0″ with “ent2″. We can do this with a sed script.

sed -e “s/ent0/ent1/g” /tmp/ent0 > /tmp/ent1.new
sed -e “s/ent1/ent2/g” /tmp/ent1 > /tmp/ent2.new
sed -e “s/ent2/ent0/g” /tmp/ent2 > /tmp/ent0.new

Then add the files back to the ODM.

odmadd /tmp/ent0.new
odmadd /tmp/ent1.new
odmadd /tmp/ent2.new

At this point, our adapters will now be redefined. Issue another lsdev command to check:

lsdev -Cc adapter -l ent*
ent0 Available 05-08 10/100/1000 Base-TX PCI-X Adapter (14106902)
ent1 Available 07-08 2-Port 10/100/1000 Base-TX PCI-X Adapter (14108902)
ent2 Available 07-09 2-Port 10/100/1000 Base-TX PCI-X Adapter (14108902)

You can see now that ent0 is now the external PCI-X adapter and ent1 and ent2 are the two onboard adapters. But, we still have no interfaces for the adapters. You can verify this by issuing the usual lsdev command again. You should only see the loopback interface.

lsdev -Cc if
lo0 Available Loopback Network Interface

To fix this ( and to make sure our changes stick upon a reboot… ), run a cfgmgr, then check for our interfaces.

cfgmgr
lsdev -Cc if
en0 Defined 05-08 Standard Ethernet Network Interface
en1 Defined 07-08 Standard Ethernet Network Interface
en2 Defined 07-09 Standard Ethernet Network Interface
et0 Defined 05-08 IEEE 802.3 Ethernet Network Interface
et1 Defined 07-08 IEEE 802.3 Ethernet Network Interface
et2 Defined 07-09 IEEE 802.3 Ethernet Network Interface
lo0 Available Loopback Network Interface

As you can see, we have successfully gotten our interfaces back. We’re almost done! All you need to do now is reboot the system.

shutdown -Fr

Once the reboot has completed, issue one last check to verify that the adapters have changed:

entstat -d ent0 | grep “Device Type”
Device Type: 10/100/1000 Base-TX PCI-X Adapter (14106902)

Looks good!

We ran into another issue this week with Kickstart. We bought a bunch of new 64 bit machines ( AMD 64 bit, to be exact ) and have been developing a Kickstart solution for them. However, we have a bunch of applications that require some 32 bit libraries in order to run. Most Kickstart suggestions were to simply install certain compat groups, however, like the bad sheep of the family, that’s not what we wanted to do. We do a very minimal install on our machines and didn’t want to install a bunch of libraries that we didn’t need. After much searching, we found this simple way of doing it. In the %packages section, simply add .i386 to the end of the package name.

For example:

libgcc.i386

That’s all it takes in order to install a 32-bit version of a library. Hope this helps.

For any of you who have ever tried to use or configure sendmail, you know it’s not a fun task. At best, you wince, try to browse some google groups as you make your way through the billions of posts regarding some tribal knowledge that has to do with this release of sendmail on that specific OS. Then throw DNS into the mix, sendmail’s partner in crime and we’ve got a real ho-down. Especially when DNS goes missing. When DNS doesn’t arrive for the party, sendmail gets mad.

Real mad.

Anyways, in alot of our networks, we have hosts that don’t have DNS. Why? Because we’re too lazy to set the shit up. Seriously tho, it’s mostly because work has me stuck on building machines ( ie: monkey work ) rather than trying to solve the real issues in our infrastructure. Since I just spent the last hour trying to figure out how to get a RedHat Enterprise 4 ( U4 ) machine to send mail without DNS to a smart relay host, I thought I’d share the results and how to do it.

First, make sure you have the sendmail-cf package installed. Either install it locally via media or use yum/up2date to get it.

rpm -ivh sendmail-cf-8.13.1-3.RHEL4.5.i386.rpm

Next, add your smart relay host to your /etc/hosts file

192.168.0.6 mailhost mailhost.domain.com

Go into /etc/mail/ and vi the sendmail.mc file. Add the following lines:

FEATURE(`accept_unresolvable_domains')dnl
FEATURE(`accept_unqualified_senders')dnl
FEATURE(nocanonify)
define(`confSERVICE_SWITCH_FILE', `/etc/mail/service.switch')dnl
define(`SMART_HOST',`mailhost')

Note: The first character after the “FEATURE(” is a tick mark ( ` ) from the tilde key, this is not a typo. The ending quotation is a single quote mark ( ‘ ).  I’ve enclosed the code in pre tags, although it looks ugly, to ease copying and pasting into the configuration. Please look at the sendmail.mc file for more clarification.

Of course, replace the mailhost with the host you entered in your /etc/hosts file. Now create a file in /etc/mail called service.switch and add the following:

hosts files

For our last file modification, edit the submit.mc file in /etc/mail and add the following line:

define(`confDIRECT_SUBMISSION_MODIFIERS', `C')dnl

Now that we’re done making our file modifications, simply type ‘make’ while in the /etc/mail directory. You shouldn’t see any output, but if you do a directory listing, you should see some .bak files now. You can compare these with the current .cf files to see what changes were made.

Restart sendmail and the sm-client:

service sendmail restart

And voila! You should now successfully be able to send mail to your smart relay host with no DNS. To test, trying something like this:

mail -s test name@domain.com < /dev/null 2>&1 >> /dev/null

Cross your fingers, sacrifice a few goats and chickens, do a little dance, make a little love and check your inbox for the test email.

Yes, powercrap! Otherwise known as Symark’s PowerPassword, PowerBroker suite. These two little products cost quite a bundle to place on every single computer in your network. They also have some limits to their functionality, for one being that they don’t integrate TOGETHER. At least, our technical team hasn’t quite figured out how to do that yet. PowerBroker, I’ll admit, could have some cool functions if we took care to use them, but we don’t. It’s like buying a Ferrari for going to the grocery markert. It’s just too big of a product for what we need. And c’mon, what the hell is with this branding? Powerthis, Powerthat. I think the Tiger could use a PowerFist. And only he knows why!

Anyways, I’m looking for some good information towards implementing sudo/ldap as a authentication/root permission scheme in a multiple multi-tier network. Man, that’s a mouthful. So if you have information, pass it my way, k? Thanks.