The following script is intended to be placed on your clients.  It allows you to search for a software policy to install, as well as for the host you wish to install it on.  If you don’t specify a host, it’ll just install locally.  It should work on HP SA 7.86 and up.

Here’s the usage:

install_policy.py -u USER [ -p PASSWORD ] -s POLICY [ --host HOSTNAME ]

And the script:

I've taken down the script for a day or so while I rewrite it! The host I develop on uses a python 2.4 binary for the agent and the script isn't compatible with the much more common 1.5.2 binary that comes with SA

#!/opt/opsware/agent/bin/python

import sys
sys.path.append("/opt/opsware/pylibs")
sys.path.append("/opt/opsware/agent/pylibs")

from pytwist import *
from pytwist.com.opsware.device import DeviceGroupRef

ts = twistserver.TwistServer()
ds = ts.device.DeviceGroupService
ss = ts.server.ServerService

# Enter your username and password here
ts.authenticate("username","password")

# Enter the device group object ID for the group you want to iterate over
# You can get this object ID from the GUI
dgref = DeviceGroupRef(#######)
srefs = ds.getDevices(dgref)
for sref in srefs:
    svo = ss.getServerVO(sref)
    print "Updating %s" % svo.name
    # Set the stage.  This has to be a valid option - these are always in caps.
    svo.stage = "PRODUCTION"
    ss.update(sref, svo, 0, 0)

There are a few ways you can go around doing this. You can use filters and the search capability, you can manually create your ref, or you can find the refs via other objects, like device groups. The last one I mentioned is actually a very good way to loop through a set of systems to perform a task.

I’ll display the first method in this article and write up the other two later.

The first thing we need to do is get our script reference. Every object you work with in SA generally has a reference object and what I call the ‘virtual object’ ( or whatever VO really stands for ). The VO contains all the important attributes where as the reference only contains the id of the object in SA and the name. There are some methods contained in the VO, but the pytwist API makes common use of a ‘Service’ interface for specific types of objects, like the ServerService, the ServerScriptService, the SearchService, JobService, DeviceGroupService and so on that contain the methods that really make use of the VOs.

This first script enables us to run a known SA script on a known host thru SA from a command line. Note that the command line can be any host that has an agent – this doesn’t need to be run from the core.

I’ll show the script first, then explain it. This script is a version that can be run from an agent that has the Python 2 API Access for Server module installed.

import sys
from optparse import OptionParser
from getpass import getpass
sys.path.append('/opt/opsware/smopylibs2')
sys.path.append('/opt/opsware/agent_tools/') 

import agenttools_common
from pytwist import *
from pytwist.com.opsware.server import ServerRef
from pytwist.com.opsware.script import ServerScriptRef
from pytwist.com.opsware.custattr import NoSuchFieldException

ts = twistserver.TwistServer()

ss = ts.server.ServerService
scs = ts.script.ServerScriptService

# This is the same basic authentication scheme used in most of
# my scripts.  You can tear this out and just use the
# ts.authenticate method with the values already filled in.
parser = OptionParser()
parser.add_option("-u","--user",dest="sauser",
                  help="SA login id")
parser.add_option("-p","--pass",dest="sapasswd",
                  help="SA login password")
parser.add_option("-r","--prompt",action="store_true",dest="readstdin",
                  help="Prompt for passwords instead of using arguments")

(options, args) = parser.parse_args()

if not options.sapasswd and not options.readstdin:
        print "You must specify a password to login to SA."
        sys.exit(1)

if not options.sauser:
        print "You must specify a username to login to SA."
        sys.exit(1)

if options.readstdin:
    options.sapasswd = getpass("Enter Opsware Password: ")

ts.authenticate(options.sauser, options.sapasswd)

# Now the real work starts

# Create a reference to a script we know the ID of
# The ID can be obtained from the SA Gui
scriptRef = ServerScriptRef(22530102)

# Create a reference to a server we know about
serverRef = ServerRef(730102)

# In order to run the script, we'll use the startServerScript method
# from the ServerScriptService.  It requires the script ref, an arugments
# object.  We can optionally give it a tag that it will put under
# the ticket ID.  There are also optional arguments for a job notification
# and job schedule that I won't cover here.
#
# We have the reference already, so we need to create the job argument
# object.  We only need to set the list of targets, although there are
# options for setting the username and password of the script

# Creates the object
scriptArgs = com.opsware.script.ServerScriptJobArgs()

# Requires a list of scriptableRefs, which can be ServerRefs,
# DeviceGroupRefs or HypervisorRefs.  In our case, ServerRefs
scriptArgs.targets = [serverRef]

# Now we can run the job
runScript = scs.startServerScript(scriptRef, scriptArgs, 'Ticket ID #001', None, None)
print "Job started as id %d" % runScript.id

So, besides the basic authentication and argument parsing, there isn’t too much to this script. We have to create the references needed and we do that by importing the references in from pytwist and then using them as constructors like so:

from pytwist.com.opsware.server import ServerRef
from pytwist.com.opsware.script import ServerScriptRef
...
scriptRef = ServerScriptRef(22530102)
serverRef = ServerRef(730102)

In order to run the script, we need to call the startServerScript, which is part of the ServerScriptService interface. But, in order to call it, we need to create one more object, the ServerScriptJobArgs object. We create the object by calling the constructor directly:

scriptArgs = com.opsware.script.ServerScriptJobArgs()

You should note that we could have done the same things with the ServerRef and ScriptRef constructors. Instead of importing them and then calling them, we could have simply done this:

scriptRef = com.opsware.script.ScriptRef(22530102)
serverRef = com.opsware.server.ServerRef(730102)

Generally I’m using alot of server refs, so I like to import to save on typing. Either way, we now have a blank ServerScriptJobArgs object that we need to fill. In our case, we only need to fill out the targets attribute, which is actually a list of any Scriptable Ref. This tells our script which targets it’s going to run on. As mentioned in the comments, a scriptable reference is either a DeviceGroupRef, a ServerRef or a HypervisorRef. Since we have a ServerRef, we just need to contain it in a list and assign it to the targets attribute:

scriptArgs.targets = [serverRef]

Everything is setup now, so I can just call the startServerScript method. The method takes 5 arguments, 3 of which are optional. The 3 optional requires are a ticket id tag, a job notification object and a job schedule object. The ticket id tag is just a string, so it’s easy to create but I’m not going to bother with the other two, so our call looks like this:

runScript = scs.startServerScript(scriptRef, scriptArgs, 'Ticket ID #001'
, None, None)

Remember that the ‘scs’ is just a shortcut to the ServerScriptService. We could have called this method another way:

runScript = scs.startServerScript(scriptRef, scriptArgs, 'Ticket ID #001', None, None)

Again, I prefer the former just to save on typing if I plan on using the service multiple times. The method returns a Run Server Script instance, which we can do other things with, but that’ll have to wait for another article. In the example, I print out the job ID so that I can find and verify the job in the GUI easily.

That’s it for now – in the next article I’ll show how you can use the filter to give the user a way to specify the name of the script they want to run and on which servers.

For our solution, we wrote up the following simple python script to do the job for us.

import sys
import time
import smtplib
from email.MIMEText import MIMEText
from optparse import OptionParser
from getpass import getpass
sys.path.append('/opt/opsware/smopylibs2')
sys.path.append('/opt/opsware/agent_tools/') 

import agenttools_common
from pytwist import *
from pytwist.com.opsware.search import Filter
from pytwist.com.opsware.server import ServerRef

ts = twistserver.TwistServer()
ses = ts.search.SearchService
ss = ts.server.ServerService
DAY = 86400

parser = OptionParser()
parser.add_option("-u","--user",dest="sauser",
                  help="SA login id")
parser.add_option("-p","--pass",dest="sapasswd",
                  help="SA login password")
parser.add_option("-r","--prompt",action="store_true",dest="readstdin",
                  help="Prompt for passwords instead of using arguments")
parser.add_option("-c","--customer",dest="customer",
                  help="Customer name to search for")
parser.add_option("-d","--days",dest="days",
                  help="Expiration limit in days")

(options, args) = parser.parse_args()

if not options.customer:
        print "You must specify a customer to search for."
        sys.exit(1)

if not options.sapasswd and not options.readstdin:
        print "You must specify a password to login to SA."
        sys.exit(1)

if not options.sauser:
        print "You must specify a username to login to SA."
        sys.exit(1)

if options.readstdin:
    options.sapasswd = getpass("Enter Opsware Password: ")

ts.authenticate(options.sauser, options.sapasswd)

filt = Filter()
filt.objectType = 'device'
filt.expression = "(ServerVO.state = UNREACHABLE) & " \
                  "(ServerVO.opswLifecycle != DEACTIVATED)  " \
                  "(device_customer_name = %s)" % options.customer

offline_servers = ses.findObjRefs(filt)
expired_servers = []

if len(offline_servers) == 0:
    print 'There are no unreachable servers'
    sys.exit(0)
else:
    print "Processing %d servers" % len(offline_servers)

now = int(time.time())
for sref in offline_servers:
    shvo = ss.getServerHardwareVO(sref)
    beginDate = shvo.beginDate
    if (now - int(beginDate)) >= (DAY * int(options.days)):
        server_data = [sref.id, sref.name, beginDate]

        # this sorts the list.  if you don't care about
        # sorting, remove this section.

        # sort while create - could have sorted
        # after create, which would probably been
        # faster, but this was an educational exercise

        # if the list is blank, just append
        if len(expired_servers) == 0:
            expired_servers.append(server_data)
            continue

        # place it in the correct spot
        inserted = None
        for i in range(0,len(expired_servers)):
            if beginDate < expired_servers[i][2]:
                expired_servers.insert(i,server_data)
                inserted = 1
                break

        if not inserted:
            expired_servers.append(server_data)
        # end of sorting

        # if you don't sort, then just uncomment this line
        # expired_servers.append(server_data)

if len(offline_servers) == 0:
    print "All unreachable servers have not been offline for more than %s" \
          "days" % options.days
    sys.exit(0)

report = "\r\n\r\nThe following servers have not registered for more than %s " \
         "days.\r\n\r\n" % options.days

for server in expired_servers:
    datestr = "%s/%s/%s %s:%s:%s" % time.gmtime(float(server[2]))[0:6]
    report += "%s : %s has not registered since %s.  \r\n" % \
            (server[0],server[1],datestr)

# If you want output to the screen
print report

# Create a text/plain message
msg = MIMEText(report)

sender = "root@example.com"
rcptto = ["user1@example.com", "user2@example.com"]
msg['Subject'] = "[opsware] Offline Servers for %s" % options.customer
msg['From'] = sender
msg['To'] = ",".join(rcptto)

# Send the message via our own SMTP server, but don't include the
# envelope header.
s = smtplib.SMTP('localhost')
s.sendmail(sender, rcptto, msg.as_string())
s.quit()

A couple of notes are probably in order. First, this was written for our environment, so it may require some massaging to get it to work in your own. The multiple ‘/r/n’ tags are so that it shows up in Outlook correctly and not as a huge, one line mess. By default, the script will sort by date. There are some comments in there if you’d like to remove that functionality.

We wanted to be able to run this from any agent, rather than from the cores, so that our internal groups that use SA could run this if they chose to. It only requires that the agent tools and the Python 2 API for Server Modules packages be installed.

To run, use something like the following:

/opt/opsware/smopython2/python ./offline.py -u username -p password -d 30 -c customername

Make sure to change the bottom of the script to have the email addresses you’d like to use and then just cron the script.

Comments welcome.

( Thanks to Trey Ratcliffe for the use of the photo used on the header of this post. )

Yeah, I thought so.

So I figured I’d write an article on one of the great things I love about Opsware – Custom Attributes! Combined with Dynamic Groups, these puppies provide the ability to create scripts that I don’t have to duplicate for different hosts.

First, however, I should explain what Dynamic Groups are. Dynamic Groups allow the user to group systems based on certain criteria. For instance, you want all the systems in a certain network to be grouped together. Whenever you add a system from that network into Opsware, it automatically becomes a member of that group. Neat by itself but nothing extraordinary.

However, you can assign Custom Attributes ( further known as CAs ) to the Dynamic Group. So for my new dynamic group that I created ( for example, a group that pulls in all hosts in the 192.168.0.0/24 network ), I can assign some CAs to the group and the CAs get assigned to each host within the group.

You may ask yourself, ‘Why is this useful?’. It’s not yet. There’s one more piece that’s missing from the puzzle. The piece that is missing is a software package that Opsware comes with – Agent Tools. When you install the Agent Tools, it comes with a set of python Opsware APIs and small scripts that use the API to make calls back to the master Opsware system and get information – including those CAs!

Armed with these 3 pieces, it becomes easy to create a script that uses the CAs that are dynamically assigned to your host to do all sorts of things. For example, let’s say you want to create a script that checks your /etc/resolv.conf on any system in that 192.168.0.0/24 network. First, we’ll create the dynamic group and assign it the correct device membership.

Next, edit the group and add a CA named something like ‘DNS_SERVERS‘. For the value, put in a DNS server on separate lines and then save your group. Make sure you’ve got the agent tools package installed and we can run a simple test.

[root@frenzy1a.star.dev:~]# /opt/opsware/agent_tools/get_cust_attr.sh DNS_SERVERS
192.168.0.20
192.168.0.21

With that information, we can create a pretty simple shell or python script ( pick your poison ) to make sure that our /etc/resolv.conf has those servers defined. For kicks, here’s an example script that checks to make sure the IPs in the DNS_SERVERS CA are set in /etc/resolv.conf. You could easily modify this so that it actually inserts the values.

#!/opt/opsware/agent/bin/python

import sys
sys.path.append('/opt/opsware/agent_tools/')
import agenttools_common
import re
from string import split
from pytwist.com.opsware.custattr import NoSuchFieldException

def searchFile(file,pattern):
    found = 0
    search = re.compile(pattern)
    try:
        f = open(file, "r")
    except IOError:
        sys.stderr.write("Could not open file %s.n" % (file))
        sys.exit(3)

    for line in f.readlines():
        if search.match(line):
            found = 1
        break

    return found

def main(args):
    ts = agenttools_common.ts
    servers = {}
    result = 0
    hostref = agenttools_common.getServerRef()

    try:
        custattr = ts.server.ServerService.getCustAttr(hostref,
        "DNS_SERVERS", 1)
    except NoSuchFieldException:
        sys.stderr.write("Could not find custom attribute DNS_SERVERS.n")
        sys.exit(3)

    servers = split(custattr)
    for s in servers:
        found = searchFile("/etc/resolv.conf","^nameservers+" + s)
        if not found:
            sys.stderr.write("The server %s was not configured in"
            " /etc/resolv.conf" % (s))
        result = 1

    return result

if __name__ == '__main__':
    sys.exit(main(sys.argv[1:]))

Now if I have another network, say, 192.168.120.0/24, I can do the same thing. Make the group, assign the membership, create the DNS_SERVERS CA and assign the script and presto – it’s done! No duplication of work involved here and I can control the contents of the file from the Opsware console.

One last thing about CAs – they do support overrides. For instance, I can override the CA by creating the same named CA on the host itself. This will override the CA at the group level. One thing you need to be careful, however, is that you don’t assign a host into two groups that define the same CA – there’s no priority between the groups and they don’t combine the contents of the CA to make one CA, so you’ll get random results.

Error 51: Unable to communicate with the VPN subsystem

Thankfully, the solution is rather simple. Open up your favorite terminal client and simply type:

sudo /System/Library/StartupItems/CiscoVPN/CiscoVPN restart

Enter your password and it’ll restart the subsystem. Then just re-open the Cisco VPN client and you’re golden.

ruby: symbol lookup error: /usr/lib64/ruby/gems/1.8/gems/rdiscount-1.3.1.1/lib/rdiscount.so: undefined symbol: RSTRING_LEN

The solution ended up being the following – In the file /usr/lib64/ruby/1.8/x86_64-linux/ruby.h ( note I’m running on 64bit, on 32bit it’d be /usr/lib/ruby/1.8/i386-linux/ ), add these lines somewhere near the top:

#define RSTRING_LEN(s) (RSTRING(s)->len)
#define RSTRING_PTR(s) (RSTRING(s)->ptr)

After that, uninstall and reinstall rdiscount

gem uninstall rdiscount
gem install rdiscount -v 1.3.1.1

And you should be good to go.

Preferably, you have a backup of the file system that you can use. If not, the filesystem you are about to try to to recover a file on must meet these requirements:

• No new files have been created on the filesystem.
• No files have been extended.
• The filesystem is able to be unmounted.
• It is a JFS filesystem, not JFS2
• It is not an overly large file, I’m not exactly sure on the byte limit

If so, then please, drink a few more beers and continue, but before you do…

BACKUP THE CURRENT FILESYSTEM!

Also, note that if you are dealing with a directory that has been deleted and would like to recover both the directory and the files under that directory, you should try Recovering a Deleted Directory ( a document I have yet to post.. ). It follows many of the same steps, but has some very important differences. Do not try and use this procedure to recover deleted directories and the files that were contained within them. You will mess up.

Before we begin, I need to note a few things. I take no responsibility if this screws up your system. Use this at your own risk. Also, the example presented here is an actual representation of me recovering a deleted file, this is not just made up numbers. Also, this only works on jfs filesystems, not jfs2. The jfs2 fsdb is much different and I haven’t had a chance to play with it to determine the proper way of doing this.

Now that I’ve said that, we can begin. We’ll use an example directory with some example files. Our directory is called /test and our filesystem is testlv, otherwise known as /dev/testlv. In our example, our Junior System Admin, Myron, has accidentally deleted a perl script called testfile.pl and needs to recover it.

Note: If you are performing this operation on a filesystem while in maintenance mode, do NOT use option 1 when asked on how to mount the filesystems. ALWAYS use option 2, which specifies to start a shell before mounting the filesystems. Otherwise, the system will force a fsck -y on the filesystem and delete your files.

Step 1.

First, run this command:

ls -id /test

Output:

[test:/]# ls -id /test
    2 /test/

This informs us that the inode for the directory /test is 2. Record this for future use.

Step 2.

Unmount /test

umount /test

Output: None

We must unmount the directory. We don’t want anyone to try and use it while we are attempting to restore the file.

Step 3

Now we’ll start up the filesystem debugger.

fsdb /dev/testlv

Output:

[test:/]# fsdb /dev/testlv

File System:                           /dev/testlv

File System Size:                         193200128  (512 byte blocks)
Disk Map Size:                                 1660  (4K blocks)
Inode Map Size:                                 831  (4K blocks)
Fragment Size:                                 4096  (bytes)
Allocation Group Size:                        16384  (fragments)
Inodes per Allocation Group:                   8192
Total Inodes:                              12075008
Total Fragments:                           24150016

This starts the filesystem debugger on our testlv filesystem.

Step 4

Now we look at our inode number.

2i

Output:

2i
i#:      2  md: d-g-rwxr-xr-x  ln:    4  uid:    3  gid:    3
szh:        0  szl:      512  (actual size:      512)
a0: 0x25d       a1: 0x00        a2: 0x00        a3: 0x00
a4: 0x00        a5: 0x00        a6: 0x00        a7: 0x00
at: Mon Jan 10 11:19:17 2005
mt: Mon Jan 10 11:11:26 2005
ct: Mon Jan 10 11:11:26 2005

The INODE in the command is the inode number we recorded in step #1. This will display the inode information for the directory. The field a0 contains the block number of the directory. The following steps assume only field a0 is used. If a value appears in a1, etc, it may be necessary to repeat steps #5 and #6 for each block until the file to be recovered is found.

Step 5

Move to the block

a0b

Output:

a0b
0x000025d000  :  0x00000000 (0)

This moves to the block pointed to by field “a0″ of this inode.

Step 6

Now we need to print out some data.

p256c

Output:

p256c

0x000025d000:   � � � ? � ? � ? .  � � � � � � ?
0x000025d010:   � ? � ? .  .  � � � � � ? � ? � n
0x000025d020:   l  o  s  t  +  f  o  u  n  d  � � � � � ?
0x000025d030:   � $  � ? m  e  m  _  r  e  p  o  r  t  _  2
0x000025d040:   0  0  4  1  1  0  1  .  d  m  p  .  g  z  � �
0x000025d050:   � � � ? � s � ? o  r  a  s  c  r  a  t
0x000025d060:   c  h  .  c  p  i  o  .  g  z  � � � � � ?
0x000025d070:   � (  � s u  s  e  r  _  a  c  t  i  v  i  t
0x000025d080:   y  _  2  0  0  4  1  1  0  1  .  d  m  p  .  g
0x000025d090:   z  � � � � � � ? � ,  � !  u  s  e  r
0x000025d0a0:   _  a  c  t  i  v  i  t  y  _  d  e  t  _  2  0
0x000025d0b0:   0  4  1  1  0  1  .  d  m  p  .  g  z  � � �
0x000025d0c0:   � ? `  � ? @  � ? E  C  R  1  X  � � �
0x000025d0d0:   � � � ? ? 0  � ? t  e  s  t  f  i  l  e
0x000025d0e0:   .  p  l  � ?    � a t  e  s  t  d  i  r  �
0x000025d0f0:   j  d  u  c  k  o  .  t  x  t  � � � � � ?

The command p256c stands for ‘print 256 bytes in character mode’. You could type ‘p128c’ and it would print 128 bytes in character mode and so on. The beginning left column is the address of the first character in that row. The important thing in this output is to find which line the file to be recovered is on. Our file ( testfile.pl ) is located on line 0x000025d0d0. Next, we have to find the address of the first character of our filename. To do this, starting at 0, count in hexidecimal until you reach the first character of the filename. In our example, the ‘t’ of testfile.pl is at address 0x000025d0d8. Record this address.

If you cannot find your filename here, issue the command again. It will print the next 256 bytes in character mode. Do this until you find your filename.

Here’s a layout to help you in figuring out how we got the address:

Address:        0  1  2  3  4  5  6  7  8  9  A  B  C  D  E  F
0x000025d0d0:   � � � ? ? 0  � ? t  e  s  t  f  i  l  e

Step 7

Reset our position.

a0b

Output:

a0b
0x000025d000  :  0x00000000 (0)

This resets our position back to the beginning of the a0 block. This is necessary whenever you want to reprint out the byte data. Remember, however, that if you had to use the ‘p’ command many times to find your filename, you will probably have to use it many times each time you reset back to the beginning.

Step 8

Print our data in decimal

p256e

Output:

p256e

0x000025d000:         0       2      12       1   11776       0       0       2
0x000025d010:        12       2   11822       0       0      16      20      10
0x000025d020:     27759   29556   11110   28533   28260       0       0      17
0x000025d030:        36      26   28005   27999   29285   28783   29300   24370
0x000025d040:     12336   13361   12592   12590   25709   28718   26490       0
0x000025d050:         0      18      28      18   28530   24947   25458   24948
0x000025d060:     25448   11875   28777   28462   26490       0       0      19
0x000025d070:        40      29   30067   25970   24417   25460   26998   26996
0x000025d080:     31071   12848   12340   12593   12337   11876   28016   11879
0x000025d090:     31232       0       0      20      44      33   30067   25970
0x000025d0a0:     24417   25460   26998   26996   31071   25701   29791   12848
0x000025d0b0:     12340   12593   12337   11876   28016   11879   31232       0
0x000025d0c0:        18   24576     320       5   17731   21041   22528       0
0x000025d0d0:         0      21     304      11   29797   29556   26217   27749
0x000025d0e0:     11888   27648     288       7   29797   29556   25705   29184
0x000025d0f0:     27236   30051   27503   11892   30836       0       0      23
0x000025d100:       260      16   27233   28005   29549   24947   29537   29281
0x000025d110:     11892   30836       0       0       0       0       0       0
0x000025d120:         0       0       0       0       0       0       0       0
0x000025d130:         0       0       0       0       0       0       0       0
0x000025d140:         0       0       0       0       0       0       0       0

The command ‘p256e’ stands for ‘print 256 bytes in decimal word format’. This output can be helpful and confusing at the same time. First, find the beginning address that our file name is on. In our example, this was 0x000025d0d0. The line in decimal format reads:

0x000025d0d0:         0      21     304      11   29797   29556   26217   27749

For each file, assume the following:

  {ADDRESS}:  x    x    x    x    x    x    x    x    x
              |    |    |    |    |---- filename -----|
    inode # --+----+    |    |
                        |    +-- filename length
        record LENGTH --+

Note that the inode # may begin on any part of the line. The reason we print the data in decimal format is to help us determine where in the line the inode number is. There are several ways to help you do this, here are some:

• Count the number of characters in your filename, then try and find that number in our address line. ( eg: There are 11 characters in the filename ‘testfile.pl’. ) You can see on our line there is a matching number 11.
• Recount to the address 0x000025d0d8, assuming each column represents two numbers. The first column is 0 and 1. The second column is 2 and 3, then 4 and 5, etc. When you reach the column that matches your address, go back one column. The number in this column should match up with your filename length. Unless, of course, your filename is over 255 characters.

Once you are sure you have the the correct column for your filename length, you are going to count back three more columns. This should put at the first column of the inode number. We’ll use our example decimal line to explain this more:

0x000025d0d0:         0      21     304      11   29797   29556   26217   27749

Like we mentioned before, testfile.pl is 11 characters. We find a matching number 11 in the 4th column. That means that the column with ’304′ is our record length field and the 0 and 21 columns make up our inode. Now, that we know which columns our inode is in ( columns 1 and 2 ), we must translate this number into our real inode number.

Step 9

Calculate our inode.

Thanks to Arthur Dent for this update

In some cases, our inode number may be a lower number and may not need any special treatment, however, for larger inodes, the information will span columns.

Here’s an example. The directory ECR1X is on an address above ours. Its inode number, like ours, is in columns 1 and 2. However, if you compare the decimal lines, you can immediately see the difference.

EC1X Decimal:
0x000025d0c0:      18  24576
testfile.pl Decimal:
0x000025d0d0:       0     21

To calculate the inode if it spans 2 columns, use the following formula:

firstcolumn * 65536 + secondcolumn = inode

In our case, we are not using column 1, so all we need is column 2 from the previous step. It was 21, so now we know the inode number of the missing file. We’re close to recovery!

Step 10

We go to our new inode number

21i

Output:

21i
i#:     21  md: f---rw-r--r--  ln:    0  uid:    0  gid:    3
szh:        0  szl:       45  (actual size:       45)
a0: 0xeff       a1: 0x00        a2: 0x00        a3: 0x00
a4: 0x00        a5: 0x00        a6: 0x00        a7: 0x00
at: Mon Jan 10 14:16:40 2005
mt: Mon Jan 10 14:16:48 2005
ct: Mon Jan 10 14:16:53 2005

From this output, you can see that we have a file.

Step 11

21i.ln=1

Output:

21i.ln=1
0x0000020a88  :  0x00000001 (1)

This sets the link count of the file back to 1. You can verify this by reissuing the command from step #10 and noticing that the ‘ln’ field has incremented.

21i
i#:     21  md: f---rw-r--r--  ln:    1  uid:    0  gid:    3
szh:        0  szl:       45  (actual size:       45)
a0: 0xeff       a1: 0x00        a2: 0x00        a3: 0x00
a4: 0x00        a5: 0x00        a6: 0x00        a7: 0x00
at: Mon Jan 10 14:16:40 2005
mt: Mon Jan 10 14:16:48 2005
ct: Mon Jan 10 14:16:53 2005

We have now told the filesystem that the link count for inode 21 should be 1. This means that there should be a filename pointing at this inode. This basically reverses what the OS actually does when deleting files. It doesn’t actually erase the file data, instead, it unlinks the filename from its inode number, effectively preventing you from seeing the data.

Step 12

Quit.

q

Output:

q
[test:/]#

This quits out of the fsdb.

Step 13

Fsck our volume

fsck /dev/testlv

Output:

[test:/]# fsck /dev/testlv

** Checking /dev/rtestlv (/test)
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
Unreferenced file  I=21  owner=root mode=100644
size=45 mtime=Jan 10 14:16 2005 ; RECONNECT? y
** Phase 5 - Check Inode Map
Bad Inode Map; SALVAGE? y
** Phase 5b - Salvage Inode Map
** Phase 6 - Check Block Map
Bad Block Map; SALVAGE? y
** Phase 6b - Salvage Block Map
18 files 21893872 blocks 171306256 free
***** Filesystem was modified *****

This does a filesystem check on /dev/testlv. As you can see, it finds an inode claiming it is linked to, but no file that links to it. We answer ‘y’ to tell it to reconnect the inode to a filename, effectively giving us our file back!

Step 14

Remount our directory.

mount /test

Output: None

We must remount our filesystem to get back at our file.

Step 15

Go into lost and found. It’s where all lost little kiddies go. Duh.

cd /test/lost+found

Output: None

Our file is now located in lost+found. If you do an ‘ls’ in this directory, you will see something like the following:

[test:/test/lost+found]# ls -l
total 8
-rw-r--r--   1 root     sys              45 Jan 10 14:16 21

And if we cat the file 21, we get the following:

[test:/test/lost+found]# cat 21
#!/usr/bin/perl

print "this is a testn";

Ta-da! It’s Myron’s missing perl script!

As a final aside, I will say that there may be different and much better ways of recovering files on AIX, however, this is the way I constructed from notes I found on various mailing lists and a few days of fooling around with it. So if you see some mistakes in this document or have some suggestions for better ways of doing this, please, let me know! I will happily update this document with better information as it is provided.

I hope this helps some of you who have to deal with certain people who accidentally delete files on your systems. Nothing beats a good backup but when you don’t have one of those, this can always be used as a fallback.

So I did. Here’s how.

First, get all the information about the adapters.

for i in ent0 ent1 ent2
do
odmget -q name="$i" CuDv >> /tmp/$i
odmget -q name="$i" CuAt >> /tmp/$i
odmget -q name="$i" CuVPD >> /tmp/$i
done 

Next, down the interfaces and detach them.

for i in en0 en1 en2 et0 et1 et2
do
ifconfig $i down
ifconfig $i detach
done 

Now, remove all the references to the devices from the ODM

for i in ent0 ent1 ent2 en0 en1 en2 et0 et1 et2
do
odmdelete -q name="$i" -o CuAt
odmdelete -q name="$i" -o CuDv
odmdelete -q name="$i" -o CuVPD
odmdelete -q value3="$i" -o CuDvDr
done 

We can verify that no adapters and no interfaces exist now by issuing the lsdev commands again. All we should see is the loopback interface.

lsdev -Cc adapter -l ent*
lsdev -Cc if

lo0 Available       Loopback Network Interface

Edit the files we created the first step and replace every instance of the adapter name with the new adapter name. For instance, I would edit /tmp/ent0 and replace all instances of “ent0″ with “ent2″. We can do this with a sed script.

sed -e "s/ent0/ent1/g" /tmp/ent0 > /tmp/ent1.new
sed -e "s/ent1/ent2/g" /tmp/ent1 > /tmp/ent2.new
sed -e "s/ent2/ent0/g" /tmp/ent2 > /tmp/ent0.new

Then add the files back to the ODM.

odmadd /tmp/ent0.new
odmadd /tmp/ent1.new
odmadd /tmp/ent2.new

At this point, our adapters will now be redefined. Issue another lsdev command to check:

lsdev -Cc adapter -l ent*

ent0 Available 05-08 10/100/1000 Base-TX PCI-X Adapter (14106902)
ent1 Available 07-08 2-Port 10/100/1000 Base-TX PCI-X Adapter (14108902)
ent2 Available 07-09 2-Port 10/100/1000 Base-TX PCI-X Adapter (14108902)

You can see now that ent0 is now the external PCI-X adapter and ent1 and ent2 are the two onboard adapters. But, we still have no interfaces for the adapters. You can verify this by issuing the usual lsdev command again. You should only see the loopback interface.

lsdev -Cc if

lo0 Available       Loopback Network Interface

To fix this ( and to make sure our changes stick upon a reboot… ), run a cfgmgr, then check for our interfaces.

cfgmgr
lsdev -Cc if

en0 Defined   05-08 Standard Ethernet Network Interface
en1 Defined   07-08 Standard Ethernet Network Interface
en2 Defined   07-09 Standard Ethernet Network Interface
et0 Defined   05-08 IEEE 802.3 Ethernet Network Interface
et1 Defined   07-08 IEEE 802.3 Ethernet Network Interface
et2 Defined   07-09 IEEE 802.3 Ethernet Network Interface
lo0 Available       Loopback Network Interface

As you can see, we have successfully gotten our interfaces back. We’re almost done! All you need to do now is reboot the system.

shutdown -Fr

Once the reboot has completed, issue one last check to verify that the adapters have changed:

entstat -d ent0 | grep "Device Type"

Device Type: 10/100/1000 Base-TX PCI-X Adapter (14106902)

Looks good!