In any case, I thought I’d go ahead and share what I told the person who asked.

Philosophy

The first thing behind a configuration management needs to be the thought of how it is supposed to interact with the admin. Is the admin supposed to be using this tool everyday? Once a week? Do they login just to check their compliance or do they login to perform tasks, modifications and installs with the tool? The answer for me is obvious – the tool needs to take over how I do my job. Let’s face it, half the day of a normal admin is fixing stuff that other people have mucked with, adding users, installing software, building systems – all that jazz.

In order to take over my job, the tool needs to be able to accomplish the things I can do on a terminal in a simple manner. If I need to install a package, the tool needs to know how to do that. If I want to add a user, it needs to know how to do that. Add a route or cronjob, edit host files or configurations, the list goes on and on. However, you can break it down, for 90% of the use cases, to two simple things – interacting with files and issuing commands. Windows is a little more difficult, because of its registry and .COM objects and other hidden magic, but in the world of unix derivatives, it largely holds true.

The second part of the philosophy is that no matter what I do in the tool, the tool knows how to make sure my changes stay. Just like a source version control system, changes that are made outside of the tool are invalid. If I edit a piece of code and forget to submit it into the SVC system, the next time I sync out the code, my changes are gone. The same should hold true for any configuration management system – if I add a user, the tool needs to make sure that user stays. If I remove a user, it should make sure that user stays deleted. Any change I would make through the tool would follow suit. The perfect tool does this automatically; by telling the tool to make a change, it automatically starts checking for and verifying that the change is in place and stays in place. I shouldn’t have to tell the tool to always check for my changes or write extra scripts and schedule them to verify my changes still exist. It’d be nice if the tool allowed me to detail how it should verify the change still exists but it should have default methods as well.

By enforcing the concept that changes done outside of the tool are invalid and will be removed, you force the admin into using the tool to make his changes, which I stated before, should be the goal of every configuration management system.

Abstraction

The next important part a configuration management tool needs to bring to the table is the idea of abstracting the change from the implementation of the change. The idea of the change, or the resource if you will, needs to be free from how it is implemented on the operating system. Face it, every OS we deal with does something in a different way – but why should I need to care about that? My task is to get the change on the system – not deal with the different flags and commands each system wants to force me to use. If I want to add a user, I just want to say ‘Add this user to this system’. I don’t want to have to tell the tool ‘Well for Linux, use the useradd command. For AIX, use the mkuser command. For Windows, do it through this COM object’. The same goes for things like packages, scheduled tasks, routes, network mounts, directories, files and many more.

Sure, there will be cases where a resource is OS specific ( again, Windows registry ), but for the most part, the change ( the resource I want configured on the system ) needs to be presented to the user of the tool as a singular entity and method. To do that, it needs to be abstracted from how the resource is actually configured on the system. An additional plus would be an API or plugin architecture that allows the admin to develop his own implementations of resources if he needs to.

Node Hierarchy

The next goal a configuration management tool needs to strive towards is some type of modularization or node hierarchy. Every system ever built can be broken into smaller pieces. The main advantage of this concept is that by breaking a system into smaller pieces, those pieces become units that you can reuse to build other hosts.

With the ability to reuse configured units ( or resources ) and by combining them into bigger units that can also be reused, you create a wonderful opportunity for a sys admin to customize and endless combinations to configure hosts with. My configuration management tool becomes a big box of legos that I can quickly use to build my systems with. And hey, every admin loves to play with legos.

Automatic Assignment and Grouping

The last tool I used accomplished this task well and it’s something that I think every configuration management tool would benefit greatly from. The idea behind this is the creation of groups that have a set of criteria defined. These criteria are used to match against nodes and if a node matches it is automatically assigned as a member to the group.

While this isn’t anything special in and of itself, when you combine the ability to assign resources and static information to the group, you instantly create a useful way of quickly getting the right resources onto the right systems. Imagine a group that pulls in all your web servers and makes sure they have the right httpd package or apache user. It’s undeniably useful.

Templating

One of the main goals of a configuration management system is just that – manage configurations. In order to do that, it needs to be able to modify the contents of files in a smart manner. Being able to create a template of a configuration file gets rid of the horror of duplication.

Imagine a scenario where you have 20 different networks to administer. That’s possibly 20 different /etc/resolv.confs to administer. I don’t want to keep 20 separate and distinct versions of the file laying around that get deployed ( do you? ) – instead, a template is a great idea. What’s even better is the idea of being able to fill in the variables of the template with information that’s stored in the management server. Imagine if I defined a group for each network ( a dynamic one that’s auto-populated! ) and attached some variables to the group, such as my DNS servers, NTP servers, file server hostnames and other such goodness. I could then have the agent deploy the template and generate the real config files based off the templates and the variables assigned to the system and/or groups the system belongs to. I’m simply providing the recipe that bakes in all my configs.

Always More!

I could go on and on and describe each of these things in detail but to sum up the rest of what I would consider necessary in a great configuration management tool, here’s a list of mostly self-explanatory items.

• A software repository for storing packages/files to deploy
• Good reporting mechanisms to detect drift and keep your manager happy
• The ability to quickly deploy/run/schedule scripts and obtain output for certain tasks across your systems
• Support for multiple platforms ( read: Windows & Unix )
• An API and plugin architecture

Additionally, here are some things that I think of ‘Nice to Have’ in a tool but not completely necessary. Why? Because they’re getting away from the main goal – management of configurations.

• Snapshots – good for comparison of systems
• Patching
• Build Infrastructure ( running your PXE/DHCP/BOOTP/GHOST systems out of the tool )
• Non-java based agents

Perfect? Not quite

Like I said, there’s always more that the perfect configuration management tool could use and more specific details for actual implementation of some of the goals and philosophies I’ve described that I feel are the most important.

I do think many of the tools out there are well on their way to accomplishing these goals – I just don’t think any are quite there yet.

Yeah, I thought so.

So I figured I’d write an article on one of the great things I love about Opsware – Custom Attributes! Combined with Dynamic Groups, these puppies provide the ability to create scripts that I don’t have to duplicate for different hosts.  

First, however, I should explain what Dynamic Groups are. Dynamic Groups allow the user to group systems based on certain criteria. For instance, you want all the systems in a certain network to be grouped together. Whenever you add a system from that network into Opsware, it automatically becomes a member of that group. Neat by itself but nothing extraordinary.

However, you can assign Custom Attributes ( further known as CAs ) to the Dynamic Group. So for my new dynamic group that I created ( for example, a group that pulls in all hosts in the 192.168.0.0/24 network ), I can assign some CAs to the group and the CAs get assigned to each host within the group.

You may ask yourself, ‘Why is this useful?’. It’s not yet. There’s one more piece that’s missing from the puzzle. The piece that is missing is a software package that Opsware comes with – Agent Tools. When you install the Agent Tools, it comes with a set of python Opsware APIs and small scripts that use the API to make calls back to the master Opsware system and get information – including those CAs!

Armed with these 3 pieces, it becomes easy to create a script that uses the CAs that are dynamically assigned to your host to do all sorts of things. For example, let’s say you want to create a script that checks your /etc/resolv.conf on any system in that 192.168.0.0/24 network. First, we’ll create the dynamic group and assign it the correct device membership.

Next, edit the group and add a CA named something like ‘DNS_SERVERS’. For the value, put in a DNS server on separate lines and then save your group. Make sure you’ve got the agent tools package installed and we can run a simple test.

[root@frenzy1a.star.dev:~]# /opt/opsware/agent_tools/get_cust_attr.sh DNS_SERVERS
192.168.0.20
192.168.0.21

With that information, we can create a pretty simple shell or python script ( pick your poison ) to make sure that our /etc/resolv.conf has those servers defined. For kicks, here’s an example script that checks to make sure the IPs in the DNS_SERVER CA are set in /etc/resolv.conf. You could easily modify this so that it actually inserts the values.

#!/opt/opsware/agent/bin/python
import sys
import re
from string import split

sys.path.append('/opt/opsware/agent_tools/')
import agenttools_common
from pytwist.com.opsware.custattr import NoSuchFieldException

def searchFile(file,pattern):
	found = 0
	search = re.compile(pattern)
	try:
		f = open(file, "r")
	except IOError:
		sys.stderr.write("Could not open file %s.\n" % (file))
		sys.exit(3)

	for line in f.readlines():
		if search.match(line):
			found = 1
			break
	return found

def main(args):
	ts = agenttools_common.ts
	servers = {}
	result = 0
	hostref = agenttools_common.getServerRef()

	try:
		custattr = ts.server.ServerService.getCustAttr(hostref, "DNS_SERVERS", 1)
	except NoSuchFieldException:
		sys.stderr.write("Could not find custom attribute DNS_SERVERS.\n")
		sys.exit(3)

	servers = split(custattr)
	for s in servers:
		found = searchFile("/etc/resolv.conf","^nameserver\s+" + s)
		if not found:
			sys.stderr.write("The server %s was not configured in /etc/resolv.conf\n" % (s))
			result = 1

	return result

if __name__ == '__main__':
	sys.exit(main(sys.argv[1:]))

Now if I have another network, say, 192.168.120.0/24, I can do the same thing. Make the group, assign the membership, create the DNS_SERVERS CA and assign the script and presto – it’s done! No duplication of work involved here and I can control the contents of the file from the Opsware console.

One last thing about CAs – they do support overrides. For instance, I can override the CA by creating the same named CA on the host itself. This will override the CA at the group level. One thing you need to be careful, however, is that you don’t assign a host into two groups that define the same CA – there’s no priority between the groups and they don’t combine the contents of the CA to make one CA, so you’ll get random results.

I had forgotten about those dislikes and while I’ve got a much cooler head on now, there are a few things I really would like to see Opsware implement.

#1. The option to remediate an audit by applying/remediating a software policy

Let’s say I want to install a piece of software called ‘p4′. I package p4 up into a zip file and create a script that creates the proper p4.sh file. All is wonderful. I can apply the software package and remediate it and presto, a system has p4 installed.

However, I want to make sure that p4 file stays in compliance, so I create an audit to check on that. I create a little script that verifies p4.sh is set correctly and the p4 binary exists. I have to use an audit because the software compliance only uses the system’s default package management software ( RPM for RedHat, for example ) to verify that the package is installed. I’ve got my audit working and again, all is well.

Well, let’s say I run my audit and find that on a system I installed p4 on, someone changed the p4.sh. I want to fix that! The problem is, the audit has no knowledge of the software policy that can do just that. Instead, I either need to know ( ie – have knowledge that the policy exists ) or I have to duplicate the scripts that are in the software policy and put them in the audit, creating a management headache if the script were ever to change.

So, please implement the ability to remediate an audit by attaching and remediating a software policy.

#2. The option to attach an audit from a software policy

Along the lines of #1, once I install that piece of software, I want to make sure that it stays compliant. To do this, I’d like to attach an audit. I can’t depend on the software policy to determine compliance because, again, it only uses the package manager on the system. However, in order to do this, I have to manually attach the audit.

#3. Provide the ability to customize how the software policy determines compliance.

Some of these complaints would go away if the software policy module had a more robust way of determining compliance. In essence, the module simply does a package check to see if the package exists. If someone removed the binary or changed something else underneath the package manager, the software policy module has no way of determining that. It’d be nice if you could specify certain scripts that would enable the software policy to determine it’s compliance and not just depend on such a blanket statement.

Those 3 things have to be, at the moment, my biggest complaints. I still don’t like the CML language and I think they should have gone with just a normal templating system, but that’s a future discussion.

For instance, OS installs. Opsware has some OS installation/sequence stuff that’s neat but nothing that we are seriously looking at using ( for many reasons, such as source control of the ks files and more complicated scripting ). Instead, we are trying to implement in such a manner that we can do the following:

* Install minimal OS
* Install Opsware agent
* Remediate policies

Once the opsware agent is installed, the system will automatically be put into some dynamic device groups. These dynamic groups have membership based off certain criteria; they also have software policies attached to them. That way, when a system is placed into a group or meets a certain criteria and automatically becomes a member of a group, it automatically is assigned to certain software policies that need to be remediated. This is really cool and can be extremely helpful for organizing your hosts.

The problem here is that the software policies have subpar methods for checking whether the policy is compliant or not. You can add quite a few items to software policies ( scripts, other policies, packages, app configs ) – but it basically only checks whether a package is installed ( and it doesn’t even do that in a satisfactory manner ). Therefore, you have to take any work that you’ve done via your software policy and then duplicate that work into an Audit that can check and verify that the work stays in place ( or is in fact, needed at all ).

This brings you to the next issue – that once an audit figures out that something is wrong, it can’t fix it by applying the software policy, so once again, you have to duplicate the software policy script into the remediation section of the audit – that or document the audit somehow to inform the operator to fix the issues by remediating a software policy onto the host.

In the Puppet world, this is an example of *one* thing. You have a resource. If you assign that resource to a host, it checks to see if that resource exists. If it doesn’t, it adds it. If someone changes the resource on the host ( not thru Puppet ), then Puppet changes it back. I don’t need to define my resource 3 times – once to install it, once to verify it and once to fix any changes; it’s all taken care of by the one resource.

I’ve heard through the rumor-mill that the new 7.8 release of Opsware SA is supposed to have a much better A&R module for auditing and remediating, so I’m eager to see what changes it brings along. Hopefully, it and I will agree with each other a little better on how things should be done in the world of configuration management.

Till then, I’ll keep on truckin’ on.

Error 51: Unable to communicate with the VPN subsystem

Thankfully, the solution is rather simple. Open up your favorite terminal client and simply type:

sudo /System/Library/StartupItems/CiscoVPN/CiscoVPN restart

Enter your password and it’ll restart the subsystem. Then just re-open the Cisco VPN client and you’re golden.

I’ll be fixing some of the formatting issues hopefully tonight and then will post my Gitorious on RHEL5 setup tutorial and the pains I went through to get there.

ruby: symbol lookup error: /usr/lib64/ruby/gems/1.8/gems/rdiscount-1.3.1.1/lib/rdiscount.so: undefined symbol: RSTRING_LEN

The solution ended up being the following – In the file /usr/lib64/ruby/1.8/x86_64-linux/ruby.h ( note I’m running on 64bit, on 32bit it’d be /usr/lib/ruby/1.8/i386-linux/ ), add these lines somewhere near the top:

#define RSTRING_LEN(s) (RSTRING(s)->len)
#define RSTRING_PTR(s) (RSTRING(s)->ptr)

After that, uninstall and reinstall rdiscount

gem uninstall rdiscount
gem install rdiscount -v 1.3.1.1

And you should be good to go.

I’ve also got to say – the further I dig into the tool, the more I enjoy how much you can customize with it. True – it’s not always the easiest thing to do and the interface leaves *alot* to be desired ( the Custom Script section in A&R is on top of my list right now.. ) but they do provide tools for you to do alot of what you can’t do with the interface.

For instance, there is a python API that you can install on your hosts that lets you pull down certain data – things like custom attributes. This is a big win because it means you can write a single script for checking something out and have it pull in some of those custom attributes. Maybe they’ll deploy a newer version of python with it next time – 1.5.2 is pretty old. ( hint, hint )

Don’t get me wrong – things aren’t all peachy yet – the custom scripts section still needs alot of work ( fixed width font, tab widths, ability to link or import scripts from the library, passing arguments to the remediation script, etc ) but as I go deeper into the tool, I also see that things aren’t quite as bad as I thought. I’m still not sold on some of the philosophy behind how they do things like managing certain resources as whole resources with methods, rather than as individual bits of config files that you have to manage separately but with some of the stuff I’ve been getting into, things seem to be getting better and will make the tool a bit easier to use.

Hopefully I’ll have more to say in the near future but I don’t want to ruin anything.

In that vein, to help, I’ve been researching online project management tools. One of the more well-known products is Basecamp by 37 Signals. It’s a subscription based service so before I signed up, I figured I’d do a little researching on them and .. wow. About 3 years ago there was a rather large altercation where one of the 37 signal members decided to fleece a few customers by posting their support emails. Although he pleaded innocent, simply because he didn’t use some specific words, the intent was plainly obvious. ( Most of the blogs are linked from here )

In any case, after about a 1 hour reading session of scouring blogs and other comments and reading the responses of the 37 Signal employees and CEO in their defense of their asinine move, I decided to move on and not bother with the product. I don’t even really care if their product could offer us what we need – their arrogance leaves a bad taste.

Yeah, I might have been just a small puny customer and they’ve got millions and an endorsement by the NY Times – It just goes to show that even after several years, bad press is still bad press and can lose customers.

I finally dove into the application configurations which try and use CML ( an Opsware templating language ) to model how a file should look. It’s not the best language by any means but we’ll skip over that small detail for now. The area of pain tonight is the use of configuration templates between audits and actually assigning the application configuration to a host.

So, for example, let’s say I want to model the /etc/yum.conf file – I create a template in CML and lets just say ( after 5 hours.. ) it works and can read a perfectly good /etc/yum.conf file. Now, I’ve got two options what to do with this template. I can create an Application Configuration and attach the yum.conf template to that, then attach the application config to a host. Once I do that, I can input values for the template and ‘push’ the template ( filled in with values ) down to the host. This would be the ideal thing to do if the Application compliance actually told you which application configuration was out of compliance but no luck there! In other words, if someone ever mucked with my yum.conf file, the host would show as not being compliant but it wouldn’t tell me that it was yum that wasn’t compliant.

So, my second option is to use an Audit. Audit’s can make use of the configuration template without the need to attach it to a host or create an application configuration object. It works with the files a bit differently however. To use the config template, I attach it to my audit and then have it pull in that file from the host and parse it. Now, unless that config is already perfect, my config template will fail. I usually have to end up going to the host I’m using as a source and modify the /etc/yum.conf till the template will read the file in.

Once that’s accomplished, we can move to the next step. All the items in /etc/yum.conf that the template understands show up as links. I can click one of these items and set a value. For example ( because that was a vague explanation ), the config value ‘tolerant=1′ shows up in the audit. I can click on the ‘1′ and say ‘Make sure in /etc/yum.conf that the value for tolerant is equal to 1′. Once I’ve sorted out all the config items and set them to their appropriate values, I can save the audit.

Next, I ran my audit against the host I had been using for the audit and all went fine. I tried to test changing a value on the host; I went over to the host and changed ‘tolerant=0′ in /etc/yum.conf. I reran the audit and sure enough, it spotted the error and asked me to remediate. So far, so good. My next test, I wanted to see what would happen if someone put something into the config that the template didn’t know about ( let’s say someone accidentally catted some characters into the file ). So I went to the end of the file and wrote ‘blah=123′ and saved it.

Another run of the audit and again, it tells me that the /etc/yum.conf has failed. This time, however, it’s at a loss to exactly say what’s wrong. Instead, when I diff the two files in Opsware it gives me a completely blank file for what the file currently is ( which is wrong ) and then it gives me what the file should look like ( which looks perfectly fine ). So I figured I’d try and remediate it. However, when I try and do that, Opsware complains that it can’t remediate the file because it doesn’t know about ‘blah=123′. In other words, it can’t fix the file, even though it gave me a good example of what it should be, because there is an item that it doesn’t know about that is causing the issue.

Huh?

I mean, it knows that the file is bad. It knows what the file should look like – it even showed me! So why in the dang-farnit-name-of-all-that-is-holy why doesn’t it just recreate the file with what it showed me? Why does it even care about ‘blah=123′? It’s not in the template, it’s invalid data, smite it! Kill it! Just rewrite my damn file.

The more ironic thing is that if I do the same tests with an Application Configuration, it works fine! I can go add ‘blah=123′ to the file, it tells me that ’something’ is out of compliance, so I push the yum.conf back down and it recreates the file on the host perfectly. Bad data gone. So again, another example where two modules of this product just do NOT work together. The audit didn’t even recognize that I have a application configuration already attached to the host. Why doesn’t it just detect that and say ‘Oh, the file isn’t like what you said it should be and you’ve got the application configuration already assigned, do you want me to repush the config down?’

At the minimum, the audit should just recreate the file how it’s specified in the audit and how it’s displayed to me in the diff – but it won’t and so now I’m stuck again.