As an adult, that is. I’m not posing this question for high school students looking to get into the university, but rather as a question to myself. I suppose with Marielle’s recent foray back to school to get her MBA, I look back at my college career and the phrase ‘meh’ comes to mind. I see my cousin Pat graduating from Art Center as valedictorian and I find myself jealous. I read and see the great things around me that people are creating, whether it be with their hands or their minds and it’s something I want to be apart of and yet, because of my educational background ( I have an associate degree in architecture.. :/ ) and my brain forgetting all the math that I was ever taught, I find it to be an uphill battle. In the snow. 5 feet deep. Backwards. Being chased by dinosaurs.

I wish that when I was younger that I had the motivation I have now, the yearning to learn and understand the most complicated of math problems and logic puzzles. ( Well, I was always in love with logic puzzles ). But I can’t sit here and wish. So I did take the first step. The first step in what may be the longest journey of my life. I’m going back to school. It’s a small step, maybe all I can afford right now to take, with everything else going on in my life ( house, work, etc ), but it’s a step in the right direction. I’m starting slow and just getting some math refreshers under my belt, get myself back up to speed.

I’ve also started watching the MIT CS department’s online courses at MIT’s OpenCourseWare. As I watch them, I ponder the question that started this topic – how does an adult, an old(er) person like me get into something like MIT? Do they even take out-of-practice students? I’m not naive enough to think I’d get in right now, but I’m curious as what it would take to be able to transfer to the school, undergraduate, graduate, whatever. If you know, please, enlighten me or is it really just too late for me to ever consider something like that?

In any case, MIT possible or not, I will walk the road I’ve set myself upon. The end goal? At least a bachelor’s degree in Computer Science. The goal, if I push myself? Well.. Dr. Hajducko has a nice ring to it.

Which brings me to another subject – how do I concentrate on getting some of these goals down? People who know me know that my personality is best described as ’scattered’. I know a little about alot and it’s something I’ve always regretted. When I have kids, I will be teaching them the exact opposite – know alot about a little. In other words, it’s great to be good at alot of things, but it’s very special to be the best at one thing.

The truth in that statement is easily supported by looking at our society. Are famous people those who are good at lots of little things or are the people that we know and respect in our communities those who excelled at one specific thing? People are famous because they are the best at a small subset of things that they do. They are the people we look up to and strive to imitate.

The sad part is, my lack of being great at any one specific thing comes not from a lack of drive or a doldrum interest in any specific category, but rather from being too interested in too many things that I don’t have enough time to dedicate to any one thing. Also, the occasional WoW gaming spree of a few months to a year effectively kills any drive to do anything else besides get more purpz and up the deepz.

Now that I’ve kicked the WoW habit again, I find myself in the sore situation of having to decide on those goals I do wish to focus on. The more I learn every day however, the more I find other subjects that interest me. One fault of mine would to be to have a lack of decision making when it comes to concentrating and driving towards some of these goals. How do I decide which ones I wish to focus on? Which ones are the most important to me?

Some of the issues I have with deciding on the goals to pursue have to deal with pressure from outside influences, such as my personal and professional life. They are rather opposing forces, as obviously a night of coding does not a ‘date night’ make.

Maybe I should go see a therapist. :/ I guess I should add that one to the goals.

The default RHEL system-auth for PAM includes the following for the ‘auth’ stack.

auth required pam_env.so
auth sufficient pam_unix.so nullok try_first_pass
auth requisite pam_succeed_if.so uid >= 500 quiet
auth required pam_deny.so

Let’s try and step through this.

auth required pam_env.so

This module basically loads up the /etc/security/pam_env.conf file and sets a bunch of environment variables. Nothing fancy.

Our second line:

auth sufficient pam_unix.so nullok try_first_pass

This is where we test their password. It’s given to the pam_unix.so module to test. Null passwords are ok, and the try_first_pass tries the password from the previous module. If pam_unix.so returns success, we stop here and state that the user is authenticated. But what happens if they are not authenticated? Well, it’s not enough to fail them, so we go to the next line.

auth requisite pam_succeed_if.so uid >= 500 quiet

This line tests whether the user is a system account or not ( system accounts usually have a UID below 500 ). If this fails ( it IS a system account ) – then fail immediately. However, let’s say it’s a normal system account, which means it succeeds. Which means it goes to the next line and it’s here where I’m becoming confused.

Our last line is simply:

auth required pam_deny.so

This basically returns a failure, straight-out. It doesn’t take any options, it doesn’t log anything, it just denies you. That’s all fine and dandy – but what the hell is the point of the line above it then? Why are we even bothering to test if they’re a system account or not, if we’re just going to deny them anyways?

The basic flow for a user account typing in the wrong password:

• Load PAM environment
• Test password
• Password failed, is UID above 500?
• Success! UID is above 500
• Hit pam_deny, required. Deny user

Compared to the wrong password for a system account:

• Load PAM environment
• Test password
• Password fail, is UID above 500?
• Failure! UID is below 500. Fail immediately

So again, what’s the point? Both pam_deny and pam_succeed_if return PAM_AUTH_ERR if they fail – so it’s not like we’re trying to give a different response. Also, the quiet on pam_succeed_if makes it so we don’t log for failure or successes for our requisite line. ( I could understand if this was quiet_success, to record a system user login attempt ).

I’m sure I’m missing something here but I don’t understand the logic behind why the ‘requisite’ line exists in this stack and I’m hoping someone can shed some light on it.

Took some quick pics – I thought they were good until I realized my ISO was set extremely high – you can see alot of noise if you look close. By the time I realized it though, the sun was already going down – I’ll just have to wait till a weekend and snap some better photos.

In any case, I thought I’d go ahead and share what I told the person who asked.

Philosophy

The first thing behind a configuration management needs to be the thought of how it is supposed to interact with the admin. Is the admin supposed to be using this tool everyday? Once a week? Do they login just to check their compliance or do they login to perform tasks, modifications and installs with the tool? The answer for me is obvious – the tool needs to take over how I do my job. Let’s face it, half the day of a normal admin is fixing stuff that other people have mucked with, adding users, installing software, building systems – all that jazz.

In order to take over my job, the tool needs to be able to accomplish the things I can do on a terminal in a simple manner. If I need to install a package, the tool needs to know how to do that. If I want to add a user, it needs to know how to do that. Add a route or cronjob, edit host files or configurations, the list goes on and on. However, you can break it down, for 90% of the use cases, to two simple things – interacting with files and issuing commands. Windows is a little more difficult, because of its registry and .COM objects and other hidden magic, but in the world of unix derivatives, it largely holds true.

The second part of the philosophy is that no matter what I do in the tool, the tool knows how to make sure my changes stay. Just like a source version control system, changes that are made outside of the tool are invalid. If I edit a piece of code and forget to submit it into the SVC system, the next time I sync out the code, my changes are gone. The same should hold true for any configuration management system – if I add a user, the tool needs to make sure that user stays. If I remove a user, it should make sure that user stays deleted. Any change I would make through the tool would follow suit. The perfect tool does this automatically; by telling the tool to make a change, it automatically starts checking for and verifying that the change is in place and stays in place. I shouldn’t have to tell the tool to always check for my changes or write extra scripts and schedule them to verify my changes still exist. It’d be nice if the tool allowed me to detail how it should verify the change still exists but it should have default methods as well.

By enforcing the concept that changes done outside of the tool are invalid and will be removed, you force the admin into using the tool to make his changes, which I stated before, should be the goal of every configuration management system.

Abstraction

The next important part a configuration management tool needs to bring to the table is the idea of abstracting the change from the implementation of the change. The idea of the change, or the resource if you will, needs to be free from how it is implemented on the operating system. Face it, every OS we deal with does something in a different way – but why should I need to care about that? My task is to get the change on the system – not deal with the different flags and commands each system wants to force me to use. If I want to add a user, I just want to say ‘Add this user to this system’. I don’t want to have to tell the tool ‘Well for Linux, use the useradd command. For AIX, use the mkuser command. For Windows, do it through this COM object’. The same goes for things like packages, scheduled tasks, routes, network mounts, directories, files and many more.

Sure, there will be cases where a resource is OS specific ( again, Windows registry ), but for the most part, the change ( the resource I want configured on the system ) needs to be presented to the user of the tool as a singular entity and method. To do that, it needs to be abstracted from how the resource is actually configured on the system. An additional plus would be an API or plugin architecture that allows the admin to develop his own implementations of resources if he needs to.

Node Hierarchy

The next goal a configuration management tool needs to strive towards is some type of modularization or node hierarchy. Every system ever built can be broken into smaller pieces. The main advantage of this concept is that by breaking a system into smaller pieces, those pieces become units that you can reuse to build other hosts.

With the ability to reuse configured units ( or resources ) and by combining them into bigger units that can also be reused, you create a wonderful opportunity for a sys admin to customize and endless combinations to configure hosts with. My configuration management tool becomes a big box of legos that I can quickly use to build my systems with. And hey, every admin loves to play with legos.

Automatic Assignment and Grouping

The last tool I used accomplished this task well and it’s something that I think every configuration management tool would benefit greatly from. The idea behind this is the creation of groups that have a set of criteria defined. These criteria are used to match against nodes and if a node matches it is automatically assigned as a member to the group.

While this isn’t anything special in and of itself, when you combine the ability to assign resources and static information to the group, you instantly create a useful way of quickly getting the right resources onto the right systems. Imagine a group that pulls in all your web servers and makes sure they have the right httpd package or apache user. It’s undeniably useful.

Templating

One of the main goals of a configuration management system is just that – manage configurations. In order to do that, it needs to be able to modify the contents of files in a smart manner. Being able to create a template of a configuration file gets rid of the horror of duplication.

Imagine a scenario where you have 20 different networks to administer. That’s possibly 20 different /etc/resolv.confs to administer. I don’t want to keep 20 separate and distinct versions of the file laying around that get deployed ( do you? ) – instead, a template is a great idea. What’s even better is the idea of being able to fill in the variables of the template with information that’s stored in the management server. Imagine if I defined a group for each network ( a dynamic one that’s auto-populated! ) and attached some variables to the group, such as my DNS servers, NTP servers, file server hostnames and other such goodness. I could then have the agent deploy the template and generate the real config files based off the templates and the variables assigned to the system and/or groups the system belongs to. I’m simply providing the recipe that bakes in all my configs.

Always More!

I could go on and on and describe each of these things in detail but to sum up the rest of what I would consider necessary in a great configuration management tool, here’s a list of mostly self-explanatory items.

• A software repository for storing packages/files to deploy
• Good reporting mechanisms to detect drift and keep your manager happy
• The ability to quickly deploy/run/schedule scripts and obtain output for certain tasks across your systems
• Support for multiple platforms ( read: Windows & Unix )
• An API and plugin architecture

Additionally, here are some things that I think of ‘Nice to Have’ in a tool but not completely necessary. Why? Because they’re getting away from the main goal – management of configurations.

• Snapshots – good for comparison of systems
• Patching
• Build Infrastructure ( running your PXE/DHCP/BOOTP/GHOST systems out of the tool )
• Non-java based agents

Perfect? Not quite

Like I said, there’s always more that the perfect configuration management tool could use and more specific details for actual implementation of some of the goals and philosophies I’ve described that I feel are the most important.

I do think many of the tools out there are well on their way to accomplishing these goals – I just don’t think any are quite there yet.

I’ve spent most of today getting one of those elements in and it’s working now – I had to rip some images from his current site but I’ll replace them in the next day or so – while reverse engineering code is one thing, I don’t condone shameless image ripping. So, Matt, if you’re reading this – I only used your images for testing because I’m truly, truly awful at divs and floats.

Anyways, enjoy the new theme and I apologize if you visit in some of those rare moments where I’ve entirely borked the CSS and everything looks entirely out of whack.

( I’ve replaced the image I was borrowing and I’ll probably change the overlay image as well, the ’shiny’ look doesn’t mesh with the theme as well as I thought it would )

Yeah, I thought so.

So I figured I’d write an article on one of the great things I love about Opsware – Custom Attributes! Combined with Dynamic Groups, these puppies provide the ability to create scripts that I don’t have to duplicate for different hosts.  

First, however, I should explain what Dynamic Groups are. Dynamic Groups allow the user to group systems based on certain criteria. For instance, you want all the systems in a certain network to be grouped together. Whenever you add a system from that network into Opsware, it automatically becomes a member of that group. Neat by itself but nothing extraordinary.

However, you can assign Custom Attributes ( further known as CAs ) to the Dynamic Group. So for my new dynamic group that I created ( for example, a group that pulls in all hosts in the 192.168.0.0/24 network ), I can assign some CAs to the group and the CAs get assigned to each host within the group.

You may ask yourself, ‘Why is this useful?’. It’s not yet. There’s one more piece that’s missing from the puzzle. The piece that is missing is a software package that Opsware comes with – Agent Tools. When you install the Agent Tools, it comes with a set of python Opsware APIs and small scripts that use the API to make calls back to the master Opsware system and get information – including those CAs!

Armed with these 3 pieces, it becomes easy to create a script that uses the CAs that are dynamically assigned to your host to do all sorts of things. For example, let’s say you want to create a script that checks your /etc/resolv.conf on any system in that 192.168.0.0/24 network. First, we’ll create the dynamic group and assign it the correct device membership.

Next, edit the group and add a CA named something like ‘DNS_SERVERS’. For the value, put in a DNS server on separate lines and then save your group. Make sure you’ve got the agent tools package installed and we can run a simple test.

[root@frenzy1a.star.dev:~]# /opt/opsware/agent_tools/get_cust_attr.sh DNS_SERVERS
192.168.0.20
192.168.0.21

With that information, we can create a pretty simple shell or python script ( pick your poison ) to make sure that our /etc/resolv.conf has those servers defined. For kicks, here’s an example script that checks to make sure the IPs in the DNS_SERVER CA are set in /etc/resolv.conf. You could easily modify this so that it actually inserts the values.

#!/opt/opsware/agent/bin/python
import sys
import re
from string import split

sys.path.append('/opt/opsware/agent_tools/')
import agenttools_common
from pytwist.com.opsware.custattr import NoSuchFieldException

def searchFile(file,pattern):
	found = 0
	search = re.compile(pattern)
	try:
		f = open(file, "r")
	except IOError:
		sys.stderr.write("Could not open file %s.\n" % (file))
		sys.exit(3)

	for line in f.readlines():
		if search.match(line):
			found = 1
			break
	return found

def main(args):
	ts = agenttools_common.ts
	servers = {}
	result = 0
	hostref = agenttools_common.getServerRef()

	try:
		custattr = ts.server.ServerService.getCustAttr(hostref, "DNS_SERVERS", 1)
	except NoSuchFieldException:
		sys.stderr.write("Could not find custom attribute DNS_SERVERS.\n")
		sys.exit(3)

	servers = split(custattr)
	for s in servers:
		found = searchFile("/etc/resolv.conf","^nameserver\s+" + s)
		if not found:
			sys.stderr.write("The server %s was not configured in /etc/resolv.conf\n" % (s))
			result = 1

	return result

if __name__ == '__main__':
	sys.exit(main(sys.argv[1:]))

Now if I have another network, say, 192.168.120.0/24, I can do the same thing. Make the group, assign the membership, create the DNS_SERVERS CA and assign the script and presto – it’s done! No duplication of work involved here and I can control the contents of the file from the Opsware console.

One last thing about CAs – they do support overrides. For instance, I can override the CA by creating the same named CA on the host itself. This will override the CA at the group level. One thing you need to be careful, however, is that you don’t assign a host into two groups that define the same CA – there’s no priority between the groups and they don’t combine the contents of the CA to make one CA, so you’ll get random results.

I had forgotten about those dislikes and while I’ve got a much cooler head on now, there are a few things I really would like to see Opsware implement.

#1. The option to remediate an audit by applying/remediating a software policy

Let’s say I want to install a piece of software called ‘p4′. I package p4 up into a zip file and create a script that creates the proper p4.sh file. All is wonderful. I can apply the software package and remediate it and presto, a system has p4 installed.

However, I want to make sure that p4 file stays in compliance, so I create an audit to check on that. I create a little script that verifies p4.sh is set correctly and the p4 binary exists. I have to use an audit because the software compliance only uses the system’s default package management software ( RPM for RedHat, for example ) to verify that the package is installed. I’ve got my audit working and again, all is well.

Well, let’s say I run my audit and find that on a system I installed p4 on, someone changed the p4.sh. I want to fix that! The problem is, the audit has no knowledge of the software policy that can do just that. Instead, I either need to know ( ie – have knowledge that the policy exists ) or I have to duplicate the scripts that are in the software policy and put them in the audit, creating a management headache if the script were ever to change.

So, please implement the ability to remediate an audit by attaching and remediating a software policy.

#2. The option to attach an audit from a software policy

Along the lines of #1, once I install that piece of software, I want to make sure that it stays compliant. To do this, I’d like to attach an audit. I can’t depend on the software policy to determine compliance because, again, it only uses the package manager on the system. However, in order to do this, I have to manually attach the audit.

#3. Provide the ability to customize how the software policy determines compliance.

Some of these complaints would go away if the software policy module had a more robust way of determining compliance. In essence, the module simply does a package check to see if the package exists. If someone removed the binary or changed something else underneath the package manager, the software policy module has no way of determining that. It’d be nice if you could specify certain scripts that would enable the software policy to determine it’s compliance and not just depend on such a blanket statement.

Those 3 things have to be, at the moment, my biggest complaints. I still don’t like the CML language and I think they should have gone with just a normal templating system, but that’s a future discussion.

For instance, OS installs. Opsware has some OS installation/sequence stuff that’s neat but nothing that we are seriously looking at using ( for many reasons, such as source control of the ks files and more complicated scripting ). Instead, we are trying to implement in such a manner that we can do the following:

* Install minimal OS
* Install Opsware agent
* Remediate policies

Once the opsware agent is installed, the system will automatically be put into some dynamic device groups. These dynamic groups have membership based off certain criteria; they also have software policies attached to them. That way, when a system is placed into a group or meets a certain criteria and automatically becomes a member of a group, it automatically is assigned to certain software policies that need to be remediated. This is really cool and can be extremely helpful for organizing your hosts.

The problem here is that the software policies have subpar methods for checking whether the policy is compliant or not. You can add quite a few items to software policies ( scripts, other policies, packages, app configs ) – but it basically only checks whether a package is installed ( and it doesn’t even do that in a satisfactory manner ). Therefore, you have to take any work that you’ve done via your software policy and then duplicate that work into an Audit that can check and verify that the work stays in place ( or is in fact, needed at all ).

This brings you to the next issue – that once an audit figures out that something is wrong, it can’t fix it by applying the software policy, so once again, you have to duplicate the software policy script into the remediation section of the audit – that or document the audit somehow to inform the operator to fix the issues by remediating a software policy onto the host.

In the Puppet world, this is an example of *one* thing. You have a resource. If you assign that resource to a host, it checks to see if that resource exists. If it doesn’t, it adds it. If someone changes the resource on the host ( not thru Puppet ), then Puppet changes it back. I don’t need to define my resource 3 times – once to install it, once to verify it and once to fix any changes; it’s all taken care of by the one resource.

I’ve heard through the rumor-mill that the new 7.8 release of Opsware SA is supposed to have a much better A&R module for auditing and remediating, so I’m eager to see what changes it brings along. Hopefully, it and I will agree with each other a little better on how things should be done in the world of configuration management.

Till then, I’ll keep on truckin’ on.