0.3 Part Four/Conclusion.

So after the apparent issue with Hong Kong, I decided to take some time and develop an installation document, detailing the steps required to setup a Func infrastructure as part of my 0.3 delivery.

This file can be found HERE

Conclusion

During these past few weeks , I have learned that even what would be considered a minor change in infrastructure can have such a large impact on installations of software in a Linux environment. Different versions of python and other software can lead to unexpected instabilities or errors that all need to be considered.  Unfortunately due to these issues and the technical issues with Hong Kong, I did not get a chance to thoroughly test my newly packaged func, with the python modifications However; A significant amount of progress has otherwise been made. I have learned a great deal about func and certmaster and their operation in regards to PKI , certificate exchange and the establishment of Secure Sockets and XML RPC over HTTPS.

O.4? What? There is a 0.4?? Onward!!

Over the semester I have become a big fan of func, and Linux Sysadmin tools, I would like and have already discussed my continuing work on this project, and helping CDOT develop a stronger centralized management strategy. My roadmap is the completion and succesful installation and deployment of func, and I would like to follow that with building on my previous colleagues work in the areas of Icinga and Puppet.

For my 0.4 my main goal is the testing of the packages I built for 0.3. After this stage is accomplished and I have reviewed and discussed it with Chris Tyler we can figure out how to manage to get it out there to all of the minions. I’m hoping to have the 0.4 up within the next two or three weeks, as testing and troubleshooting should be a fairly straightforward process.

I wanted to send a Thanks to Paul Whalen and Chris Tyler for the many extra hours the put in fixing up Hong Kong, and the great advice during the semester.

Stay Tuned Folks!!

0.3 Part Three

Continuing on our path of understating, the way certmaster and func work, is using a PKI or Public Key Infrastructure. Public Key encryption revolves around the generation of key pairs. Each participating party generates their own Private and Public key using a one way algorithm. Each party can distribute their public key to various computers in our case func clients send their keys to the Overlord machine.  In our case, Certmaster also signs the public keys sent by func. This generates trust between machines. In func’s case, it lies on top of the Certmaster infastructure utilizing the key-signing process done by this application.

Establishing of a ‘Secure Socket’ (which is the trust and connection establishment that comes with SSL encryption) Revolves around the utilization of Public keys to encrypt data; and only their respective Public Key can decrypt data.

In both Func and Certmasters case, they use an XML based RPC (remote procedure calls) over HTTPS (HTTPS being our SSL communication) these calls are created in Python using the xmlrpclib module. Our error arises out of the newer version of func and certmasters handling of these calls using newer versions of Python , on our Honk Kong machine, we now run Python 2.7 which is the version of Python that ships by default with Fedora 15 , a change from what we were previously working with.

Creation and handling of the SSL connections are handled during the program installations on both certmaster and func, these files are respectively

/etc/func/overlord/sslclient.py  in func

and

/etc/certmaster/SSLConnection.py in certmaster.

The changes however are different;

After scouring the internet  I came across and open bug on func, here is the link to it:

Func Bug

The Following lines were removed and added to each file

/etc/func/overlord/sslclient.py

/etc/certmaster/SSLConnection.py

Since I didn’t want to have to make these modifications every time I installed a new instance, So i decided to take a Source RPM, make the modification and repackage it. This would give me the opportunity to use the same RPM for every installation.

I built 4 instances of the same package,

FUNC- FC13 , NOARCH on ARM Koji

CERTMASTER- FC13 , NOARCH on ARM Koji

FUNC- FC15 , NOARCH on the main Fedora Koji

CERTMASTER- FC15 , NOARCH on the main Fedora Koji

Here is a link to all of the RPMs I have built for my 0.3 and the system , feel free to use them anyway you can (NOTE as of 0.3 they are UNTESTED due to Hong Kong’s technical failure)

http://matrix.senecac.on.ca/~tefurzer/packages/

Unfortunately, I had successfully completed my initial install of the func and certmaster, but when I went to install the final updated packages the rpm database had become corrupted (we think after a yum update). After having a talk with Paul Whalen we thought it best to leave it and wait for Chris Tyler to have a look at it , as something as important as the rpm is not something in the realm of my given authority to handle. The database went down at about 2pm on Thursday, and with Friday being a holiday it remains unavailable. As such, the progress on my 0.3 stopped before the second installation and my packages remain untested.

0.3 Part Two

Continuing the Installation Process

Now that we have our two packages build and verified to work correctly on the ARM system we can continue with the Install process. Initially I had thought I would just have to install these new packages on the ARM machines , However; Hong-Kong has had some recent hardware problems and was upgraded to a Fedora 15 Beta Version , which meant the infrastructure I had previously set up was no longer available I thought (which was later proven very wrong) This was a good thing, as Certmaster was fairly easy to set up and Fedora 15 ships with the most current 0.27 version.

Following the steps I have outlined in the previous releases I installed func, and certmaster with YUM on Honk-Kong, after which I started the certmaster service.

Since I had completed two RPMs I needed a way to get them on to the ARM machine, This seemed an easy enough solution and I just SCPed them to my Matrix Account, Here are download links to the two completed RPMs

Func

Certmaster

Everything was going smoothy so far, was quite pleased.  However; When I woke up the next day I heard that Hong Kong, and connectivity to all of the ARM machines we’re not functional yet, and I could no longer connect to my 0-4 arm machine,  NFS was broken, as well as DHCP settings on Honk Kong, Paul was swamped with work and Chris Tyler was backed up with marking.  I managed to find one what was working (cdot-beaglexm-0-3) and since func leaves such a small footprint and does very little to the system I wasn’t too worried about using it.

I downloaded the two packages with the

wget http://matrix.senecac.on.ca/~tefurzer/certmaster-0.27-2.fc13.noarch.rpm

and

wget http://matrix.senecac.on.ca/~tefurzer/func-0.27-2.fc13.noarch.rpm

commands respectively.

I installed certmaster and then installed func using the rpm -ivh {package} command, and was given a dependency error on smolt, which was quickly resolved with a yum install.  After installing func, I went to start the service with the service funcd start.. and naturally the service started and died. Checking the func.log again, I noticed the same issue as my previous installation, so It appears that the newer version had not solved our problem for us. That was quite disappointing, and as of yet I am not extremely proficient with python and nowhere near talented enough to attempt to rewrite the erroneous code.

After adding the following lines:

minionname=cdot-beaglexm-0-3

listenaddr=192.168.1.103

to the /etc/func/minion.config file , the func daemon started succesfully. So at this point, I had two working services running. I went ahead and signed the certificate on the Overlord (Hong Kong) Server and attempted to run a simple func command. I was presented with this error

I had no idea what this error was, what it meant or why it was happening. This was a solid 3-4 hours trying to figure out what was wrong with it, Stay tuned tomorrow for the post on how I fixed it

0.3 Part One

Introduction

So, proceeding from the conclusion of our 0.2 we need to look at fixing the issues we had with the previous installation of func-0.25-1. Toward the end of 0.2 I did mention the availability of a newer version of func, func-0.27-1. After talking with Chris Tyler, we decided it might be a good idea to attempt to install the newer version of func.

Our first step in this process was determining the availability of the package for ARM fedora, and whether this package was readily available. A quick search on the ARM koji site, and we can see that only 0.25 was built for the ARM. This means our next step is packaging and running our 0.27 build through the Koji Arm. The problem is, we currently don’t have the package source to build into our new RPM. We also needed to remove and clean the environment we previously had created to prepare for the newer verision

Building Our New 0.27 Packages

So, what I did was go and grab the Package from the Fedora Main koji, I couldn’t find a func or certmaster version 0.27 built for fedora13 yet, so I grabbed one of the newer fedora 15 builds. After grabbing the newer distro build I simply edited the spec file, and increment the version by 1.

It’s worth noting that I initially tried to just build func and install it, this failed as they changed the build requires to specify that the newest version of certmaster must be present, which means I had to go and build both certmaster and func.

The spec files will be as follows.

• Certmaster Spec
• Func Spec

After we complete editing our spec files, we need to make the RPM’s for the two programs this is accomplished by executing the following commands

rpmbuild -ba func.spec

rpmbuild -ba certmaster.spec

commands respectively.

Upon Successful build ( I did not encounter any problems building this this RPM) you should be greeted with an exit status of 0. As per the following two screenshots, these are what your finished build should look like

Func:

Certmaster:

Next, we need to build these packages on the ARM koji to make sure this will deploy correctly. This is done by executing the following commands to send these two packages to the Koji Hub

arm-koji build dist-f13 –scratch certmaster-0.27-2.fc14.src.rpm

arm-koji build dist-f13 –scratch func -0.27-2.fc14.src.rpm

Here are links to my two task ID’s specifying the successful completion of the build on ARM.

Certmaster

Func

YUM Repository

In this lab, we were responsible for the creation of our own YUM repository.  This consists of 3 major stages. In the first stage, we need to generate a GPG key in order for us to sign our packages. What is the purpose of this? Signing packages allows users to verify the source of a package, its important to note that signing a package does not ensure its integrity, just verifies the source of the package.

First we generate the key

gpg --gen-key

Then we need to put the link to they key in our user macro. This ensures that when we do the signing command it will use this key. This line will be located in the ~/.rpmmacros file

The two packages I chose to put in my repository were nled and gnucash. On each package, I execute the command

rpm --addsign {package rpm}


I then created a directory in /var/www/html called repo, put my packages in and executed the create repo command

Then set my machine to use this repository

After that, I threw together an RPM for my repository data

Here is the RPM link

TIMREPO RPM link.

0.2 Part Four – Testing/Conclusion

After I had completed the installation and had a functional client/server func environment in CDOT, I wanted to test func and see if there were any module issues with the change to the arm machines ( Im not totally sure considering its such a drastic change in architecture.

I did several quick func module tests and I found out that for the most part all of the modules that are solely related to the operating system platform such as PS, RPM and those various modules are successful. However; many of the modules that relate to hardware calls and information seem to fail to operate and upon looking at them it appears to be client-side based.

Conclusion and Toward 0.3

While this was successful for 0.2 purposes , installing a version of func on the arm machines , it is not 100% entirely functional and without flaws. Funnily enough after doing some further research I noticed that the version of func installed by yum is 0-25, and the current new release (which has been tested on the arm koji hub) is 0-27. Toward 0.3 and beyond (I would like to continue with this project after graduating) I would like to install the 0-27 version and see if there has been any changes to the python modules that fix the issues I’m having. If not, I would love to become proficient enough in python to be able to contribute and rememdy these issues.

 

 

Stay Tuned.

0.2 Part Three – Installing Func in CDOT

Now that everything outside of actually installing the software has been looked at we can continue on to the actual installation of func. As per my previous post, we can use those steps to install func. In our situation Hong Kong already has the certmaster service installed, as well as func.  So there is no need to set that up for us. Func is present in the repository on the arm machine, so it is as simple as executing a YUM install func on the arm system.

The func package installs correctly on the system, then we start the service and we get the “OK” message from the service command.  However; I notice no certificate request is being sent to the certmaster on Hong Kong.  I wonder why no certificate request is being sent, so I run ps -A | grep ‘funcd’ on the arm machine and see no funcd service running. I wonder why the service would be dying so I attempt to restart and get the following

Peculiar why the service would start correctly but fail upon restart. Conveniently enough func maintains its own set of logs at /var/log/func/func.log. Upon inspecting the logs, I see this.

My understanding and familiarity with python is somewhat limited, but it sounds to me like func was/is somewhere having issues binding itself to the IP or Hostname of the arm machine (present in the  variable ip is referenced before assignment statement in the logs). Which a lesson can be learnt from, while something may BUILD on Koji correctly, the full functionality may not be available. So, my first remedy was an attempt to assign these variables to nothing, and hope that farther down in the python script they are assigned correctly, something suggested while reading the func mailing list. So after making that slight modification as followed

I am presented with this further error.

However; I further read the script and noticed that it does a quick check to see if a minion name and listen address are specified in the minion config file.  To accomplish this, we make the quick changes to the config file, the change is specifying the listen addr = and the minion_name= .

and restart the funcd service. Now we check the funcd log files to double check and make sure all of the modules are loaded correctly and the funcd service has successfully started

Now the funcd service is started and running.

Since we are trying to run this as securely as possible, we have turned off the auto sign option for the certmaster on Hong Kong.  This means we have to manually  sign requested certificated to allow func to work properly. In this situation this is best practice because Seneca is a reasonably easily accessible network, we want to limited the possibility of rogue machines getting their keys signed by our certmaster.

To list pending key requests we use the following

Next we can sign keys using the following command. certmaster-ca –sign {hostname}, where hostname is in the list of certificates waiting to be signed. and the final step in this process is to check the keys that we have signed on the certmaster.