This file can be found HERE

Conclusion

During these past few weeks , I have learned that even what would be considered a minor change in infrastructure can have such a large impact on installations of software in a Linux environment. Different versions of python and other software can lead to unexpected instabilities or errors that all need to be considered.  Unfortunately due to these issues and the technical issues with Hong Kong, I did not get a chance to thoroughly test my newly packaged func, with the python modifications However; A significant amount of progress has otherwise been made. I have learned a great deal about func and certmaster and their operation in regards to PKI , certificate exchange and the establishment of Secure Sockets and XML RPC over HTTPS.

O.4? What? There is a 0.4?? Onward!!

Over the semester I have become a big fan of func, and Linux Sysadmin tools, I would like and have already discussed my continuing work on this project, and helping CDOT develop a stronger centralized management strategy. My roadmap is the completion and succesful installation and deployment of func, and I would like to follow that with building on my previous colleagues work in the areas of Icinga and Puppet.

For my 0.4 my main goal is the testing of the packages I built for 0.3. After this stage is accomplished and I have reviewed and discussed it with Chris Tyler we can figure out how to manage to get it out there to all of the minions. I’m hoping to have the 0.4 up within the next two or three weeks, as testing and troubleshooting should be a fairly straightforward process.

I wanted to send a Thanks to Paul Whalen and Chris Tyler for the many extra hours the put in fixing up Hong Kong, and the great advice during the semester.

Stay Tuned Folks!!

Establishing of a ‘Secure Socket’ (which is the trust and connection establishment that comes with SSL encryption) Revolves around the utilization of Public keys to encrypt data; and only their respective Public Key can decrypt data.

In both Func and Certmasters case, they use an XML based RPC (remote procedure calls) over HTTPS (HTTPS being our SSL communication) these calls are created in Python using the xmlrpclib module. Our error arises out of the newer version of func and certmasters handling of these calls using newer versions of Python , on our Honk Kong machine, we now run Python 2.7 which is the version of Python that ships by default with Fedora 15 , a change from what we were previously working with.

Creation and handling of the SSL connections are handled during the program installations on both certmaster and func, these files are respectively

/etc/func/overlord/sslclient.py  in func

and

/etc/certmaster/SSLConnection.py in certmaster.

The changes however are different;

After scouring the internet  I came across and open bug on func, here is the link to it:

Func Bug

The Following lines were removed and added to each file

/etc/func/overlord/sslclient.py

/etc/certmaster/SSLConnection.py

Since I didn’t want to have to make these modifications every time I installed a new instance, So i decided to take a Source RPM, make the modification and repackage it. This would give me the opportunity to use the same RPM for every installation.

I built 4 instances of the same package,

FUNC- FC13 , NOARCH on ARM Koji

CERTMASTER- FC13 , NOARCH on ARM Koji

FUNC- FC15 , NOARCH on the main Fedora Koji

CERTMASTER- FC15 , NOARCH on the main Fedora Koji

Here is a link to all of the RPMs I have built for my 0.3 and the system , feel free to use them anyway you can (NOTE as of 0.3 they are UNTESTED due to Hong Kong’s technical failure)

http://matrix.senecac.on.ca/~tefurzer/packages/

Unfortunately, I had successfully completed my initial install of the func and certmaster, but when I went to install the final updated packages the rpm database had become corrupted (we think after a yum update). After having a talk with Paul Whalen we thought it best to leave it and wait for Chris Tyler to have a look at it , as something as important as the rpm is not something in the realm of my given authority to handle. The database went down at about 2pm on Thursday, and with Friday being a holiday it remains unavailable. As such, the progress on my 0.3 stopped before the second installation and my packages remain untested.

Now that we have our two packages build and verified to work correctly on the ARM system we can continue with the Install process. Initially I had thought I would just have to install these new packages on the ARM machines , However; Hong-Kong has had some recent hardware problems and was upgraded to a Fedora 15 Beta Version , which meant the infrastructure I had previously set up was no longer available I thought (which was later proven very wrong) This was a good thing, as Certmaster was fairly easy to set up and Fedora 15 ships with the most current 0.27 version.

Following the steps I have outlined in the previous releases I installed func, and certmaster with YUM on Honk-Kong, after which I started the certmaster service.

Since I had completed two RPMs I needed a way to get them on to the ARM machine, This seemed an easy enough solution and I just SCPed them to my Matrix Account, Here are download links to the two completed RPMs

Func

Certmaster

Everything was going smoothy so far, was quite pleased.  However; When I woke up the next day I heard that Hong Kong, and connectivity to all of the ARM machines we’re not functional yet, and I could no longer connect to my 0-4 arm machine,  NFS was broken, as well as DHCP settings on Honk Kong, Paul was swamped with work and Chris Tyler was backed up with marking.  I managed to find one what was working (cdot-beaglexm-0-3) and since func leaves such a small footprint and does very little to the system I wasn’t too worried about using it.

I downloaded the two packages with the

wget http://matrix.senecac.on.ca/~tefurzer/certmaster-0.27-2.fc13.noarch.rpm

and

wget http://matrix.senecac.on.ca/~tefurzer/func-0.27-2.fc13.noarch.rpm

commands respectively.

I installed certmaster and then installed func using the rpm -ivh {package} command, and was given a dependency error on smolt, which was quickly resolved with a yum install.  After installing func, I went to start the service with the service funcd start.. and naturally the service started and died. Checking the func.log again, I noticed the same issue as my previous installation, so It appears that the newer version had not solved our problem for us. That was quite disappointing, and as of yet I am not extremely proficient with python and nowhere near talented enough to attempt to rewrite the erroneous code.

After adding the following lines:

minionname=cdot-beaglexm-0-3

listenaddr=192.168.1.103

to the /etc/func/minion.config file , the func daemon started succesfully. So at this point, I had two working services running. I went ahead and signed the certificate on the Overlord (Hong Kong) Server and attempted to run a simple func command. I was presented with this error

I had no idea what this error was, what it meant or why it was happening. This was a solid 3-4 hours trying to figure out what was wrong with it, Stay tuned tomorrow for the post on how I fixed it

So, proceeding from the conclusion of our 0.2 we need to look at fixing the issues we had with the previous installation of func-0.25-1. Toward the end of 0.2 I did mention the availability of a newer version of func, func-0.27-1. After talking with Chris Tyler, we decided it might be a good idea to attempt to install the newer version of func.

Our first step in this process was determining the availability of the package for ARM fedora, and whether this package was readily available. A quick search on the ARM koji site, and we can see that only 0.25 was built for the ARM. This means our next step is packaging and running our 0.27 build through the Koji Arm. The problem is, we currently don’t have the package source to build into our new RPM. We also needed to remove and clean the environment we previously had created to prepare for the newer verision

Building Our New 0.27 Packages

So, what I did was go and grab the Package from the Fedora Main koji, I couldn’t find a func or certmaster version 0.27 built for fedora13 yet, so I grabbed one of the newer fedora 15 builds. After grabbing the newer distro build I simply edited the spec file, and increment the version by 1.

It’s worth noting that I initially tried to just build func and install it, this failed as they changed the build requires to specify that the newest version of certmaster must be present, which means I had to go and build both certmaster and func.

The spec files will be as follows.

• Certmaster Spec
• Func Spec

After we complete editing our spec files, we need to make the RPM’s for the two programs this is accomplished by executing the following commands

rpmbuild -ba func.spec

rpmbuild -ba certmaster.spec

commands respectively.

Upon Successful build ( I did not encounter any problems building this this RPM) you should be greeted with an exit status of 0. As per the following two screenshots, these are what your finished build should look like

Func:

Certmaster:

Next, we need to build these packages on the ARM koji to make sure this will deploy correctly. This is done by executing the following commands to send these two packages to the Koji Hub

arm-koji build dist-f13 –scratch certmaster-0.27-2.fc14.src.rpm

arm-koji build dist-f13 –scratch func -0.27-2.fc14.src.rpm

Here are links to my two task ID’s specifying the successful completion of the build on ARM.

Certmaster

Func

First we generate the key

gpg --gen-key

Then we need to put the link to they key in our user macro. This ensures that when we do the signing command it will use this key. This line will be located in the ~/.rpmmacros file

The two packages I chose to put in my repository were nled and gnucash. On each package, I execute the command

rpm --addsign {package rpm}


I then created a directory in /var/www/html called repo, put my packages in and executed the create repo command

Then set my machine to use this repository

After that, I threw together an RPM for my repository data

Here is the RPM link

TIMREPO RPM link.

I did several quick func module tests and I found out that for the most part all of the modules that are solely related to the operating system platform such as PS, RPM and those various modules are successful. However; many of the modules that relate to hardware calls and information seem to fail to operate and upon looking at them it appears to be client-side based.

Conclusion and Toward 0.3

While this was successful for 0.2 purposes , installing a version of func on the arm machines , it is not 100% entirely functional and without flaws. Funnily enough after doing some further research I noticed that the version of func installed by yum is 0-25, and the current new release (which has been tested on the arm koji hub) is 0-27. Toward 0.3 and beyond (I would like to continue with this project after graduating) I would like to install the 0-27 version and see if there has been any changes to the python modules that fix the issues I’m having. If not, I would love to become proficient enough in python to be able to contribute and rememdy these issues.

Stay Tuned.

The func package installs correctly on the system, then we start the service and we get the “OK” message from the service command.  However; I notice no certificate request is being sent to the certmaster on Hong Kong.  I wonder why no certificate request is being sent, so I run ps -A | grep ‘funcd’ on the arm machine and see no funcd service running. I wonder why the service would be dying so I attempt to restart and get the following

Peculiar why the service would start correctly but fail upon restart. Conveniently enough func maintains its own set of logs at /var/log/func/func.log. Upon inspecting the logs, I see this.

My understanding and familiarity with python is somewhat limited, but it sounds to me like func was/is somewhere having issues binding itself to the IP or Hostname of the arm machine (present in the  variable ip is referenced before assignment statement in the logs). Which a lesson can be learnt from, while something may BUILD on Koji correctly, the full functionality may not be available. So, my first remedy was an attempt to assign these variables to nothing, and hope that farther down in the python script they are assigned correctly, something suggested while reading the func mailing list. So after making that slight modification as followed

I am presented with this further error.

However; I further read the script and noticed that it does a quick check to see if a minion name and listen address are specified in the minion config file.  To accomplish this, we make the quick changes to the config file, the change is specifying the listen addr = and the minion_name= .

and restart the funcd service. Now we check the funcd log files to double check and make sure all of the modules are loaded correctly and the funcd service has successfully started

Now the funcd service is started and running.

Since we are trying to run this as securely as possible, we have turned off the auto sign option for the certmaster on Hong Kong.  This means we have to manually  sign requested certificated to allow func to work properly. In this situation this is best practice because Seneca is a reasonably easily accessible network, we want to limited the possibility of rogue machines getting their keys signed by our certmaster.

To list pending key requests we use the following

Next we can sign keys using the following command. certmaster-ca –sign {hostname}, where hostname is in the list of certificates waiting to be signed. and the final step in this process is to check the keys that we have signed on the certmaster.

• Since  we hope to eventually be able to use puppet to deploy both func and Icinga , It only makes sense to  have administration of these services all on the same machine, which is currently the Hong Kong machine. The only concern I have with running all these services from Hong Kong is that it’s already fairly convoluted with running the Koji build system and many other  systems within CDOT, appropriate distribution of core services is something that needs to be appropriately considered when developing such an infrastructure.

• Because we are currently not looking at full-scale deployment until we can appropriately test the system, we took a single arm machine out of the Koji Build farm to allow us to install and configure a single system and test funcs ability on ARM machines. After we allow for appropriate hardware requirements and commit appropriate resources. Now, we need to ensure that there is appropriate software available for our installation , this is a fairly simple task of checking the Koji Build system.

http://koji.fedoraproject.org/koji/packageinfo?packageID=5212

as such, we can see that func has been successfully built for the arm architecture.

• Upon inspecting both the arm machine that I was allocated and Hong Kong, I could see that someone had installed func on Hong Kong, but not the arm system.  Kind of made me wonder if the previous person had some installation issues.

• The final test in ensuring that we will be able to install func with little or no netwrok problems is ensuring that ports are open an accepting connections, the following are the ports used for communication by the various services.

Certmaster – 51235

Func Minion – 51234

According to the iptables -L -v -n command, the appropriate ports were open for             communication.

**NOTE**  I left out IPTABLES Screenshots for security purposes

This is an immensely important step in the installation process. Its fair enough to simply stick func on two machines and hope they work, but it’s better to spend a little time now preparing and inspecting the systems pre install

0.1 seemed to be a reasonably simple straightforward install process, and upon starting this second part of the project I saw very little possibility of Issues with the install and configuration of func. However; we had to do some simple steps first to ensure its functionality and proper installation on the systems. For my 0.2 I created an outline of steps required in the process and how the project would flow and be completed.

• Determine where to locate the service, which machine should be the overlord and what are its clients
• Evaluate existing installation.
• Since the ARM arch is relatively new to Fedora , we need to check if it has been successfully build and if we can get it from the YUM repo, or are we going to have to build from source
• Test network infrastructure to ensure appropriate configuration.
• Install the software and troubleshoot any issues that arise during installation
• Test modules
• Conclude

After 0.2 is completed I should have a working func overlord client/server setup that can be demonstrated and  more concise understanding of the complications and problems with a full roll-out of func on ALL arm machines and an approach to potentially solving problems within my ability for 0.3

To start the evaluation of func and its eventual deployment in the CDOT system I wanted to start with a working infrastructure and successful installation of the tool  in a sand boxed environment. I currently have my own installation of VMware ESXi 4.1.0 at home and I decided this would be a great opportunity to put it to use, the specs on the system are as follows

VMware ESXi 4.1.0

AMD Phenom x4 965 3.4GHz

8.0 GB DDR3 1333Hz RAM

320GB SATA HDD

Intel MT 1000/100 Pro NIC

I created 3 VM’s  with the following specs

Fedora Core 13 x64

Dual Core Processors

2.0 GB RAM

8GB Disk Space (Thin Provisioning)

Func installation

Func consists of two parts. Certmaster, which is the mechanism for implementing the PKI encrypted communication between master and minion. The second part of the install is the func command and API system. The install on my little test infrastructure was fairly straightforward in nature.

On all machines involved we need to install the func package, (works on FC7 and above, NOARCH). Since we know CDOT has DNS established for our environment with only 3 machines we simply add the machines to the /etc/hosts file. the three machines are names, certmaster, certminion1 and certminion2 respectively.

yum install func

Now, we need to correctly configure what is known as the ‘Overlord’ or the machine responsible for signing certificates and executing the func commands which will be executed on external machines. This is accomplished by turning the certmaster service on. I have not yet decided if auto-signing certificates is something that is secure and a wise decision but this can be accomplished by editing the overlord config file found in /etc/certmaster/certmaster.conf

/sbin/chkconfig --level 345 certmaster on
/sbin/service certmaster start

next we need to configure our minion machines. we do this by edition the /etc/certmaster/minion.conf

[main]
certmaster = certmaster
log_level = DEBUG
cert_dir = /etc/pki/certmaster

then we start the funcd service, which will allow the machines to be managed by the overlord server

/sbin/chkconfig --level 345 funcd on
/sbin/service funcd start

Based on whether or not we have turned on certificate auto-signing we can use the following commands to sign certificates

certmaster-ca –list (list certificates needed for signing)

certmaster-ca –sign {hostname}

Another important security consideration we will need to look into when running func is that it is best practice to run func commands outside of root. This can be accomplished by the following series of commands.

setfacl -d -R -m 'u:MYUSER:rX' /etc/pki/certmaster/
setfacl -R -m 'u:MYUSER:rX' /etc/pki/certmaster/
setfacl -d -R -m 'u:MYUSER:rX' /var/lib/certmaster
setfacl -R -m 'u:MYUSER:rX' /var/lib/certmaster
setfacl -d -R -m 'u:MYUSER:rX' /var/lib/certmaster/certmaster
setfacl -R -m 'u:MYUSER:rX' /var/lib/certmaster/certmaster
setfacl -d -R -m 'u:MYUSER:rX' /var/lib/certmaster/certmaster/certs
setfacl -R -m 'u:MYUSER:rX' /var/lib/certmaster/certmaster/certs
setfacl -d -R -m 'u:MYUSER:rX' /var/lib/certmaster/peers
setfacl -R -m 'u:MYUSER:rX' /var/lib/certmaster/peers
setfacl -d -R -m 'u:MYUSER:rwX' /var/lib/func
setfacl -R -m 'u:MYUSER:rwX' /var/lib/func
setfacl -d -R -m 'u:MYUSER:rwX' /var/log/func/
setfacl -R -m 'u:MYUSER:rwX' /var/log/func/

MYUSER= The username you wish to allow access to func command.

So now that func has been installed, lets run a few simple commands from the func CLI ( I have not yet gotten the Python API down pat yet) heres a few examples and samples to illustrate functionality and some of the usefulness of func , mind you we can further refine the output with various command, func can spit some ugly output sometimes.

The Following is the successful output of the command, which will gather ALL rpms installed on the certminion machine

rpm

The process info command essentially runs the PS command on the target minion. In this case, I am passing it the -x switch. The following link is the successful output

psx

Where from here?

Over the next week or so, I would like to do some experimenting with the python API, potentially giving me the ability to gain more valuable information and format it in such a way that it would be useful. It also gives us the opportunity to chain calls. After that the next step is to deploy func within CDOT with the help of my colleagues, working together to efficiently deploy these tools in the least disruptive manner.

Resources.

https://fedorahosted.org/func/wiki/

https://fedorahosted.org/certmaster/

Func Man Page