New Blog – awaseroot

AwaseConfigurations project has ended but we’ll continue with many of the same topics on our new blog:

http://awaseroot.wordpress.com/

See you there!

Testing with 65 Hosts

We have now tested our system with 65 machines using three of our school labs simultaneously. The time required to run the full installation of our Ubuntu workstations and servers took about the same time as with just 25 machines. We have a ten second delay between machine wake ups so it takes a bit longer due to that but we didn’t notice any delay otherwise. We also ran some Fabric tasks to see if there was any difference but everyhing went just as well as before. So I think it’s safe to say that you can control at least 65 remote hosts simultaneously without any problems with Fabric. I wish we could do tests with more machines but I think this is all we can do at this point.

Here’s a short video of us rebooting our 65 machines:

Feedback

Have you tested our guide, the complete system or a maybe a small part of it? Feel free to give us some feedback by adding a comment to this post.

How did you test it?
In what kind of network environment?
Was our material helpful?
Was there any problems?
Improvement suggestions?

The End of AwaseConfigurations Project

The project has now come to it’s end and we have published our closing report. It’s available in our git.

Although the project is now closed, we will still answer any comments or emails regarding this project. Also there might still be some updates here on the blog. We’re planning on doing a short video about the system and it will be published here when it’s finished.

Thanks to our readers, project team members and of course big thanks to our coordinator.

The Guide published

The project is nearing it’s end quite soon and there won’t be many new features added anymore. So we made a complete guide for setting up our system and using it in a new environment. This includes installing and configuring cobbler, fabric and configuration packages using our scripts. The guide can be found here: http://awaseconfigurations.wordpress.com/guide/

Fabric file updated

Greetings Everyone,

We have been working hard on the closing touches to our project and I am pleased to bring out news about the updated the fabric control file which now has a more defined structure. The naming is now more consistent and the listing provides some information on each command. We also have an improved readme file to give the general outlines on how to use it. The files can be found as usual from our git.

setup squid for apt caching

After fabric has introduced parallel execution our team decided to try out this feature within our projects environment. One of the tasks included installation of gnome onto all the workstation in the lab (abt 20 machines). This task seemed interesting to parallelize, since both – the data processing load and the number of computers seems challenging enough [for a small network environment].

Running such an installation without any caching server seemed somewhat suicidal. We considered apt-mirror, we tried apt-cacher-ng, but the solution was found in squid, a well supported extensively documented caching server. The installation on an Ubuntu 11.04 server went with:

sudo apt-get install squid

To adapt it to our network (172.28.0.0) I edited / uncommented a line in the /etc/squid/squid.conf Access Control section:

acl localnet src 172.28.0.0/16

On workstations’ side I added a pointer to proxy:

sudoedit /etc/apt/apt.conf

and added the following line there:

Acquire { Retries "0"; HTTP { Proxy "http://host1.local:3128"; }; };

Idempotence with Fabric

One important part of our project is to have our fabfile idempotent. Meaning that we can execute same fabric tasks multiple times and always expect the same result. Most of it is already handled by normal Linux features like in this case:

def add_user(new_user):
	if sudo("useradd -m %s" % new_user).failed:
		print("User %s already exists!" % new_user)
		return

Here we can run fab add_user:simo as many times as we want and there will still be only one simo user on each machine. The useradd command won’t allow adding users that already exist.

Apt takes care of idempotence issues with packages like for example in this one:

def install(package):
	sudo("apt-get update")
        sudo("apt-get -y install %s" % package)

Apt will install the package if it’s not already installed and won’t if it’s already there. So we can run this and expect to always have the package installed and not have multiple copies of it every time we run the task again.

Then we have rest of the tasks that are not handled by default Linux commands or apt. We take care of those by adding a few lines of code in each task so that it will make sure they behave as expected. Most of it is done with simple if-else checks or by removing duplicate files. Here’s an example:

Automated Ubuntu Release Upgrade

Ever wondered how to do a Ubuntu release upgrade from command line automatically so that it doesn’t ask any questions during the upgrade process? It really gets quite boring to answer those questions if you’re upgrading multiple machines. In our case the upgrade was from 11.04 to 11.10 (natty to oneiric) and this is how we did it:

1) Change the sources from natty to oneiric in the /etc/apt/sources.list. (We use our configuration package to do it efficiently on multiple machines)
UPDATE: You can use this command to change the sources:
sudo sed -i 's/natty/oneiric/' /etc/apt/sources.list

2) Run these two commands:
sudo apt-get update
sudo DEBIAN_FRONTEND=noninteractive /usr/bin/apt-get dist-upgrade -o Dpkg::Options::="--force-confold" --force-yes -y

The –force-confold option is used to keep the old configuration files and not replace them with new ones.

3) Reboot the machine:
sudo reboot

Or you can just use this simple script for full automation. The script does all of the above tasks. Works only when upgrading from Ubuntu 11.04 to 11.10.

We are using this with Fabric so that we can remotely upgrade every machine from 11.04 to 11.10, and we do it simultaneously. Takes about 15 minutes total to upgrade all of our 20 machines. Here’s our fabric task for it:

@parallel
@roles('workstations')
@with_settings(warn_only=True)
def auto_dist_upgrade():
        if not _is_host_up(env.host):
		return
	config('oneiric-sources')
	sudo("apt-get update")
        sudo('DEBIAN_FRONTEND=noninteractive /usr/bin/apt-get dist-upgrade -o Dpkg::Options::="--force-confold" --force-yes -y')
	sudo("reboot")

We haven’t tested this on other Ubuntu releases but in theory it should work on them as well.

UPDATE 29.4.2012: It is recommended to use do-release-upgrade instead of dist-upgrade, see my new guide to try the recommended way.

Problems with apt-cacher-ng and parallel fabric execution (or ‘the crash of the cacher’)

This Thursday we installed apt-cacher-ng on our project’s webserver. The idea was to test [under a 'relative amount of stress'] both:

• how fabric performs parallel execution
• how apt-cacher-ng copes with getting the packages, caching them, and providing upon request, in parallel
  execution

As ‘relative amount of stress’ we set: parallel installation of Gnome onto 20 Ubuntu 11.04 machines. As a result of running our fabfile, which included installing gnome on all the 20 workstations in the LAN nothing happend, i.e. no gnome installed. From the output we saw that packages could not be found, apt-cacher-ng was crashing every time we tried to install gnome on all the machines. Changing its configuration to provide up to 25 parallel threads did not help.

The workaround which did help was to first run the installation of gnome on one machine, and only then (after the operation was completed) to let fabric install gnome on all machines in parallel.

Conclusion: apt-cacher-ng could not at the same time manage fetching packages, caching them and serving the 20 machines, however, it managed to fetch packages, cache them and serve one machine and to then serve [from its cache] another 19 machines in parallel.

This leaves us with either / or:

• constructing our fabfile so that every time we want to install packages in parallel we first do it on one machine and then on rest
• instead of apt-cacher-ng use some other tool for proxy caching e.g. our project supervisor Tero Karvinen was suggesting to use Squid, a well-known, well-developed, well-supported, and widely used web-cache / proxy server.

The group has decided on the latter, so tests with Squid have begun.

apt-mirror vs apt-cacher-ng

Last friday we were making good advancements with fabric and thought its now time to set a mirror for our packages. The first candidate on the list was apt-mirror. Initially it seemed we’d set it up on our Ubuntu 11.04 server in no time:

New Scripts – Fabric & Config Packages

Since we need to set up fabric each time we do these tests at the lab, I thought it would be best to create a script for it. So I made this small script for installing everything we need for our fabric tests including the latest version of our fabfile:

#!/bin/bash

# this script will download fabric, enable universe sources, update source list,
# install packages that are needed for fabric, install fabric, download the
# project fabfile.

wget http://git.fabfile.org/cgit.cgi/fabric/snapshot/fabric-1.3.1.tar.gz
wget http://myy.haaga-helia.fi/~a0900094/awasebg.jpg
tar -xf fabric-1.3.1.tar.gz
sudo software-properties-gtk -e universe
sudo apt-get update
sudo apt-get install python python-pycryptopp python-setuptools python-paramiko
cd fabric-1.3.1/
sudo python setup.py install
cd
wget https://raw.github.com/AwaseConfigurations/main/master/fabric/fabfile.py
fab -l

It’s also available in our git here.

Setting up Reprepro repository

So we wanted to create a repository for our configuration packages and reprepro seemed like a good and easy way to go. So I started making a script to the fabfile for installing and configuring it using this tutorial. I got the script done, tested it, and soon realised that it didn’t work. The distributions file was missing a line and the packages didn’t get added to the repository. So with the help of  Armens I made some improvements to the script and got it working. Here’s our new working reprepro conf/distributions file:

Fabric – Hostnames and initial tasks

Our Cobbler setup seems to be in a stable state right now so we’ve focused on testing fabric a bit more at the lab. First thing was to make our fabfile use hostnames instead of IP addressess so we can keep track of the machines even if they’re getting new IP addresses from a DHCP-server. Our coordinator Tero advised us to use avahi-browse tool which lists all the machines in the LAN. Here’s the command that we tested on our control machine:
avahi-browse -at

This should list all of our machines since at this point we had already installed them using Cobbler. Well it didn’t show anything besides our cobbler machine and the control machine. Soon we figured out the machines might be missing something since we there’s not very much software installed on our alternate-ubuntu installs. The missing part was libnss-mdns and we installed it to each of our machines with:
fab install:libnss-mdns

The test environment

I thought it might be useful to show a figure of our test environment that we have in the school lab. This way it’s easier to understand what we’re doing and how we’ve set up the system. This is an image from a network simulation that was done with Cisco Packet Tracer 5.3.2.

Cobbler setup script – problem with restarting service and rebuilding configuration

Writing a bash script for an easy and fast setup of Cobbler environment took a while. But the need to get it done – our project needs frequent and painless network installations of Ubuntu on 20-30 machines, took over my ambitions for the afternoon as a seemingly easy task and just swallowed my whole evening.

The problem: running the script was not completely setting up the environment, although it contained same commands as i have been manually inserting into the terminal, the output was returning that cobblerd does not appear to be running/accessible, though the script was deliberately staring the service after the installation with
sudo service cobbler start

Fabric – Parallel execution

Fabric 1.3.0 comes with a great new feature that allows parallel execution of fabric tasks. Also the serial execution is still available which I think is still very useful. I did a small test to see how the new feature behaves.

This was tested on virtual Xubuntu 11.04 running on VirtualBox 4.1.2.

Reprepro

What’s up ladies and gentlemen! It’s time for yet another tutorial here at the awaseConfigurations. This time it’s a match up between the DebAthenas Config-Package-Dev in the upper right corner versus Reprepo apt-repositories of a haaga-helia student at the bottom left position.

Continuing tests on wakeonlan and cobbler automation problems / limitations

Continuing tests on wakeonlan and cobbler automation problems / limitations

• Set up Cobbler just like in this article. Added these two lines to var/lib/cobbler/kickstarts/ubuntu-nqa.seed:

  d-i mirror/country string FI
  d-i mirror/http/proxy string

• Powered on 23 computers with a similar wakeup command (23 MAC addresses were replaced for security reasons):
  for MAC in 12:34:56:78:9a:cc 12:34:56:78:9a:ff; do wakeonlan $MAC; sleep 10; done

Results:

• All 23 machines switched on.
• 10 machines initiated network installation of Ubuntu 11.04 Alternate, out of these:
  • 7 machines successfully completed installation with no questions asked (fully automated)
  • 3 machines did not complete network configuration, however, manually choosing “Retry network autoconfiguration” from the given menu did sort it out, after which these computers also successfully completed the installation (say, 97% automated)
• 13 machines just booted straight into Ubuntu 11.04 Alternate, which we apparently installed onto them in the previous days. I am suspecting that Cobbler still ‘remembers’ them as systems in /var/lib/cobbler/config/systems.d/. Replacing these systems’ names with other [names] should solve this problem if in the future we want / need clean installations on all the machines -> still needs to be tested.

Wakeonlan and Cobbler – some automation problems / limitations

During last few days deployment of Ubuntu 11.04 Alternate over the LAN using cobbler and wakeonlan was causing some ‘automation’ problems / limitations that are worth mentioning:

• machines can not complete network autoconfiguration
• machines are stuck on loading additional components

Fun with Fabric – First lab tests

For couple of days now I’ve been testing fabric on virtual machines at home and a lot of it has been just hunting for solutions for weird errors which in retrospect are seemingly caused by the virtual environment. So I was happy to finally get to our school lab to test the system in a proper test environment.

Armens was with me so he set up about twenty machines with cobbler as I set up one control machine with fabric and our project fabfile. We needed the IP-addresses of all our machines and first I tried to add them directly to the fabfile just like this:

webserver='172.28.212.1'
env.hosts=['webserver']

Soon to find out it didn’t work… “unknown host webserver”
So as I wanted to get to testing I just did what I knew would certainly work and added the IP-addresses to the /etc/hosts file. Here are the contents:

Arpwatch, wakeonlan, cobbler

In order to collect mac addresses of all the NICs in one of the labs (25 pcs per lab) install arpwatch on one of the Ubuntu 11.04 machines, but just before that open a terminal and start following on what is going on in syslog
tail -F /var/log/syslog

Then, in another terminal install arpwatch with
sudo apt-get install arpwatch

After that, start switching on all the machines in the lab, after a few minutes they will all be collected in the output of syslog.

Now, it should be possible to send the magic packet to all these machines with wakeonlan with just one for-loop

for MAC in 12:34:56:78:9a:cc 12:34:56:78:9a:ff; do wakeonlan $MAC; done
*instead of the two MACs my command contained 24 MACs, which I cannot publish for security reasons.

As a result:
- all (24) machines caught the magic packet and started booting
- almost all (21) machines initiated network boot
- 3 machines started up from hard drive (i.e. woke up but skipped the network boot)

Milestones

Here’s the milestones we’ve set for the project. They’re also listed in our project plan but I think you get a better idea from this one.

Project Milestones

Git repository moved

We have moved our git repository to a new github account and we won’t commit updates to the old one any more. You can find the new git repository here: AwaseConfigurations

Magic Packet – wake on LAN

In order to speed up the network installation of Ubuntu 11.04 onto about 30 machines at our university lab, I tested if sending the Magic Packet would help dealing with this quantity.

On an Ubuntu 11.04 machine I installed wakeonlan:
sudo apt-get install wakeonlan

and ran a single command:
wakeonlan AA:AA:AA:C0:8A:7F

where 78:AC:C0:C0:8A:7F is a MAC address of one of our lab computers, which was switched off.. until I sent the Magic Packet – after that it switched on and initialized network boot!.. Magic!..

Project Plan

The project plan is now out and available from our repository.

Git Repository

We’ve set up a git repository in Github for our project. We’re going to use it for our version management and there you can (soon) access all of our project files. There’s not much content yet but we’ll add more soon.

The repository can be found here:  AwaseConfigurations

Cobbler – Ubuntu 11.04 Network Installation (4PCs in 7min)

Setting up Cobbler for Ubuntu 11.04 network installation at home was fun (here is the report), setting it up at Haaga-Helia labs – the actual project’s operational environment turned out to be even greater fun. Especially for Henri, who stayed in the lab (while I went for some coffee) and had to very soon be explaining to some of the IT personnel why instead of their network boot they have ours. When I came back with my coffee the network cable was unplugged from the Cobbler machine (so that our DHCP server managed by Cobbler would not interfere with the labs DHCP server) so, after some good laugh we sat down to reshuffle. So, initially, we found two possibilities:

• operate with Cobbler in the labs when [absolutely] nobody else is there
• use Cobbler to pin an installation to particular MAC addresses

But neither of these seemed useful: we cant be dependent on labs being completely empty (there is a good chance someone is always going to be at one of the 6 lab-rooms’ roughly 150 computers); and having Cobbler to pin MAC addresses to the installation didnt solve the problem, it did force the rebooted machine with that MAC to choose a specified installation, but the DHCP problem remained, it was still interfering with the lab’s DHCP server, so other machines catching lab’s network boot or ours would be of pure luck (or no-luck).

An idea for the solution came from Tero Karvinen and his article on PXE Network Boot. In essence, we needed to edit the dhcpd.conf (or in our case – Cobbler’s dhcpd.template) to:

• make our DHCP server authoritative
• make it authoritative for only certain MAC addresses

And it worked, for only those MAC addresses that were mentioned in the Cobbler’s dhcpd.template, without interfering with any other machines. Below are my brief steps for setting up Cobbler to install Ubuntu 11.04 Alternate onto 4 machines. The whole installation took only 7 (european) minutes!

Tools & Tasks

Here’s a short list of administrative tasks we can automate using different tools.

Tasks	Fabric	config-package-dev	Cobbler
OS installations	No	No	Yes
Install using PXE-server	No	No	Yes
User account management	Yes	No	No
Add users	Yes	No	No
Configuration file management	Yes	Yes	Yes
Create config packages	No	Yes	No
Send config packages	Yes	No	No
Send config files	Yes	No	Yes
Modify config files	Yes	Yes	No
Backups	Yes	No	No
Copy backup files	Yes	No	No
Package management	Yes	No	No
Send packages	Yes	No	No
Install packages	Yes	No	No
Run apt	Yes	No	No

Config-package-dev – Building configuration packages

Due to my original test becoming too messy and confusing, I made this tutorial on how a debian config package can be done using config-package-dev.

This configuration package will add # to the beginning of a certain line in a config file, in other words it will comment a line in the file. Commenting lines this way in a config file is a very common task for modifying configurations. Using packages like these is essential for large networks instead of modifying the files manually. I have marked a timestamp on these as I tested them so you can see a time estimate for how long this takes.

This was tested on Ubuntu 11.04

Cobbler – set up network installation of Ubuntu 11.04

At our project we’ll need to deploy Ubuntu desktops and servers onto abt 30 machines, and because the operational environment will be within our university premises, i doubt that the school will provide us with a permanent access to 30 machines on which we could roll out our systems and keep them exclusively for us, therefore, we need an installation server that allows for fast and frequent deployment of operating systems. Cobbler, described as “a Linux installation server that allows for rapid setup of network installation environments” seems to be the tool for setting up that environment for us.

Primary objective of the test: set up Cobbler environment for network installation of Ubuntu 11.04 (alternate and server) with no-question-asked
On the target machine: enable PXE in BIOS settings
On the server (Ubuntu 11.04): see bellow [unless explicitly mentioned otherwise]

Config-package-dev

For our configuring needs we’re propably going to need to make configuration packages and config-package-dev seems like the way to go. Here’s some research on that.

Config-package-dev is a set of modules for the Common Debian Build System(CDBS) and is used for creating configuration packages on debian systems. It’s part of MIT’s Debathena project but it can be used without the whole Debathena system. Purpose of configuration packages is to configure Debian systems. Config-package-dev is safe to use since the installed configuration packages can be uninstalled to restore a machine to it’s previous state. (MIT Debathena)

Fabric Test 2: Operations – File transfer

Here’s my second test with Fabric. This time I tried uploading and downloading files. The same virtual machines were used as in the first Fabric test.

Testing Fabric with two virtual machines

This is my first test with Fabric. It’s a very simple one where I just try to run a command on two virtual machines using Fabric.

Fabric

Fabric seemed like a useful tool for our deployment tasks so I did some research on that.

Fabric is a tool for running commands and administrative tasks on a remote computer. It can also be used to upload and download files to and from the remote machine. Fabric uses python modules which include tasks that can operate on local and remote computer. The taks are defined in a python file called fabfile and it’s possible to use fabfile to define multiple remote hosts that run all or some of the tasks in the fabfile. (fabfile.org)

Tasks can be simple commands that send the output back to the fabric user or they  can be more complex operations like for example collecting configuration files from a local computer to a single tar packet, sending it to a remote machine, unpacking it and finally replacing old configuration files with the ones that were sent. All in all it really comes down to these basic fabric commands  and their combinations:

run(’command’)                         # Run a command on a remote machine
sudo(’command’)                      # Sudo a command on a remote machine
local(’command’)                       # Run a command on the local machine
put(’path1, path2’)                   # Send a file in local path1 to path2 in a remote machine
get(’path1, path2’)                    # Download a file from remote machine’s path1 to local path2

Of course the complexity of your fabfile tasks depend on the amount of python scripting you put in to it.

So what’s this project about?

This project is about managing a network of 30 workstations including a few servers. The goal is to centralize management and make it as well automated as we can. The project is coordinated by Tero Karvinen.

For now we’ll be testing various tools that hopefully will help us to achieve our goal. Some tests have already been done and they will be published later this week.

What’s up!

Art of networking is here with AwaseConfigurations! Project includes the talent’s of 3 students from Haaga-Helia university of applied sciences. The coming months we will be updating this blog with news and insights from the world of network configuration.

Some more insight in finnish.

Projekti tulee käsittelemään testein ja tutoriaalein erilaisia työkaluja, joita voi käyttää tehokkaaseen tietokoneverkoston kontrollointiin. Tärkeintä projektin kannalta on tehokkuus ja järjestelmien vakaa tila. Pyrkimyksenä on vähentää huomattavasti aikaa, joka kuluisi järjestelmän konfigurointiin useamman koneen kesken ja samalla varmistaa ettei yksikään kone ole eri tilassa kuin sen pitäisi. Järjestelmän voi asentaa tyhjästä tilasta, jossa koneet käynnistetään ja niihin asentuu haluttu käyttöjärjestelmä ilman, että käyttäjän tarvitsee puuttua toimenpiteisiin asennuksen aikana.