Cluster mode

From VoxBoWiki

Contents 1 Introduction 1.1 Cluster configuration requirements 1.2 Unpacking the distribution 2 Scheduler configuration 3 Server configuration 3.1 Testing the server installation 4 Putting VoxBo in a non-standard location 5 Mixed operating systems 6 User configuration

Introduction

Installing VoxBo on cluster takes a little bit of work, but it's worth it. The basic idea is that you want the scheduler running on one machine, and the server (that runs the jobs) installed on all the machines. We'll first cover a typical installation, and then cover special circumstances in a later section.

Cluster configuration requirements

VoxBo doesn't ship files around, and it doesn't rewrite path names. In order for VoxBo jobs to work, they can only refer to files that have the same full path names on all your machines. Typically this means you'd be running an NFS server with the same mount point or symbolic link on all your cluster machines.

Unpacking the distribution

The VoxBo distribution is in a gzipped tar file called something like

voxbo-1.8.4-linux.tar.gz

This file just contains a directory called "voxbo-1.8.2" that is typically renamed as /usr/local/VoxBo. If you have an existing VoxBo install, we recommend you rename it to something like OldVoxBo. Then if something goes wrong with the new install, and you need to get some work done, you can just rename it back and you're back in business.

If this is your first voxbo install on this machine, you will want to create a user "voxbo" to own the VoxBo queue files.

As user voxbo, unpack the new version with something like the following:

tar xzvf voxbo-1.8.4-linux.tar.gz

This will give you a VoxBo directory, called voxbo-1.8.4. Now, as root, do the following:

mv /usr/local/VoxBo /usr/local/OldVoxBo    # if needed
mv voxbo-1.8.4 /usr/local/VoxBo
chown -R voxbo /usr/local/VoxBo            # just to be sure
chmod u+s /usr/local/VoxBo/bin/vbsubmit

Next, you will want to edit a handful of VoxBo configuration files, and a few system configuration files. The VoxBo ones are self-documenting (or contain README files), so here's the list

• VoxBo/etc/defaults (most of the main VoxBo configuration)
• VoxBo/etc/serverlist (the list of servers that will be included in the queue when the scheduler starts up)
• VoxBo/etc/servers/README (this file will tell you how to configure each server)

Scheduler configuration

The scheduler is a process that runs constantly, interacting with users, monitoring the queue and telling the servers what to do when. If you edited the "defaults" file above, it's already configured. So you should be able to start the scheduler by doing:

voxbo -d

as user voxbo. The -d lets the process detach, and sends the scheduler log to VoxBo/etc/logs/voxbo.log. If you omit it, the scheduler will run in your terminal, which can be useful for debugging.

Server configuration

The server runs as an xinetd process on every machine you want to run VoxBo jobs. Installing it is usually simple if you have xinetd (or some variant) on your machine. You do have to do this on every machine. First edit your /etc/services file and add the following line:

vbsrvd     6004/tcp

Second, copy the xinetd configuration file from VoxBo/misc/vbsrvd to /etc/xinetd.d/vbsrvd. Edit the file if needed to reflect the correct path to the VoxBo binary directory.

Finally, start or restart xinetd. If it wasn't already started, you might want to make sure it's set to start on boot.

Testing the server installation

To see if your server is responding locally, telnet to the port, type "test," and see what happens. Here's what should happen:

bash> telnet localhost 6004
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
test
ACK

If this doesn't go smoothly, you may need to adjust your security settings. If it does go smoothly, try it again from another machine (remember to replace "localhost" with the machine name).

Putting VoxBo in a non-standard location

We normally assume the VoxBo tree is at /usr/local/VoxBo, and that at least the queue subdirectory is shared among machines on the cluster. In fact, the VoxBo components that care about where the tree is located check the following locations, in order:

• whatever location is in the environment variable VOXBO_ROOT
• /usr/local/VoxBo
• /usr/local/voxbo
• directory named on the first line of the file /etc/voxbo.dir

Using the VOXBO_ROOT variable can be tricky, because it must be set not only for all users, but also for the xinetd server process.

Mixed operating systems

When you have mixed operating systems in your cluster, the important consideration is that the VoxBo queue directory be shared among them. It may be easiest to install the distribution separately on each machine (or at least for each architecture), create one global queue directory, and use symbolic links to make sure each machine is looking at the same queue.

If you want to share the entire VoxBo tree among multiple architectures, it may be easiest if you put the binary stuff elsewhere. It's often easiest just to put them in /usr/local/bin (for the binaries) and /usr/local/lib (for the plug-ins) on each machine.

User configuration

It will help everyone if the VoxBo binaries are in everyone's path. This usually means either putting /usr/local/VoxBo/bin in everyone's path or placing the VoxBo binaries someplace else, like /usr/local/bin. If you move the binaries around, make sure vbsubmit is always owned by user voxbo and has the setuid bit set. i.e.,

chown voxbo vbsubmit ; setuid u+s vbsubmit

Users will most likely want to set where their email should go, using the vbprefs program.