I recently found myself putting together a perl script that performs an rsync of one server in the Rackspace cloud to another server on another account. Why? See my next post about restrictions found with Rackspace for details. Long story short, I needed a way to copy an image from one account to another and Rackspace recommends using rsync.
Since I will be needing to do this at least once a month for my testing, I thought it best to write up a script using perl that does the rsync. The first thing was to figure out to successfully do this rsync before putting things together in a script. I followed the instructions and when I rebooted my new server with the files rsync'ed over, it failed to accept ssh anymore. On a whim, I decided to go through the basic exclude file from the instructions to see if those directories/files really lived where the file says they would be.
Ah-ha! That was my problem. I'm not sure which distro was used to create the exclude file, but some of the recommended files/directories to be excluded were located in different places.
#Suggested by Rackspace /etc/hostname #Actual location on Fedora 14 64bit /bin/hostname #Suggested by Rackspace /etc/modules #Actual location on Fedora 14 64bit /etc/sysconfig/modules
What I did was to do a
find / -name [sometext]
Where [sometext] I started from the end of the path and worked my way in to find the location. It was definitely trial and error. Here is what I have for my final exclude file for Fedora 14 64bit:
#exclude.txt /proc /sys /tmp /dev /root/copyImage.log /root/.ssh/ /var/lock /etc/fstab /etc/mtab /etc/resolv.conf /usr/share/dbus-1/interfaces /etc/networks /etc/sysconfig/network /etc/sysconfig/network-scripts /etc/sysconfig/iptables-config /lib/modules/ /usr/share/selinux/devel/include/kernel /bin/hostname /usr/lib64/gettext/hostname /etc/hosts /etc/modprobe* /etc/selinux/targeted/modules /etc/selinux/targeted/modules/active/modules /etc/sysconfig/modules /etc/selinux/targeted/modules /etc/selinux/targeted/modules/active/modules /usr/lib64/gio/modules /lib/udev/devices/net /usr/lib/ruby/1.8/net /etc/init/
So now that I have a working exclude file, the next thing was to code up a perl script to do this for me. In order to automate this through the script I needed to be able to:
- Create an ssh tunnel from the "sending" server to the "receiving" server and be able to generate an ssh key on the "receiving" server.
- Add the "sending" server's public key to the "receiving" server's authorized_key file
- In my case, as part of the image, there is an authorized_key file with some public keys in it, which I want available to all servers generated with the image. I added the "sending" server's public key to this file and rsync'ed this file over first
- Rsync the "sending" server data to the "receiving" server.
- Open the ssh connection to the "receiving" server
- Know when the "receiving" server is asking for the password and supply it
Neither of these modules are included with the Perl package that comes with Fedora 14 64 bit. In trying to add these modules, I learned some more things. For example, I learned about CPAN. This is a little piece of software that manages installation and dependency resolution for perl modules. Other benefits I found are:
- Access to modules that may not be in repos yet, i.e. Net::OpenSSH is not available via yum on Fedora 14
- Type in instmodsh at the command prompt and you can see a list of all installed modules and get information about the modules
- If module is installed via yum install , will not show up in instmodsh
ERROR: cannot run the configured compiler 'gcc' (see conf/compilerok.log). Suggestions: 1) The complier 'gcc' is not in your PATH. Add it to the PATH and try again. OR 2) The compiler isn't installed on your system. Install it. OR 3) You only have a different compiler installed (e.g. 'gcc'). Either fix the compiler config in the perl Config.pm or install a perl that was built with the right compiler (you could build perl yourself with the available compiler). Note: this is a system-administration issue, please ask your local admin for help. Thank you. Warning: No success on command[/usr/bin/perl Makefile.PL] 'YAML' not installed, will not store persistent state TODDR/IO-Tty-1.10.tar.gz /usr/bin/perl Makefile.PL -- NOT OK Could not read metadata file. Falling back to other methods to determine prerequisites Failed during this command: TODDR/IO-Tty-1.10.tar.gz : writemakefile NO '/usr/bin/perl Makefile.PL' returned status 6400
Finally, my last hurdle for putting together this script is trouble using the open function in perl.
#DOES *NOT* WORK open(CAT_FILE,"cat /root/test.txt >>/root/test2.txt ") || die "Failed: $!\n"; #DOES WORK open(CAT_FILE,"| cat /root/test.txt >>/root/test2.txt ") || die "Failed: $!\n";
Can you spot the difference?
So apparently that "|" is required ... oops...guess the documentation was right. *insert chagrin here*.
So without further ado, please find below my full script to do the rsync step of copying an image to another server on Rackspace:
#!/usr/bin/perl
use strict;
use Expect;
use Net::OpenSSH;
use POSIX qw(strftime);
# ---- start of main ---- #
# Input to script should be ipAddress pwd ipAddress pwd
# take arguments and put into a hash, {key=ipAddress, value=pwd}
my %args = @ARGV;
my @bad_args;
my $log_fname = "/root/copyImage.log";
my $user = "root";
open(LOG_FILE,">>".$log_fname);
# Do a simple check to make sure that the key of hash is
# IP-like (just checking it is ###.###.###.###) and the
# password is only alphanumeric
foreach my $key (keys(%args)) {
if (not $key =~ m/^\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}$/) {
print LOG_FILE "[".get_timestamp()."]\t";
print LOG_FILE "Invalid IP address: ".$key."\n";
push(@bad_args,"Invalid IP, ".$key);
}
if (not $args{$key} =~ m/^[a-z0-9.-]+$/i) {
print LOG_FILE "[".get_timestamp()."]\t";
print LOG_FILE "Invalid password: ".$args{$key}."\n";
push(@bad_args,"Invalid password, ".$args{$key});
}
}
if (scalar(@bad_args) > 0) {
# at least one of the arguments is bad, safer to stop now
# close log file and exit
close(LOG_FILE);
exit(1);
}
# Comment out below to disable debugging
$Net::OpenSSH::debug |= 16;
# add local host pub key to authorized_keys file, so local host pub key is
# transferred when transferring authorized_keys file to remote host
my $cat_out = `cat /root/.ssh/id_rsa.pub >>/root/.ssh/authorized_keys`;
print "Concatenate local pubId to authorized keys file. [".$cat_out."]\n";
# now loop through the hash and do the work
foreach my $key (keys(%args)) {
# reassign key & value to make rest of code more readable
my $host = $key;
my $pwd = $args{$key};
my $opts = "StrictHostKeyChecking=no";
# set up the ssh connection to receiving host to do
# ssh key generation
my $ssh = Net::OpenSSH->new($host,user=>$user,password=>$pwd,
master_opts=>[-o=>$opts]);
$ssh->error and die "Couldn't establish connection: ".$ssh->error;
# send the ssh key generation down the pipe
my ($rout,$pid) = $ssh->pipe_out("ssh-keygen -t rsa -f /root/.ssh/id_rsa -P \"\"") or
die "Pipe out failed: ".$ssh->eror;
wait_pid_exit($rout, $pid, "Key generation");
# use rsync_put method to send rsync command via ssh
# send over the authorized keys file
my ($rout, $pid) = $ssh->rsync_put({stdout_fh => *STDOUT,
quiet=>0,
progress=>1,
archive=>1,
compress=>1,
partial=>1,
one_file_system=>1,
delete_after=>1},
"/root/.ssh/authorized_keys","/root/.ssh/authorized_keys");
wait_pid_exit($rout, $pid, "Rsync: authorized_keys");
# now rsync the rest of the files needed
my ($rout, $pid) = $ssh->rsync_put({stdout_fh => *STDOUT,
quiet=>0,
progress=>1,
archive=>1,
compress=>1,
partial=>1,
one_file_system=>1,
delete_after=>1,
exclude_from=>"/root/rsyncExclude.txt"},
"/","/");
wait_pid_exit($rout, $pid, "Rsync: rest of files");
# send command to delete the last line of the authorized_keys file
# down the pipe
my ($rout,$pid) = $ssh->pipe_out("sed '\$d' < /root/.ssh/authorized_keys > /root/.ssh/temp && mv -f /root/.ssh/temp /root/.ssh/authorized_keys") or die "Pipe out failed: ".$ssh->eror;
wait_pid_exit($rout, $pid, "Editing authorized_keys file");
my ($rout, $pid) = $ssh->pipe_out("exit") or
die "Pipe out failed: ".$ssh->error;
wait_pid_exit($rout, $pid, "Exiting ssh");
print "Exiting ssh.\n";
}
# Clean local authorized_key file by removing self from it
my $cat_out = `sed '\$d' < /root/.ssh/authorized_keys > /root/.ssh/temp && mv -f /root/.ssh/temp /root/.ssh/authorized_keys`;
print $cat_out."\nC'est fin!\n";
close(LOG_FILE);
exit(0);
# ---- end of main ----#
# ----sub routines start here ----#
# This method uses Expect to spawn a command line process on the local/this host/server
# that communicates to a remote/receiving host/server (i.e. rsync, ssh, etc.).
# The command will result result in a prompt on the local host for the remote host's password.
# and the method will wait for the prompt and provide the given password.
#
# Takes 2 parameters:
# 1 - The command string to be executed on the remote/receiving host/server
# 2 - The password to use
sub spawn_expect {
my ($command, $pwd, $ssh) = @_;
my @cmd = $command.split(/ /);
print "[".@cmd."]";
#my $exp = new Expect;
my ($pty, $pid) = ssh->open2pty(@cmd);
my $exp = Expect->init($pty);
#$exp->spawn($command);
$exp->expect(10,[ qr/password:\s*\z/ => sub { my $exp = shift;
$exp->send($pwd."\n"); } ] );
$exp->soft_close();
# wait for the process to complete
print "Waiting for pid [".$pid."] to exit...";
waitpid($pid, 0);
print "Spawning done.";
}
# Returns a local formatted timestamp string
sub get_timestamp {
return strftime "%a %b %e %Y %H:%M:%S", gmtime;
}
# waits for the given process id to exit
# uses given string to display helpful print out
sub wait_pid_exit {
my ($rout, $pid, $str_out) = @_;
while (<$rout>) { print } close $rout;
print "\nWaiting for pid [".$pid."] to exit...\n";
waitpid($pid, 0);
print $str_out." done.\n";
}
# ---- end of sub routines ---- #
No comments:
Post a Comment