Installing latest Python and Scrapy on Planetlab

This post handles multiple issues, which need to be applicable to PlanetLab alone. I was following many instruction references from CentOS, so I believe the instructions are applicable there too.

Running Latest version of Python(>=2.7) on PlanetLab:
PlanetLab uses fedora 8, and yum update installs Python 2.5. So we need to install Python from source. Initially install the dependancies.

sudo yum groupinstall "Development tools"
sudo yum install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel

Create a directory for installing Python in your home directory. Then download and extract the Python source.

mkdir ~/python
wget http://www.python.org/ftp/python/2.7.2/Python-2.7.2.tgz
tar zxfv Python-2.7.2.tgz
find ~/python -type d | xargs chmod 0755

Install from the source code.

cd Python-2.7.2
./configure --prefix=/home/user/python
make
sudo make install

Then edit the PATH environment variable, so Python always refers to the one installed in the home directory. Make sure the addition is before, the default /usr/bin (containing the old Python) should have lesser preference than the one in our home directory.

vi ~/.bashrc
export PATH=/home/user/python/bin/:$PATH
source ~/.bashrc

NOTE: Sometimes you might need to logout and login to observe the change in PATH variable.

Installing easy_install for the version in local home directory:
The easiest way to install "easy_install" is using the egg script. Download the egg script and run it using "sh".

su
wget https://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.egg
sh setuptools-0.6c11-py2.7.egg

Now when you type "which python" and "which easy_install" you should be able to see they point to the versions in the home directory.

Installing Scrapy on PlanetLab:
Scrapy depends on many packages (installation using yum) which makes it hard to run using latest Python2.7 that we installed locally. The trick is to setup a "virtualenv" - python virtual environment, which enables us to have multiple Python installations run parallelly and let us use different installations for different projects.

Install virtualenv, and create a project space.

easy_install virtualenv
virtualenv --distribute project_name
source project_name/bin/activate

After the last instruction, your shell prompt will look like "(project_name) user@server$", indicating you are now in the virtual environment. Also, there should be a local folder titled "project_name".

Now install the Scrapy dependencies. We cannot simply use "easy_install Scrapy", because easy_install would then install latest versions of dependencies (like pyOpenSSL) which do not work on PlanetLab. When I say they do not work, I meant I couldn't make them work. Installing latest version of pyOpenSSL gave a gcc error saying some symbols are missing like in [2]. So we use a hack - install earlier version. These are sufficient for using Scrapy, so we dont break any functionality (atleast I did not come across any case so far).

Install pyOpenSSL 0.12 (latest is 0.13) from egg file.

wget https://pypi.python.org/packages/2.7/p/pyOpenSSL/pyOpenSSL-0.12-py2.7-win32.egg#md5=c343e3833b725e060c094bbf33349349
easy_install pyOpenSSL-0.12-py2.7-win32.egg

Install other dependencies. I guess now yum uses Python 2.7.2 since we are in the virtualenv. Because of that libxml2 being installed was for Python 2.7.

sudo yum install libxml2-devel
sudo yum install libxslt-devel

Now install Scrapy.

easy_install Scrapy

Now to test if it is properly installed, type "python" on the shell prompt, and when you launch Python type "import scrapy" after the python prompt ">>>" to test the import the successful.

References:
1. http://toomuchdata.com/2012/06/25/how-to-install-python-2-7-3-on-centos-6-2/
2. http://stackoverflow.com/questions/11084863/istalling-scrapy-openssl
3. https://pypi.python.org/pypi/pyOpenSSL/0.12
4. https://pypi.python.org/pypi/setuptools#rpm-based-systems
5. http://stackoverflow.com/questions/10927492/getting-gcc-failed-error-while-installing-scrapy

Codeploy multicopy large files timedout ssh

When using multicopy command in Codeploy to transfer large files from/to to planetlab nodes, we might face "timedout ssh" problem. This can be avoided by setting up the environment variable MQ_TIMEOUT to be a large value like 100000. So along with MQ_SLICE and MQ_NODES, we need to set this third environment variable.

References:
1. http://lists.planet-lab.org/pipermail/users/2007-November/002599.html

Running GUI applications in PlanetLab

I had to run GUI applications from PlanetLab for a Project. X11 forwarding (necessary to run GUI applications over SSH connections) is not enabled on the PlanetLab machines due to security reasons. Another option was to install a VNC server on each of the nodes, which provides a GUI enabled remote access to these machines.

I had setup a vnc-server on the planet lab nodes using the instructions given in http://www.g-loaded.eu/2005/11/10/configure-vnc-server-in-fedora/.

Step-1:
First you need to setup vnc-server on the planetlab node.

sudo yum install vnc-server

You need to have a remote user account that you can use with VNC. Your planetlab account should suffice.

Step-2:
Next you need to configure the vncservers config file. In Fedora Core or Red Hat based distros in general, all we have to do is define the VNC server instances in /etc/sysconfig/vncservers. These will be started by the vncserver initscript. This has to be done as root. Edit this file so that it contains the following lines. Below planetlabaccount is your slice name that you use for logging into planetlab.

VNCSERVERS="3:planetlabaccount"
VNCSERVERARGS[3]="-geometry 1024x768 -depth 16"

With these we define that a vnc server instance should be started as user 'planetlabaccount' on display 3 and we also set some options for this server such as resolution and color depth. Each VNC server instance listens on port 5900 plus the display number on which the server runs. In our case, planetlabaccount’s vnc server would listen on port 5903.

For multiple vnc instances /etc/sysconfig/vncservers would look like this:

VNCSERVERS="1:tiger 2:albatros 3:leopard"
VNCSERVERARGS[1]="-geometry 1024x768 -depth 16"
VNCSERVERARGS[2]="-geometry 800x600 -depth 8"
VNCSERVERARGS[3]="-geometry 1024x768 -depth 16"

These would listen on ports 5901, 5902, 5903 respectively.

Step-3:
You need to setup planetlabaccount’s vnc password. So, as user planetlabaccount give the command:

vncpasswd

We are prompted for a password. This is the password that we will use when we connect to planetlabaccount’s vnc server instance. This password is saved in /home/<planetlabaccount>/.vnc/passwd.

Step-4:
After the initial configuration is done we restart the vnc service. As root:

sudo /etc/init.d/vncserver restart

You can make VNC server to start on boot. The command looks like (I haven't tested it out):

chkconfig vncserver on

Step-5:
When vncserver is restarting you will see a message like

New '<planetlabmachine>:3 (planetlabaccount)' desktop is <planetlabmachine>:3

Here "planetlabmachine" is your planetlab node name, eg: "planetlab1.poly.edu"

You need to check if the DISPLAY environment variable is set using the command:

echo $DISPLAY

If you see a blank like, try setting it with the desktop name seen above "<planetlabmachine>:3"

export DISPLAY=<planetlabmachine>:3

Now any GUI applications like xterm or Firefox you run from the terminal will be active in your vncserver's desktop.

Step-6:

Now install a vncviewer like xvnc4viewer or xtightvncviewer on your local machine. If using windows, use realvnc's vnc viewer. Once you login to the planetlab's vnc server, you should be able to see all your GUI applications (like xclock and Firefox) running. Though the display is not so colorful and not so fast, it might serve the purpose.

Step-7 (OPTIONAL):
If working with Firefox, remember to install x11 fonts for Firefox to work. The command you need to give is:

sudo yum install xorg-x11-fonts-Type1

With regular VNC, you wont be able to move or resize the windows as there is no window manager running. You need to run "Metacity" a GNOME window manager. Run the command from the terminal:

metacity &

Reference:
1. http://www.g-loaded.eu/2005/11/10/configure-vnc-server-in-fedora/.
2. http://forums.fedoraforum.org/showthread.php?t=201885