Migrate VM from VMware Player to VirtualBox

I had to migrate a VM from VMware player to VirtualBox once. VMware player does not provide any "Export" option from the GUI, but there is a conversion tool called OVF tool in the C:\Program Files (x86)\VMware\VMware Player\OVFTool location.

Open the command prompt and go to the location of the OVFTool. This tool takes two arguments - the source VM (its .vmx file) and the destination VM (a .ovf file). Type in the source .vmx file (the VMware VM's) file location as source and provide the destination path where the .ovf file should be created.

ovftool "C:\VMWare VMs\Ubuntu.vmx" "C:\VirtualBox VMs\Ubuntu.ovf"

Its takes a long time to create the OVF file. Once done you should be able to load the VM in VirtualBox using "Import Appliance" feature.

While importing I encountered an error:


....unknown element "Config" under Item element, line 47.
Result Code: VBOX_E_FILE_ERROR (0x80BB0004)
Component: Appliance
Interface: IAppliance {3059cf9e-25c7-4f0b-9fa5-3c42e441670b}


This happens because there are some additional configurations embedded in the created OVF file which VirtualBox does not recognise. To solve this error, just open the .ovf file using a text editor (its a simple xml file) and delete the lines containing the "Config" tag. Now you should be able to import the VM to VirtualBox.

References:
1. http://www.howtogeek.com/125640/how-to-convert-virtual-machines-between-virtualbox-and-vmware/
2. http://sharepointtherapy.blogspot.com/2012/12/converting-vmware-workstation-machines.html

Installing latest Python and Scrapy on Planetlab

This post handles multiple issues, which need to be applicable to PlanetLab alone. I was following many instruction references from CentOS, so I believe the instructions are applicable there too.

Running Latest version of Python(>=2.7) on PlanetLab:
PlanetLab uses fedora 8, and yum update installs Python 2.5. So we need to install Python from source. Initially install the dependancies.

sudo yum groupinstall "Development tools"
sudo yum install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel

Create a directory for installing Python in your home directory. Then download and extract the Python source.

mkdir ~/python
wget http://www.python.org/ftp/python/2.7.2/Python-2.7.2.tgz
tar zxfv Python-2.7.2.tgz
find ~/python -type d | xargs chmod 0755

Install from the source code.

cd Python-2.7.2
./configure --prefix=/home/user/python
make
sudo make install

Then edit the PATH environment variable, so Python always refers to the one installed in the home directory. Make sure the addition is before, the default /usr/bin (containing the old Python) should have lesser preference than the one in our home directory.

vi ~/.bashrc
export PATH=/home/user/python/bin/:$PATH
source ~/.bashrc

NOTE: Sometimes you might need to logout and login to observe the change in PATH variable.

Installing easy_install for the version in local home directory:
The easiest way to install "easy_install" is using the egg script. Download the egg script and run it using "sh".

su
wget https://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.egg
sh setuptools-0.6c11-py2.7.egg

Now when you type "which python" and "which easy_install" you should be able to see they point to the versions in the home directory.

Installing Scrapy on PlanetLab:
Scrapy depends on many packages (installation using yum) which makes it hard to run using latest Python2.7 that we installed locally. The trick is to setup a "virtualenv" - python virtual environment, which enables us to have multiple Python installations run parallelly and let us use different installations for different projects.

Install virtualenv, and create a project space.

easy_install virtualenv
virtualenv --distribute project_name
source project_name/bin/activate

After the last instruction, your shell prompt will look like "(project_name) user@server$", indicating you are now in the virtual environment. Also, there should be a local folder titled "project_name".

Now install the Scrapy dependencies. We cannot simply use "easy_install Scrapy", because easy_install would then install latest versions of dependencies (like pyOpenSSL) which do not work on PlanetLab. When I say they do not work, I meant I couldn't make them work. Installing latest version of pyOpenSSL gave a gcc error saying some symbols are missing like in [2]. So we use a hack - install earlier version. These are sufficient for using Scrapy, so we dont break any functionality (atleast I did not come across any case so far).

Install pyOpenSSL 0.12 (latest is 0.13) from egg file.

wget https://pypi.python.org/packages/2.7/p/pyOpenSSL/pyOpenSSL-0.12-py2.7-win32.egg#md5=c343e3833b725e060c094bbf33349349
easy_install pyOpenSSL-0.12-py2.7-win32.egg

Install other dependencies. I guess now yum uses Python 2.7.2 since we are in the virtualenv. Because of that libxml2 being installed was for Python 2.7.

sudo yum install libxml2-devel
sudo yum install libxslt-devel

Now install Scrapy.

easy_install Scrapy

Now to test if it is properly installed, type "python" on the shell prompt, and when you launch Python type "import scrapy" after the python prompt ">>>" to test the import the successful.

References:
1. http://toomuchdata.com/2012/06/25/how-to-install-python-2-7-3-on-centos-6-2/
2. http://stackoverflow.com/questions/11084863/istalling-scrapy-openssl
3. https://pypi.python.org/pypi/pyOpenSSL/0.12
4. https://pypi.python.org/pypi/setuptools#rpm-based-systems
5. http://stackoverflow.com/questions/10927492/getting-gcc-failed-error-while-installing-scrapy

Installing Python SetupTools on 64-bit Windows

I already installed Python 2.7 on Windows 7 64-bit. When I try to run the installer for setuptools it tells me that Python 2.7 is not installed. The specific error message is:

"Python Version 2.7 required which was not found in the registry"

This seems to be a known error. As indicated in reference [2].
"Apparently (having faced related 64- and 32-bit issues on OS X) there is a bug in the Windows installer. I stumbled across this workaround, which might help - basically, you create your own registry value HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Python\PythonCore\2.6\InstallPath and copy over the InstallPath value from HKEY_LOCAL_MACHINE\SOFTWARE\Python\PythonCore\2.6\InstallPath. See the answer below for more details.


If you do this, beware that setuptools may only install 32-bit libraries."

I followed instructions in reference [1].
"Apparently, the setuptools msi is looking for the Python installation registry value InstallPath in HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Python\PythonCore\2.6\InstallPath. Notice the Wow6432Node, which is a registry compatibility layer used for 32-bit apps in Windows 7 64-bit.

As far as I can tell, InstallPath is the only value that this installer looks for. Therefore, using regedit, you can create your own registry value HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Python\PythonCore\2.6\InstallPath, and copy over the InstallPath value from HKEY_LOCAL_MACHINE\SOFTWARE\Python\PythonCore\2.6\InstallPath. To be paranoid, you can try replicating the entire cluster, not just InstallPath.

After this, installation for setuptools seems to proceed correctly. Note that this may only install 32-bit libraries -- WoW6432 is a compatibility layer. Check the other documented solution to this problem if this is not sufficient."



Following the instructions seems to solve the issue, and setuptools installation succeeded.

References:
1. http://selfsolved.com/problems/setuptools-06c11-fails-to-instal/s/63
2. http://stackoverflow.com/questions/3652625/installing-setuptools-on-64-bit-windows

Listing files containing all the words

To find files which contain all words in a set, you can use the below awk script. Here I am searching for files containing three words (word1, word2 and word3). The script output all filenames which contain all three words.

 find  -type f -exec awk 'BEGIN{word1=0;word2=0;word3=0}/word1/{word1++}/word2/{word2++}/word3/{word3++}END{if(word1>0 && word2>0 && word3>0){print FILENAME}}' {} \;

Reference:
1. http://www.linuxquestions.org/questions/linux-newbie-8/grep-an-entire-file-but-must-contain-multiple-words-705681/

Automatic/periodic FTP download using cron jobs

I came across a situation where I had to download files from an FTP server every week. Initially I was doing it manually, but due to human errors I missed some data. I then realized it must be possible to automate the download.

I initially created a shell script to enable FTP download[2]. The script looks like:

#!/bin/bash
HOST='ftp.server.com'   # change the ipaddress accordingly
USER='username'   # username also change
PASSWD='password'    # password also change
ftp -inv $HOST<<EOF
quote USER $USER
quote PASS $PASSWD
bin
cd /move/to/remote/directory        
lcd "/local/directory/" 
mget filename*
cd /move/to/remote/directory2
lcd "/local/directory2/"
mget filename*     
bye
EOF

Using [1], I setup a cron job using the command:

crontab -e

The job entry format is pretty self-explanatory in the reference [1], and there are some commonly used job examples too.
I had to launch a job at the beginning of every week, so my entry in the file looks like:

0 10 * * 1 ~/ftp_download_script.sh

This line states that the script should be launched at 10am every monday.

References:
1. http://www.cyberciti.biz/faq/how-do-i-add-jobs-to-cron-under-linux-or-unix-oses/
2. https://blogs.oracle.com/SanthoshK/entry/automate_ftp_download_using_sh

Python - Value Error Unsupported Format Character

I had a simple python code working with URLs which caused an error of the form: "unsupported format character 'p' (0x70) at index 72".

The code looks like:
num = 1
abc = "http://<site>?value=[abc%20def],value2=%d"
URL = abc % (num)

This is caused because of using the % sign in the string. We need to escape the % sign with another % sign.
So the new string looks like:
abc = "http://<site>?value=[abc%%20def],value2=%d"

References:
1. http://yuji.wordpress.com/2009/01/09/python-valueerror-unsupported-format-character-percent-sign-python-format-string/

GNU plot CDF

you can plot CDF using the following gnuplot command. 

plot "data" u 1:(1./100.) smooth cumulative

Example: plot "data.txt" using $1:(1./100.) smooth cumulative title "CDF" with lines lw 3

References:
1. http://morforma.blogspot.com/2009/07/plotting-cumulative-distribution.html