Wednesday, November 13, 2013

Latex hyperref, url packages: Break URL

When using both hyperref and url packages, and you get a package conflict, make sure the url package is listed before the hyperref package.

Also, if you need to break the URL into multiple lines, then you need to use the hyphens option in the url package. So your latex configuration should look like:

\usepackage[hyphens]{url}
\usepackage{hyperref}  

Reference:
1. http://stackoverflow.com/questions/2625714/linebreak-in-url-with-bibtex-and-hyperref-package

Sunday, August 4, 2013

R - commands cheat sheet

To read a text file
> data <- read.csv('R - anon answers to total answers.txt',sep=",",header=TRUE)

To convert cells (row-column pairs) satisfying a certain condition to NA.
> data$Column[data$Column == 0] <- NA
> data$Column[data$Column == -Inf] <- NA

To generate a linear model from the data
> linear=lm(data[,2]~data[,1])
> polynomial=lm(data[,2]~data[,1]+I(data[,1]^2))
> loglinear=lm(logdata[,2]~logdata[,1])

To get a summary of the generated models
> summary(linear)
> summary(polynomial)

To plot data
> plot(data[,1],data[,2])
> plot(data[,2]~data[,1])
> plot(log(data[,1]),log(data[,2])) 

Adding text to plot
> text(c(300,300),c(0.8,0.82),labels=c("R^2 = 0.0001","y = 0.00072 * x - 0.163"))

To draw the linear model generated using "lm" command
> abline(linear)

We can draw polynomial curve generated using lm/glm using
> lines(sort(x), linear$fitted.values[order(x)])

To generate log-linear models using the glm package
> glm_test2=glm(data[,2]~data[,1],family=poisson())
> glm_test2=update(glm_test,.~.+I(data[,1]^3),family=poisson())

We can compare different linear models generated using lm or glm commands through anova tests. When there are more than 2 models, model-1 is compared to null hypothesis, model-2 with model-1, model-3 with model-2 and so on.
> anova(glm_test,glm_test2,glm_test3,glm_test4,glm_test5,glm_test6,glm_test7,glm_test8,glm_test9,glm_test10,test = "Chisq")


vmware tools copy-paste feature in Ubuntu

You will need the vmware tools for having the feature to copy-paste feature between the VM and the host machine. Installing vmware tools on an Ubuntu machine was a little tricky.

You can find the instructions at: Installing VMware Tools in an Ubuntu virtual machine (1022525)

References:
1. http://communities.vmware.com/thread/395491?start=0&tstart=0
2. http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1014294
3. http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&docTypeID=DT_KB_1_1&externalId=1022525

Saturday, August 3, 2013

Extract range of lines from text file in unix

It is possible to use command line sed to easily extract a range of lines from a text file. The command to be used is:
sed -n 16224,16482p filename > newfile


Where 16224,16482 are the start line number and end line number, inclusive. This is 1-indexed. 
-nsuppresses echoing the input as output, which you clearly don't want; 
the numbers indicate the range of lines to make the following command operate on; 
the command p prints out the relevant lines.

References:
http://stackoverflow.com/questions/83329/how-can-i-extract-a-range-of-lines-from-a-text-file-on-unix

Motorola Atrix - Unlocking bootloader and reflashing ICS

I recently tried unlocking the boot loader to load Ice Cream Sandwich onto my Atrix MB860 phone. During the process, I had some difficulties, and wanted to document the steps for future reference (and for other's benefit).

Unlocking bootloader

This webpage(Installing Atrix-MROM on an Atrix 4G with 4.5.145 update) mentioned it is possible to unlock Atrix4G with 2.3.6 Gingerbread and 4.5.145 update using the unlock instructions given here: Automatic bootloader UNLOCK/ROOT for AT&T 1.26/1.57/1.83/4.5.91 [UPDATE 07/31/11]

I followed the instructions in Automatic bootloader UNLOCK/ROOT for AT&T 1.26/1.57/1.83/4.5.91 [UPDATE 07/31/11]. To unlock I used option-1 (Automatic Bootloader Unlock for ATT ATRIX 4G Only), and then selected the option 3 (4.5.91). It threw a "unable to boot" screen, and gave around 10 options (first option was RSD and second was fastboot, and some others followed) and automatically started the fastboot mode - but windows was not able to recognize the device. Since the instructions on the terminal were asking us to select the fastboot mode by pressing  "volume down+power" combo, I removed the battery for 30 seconds and tried it multiple times but the phone was not booting. At last after few trials it booted to fastboot mode. the terminal instructions showed an unlock code and asked me to enter it (probably option3 - Apply FIX for "Failed to boot 0x1000"/ NO OS Users (fastboot) must have been initiated). Later the phone restarted automatically and I reached the "Welcome to MOTOBLUR" screen. During the motorola boot screen, I saw "unlocked" on the top left corner.

This video should help out too: http://www.youtube.com/watch?v=S6a0BWquqAc

Flashing ROM

I then selected "Automatic ROOT/CWM for Unlocked ATT ATRIX 4G Only (Tenfar CWM)" for backing up my old ROM.

It is to be noted that the cell battery does not get charged while in the boot menu and connected to a power source (either a computer or the power port on the wall). While trying to flash ROM, I received a "Low battery to flash error". For the battery to charge, I need to flash a ROM and flashing was not possible at the moment.

I purchased an external battery charger like the enercell universal external battery charger and charged the battery completely. Now I was able to flash a new ROM.

I followed the instructions at this XDA forum webpage to solve the soft brick issue during the flash process.

After launching ICS, I was unable to download apps from playstore. So removed the gmail account and re-added it without sync as per this forum .

References:
1. http://forum.xda-developers.com/showthread.php?t=1182871
2. http://forum.xda-developers.com/showthread.php?t=2230432&highlight=unlock+bootloader
3. http://forum.xda-developers.com/showthread.php?t=2262869&highlight=soft+brick
4. http://forums.androidcentral.com/google-nexus-7-tablet/224884-google-play-error-solution-rpc-s-5-aec-0-a.html

Saturday, May 11, 2013

Migrate VM from VMware Player to VirtualBox

I had to migrate a VM from VMware player to VirtualBox once. VMware player does not provide any "Export" option from the GUI, but there is a conversion tool called OVF tool in the C:\Program Files (x86)\VMware\VMware Player\OVFTool location.

Open the command prompt and go to the location of the OVFTool. This tool takes two arguments - the source VM (its .vmx file) and the destination VM (a .ovf file). Type in the source .vmx file (the VMware VM's) file location as source and provide the destination path where the .ovf file should be created.
ovftool "C:\VMWare VMs\Ubuntu.vmx" "C:\VirtualBox VMs\Ubuntu.ovf"

Its takes a long time to create the OVF file. Once done you should be able to load the VM in VirtualBox using "Import Appliance" feature.

While importing I encountered an error:


....unknown element "Config" under Item element, line 47.
Result Code: VBOX_E_FILE_ERROR (0x80BB0004)
Component: Appliance
Interface: IAppliance {3059cf9e-25c7-4f0b-9fa5-3c42e441670b}


This happens because there are some additional configurations embedded in the created OVF file which VirtualBox does not recognise. To solve this error, just open the .ovf file using a text editor (its a simple xml file) and delete the lines containing the "Config" tag. Now you should be able to import the VM to VirtualBox.

References:
1. http://www.howtogeek.com/125640/how-to-convert-virtual-machines-between-virtualbox-and-vmware/
2. http://sharepointtherapy.blogspot.com/2012/12/converting-vmware-workstation-machines.html

Monday, April 15, 2013

Installing latest Python and Scrapy on Planetlab

This post handles multiple issues, which need to be applicable to PlanetLab alone. I was following many instruction references from CentOS, so I believe the instructions are applicable there too.

Running Latest version of Python(>=2.7) on PlanetLab:
PlanetLab uses fedora 8, and yum update installs Python 2.5. So we need to install Python from source. Initially install the dependancies.
sudo yum groupinstall "Development tools"
sudo yum install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel

Create a directory for installing Python in your home directory. Then download and extract the Python source.
mkdir ~/python
wget http://www.python.org/ftp/python/2.7.2/Python-2.7.2.tgz
tar zxfv Python-2.7.2.tgz
find ~/python -type d | xargs chmod 0755

Install from the source code.
cd Python-2.7.2
./configure --prefix=/home/user/python
make
sudo make install

Then edit the PATH environment variable, so Python always refers to the one installed in the home directory. Make sure the addition is before, the default /usr/bin (containing the old Python) should have lesser preference than the one in our home directory.
vi ~/.bashrc
export PATH=/home/user/python/bin/:$PATH
source ~/.bashrc

NOTE: Sometimes you might need to logout and login to observe the change in PATH variable.

Installing easy_install for the version in local home directory:
The easiest way to install "easy_install" is using the egg script. Download the egg script and run it using "sh".


su
wget https://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.egg
sh setuptools-0.6c11-py2.7.egg

Now when you type "which python" and "which easy_install" you should be able to see they point to the versions in the home directory.

Installing Scrapy on PlanetLab:
Scrapy depends on many packages (installation using yum) which makes it hard to run using latest Python2.7 that we installed locally. The trick is to setup a "virtualenv" - python virtual environment, which enables us to have multiple Python installations run parallelly and let us use different installations for different projects.

Install virtualenv, and create a project space.
easy_install virtualenv
virtualenv --distribute project_name
source project_name/bin/activate

After the last instruction, your shell prompt will look like "(project_name) user@server$", indicating you are now in the virtual environment. Also, there should be a local folder titled "project_name".

Now install the Scrapy dependencies. We cannot simply use "easy_install Scrapy", because easy_install would then install latest versions of dependencies (like pyOpenSSL) which do not work on PlanetLab. When I say they do not work, I meant I couldn't make them work. Installing latest version of pyOpenSSL gave a gcc error saying some symbols are missing like in [2]. So we use a hack - install earlier version. These are sufficient for using Scrapy, so we dont break any functionality (atleast I did not come across any case so far).

Install pyOpenSSL 0.12 (latest is 0.13) from egg file.
wget https://pypi.python.org/packages/2.7/p/pyOpenSSL/pyOpenSSL-0.12-py2.7-win32.egg#md5=c343e3833b725e060c094bbf33349349
easy_install pyOpenSSL-0.12-py2.7-win32.egg

Install other dependencies. I guess now yum uses Python 2.7.2 since we are in the virtualenv. Because of that libxml2 being installed was for Python 2.7.
sudo yum install libxml2-devel
sudo yum install libxslt-devel

Now install Scrapy.
easy_install Scrapy

Now to test if it is properly installed, type "python" on the shell prompt, and when you launch Python type "import scrapy" after the python prompt ">>>" to test the import the successful.

References:
1. http://toomuchdata.com/2012/06/25/how-to-install-python-2-7-3-on-centos-6-2/
2. http://stackoverflow.com/questions/11084863/istalling-scrapy-openssl
3. https://pypi.python.org/pypi/pyOpenSSL/0.12
4. https://pypi.python.org/pypi/setuptools#rpm-based-systems
5. http://stackoverflow.com/questions/10927492/getting-gcc-failed-error-while-installing-scrapy

Saturday, April 6, 2013

Installing Python SetupTools on 64-bit Windows


I already installed Python 2.7 on Windows 7 64-bit. When I try to run the installer for setuptools it tells me that Python 2.7 is not installed. The specific error message is:

"Python Version 2.7 required which was not found in the registry"

This seems to be a known error. As indicated in reference [2].
"Apparently (having faced related 64- and 32-bit issues on OS X) there is a bug in the Windows installer. I stumbled across this workaround, which might help - basically, you create your own registry value HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Python\PythonCore\2.6\InstallPath and copy over the InstallPath value from HKEY_LOCAL_MACHINE\SOFTWARE\Python\PythonCore\2.6\InstallPath. See the answer below for more details.


If you do this, beware that setuptools may only install 32-bit libraries."

I followed instructions in reference [1].
"Apparently, the setuptools msi is looking for the Python installation registry value InstallPath in HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Python\PythonCore\2.6\InstallPath. Notice the Wow6432Node, which is a registry compatibility layer used for 32-bit apps in Windows 7 64-bit.

As far as I can tell, InstallPath is the only value that this installer looks for. Therefore, using regedit, you can create your own registry value HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Python\PythonCore\2.6\InstallPath, and copy over the InstallPath value from HKEY_LOCAL_MACHINE\SOFTWARE\Python\PythonCore\2.6\InstallPath. To be paranoid, you can try replicating the entire cluster, not just InstallPath.

After this, installation for setuptools seems to proceed correctly. Note that this may only install 32-bit libraries -- WoW6432 is a compatibility layer. Check the other documented solution to this problem if this is not sufficient."



Following the instructions seems to solve the issue, and setuptools installation succeeded.

References:
1. http://selfsolved.com/problems/setuptools-06c11-fails-to-instal/s/63
2. http://stackoverflow.com/questions/3652625/installing-setuptools-on-64-bit-windows

Friday, March 15, 2013

Listing files containing all the words

To find files which contain all words in a set, you can use the below awk script. Here I am searching for files containing three words (word1, word2 and word3). The script output all filenames which contain all three words.

 find  -type f -exec awk 'BEGIN{word1=0;word2=0;word3=0}/word1/{word1++}/word2/{word2++}/word3/{word3++}END{if(word1>0 && word2>0 && word3>0){print FILENAME}}' {} \;

Reference:
1. http://www.linuxquestions.org/questions/linux-newbie-8/grep-an-entire-file-but-must-contain-multiple-words-705681/

Wednesday, March 13, 2013

Automatic/periodic FTP download using cron jobs

I came across a situation where I had to download files from an FTP server every week. Initially I was doing it manually, but due to human errors I missed some data. I then realized it must be possible to automate the download.

I initially created a shell script to enable FTP download[2]. The script looks like:

#!/bin/bash
HOST='ftp.server.com'   # change the ipaddress accordingly
USER='username'   # username also change
PASSWD='password'    # password also change
ftp -inv $HOST<<EOF
quote USER $USER
quote PASS $PASSWD
bin
cd /move/to/remote/directory        
lcd "/local/directory/" 
mget filename*
cd /move/to/remote/directory2
lcd "/local/directory2/"
mget filename*     
bye
EOF

Using [1], I setup a cron job using the command:
crontab -e

The job entry format is pretty self-explanatory in the reference [1], and there are some commonly used job examples too.
I had to launch a job at the beginning of every week, so my entry in the file looks like:
0 10 * * 1 ~/ftp_download_script.sh
This line states that the script should be launched at 10am every monday.

References:
1. http://www.cyberciti.biz/faq/how-do-i-add-jobs-to-cron-under-linux-or-unix-oses/
2. https://blogs.oracle.com/SanthoshK/entry/automate_ftp_download_using_sh

Sunday, March 10, 2013

Python - Value Error Unsupported Format Character

I had a simple python code working with URLs which caused an error of the form: "unsupported format character 'p' (0x70) at index 72".

The code looks like:
num = 1
abc = "http://<site>?value=[abc%20def],value2=%d"
URL = abc % (num)

This is caused because of using the % sign in the string. We need to escape the % sign with another % sign.
So the new string looks like:
abc = "http://<site>?value=[abc%%20def],value2=%d"

References:
1. http://yuji.wordpress.com/2009/01/09/python-valueerror-unsupported-format-character-percent-sign-python-format-string/

Friday, February 8, 2013

GNU plot CDF

you can plot CDF using the following gnuplot command. 

plot "data" u 1:(1./100.) smooth cumulative

Example: plot "data.txt" using $1:(1./100.) smooth cumulative title "CDF" with lines lw 3

References:
1. http://morforma.blogspot.com/2009/07/plotting-cumulative-distribution.html

Saturday, January 19, 2013

Passwordless SSH login

It is good to have authentication using SSH keys with password. However, when using scripts like multicopy/multiquery in Codeploy for simultaneously sending files/commands to multiple planetlab servers, it becomes a issue to keep retyping password.

We can use ssh-copy-id command to add our public key to authorized_keys on the target machine. This allows us to login without entering the passphrase when logging to the target server.

To do this, copy your public and private keys as id_rsa.pub and id_rsa in the ~/.ssh directory. Then type
ssh-copy-id -i ~/.ssh/id_rsa.pub username@remote_host

The reason why we need to copy public and private keys as id_rsa to .ssh directory is because there is no way (or I am not aware of) to specify the private key to use with ssh-copy-id command when logging to the remote server. The "-i" argument specifies the public key to be copied to authorized_keys on the target machine, and is not an equivalent of "-i" argument you use for ssh.

References:
1. http://everydaylht.com/howtos/system-administration/loggin-in-via-ssh-without-a-password/

Codeploy multicopy large files timedout ssh

When using multicopy command in Codeploy to transfer large files from/to to planetlab nodes, we might face "timedout ssh" problem. This can be avoided by setting up the environment variable MQ_TIMEOUT to be a large value like 100000. So along with MQ_SLICE and MQ_NODES, we need to set this third environment variable.

References:
1. http://lists.planet-lab.org/pipermail/users/2007-November/002599.html

Tuesday, January 15, 2013

Google NativeClient on Ubuntu

Initially I tried installing NativeClient (NaCl) on a Ubuntu 32bit VM, but some NaCl tests failed saying specific CPU features needed for NaCl are not available. But later on when setting up a 64bit Ubuntu VM I had to enable Intel VT-X features in BIOS, and this seemed to enable 64bit VM to pass NaCl tests.

Installing NaCl on a 32bit Ubuntu:
Follow the instructions here to download NaCl source using gclient/depot tools: http://code.google.com/p/nativeclient/wiki/Source?tm=4

Once you download everything to nativeclient, you need to run:
make

I had to install g++ libraries too:
sudo apt-get install g++

make failed with some errors:
#include<gelf.h> was throwing an error saying gelf.h was not found.

Installing gelf and libelf did not help.
- sudo apt-get install gelf
- installing libelf from Software center

I had to install libelf-dev from the Ubuntu software center, then the make succeeded.

After make, go to native_client sub directory and run
./scons MODE=opt-linux,nacl
Compiling was successful.
When testing NaCl, just try small_tests, medium_tests and large_tests. run_all_tests argument has some tests specific for 64bit linux, and hence is supposed to fail.

Installing NaCl on a 64bit Ubuntu:
Most of the instructions are same as for 32 bit. When doing a make, I got this error:
make: *** [out/Debug/obj/gen/tc_newlib/lib32/crti.o] Error 1

As per this post: https://groups.google.com/forum/#!topic/native-client-discuss/NJZ2JKWCCCY
we need to install the capability to run x86_32 binaries on a x86_64 machine.

So I had to install :
sudo apt-get install ia32-libs

Then make threw another error:
bits/predefs.h: No such file or directory

This thread gives the solution: http://ubuntuforums.org/showthread.php?t=1877944
we need 32bit libc. We can get it using:
sudo apt-get install libc6-dev-i386

I then got this error:
/usr/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-linux-gnu/4.7/libstdc++.so when searching for -lstdc++
/usr/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-linux-gnu/4.7/libstdc++.a when searching for -lstdc++
/usr/bin/ld: cannot find -lstdc++

As per this link, i had to install "g++-multilib"
http://ubuntuforums.org/archive/index.php/t-1604953.html
sudo apt-get install g++-multilib


run_all_tests  does not work unless you put platform=x86-64 as argument for scons

I had to install python-dev package when giving platform=x86-64 for scons.

Git to SVN

Sometimes cases might arise when you need to transfer projects from Git to SVN, like when you are required to maintain an in-house SVN repository of multiple external Git repositories. In my case I had to transfer two Git repositories (say git1 and git2) to SVN.

Since there are two Git repos, I wanted to set them as different projects in SVN repo. SVN project is simply a folder in the repo server (with trunk/branches/tags subfolders when following standard svn project format).

First step would be to checkout the Git repos to the system.
git clone https://github.com/your-project1 git1
git clone https://github.com/your-project2 git2

The next step would be to follow these set of instructions from each local Git directory to import them to SVN project.
1. cd /path_to/git1
2. svn mkdir --parents protocol:///path/to/repo/PROJECT1/trunk -m "Importing git repo"
3. git svn init protocol:///path/to/repo/PROJECT1 -s
4. git svn fetch
5. git rebase trunk
5.1.  git status
5.2.  git add (conflicted-files)
5.3.  git rebase --continue
5.4.  (repeat 5.1.)
6. git svn dcommit

In step-2, svn mkdir would create both the PROJECT1 and trunk folders and commits the change.
Here protocol:///path/to/repo/ is the http/https location of the SVN repo like http://IP/svn  or the local SVN repo like file:///home/user/svn-repo.

In step-3, "-s" indicates that project follows standard svn trunk/branches/tags folder structure. If not following the non-standard format, we can use --trunk, --branches and --tags flags. Following a non-standard folder structure is complicating the Git-to-SVN process. So prefer the standard structure.

After #3 you'll get a cryptic message like this:
Using higher level of URL: protocol:///path/to/repo/PROJECT1 => protocol:///path/to/repo
Just ignore that.

When you run #5, you might get conflicts. Resolve these by adding/removing files with state "unmerged" and resuming rebase. You might also need to solve content conflicts. To identify the conflicts when the rebase stops at a patch #, use the command:
git status

If there is CONFLICT(add/add) for a file or "added" message in the git status's contents in the unmerged section, then add the file using:
git add conflict-file

If there is a delete/modify message in the status, then we need to remove the file using:
git rm conflict-file

If there is content conflict, then we need to perform the following operations:
git checkout --theirs conflict-file
git add conflict-file
As per [3], when we perform git rebase trunk, our HEAD would be on the svn trunk branch and our git master commits would be on other branch. Hence we need to checkout the file from theirs branch and add the file to SVN.

After we resolve the conflicts as above, type command git rebase --continue.
Eventually, you'll be done; Then sync back to the svn-repo, using dcommit. That's all.

Keeping repos in sync

You can now sync from svn -> git, using the following commands:
git svn fetch
git rebase trunk
And to sync from git -> svn, use:
git svn dcommit

Final note

You might want to try this out on a local copy, before applying to a live repo. You can make a copy of your git-repo to a temporary place, simply using cp -r, as all data is in the repo itself. You can then set up a file-based testing repo, using:
svnadmin create /home/name/tmp/test-repo

And check a working copy out, using:
svn co file:///home/name/tmp/test-repo svn-working-copy

That'll allow you to play around with things before making any lasting changes.

Addendum: If you mess up git svn init

If you accidentally run git svn init with the wrong url, and you weren't smart enough to take a backup of your work (don't ask ...), you can't just run the same command again. You can however undo the changes by issuing:
rm -rf .git/svn
edit .git/config
And remove the section [svn-remote "svn"] section.

You can then run git svn init anew.

References:
1. http://stackoverflow.com/questions/661018/pushing-an-existing-git-repository-to-svn
2. http://stackoverflow.com/questions/8146289/git-how-to-get-theirs-in-the-middle-of-conflicting-rebase
3. http://stackoverflow.com/questions/2959443/why-is-the-meaning-of-ours-and-theirs-reversed-with-git-svn

APPENDIX
Another way of achieving the same would be using https://github.com/guilhermechapiewski/git2svn, where you run a script to transfer the commits from git to svn. But this completely shifts you to SVN and your SVN and Git are delinked. So future commits to SVN will not be reflected to Git and vice versa.





Monday, January 14, 2013

Get the path of a running jar file

When code is running inside a jar file, say foo.jar, and we need to know, in the code, in which folder the running foo.jar is use this command in the Java code.


 return new File(MyClass.class.getProtectionDomain().getCodeSource().getLocation().getPath());
Another of achieving the same is using the following code:
String path = Test.class.getProtectionDomain().getCodeSource().getLocation().getPath();
String decodedPath = URLDecoder.decode(path, "UTF-8");
Reference:
http://stackoverflow.com/questions/320542/how-to-get-the-path-of-a-running-jar-file