Saturday, January 19, 2013

Passwordless SSH login

It is good to have authentication using SSH keys with password. However, when using scripts like multicopy/multiquery in Codeploy for simultaneously sending files/commands to multiple planetlab servers, it becomes a issue to keep retyping password.

We can use ssh-copy-id command to add our public key to authorized_keys on the target machine. This allows us to login without entering the passphrase when logging to the target server.

To do this, copy your public and private keys as id_rsa.pub and id_rsa in the ~/.ssh directory. Then type
ssh-copy-id -i ~/.ssh/id_rsa.pub username@remote_host

The reason why we need to copy public and private keys as id_rsa to .ssh directory is because there is no way (or I am not aware of) to specify the private key to use with ssh-copy-id command when logging to the remote server. The "-i" argument specifies the public key to be copied to authorized_keys on the target machine, and is not an equivalent of "-i" argument you use for ssh.

References:
1. http://everydaylht.com/howtos/system-administration/loggin-in-via-ssh-without-a-password/

Codeploy multicopy large files timedout ssh

When using multicopy command in Codeploy to transfer large files from/to to planetlab nodes, we might face "timedout ssh" problem. This can be avoided by setting up the environment variable MQ_TIMEOUT to be a large value like 100000. So along with MQ_SLICE and MQ_NODES, we need to set this third environment variable.

References:
1. http://lists.planet-lab.org/pipermail/users/2007-November/002599.html

Tuesday, January 15, 2013

Google NativeClient on Ubuntu

Initially I tried installing NativeClient (NaCl) on a Ubuntu 32bit VM, but some NaCl tests failed saying specific CPU features needed for NaCl are not available. But later on when setting up a 64bit Ubuntu VM I had to enable Intel VT-X features in BIOS, and this seemed to enable 64bit VM to pass NaCl tests.

Installing NaCl on a 32bit Ubuntu:
Follow the instructions here to download NaCl source using gclient/depot tools: http://code.google.com/p/nativeclient/wiki/Source?tm=4

Once you download everything to nativeclient, you need to run:
make

I had to install g++ libraries too:
sudo apt-get install g++

make failed with some errors:
#include<gelf.h> was throwing an error saying gelf.h was not found.

Installing gelf and libelf did not help.
- sudo apt-get install gelf
- installing libelf from Software center

I had to install libelf-dev from the Ubuntu software center, then the make succeeded.

After make, go to native_client sub directory and run
./scons MODE=opt-linux,nacl
Compiling was successful.
When testing NaCl, just try small_tests, medium_tests and large_tests. run_all_tests argument has some tests specific for 64bit linux, and hence is supposed to fail.

Installing NaCl on a 64bit Ubuntu:
Most of the instructions are same as for 32 bit. When doing a make, I got this error:
make: *** [out/Debug/obj/gen/tc_newlib/lib32/crti.o] Error 1

As per this post: https://groups.google.com/forum/#!topic/native-client-discuss/NJZ2JKWCCCY
we need to install the capability to run x86_32 binaries on a x86_64 machine.

So I had to install :
sudo apt-get install ia32-libs

Then make threw another error:
bits/predefs.h: No such file or directory

This thread gives the solution: http://ubuntuforums.org/showthread.php?t=1877944
we need 32bit libc. We can get it using:
sudo apt-get install libc6-dev-i386

I then got this error:
/usr/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-linux-gnu/4.7/libstdc++.so when searching for -lstdc++
/usr/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-linux-gnu/4.7/libstdc++.a when searching for -lstdc++
/usr/bin/ld: cannot find -lstdc++

As per this link, i had to install "g++-multilib"
http://ubuntuforums.org/archive/index.php/t-1604953.html
sudo apt-get install g++-multilib


run_all_tests  does not work unless you put platform=x86-64 as argument for scons

I had to install python-dev package when giving platform=x86-64 for scons.

Git to SVN

Sometimes cases might arise when you need to transfer projects from Git to SVN, like when you are required to maintain an in-house SVN repository of multiple external Git repositories. In my case I had to transfer two Git repositories (say git1 and git2) to SVN.

Since there are two Git repos, I wanted to set them as different projects in SVN repo. SVN project is simply a folder in the repo server (with trunk/branches/tags subfolders when following standard svn project format).

First step would be to checkout the Git repos to the system.
git clone https://github.com/your-project1 git1
git clone https://github.com/your-project2 git2

The next step would be to follow these set of instructions from each local Git directory to import them to SVN project.
1. cd /path_to/git1
2. svn mkdir --parents protocol:///path/to/repo/PROJECT1/trunk -m "Importing git repo"
3. git svn init protocol:///path/to/repo/PROJECT1 -s
4. git svn fetch
5. git rebase trunk
5.1.  git status
5.2.  git add (conflicted-files)
5.3.  git rebase --continue
5.4.  (repeat 5.1.)
6. git svn dcommit

In step-2, svn mkdir would create both the PROJECT1 and trunk folders and commits the change.
Here protocol:///path/to/repo/ is the http/https location of the SVN repo like http://IP/svn  or the local SVN repo like file:///home/user/svn-repo.

In step-3, "-s" indicates that project follows standard svn trunk/branches/tags folder structure. If not following the non-standard format, we can use --trunk, --branches and --tags flags. Following a non-standard folder structure is complicating the Git-to-SVN process. So prefer the standard structure.

After #3 you'll get a cryptic message like this:
Using higher level of URL: protocol:///path/to/repo/PROJECT1 => protocol:///path/to/repo
Just ignore that.

When you run #5, you might get conflicts. Resolve these by adding/removing files with state "unmerged" and resuming rebase. You might also need to solve content conflicts. To identify the conflicts when the rebase stops at a patch #, use the command:
git status

If there is CONFLICT(add/add) for a file or "added" message in the git status's contents in the unmerged section, then add the file using:
git add conflict-file

If there is a delete/modify message in the status, then we need to remove the file using:
git rm conflict-file

If there is content conflict, then we need to perform the following operations:
git checkout --theirs conflict-file
git add conflict-file
As per [3], when we perform git rebase trunk, our HEAD would be on the svn trunk branch and our git master commits would be on other branch. Hence we need to checkout the file from theirs branch and add the file to SVN.

After we resolve the conflicts as above, type command git rebase --continue.
Eventually, you'll be done; Then sync back to the svn-repo, using dcommit. That's all.

Keeping repos in sync

You can now sync from svn -> git, using the following commands:
git svn fetch
git rebase trunk
And to sync from git -> svn, use:
git svn dcommit

Final note

You might want to try this out on a local copy, before applying to a live repo. You can make a copy of your git-repo to a temporary place, simply using cp -r, as all data is in the repo itself. You can then set up a file-based testing repo, using:
svnadmin create /home/name/tmp/test-repo

And check a working copy out, using:
svn co file:///home/name/tmp/test-repo svn-working-copy

That'll allow you to play around with things before making any lasting changes.

Addendum: If you mess up git svn init

If you accidentally run git svn init with the wrong url, and you weren't smart enough to take a backup of your work (don't ask ...), you can't just run the same command again. You can however undo the changes by issuing:
rm -rf .git/svn
edit .git/config
And remove the section [svn-remote "svn"] section.

You can then run git svn init anew.

References:
1. http://stackoverflow.com/questions/661018/pushing-an-existing-git-repository-to-svn
2. http://stackoverflow.com/questions/8146289/git-how-to-get-theirs-in-the-middle-of-conflicting-rebase
3. http://stackoverflow.com/questions/2959443/why-is-the-meaning-of-ours-and-theirs-reversed-with-git-svn

APPENDIX
Another way of achieving the same would be using https://github.com/guilhermechapiewski/git2svn, where you run a script to transfer the commits from git to svn. But this completely shifts you to SVN and your SVN and Git are delinked. So future commits to SVN will not be reflected to Git and vice versa.





Monday, January 14, 2013

Get the path of a running jar file

When code is running inside a jar file, say foo.jar, and we need to know, in the code, in which folder the running foo.jar is use this command in the Java code.


 return new File(MyClass.class.getProtectionDomain().getCodeSource().getLocation().getPath());
Another of achieving the same is using the following code:
String path = Test.class.getProtectionDomain().getCodeSource().getLocation().getPath();
String decodedPath = URLDecoder.decode(path, "UTF-8");
Reference:
http://stackoverflow.com/questions/320542/how-to-get-the-path-of-a-running-jar-file