Dec 28 2007

I've been doing a lot of hacking on Pieforms recently, which has reminded me again of how annoying it is to work with Subversion. With the help of git-svn, I can now use git for revision control, which grants me the advantages of using git with the benefits of continuing to gain sourceforge activity percentile for coding activity. This technique will work for any project hosted at sourceforge, google code or any other code hosting where you're stuck with Subversion but would rather use git. While I'm sure it's well known to those people who know of git-svn, I thought I'd publish my experiences so others looking to do the same thing will have a reference point.

First, some background: Pieforms was up to revision 249 at the time I did this, which isn't a lot of history really. Due to a historic project decision, the code repository is not layed out with the standard Subversion branches/tags/trunk scheme - instead, these directories live inside a pieforms-php5/ directory (in theory, so implementations in other languages can be created). Thankfully, this doesn't really complicate issues.

The first thing I did was created an authors mapping file. This is a file that maps subversion usernames to the names that will be used in git. For Pieforms, this was easy:

nigel@bourdon:~/work/pieforms$ echo 'oracleshinoda = Nigel McNie <nigel@mcnie.name>' > authors.txt

Then, I cloned the repo with git svn clone. Read the man page for more information about the options, and remember the -A option should specify the authors mapping file you created previously:

nigel@bourdon:~/work/pieforms$ time git svn clone -t pieforms-php5/tags -b pieforms-php5/branches \
-T pieforms-php5/trunk -A authors.txt --username oracleshinoda https://pieforms.svn.sourceforge.net/svnroot/pieforms
Initialized empty Git repository in .git/
        A       INSTALL
        A       src/pieform/renderers/multicolumntable.php
        A       src/pieform/renderers/div.php
        A       src/pieform/renderers/table.php
        A       src/pieform/rules/required.php
        A       src/pieform/rules/minlength.php
        A       src/pieform/rules/validateoptions.php
        A       src/pieform/rules/email.php
        A       src/pieform/rules/maxlength.php
        A       src/pieform/elements/hidden.php
        A       src/pieform/elements/radio.php
        A       src/pieform/elements/fieldset.php
        A       src/pieform/elements/password.php
        A       src/pieform/elements/submit.php
        A       src/pieform/elements/html.php
        A       src/pieform/elements/submitcancel.php
        A       src/pieform/elements/button.php
        A       src/pieform/elements/textarea.php
        A       src/pieform/elements/date.php
        A       src/pieform/elements/select.php
        A       src/pieform/elements/file.php
        A       src/pieform/elements/expiry.php
        A       src/pieform/elements/wysiwyg.php
        A       src/pieform/elements/text.php
        A       src/pieform/elements/cancel.php
        A       src/pieform/elements/checkbox.php
        A       src/pieform.php
        A       src/JSON/JSON.php
        A       src/JSON/LICENSE
        A       src/js/MochiKit/LICENSE.txt
        A       src/js/MochiKit/MochiKit.js
        A       COPYING
        A       README
W: +empty_dir: pieforms-php5/trunk/doc
r48 = ac4f97630529b20cdb59edfacb0eac224657ad74 (trunk)
Found possible branch point: https://pieforms.svn.sourceforge.net/svnroot/pieforms/pieforms-php5/trunk => https://pieforms.svn.sourceforge.net/svnroot/pieforms/pieforms-php5/tags/release-0.1.0, 49
Found branch parent: (tags/release-0.1.0) ac4f97630529b20cdb59edfacb0eac224657ad74
Following parent with do_switch
Successfully followed parent
r50 = e50c12d19585bb1eaacfe8f1d8d9f029a1621d11 (tags/release-0.1.0)
        M       src/pieform/rules/email.php
r52 = e077be23cf127684b0cda6a0365a9809bee25b33 (trunk)
        A       src/pieform/rules/integer.php
        M       src/pieform.php
r53 = b7a3c80055b52706bbad4a2fdf84f234db043bcd (trunk)
        M       src/pieform.php
r54 = 9116b59ea3ff6f20411bb0bb71ea5b48f9e6ae3e (trunk)

...

        M       src/pieform.php
r248 = 5b03c757f682c93bfaeebbcf09aa596b08280146 (trunk)
        M       src/pieform.php
r249 = 845a210e97f2e791eb907145a528148b697decc5 (trunk)
Checked out HEAD:
  https://pieforms.svn.sourceforge.net/svnroot/pieforms/pieforms-php5/trunk r249

real    3m42.527s
user    0m5.460s
sys     0m8.533s
nigel@bourdon:~/work/pieforms$

As you can see, just ~250 revisions took nearly four minutes to import. When I ran this, my download speed was ~700K/s. No doubt the math will be different for you, but be prepared for a wait regardless.

After the import, we tell git-svn about the authors mapping file permanently, and have a look at the history:

nigel@bourdon:~/work/pieforms$ cd pieforms
nigel@bourdon:~/work/pieforms/pieforms$ git config svn.authorsfile /home/nigel/work/pieforms/authors.txt
nigel@bourdon:~/work/pieforms/pieforms$ gitk --all
What gitk shows for the imported Pieforms project

After that, things are easy. You commit your changes with git commit as usual, and push changes to the SVN repository with git svn dcommit. In my case, I should never need to pull from the repository, but if I did I would use git svn rebase.

Setting up a Remote Repository

Now we're using git, we can set up a remote repository and push to it when we git svn dcommit. I'm no git ninja, so I have little idea on how to set things up so you or others can push to the remote and then you merge changes to the SVN repository (i.e. I'm not sure how to make the remote the "master" repository), but if I figure it out then I'll update this document. Though one of the nice things about it being readonly is that I can blow it away and re-do the process if I find a better way to do it.

These instructions are for Debian Etch (with a 1.5 git backport) or newer, they might work on Ubuntu too.

You have to set up the server so that it can be pushed to/have gitweb etc. For git.nigel.mcnie.name, which is hosted on an etch box, I did the following:

hahn:~# vim /etc/apt/sources.list
hahn:~# cat /etc/apt/sources.list
deb http://ftp.us.debian.org/debian etch main contrib non-free
deb http://security.debian.org/ etch/updates main contrib non-free

# For git-core 1.5+ and other backports if required
deb http://www.backports.org/debian etch-backports main contrib non-free

hahn:~# apt-get update
<snip>
hahn:~# apt-get install debian-backports-keyring
<snip>

hahn:~# vim /etc/apt/preferences
hahn:~# cat /etc/apt/preferences
Package: git-core
Pin: release a=etch-backports
Pin-Priority: 999

Package: gitweb
Pin: release a=etch-backports
Pin-Priority: 999

hahn:~# apt-get install git-core curl gitweb
<snip>

At this point, I noted that gitweb's README.Debian was suggesting that the repositories be stored in /var/cache/git, which seems like a violation of the Debian FHS. I started to file a bug and then realised I couldn't send e-mail from where I was, so gave up. A more sensible location is /srv/git:

hahn:~# mkdir /srv/git
hahn:~# addgroup git
<snip>
hahn:~# adduser nigel git
<snip>
hahn:~# chgrp git /srv/git
hahn:~# chmod g+s /srv/git
hahn:~# ln -s /srv/git /git

Now all that is done, it's easy enough to set up remotes to push to:

hahn:~# export GIT_DIR=/srv/git/pieforms-php5.git
hahn:~# git init --shared=group
<snip>
hahn:~# chmod +x $GIT_DIR/hooks/post-update
hahn:~# vim $GIT_DIR/description
hahn:~# unset GIT_DIR

Whew! Now we can tell our local git repository about the remote, and do the first push to it:

nigel@bourdon:~/work/pieforms/pieforms$ git remote add origin git+ssh://git.nigel.mcnie.name/git/pieforms-php5.git
nigel@bourdon:~/work/pieforms/pieforms$ git push origin master:refs/heads/master
updating 'refs/heads/master'
  from 0000000000000000000000000000000000000000
  to   845a210e97f2e791eb907145a528148b697decc5
 Also local refs/remotes/origin/master
Generating pack...
Done counting 1183 objects.
Deltifying 1183 objects...
 100% (1183/1183) done
Writing 1183 objects...
 100% (1183/1183) done
Total 1183 (delta 695), reused 0 (delta 0)
refs/heads/master: 0000000000000000000000000000000000000000 -> 845a210e97f2e791eb907145a528148b697decc5
nigel@bourdon:~/work/pieforms/pieforms$

Now my workflow is as before, but after doing a git svn dcommit I also do a git push so the remote is up to date.

My limited experience thus far is that you shouldn't mess too much with the ordering of things - git commit whatever you are committing (in as many stages as you like), then when you're done, do a git svn dcommit then a git push. And never git pull!

Back on the server, install openbsd-inetd if it is not installed already. Add the following line to /etc/inetd.conf and then restart it:

git     stream  tcp     nowait  nobody  /usr/bin/git-daemon git-daemon --inetd --verbose --export-all --base-path=/srv/git /srv/git

Drop the verbose when you know things are working.

Then cloning works fine:

nigel@bourdon:~$ git clone git+ssh://git.nigel.mcnie.name/git/pieforms-php5.git pieforms-git+ssh
Initialized empty Git repository in /home/nigel/temp/pieforms-git+ssh/.git/
remote: Generating pack...
remote: Done counting 1183 objects.
remote: Deltifying 1183 objects...
remote:  100% (1183/1183) done
Indexing 1183 objects...
remote: Total 1183 (delta 695), reused 1183 (delta 695)
 100% (1183/1183) done
Resolving 695 deltas...
 100% (695/695) done
nigel@bourdon:~$

Non-ssh based cloning:

nigel@bourdon:~$ git clone git://git.nigel.mcnie.name/pieforms-php5.git pieforms-git
Initialized empty Git repository in /home/nigel/temp/pieforms-git/.git/
remote: Generating pack...
remote: Done counting 1183 objects.
remote: Deltifying 1183 objects...
remote:  100% (1183/1183) done
Indexing 1183 objects...
remote: Total 1183 (delta 695), reused 1183 (delta 695)
 100% (1183/1183) done
Resolving 695 deltas...
 100% (695/695) done
nigel@bourdon:~$

If we set up a virtualhost for apache and muck around in /etc/gitweb.conf, we can have gitweb and HTTP checkouts working. Add the following virtualhost to apache:

<VirtualHost *>
    ServerName git.nigel.mcnie.name
    ServerAdmin nigel@mcnie.name
    DocumentRoot /srv/git
    ErrorLog /var/log/apache2/git.nigel.mcnie.name-error.log
    CustomLog /var/log/apache2/git.nigel.mcnie.name-access.log combined

    Alias /gitweb.css /var/www/gitweb.css
    Alias /git-favicon.png /var/www/git-favicon.png
    Alias /git-logo.png /var/www/git-logo.png
    Alias /git /srv/git

    ScriptAlias / /usr/lib/cgi-bin/gitweb.cgi

    <Directory /srv/git>
        AllowOverride None
        Options Indexes FollowSymlinks
        Order allow,deny
        Allow from all
    </Directory>
</VirtualHost>

And edit /etc/gitweb.conf to change the $projectroot to "/srv/git".

After restarting apache, HTTP checkouts should work fine, and gitweb should be working:

nigel@bourdon:~$ git clone http://git.nigel.mcnie.name/pieforms-php5.git pieforms-http
Initialized empty Git repository in /home/nigel/temp/pieforms-http/.git/
Getting alternates list for http://git.nigel.mcnie.name/pieforms-php5.git
Getting pack list for http://git.nigel.mcnie.name/pieforms-php5.git
Getting index for pack a9a21ed56e44f62c809b18755e056306caeb6692
Getting pack a9a21ed56e44f62c809b18755e056306caeb6692
 which contains 845a210e97f2e791eb907145a528148b697decc5
walk 845a210e97f2e791eb907145a528148b697decc5
walk 5b03c757f682c93bfaeebbcf09aa596b08280146
walk 544b58f82ced2e06fb06e06c4b65c5ce9e4c1811
...
walk b7a3c80055b52706bbad4a2fdf84f234db043bcd
walk e077be23cf127684b0cda6a0365a9809bee25b33
walk ac4f97630529b20cdb59edfacb0eac224657ad74
nigel@bourdon:~$

Congratulations, you're done!

What have you gained:

  • Ability to use git for your sourceforge/google code/other project which is stored in SVN
  • Continued gain of activity percentile for commit activity (at least in the case of sourceforge, I don't know about google code)

Of course, someone more enterprising than myself could probably come up with a way to have the git repository as the 'master' (which you could create initially with git-svnimport, and then commit to the "slave" SVN repository from time to time. I'd love to hear about it if you know how to do it. This method will do me for now.

Sam Vilain's Introduction to git-svn for Subversion/SVK users and deserters was helpful to me while working through this, as was the man himself.

Like this post? Subscribe to my RSS feed and follow me on twitter to hear about new posts early.

Want to share this post?

blog comments powered by Disqus