Backup and Synchronization with Git

Posted by Ryan Coyner on November 24, 2008 under Git - 3 Comments

I have a desktop and a laptop both running Arch Linux. The desktop used to run Ubuntu, but I was a little frustrated with its bloat and complexity. Arch is faster, leaner and easier to configure thanks to its simple model. Being a power user, it didn't make sense for me to keep Ubuntu.

I've started to use my desktop more regularly since installing Arch on it. This eventually became a problem because when I am on my desktop there is bound to be a file on my laptop that I desperately need, and vice versa. No, not porn. Usually homework assignments or a critical piece of a project I'm working on.

I was determined to solve this problem because it hindered my productivity significantly. The first solution that immediately came to mind was rsync. I did a little reading up on it and concluded that it wasn't the tool for me. In order for the synchronization with rsync to be practical you have to synchronize at the end of every session because rsync doesn't handle conflicting merges like a VCS does. When I'm working from my laptop I'm not guaranteed to be connected to the net to complete synchronization. If I don't complete synchronization at the end of each session, rsync won't be able to include separate modifications I make to a single file from different machines when I synchronize later.

So why the hell not just use a VCS? There is no reason not to, so that's exactly what I did - I put my entire home directory into a Git repository. Not only does it give me fast and reliable synchronization, I also get version control. Sweet.

Setting Up a Remote Repository

I hosted the repository on my server and setup a remote repository:

$ mkdir /var/git/ryan.git
$ cd /var/git/ryan.git
$ git --bare init

Depending on your system setup you may have to change some ownerships and privileges.

Adding Only Necessary Files

I initialized an empty repository from one of my machine's home directory:

$ git init

And added the remote repository:

$ git remote add origin ssh://username@serveraddress:port/var/git/ryan.git

At this point I had to be careful as to what I added. I didn't want to add any directories that are already under version control because that would be redundant. The directory containing all my music was also excluded; otherwise the repository would have become insanely large and exploded. I also had to be careful as to which hidden files I added. Some hidden files contain hardware information, cached information, or information specific to a machine that I didn't want to synchronize. I listed those files and directories in .gitignore:

.Xauthority
.bash_history
.config/epdfview/
.config/gtk-2.0/
.easytag/
.dbus/
.fehbg
.fontconfig/
.gegl-0.0/
.gimp-2.6/
.java/
.lesshst
.local/
.macromedia/
.mozilla/
.mpd/
.openoffice.org/
.opera/
.recently-used
.recently-used.xbel
.ssh/
.subversion/
.thumbnails/
.vimbackup/
.viminfo
/media/music/
/projects

Now add the entire home directory and commit:

$ git add .
$ git commit

To make committing simple, rename .git/hooks/prepare-commit-msg.sample to .git/hooks/prepare-commit-msg and append this at the end of the file:

NODE=`uname -n`
sed -i "1 s/^/$NODE/" "$1"

This will put the machine name at the beginning of the commit message. I'm too lazy to write a proper commit message each time just for synchronization, and you can't commit without writing a message. This lets me commit without having to have to type a single letter.

Synchronization

Once it's been committed, push to the repository:

$ git push origin master

From the other machine, clone the repository:

$ git clone ssh://username@serveraddress:port/var/git/ryan.git

If you've already cloned it and just want to update it, pull instead:

$ git pull origin master

I've used this setup for about two weeks now and it's been working great. I've been consistent about pushing at the end of my sessions and pulling at the beginning of my sessions so I haven't faced any major file conflicts yet.

Pros

Cons

Configuring Special Keys on a ThinkPad T60 (Arch Linux)

Posted by Ryan Coyner on November 4, 2008 under Arch Linux - 0 Comments

Life has kept me on my toes the last month but I finally managed to find some time to hack around on my laptop. I never bothered to configure the function key combinations on my ThinkPad T60 so I spent a couple of hours setting that up.

When I installed Arch Linux back in May, it automatically detected and loaded the sound and ACPI modules snd-hda-intel and thinkpad_acpi, respectively. Without any hacking, the following buttons and function key combinations worked thanks to the automated setup:

That left only a few special keys and function key combinations that I had to manually configure for my own needs. More specifically, I wanted to:

  1. Get the forward and back buttons to flip through the workspaces in Fluxbox.
  2. Get the function key combination for sleep and hibernation to work.
  3. Get the function key combination for toggling bluetooth to work since disabling bluetooth saves some battery power.

Forward and Back Buttons

Getting the Forward and Back buttons to work was simple. I first looked up the keycodes for both keys by running xev and pressing the buttons. For instance, the Back key returned this:

KeyRelease event, serial 34, synthetic NO, window 0x1a00001,
    root 0x5b, subw 0x0, time 30475365, (377,169), root:(622,439),
    state 0x0, keycode 234 (keysym 0xffd0, F19), same_screen YES,
    XLookupString gives 0 bytes: 
    XFilterEvent returns: False

The keymap needs to be modified so that the keycode is mapped to a specific key. This can be done by creating a ~/.xmodmaprc file. In my case the keycodes need to be mapped to F19 and F20 which are the Forward and Back keys. My ~/.xmodmaprc file contains the following:

keycode 234 = F19
keycode 233 = F20

You need to tell Fluxbox to automatically load this file when it starts up. Add this in your ~/.fluxbox/startup file:

xmodmap ~/.xmodmaprc
exec /usr/bin/fluxbox

Add these declarations in ~/.fluxbox/keys:

None F19 :PrevWorkspace
None F20 :NextWorkspace

Restart Fluxbox and the keys should work!

Bluetooth, Sleep and Hibernation

Getting sleep and hibernation to work was not difficult but was a little bit more involved. First, install the following packages via pacman:

$ sudo pacman -S acpid hal pm-utils

acpid is a daemon that listens for ACPI events in the /proc/acpi/event file and handles the event. I manually configured the handler script to get the function key cominations to work.

pm-utils provides the programs for sleeping and hibernating but requires hal for it to work properly. I don't think pacman recognized the dependency so you will have to manually tell pacman to download it.

With acpid and hal installed, they should always be started when the system boots. Add them into the DAEMONS array in rc.conf:

 
DAEMONS=(syslog-ng !network netfs @crond @alsa @postgresql @acpid @hal)

For hibernation to work, the boot loader needs to know which partition to restore the current session from when waking up. Hibernation places the current session in the swap partition, so figure out the name of your swap partition and add the following in /boot/grub/menu.lst if you are using GRUB:

# (0) Arch Linux
title  Arch Linux
root   (hd0,1)
kernel /vmlinuz26 root=/dev/disk/by-uuid/d4a15d35-8c13-4971-939b-884e93d7fc01 resume=/dev/sda1 ro
initrd /kernel26.img

At this point your computer should be able to sleep and hibernate. You can test this by running pm-suspend and pm-hibernate respectively.

Now the ACPI handler script needs to be modified so that it recognizes the ACPI calls made by the function key combinations for sleeping and hibernating. Make the following additions in /etc/acpi/handler.sh:

    ibm/hotkey)
        case "$4" in
            00001004)
                logger "acpid: sleep button (Fn+F4) pressed; suspending"
                /usr/sbin/pm-suspend
            ;;
            00001005)
                logger "acpid: bluetooth button (Fn+F5) pressed; toggling bluetooth"
                if grep -q enabled /proc/acpi/ibm/bluetooth ; then
                    echo disable > /proc/acpi/ibm/bluetooth
                else
                    echo enable > /proc/acpi/ibm/bluetooth
                fi
            ;;
            0000100c)
                logger "acpid: hibernate button (Fn+F12) pressed; hibernating"
                /usr/sbin/pm-hibernate
            ;;
            *)
                logger "acpid: action undefined: $4"
                ;;
        esac
        ;;

The function key combinations should now work.

Other Keys

Some other function key combinations that are on the ThinkPad T60 that I haven't touched are:

I may configure these sometime in the future but I have no immediate need for them. Here are a list of events if you want to do it yourself.

Building Packages in Arch Linux

Posted by Ryan Coyner on September 16, 2008 under Arch Linux - 2 Comments

Recently I have been becoming more involved in the Arch Linux community. I have adopted a number of packages in the AUR and am seriously thinking about applying to become a Trusted User. I'm specifically interested in improving the AUR which can certainly use some additional features. Also, the development of AUR2 is in Django which is right up my alley.

Anyhow, I noticed while I was learning how to build packages that a lot of good how to information on building them is scattered all over the Wiki. It is a little disorganized and there is a lot of repeated, duplicate information. Although this is not a huge problem, learning how to build packages was a little confusing and overwhelming at first. I much prefer having a single, detailed reference that I can consistently refer to. This post is my attempt at creating such a reference. Most of this information is adapted from the Wiki.

Setting Up

First, verify that you have the necessary tools installed. You need the following packages:

If you do not have them, install them using pacman:

$ pacman -S abs base-devel fakeroot

One of the key tools for building packages is makepkg. It does the following:

  1. Checks if package dependencies are installed.
  2. Downloads the source file(s) from the specified server(s).
  3. Unpacks the source file(s).
  4. Compiles the software and installs it under a fakeroot environment.
  5. Strips symbols from binaries and libraries.
  6. Generates the package meta file which is included with each package.
  7. Compress the fakeroot environment into a package file.
  8. Stores the package file in the configured destination directory, which is the present working directory by default.

abs is optional to use, but it gives you the option to build official packages directly from source instead of downloading the pre-built, binary copy. It also helps you keep all of your PKGBUILD files, both official and unofficial in a centralized location. A PKGBUILD file contains metadata about a package, and is the essential component for building packages. You can specify which repositories to include in your ABS tree in the /etc/abs.conf file. To obtain the ABS tree which contains all of the PKGBUILD files, just run the abs command as root:

$ abs

Although you can build packages in any directory, the most flexible and organized place to build them is in /var/abs/local. /var/abs/local is owned by root:root, but a normal user needs write access to the directory to be able to run makepkg on a PKGBUILD file. To accomplish this and maintain security, create an abs group and make yourself a member of that group:

$ groupadd abs
$ usermod -G abs [user]

If you are already logged in, log out for the changes to take place. Next, change the group ownership of the /var/abs/local directory to abs:

$ chown root:abs /var/abs/local
$ chmod 775 /var/abs/local

Preparing the Files

Now download the source tarball of the software you want to package, extract it, and note all commands needed to compile and install it. You will be repeating the same commands in the PKGBUILD file. Most software authors stick to the 3-step build cycle:

./configure
make
make install

When you run makepkg, it will look for a PKGBUILD file in the present working directory. If a PKGBUILD file is found it will download the software's source code and compile it according to the instructions specified in the PKGBUILD file. The instructions must be fully interpretable by Bash. After successful compilation, the resulting binaries and metadata of the package, i.e. package version and dependencies, are packed in a pkgname.pkg.tar.gz package file that can be cleanly installed with pacman -Up [package file].

To start out with a new package, you should first create an empty working directory, preferably named /var/abs/local/pkgname. That way it's nicely integrated into the normal ABS-tree and unaltered whenever you synchronize the tree. Once you make the directory and change into it, create a PKGBUILD file to work with either by copying the prototype from /usr/share/pacman/PKGBUILD.proto or by copying a PKGBUILD from another package. The latter is quite useful if you want to modify compilation options of a package instead of creating an entirely new one.

Defining PKGBUILD Variables

The following are variables that can be filled out in the PKGBUILD file:

pkgname
The name of the package. It should consist of alphanumeric characters only and all letters should be lowercase. For the sake of consistency, pkgname should match the name of the source tarball of the software you are packaging. For instance, if the software is in foobar-2.5.tar.gz, the pkgname value should be foobar. The present working directory the PKGBUILD file is in should also match the pkgname.
pkgver
The version of the package. The value should be the same as the version released by the author of the package. It can contain letters, numbers and periods but CANNOT contain a hyphen. If the author of the package uses a hyphen in their version numbering scheme, replace it with an underscore. For instance, if the version is 0.99-10, it should be changed to 0.99_10.
pkgrel
The release number of the package specific to Arch Linux. This value allows users to differentiate between consecutive builds of the same version of a package. When a new package version is first released, the release number starts at 1. As fixes and optimizations are made to the PKGBUILD file, the package will be re-released and the release number will increment by 1. When a new version of the package comes out, the release number resets to 1.
pkgdesc
The description of the package. The description should be about 80 character or less and should not include the package name in a self-referencing way. For instance, "Nedit is a text editor for X11" should be written as "A text editor for for X11."
arch
An array of architectures that the PKGBUILD file is known to build and work on. Currently, it should contain i686 and/or x86_64. The value any can also be used for independent packages, but the official repositories of Arch does not support this value.
url
The URL of the official site of the software being packaged.
licenses
The license under which the software is distributed. A licenses package has been created in [core] that stores common licenses in /usr/share/licenses/common, i.e. /usr/share/licenses/common/GPL. If a package is licensed under one of these licenses, the value should be set to the directory name, e.g. license=('GPL'). If the appropriate license is not included in the official licenses package, several things must be done:
  1. The license file(s) should be included in: /usr/share/licenses/pkgname/, e.g. /usr/share/licenses/foobar/LICENSE.
  2. If the source tarball does NOT contain the license details and the license is only displayed elsewhere, e.g. a website, then you need to copy the license to a file and include it.
  3. Add custom to the licenses array. Optionally, you can replace custom with custom:name of license. Once a license is used in two or more packages in an official repository (including [community]), it becomes a part of the licenses package.
The BSD, MIT, zlib/libpng and Python licenses are special cases and could not be included in the licenses package. For the sake of the license array, it is treated as a common license (license=('BSD'), license=('MIT'), license=('ZLIB') and license=('Python')) but technically each one is a custom license because each one has its own copyright line. Any packages licensed under these four should have its own unique license stored in /usr/share/licenses/pkgname. Some packages may not be covered by a single license. In these cases, multiple entries may be made in the license array, e.g. license=('GPL' 'custom:name of license'). Additionally, the (L)GPL has many versions and permutations of those versions. For (L)GPL software, the convention is:
  • (L)GPL - (L)GPLv2 or any later version
  • (L)GPL2 - (L)GPL2 only
  • (L)GPL3 - (L)GPL3 or any later version
groups
The group the package belongs in. For instance, when you install the kdebase package, it installs all packages that belongs in the kde group.
depends
An array of package names that must be installed before this software can be run. If a software requires a minimum version of a dependency, the >= operator should be used to point this out, e.g. depends=('foobar>=1.8.0'). You do not need to list packages that your software depends on if other packages your software depends on already have those packages listed in their dependency. For instance, gtk2 depends on glib2 and glibc. However, glibc does not need to be listed as a dependency for gtk2 because it is a dependency for glib2.
makedepends
An array of package names that must be installed to build the software but unnecessary for using the software after installation. You can specify the minimum version dependency of the packages in the same format as the depends array.
optdepends
An array of package names that are not needed for the software to function but provides additional features. A short description of what each package provides should also be noted. An optdepends may look like this: 
optdepends=('cups: printing support'
            'sane: scanners support'
            'libgphoto2: digital cameras support'
            'alsa-lib: sound support'
            'giflib: GIF images support'
            'libjpeg: JPEG images support'
            'libpng: PNG images support')
provides
An array of package names that this package provies the features of (or a virtual package such as cron or sh). If you use this variable, you should add the version (pkgver and perhaps the pkgrel) that this package will provide if dependencies may be affected by it. For instance, if you are providing a modified qt package named qt-foobar version 3.3.8 which provides qt then the provides array should look like provides=('qt=3.3.8'). Putting provides=('qt') will cause dependencies that require a specific version of qt to fail.
conflicts
An array of package names that may cause problems with this package if installed. You can also specify the version properties of the conflicting packages in the same format as the depends array.
replaces
An array of obsolete package name that are replaced by this package, e.g. replaces=('ethereal') for the wireshark package. After syncing with pacman -Sy, it will immediately replace an installed package upon encountering another package with the matching replaces in the repositories. If you are providing an alternate version of an already existing package, use the conflicts variable which is only evaluated when actually installing the conflicting package.
backup
An array of files to be backed up as file.pacsave when the package is removed. This is commonly used for packages placing configuration files in /etc.
options
This array allows you to override some of the default behavior of makepkg. To set an option, include the option name in the array. To reverse the default behavior, place an ! at the front of the option. The following options may be placed in the array:
  • strip - Strips symbols from binaries and libraries.
  • docs - Save /doc directories.
  • libtool - Leave libtool (.la) files in packages.
  • emptydirs - Leave empty directories in packages.
  • zipman - Compress man and info pages with gzip.
  • ccache - Allow the use of ccache during build. More useful in its negative form !ccache with select packages that have problems building with ccache.
  • distcc - Allow the use of distcc during build. More useful in its negative form !distcc with select packages that have problems building with distcc.
  • makeflags - Allow the use of user-specific makeflags during build. More useful in its negative form !makeflags with select packages that have problems building with custom makeflags.
  • force - Force the package to be upgraded by a pacman system upgrade operation, even if the version number would normally not trigger such an upgrade. This is useful when the version numbering scheme of a package changes (or is alphanumeric).
install
The name of the .install script to be included in the package. pacman has the ability to store and execute a package-specific script when it installs, removes or upgrades a package. The script contains the following functions which run at different times:
  • pre_install - The script is run right before files are extracted. One argument is passed: new package version.
  • post_install - The script is run right after files are extracted. One argument is passed: new package version.
  • pre_upgrade - The script is run right before files are extracted. Two arguments are passed in the following order: new package version, old package version.
  • post_upgrade - The script is run after files are extracted. Two arguments are passed in the following order: new package version, old package version.
  • pre_remove - The script is run right before files are removed. One argument is passed: old package version.
  • post_remove - The script is run right after files are removed. One argument is passed: old package version.
A prototype .install is provided at /usr/share/pacman/proto.install.
source
An array of files which are needed to build the package. It must contain the location of the software source, which in most cases is a full HTTP or FTP URL. The previously set variables pkgname and pkgver can be used effectively here: 
source=(http://example.com/$pkgname-$pkgver.tar.gz)
If you need to supply which are not downloadable on the fly, e.g. self-made patches, you simply put those into the same directory where your PKGBUILD file is in and add the filename to this array. Any paths you add here are resolved relative to the directory where the PKGBUILD lies. Before the actual build process is started, all of the files referenced in this array will be downloaded or checked for existence, and makepkg will not proceed if any are missing.
noextract
An array of files listed under the source array which should not be extracted from their archive format by makepkg. This most commonly applies to certain zip files which cannot be handled by bsdtar because libarchive processes all files as streams rather than random access as unzip does. In these situations unzip should be added in the makedepends array and the first line of the build() function should contain: 
cd $srcdir/$pkgname-$pkgver
unzip [source].zip
md5sums
An array of md5 checksums of the files listed in the source array. Once all files in the source array are available, an md5 hash of each file will be automatically generated and compared with the values of this array in the same order they appear in the source array. While the order of the source files itself does not matter, it is important that it matches the order of this array since makepkg cannot guess which md5sum belongs to what source file. You can generate this array quickly and easily using the command makepkg -g in the directory that contains the PKGBUILD file.

It is common practice to preserve the order of the PKGBUILD variables as shown above. However, this is not mandatory, as the only requirement in this context is correct Bash syntax.

The build() function

Now you need to implement the build() function in the PKGBUILD file. This function uses common shell commands in the Bash syntax to automatically compile software and create a pkg/ directory to install the software to. This allows makepkg to pack it up without having to pick all the necessary files from your actual filesystem.

The first step in the build() function is to change into the directory created by uncompressing the source tarball. In most common cases the first command will look like this:

cd $srcdir/$pkgname-$pkgver

Now, you need to list the same commands you used when you manually compiled the software. The build() function in essence automates everything you did by hand and compiles the software in the fakeroot build environment. If the software you are packaging uses a configure script, it is good practice to use --prefix=/usr when building packages for pacman. A lot of software install their files relative to the /usr/local directory, which should only be done if you are manually building from source. All Arch Linux pakcages should use the /usr directory. As seen in the /usr/share/pacman/PKGBUILD.proto file, the next two lines should look like this:

./configure --prefix=/usr
make || return 1

The final step in the build() funciton is to put the compiled files in a directory where makepkg can retrieve to create a package. This by default is the pkg/ directory - a simple fakeroot environment. It imitates the root of your filesystem to the software's installation procedure. If you have to manually install files under the root of your filesystem, you should instead install it in the pkg/ directory under the same directory structure, e.g. if you want to install a file to /usr/bin, it should instead be placed under $pkgdir/usr/bin. Very few software require the user to copy dozens of files manually, and you usually will not have to worry about this. Instead, for most software you will have to call make install. The final line should look like the following in order to correctly install the software in the pkg/ directory:

make DESTDIR=$pkgdir install

In some odd cases, the software expects to be run from a single directory. In such cases it is wise to simply copy these to $pkgdir/opt.

More often than not the installation process of the software will create any subdirectories below the pkg/ directory. If it does not, however, makepkg will generate a lot of errors and you will need to manually create subdirectories by adding the appropriate mkdir -p commands in the build() function before the installation procedure is run.

Also, makepkg defines three variables that you should use as part of the build and install process:

Additional Guidelines

Here are some additional guidelines that you should adhere to:

Testing the Package

As you are writing the build() function, you will want to test your changes frequently to ensure there are no bugs. You can do this using the makepkg command in the directory containing the PKGBUILD file. With a properly formatted PKGBUILD, this will create a package, but with a broken or unfinished one it will throw an error.

If makepkg finishes successfully, it will place a file named pkgname-pkgver.pkg.tar.gz in your working directory. This is package that can be installed with the pacman -U command. However, just because a package file was built does not imply that it is fully functional. It might conceivably contain only the directory and no files whatsoever if, for example, a prefix was specified improperly. You can use pacman's query functions to display a list of files contained in the package and the dependencies it requires with pacman -Qlp [package file] and pacman -Qip [package file] respectively.

If the package looks sane, then you are done! However, if you plan on release the PKGBUILD file, it is imperative that you check and double check the contents of the depends array.

ldd and namcap

Dependencies are the most common packaging error. There are two excellent tools you can use to check dependencies. The first one is ldd, which will show you the shared library dependencies of dynamic executables:

$ ldd gcc
        linux-gate.so.1 =>  (0xb7f33000)
        libc.so.6 => /lib/libc.so.6 (0xb7de0000)
        /lib/ld-linux.so.2 (0xb7f34000)

The other tool is namcap which not only checks for dependencies but the overall sanity of your package. It can be used on both the PKGBUILD file and the pkg.tar.gz package. It will report any bad permissions, missing dependencies, unnecessary dependencies and other common mistakes. You can install namcap via pacman:

$ pacman -S namcap

To use namcap simply supply a PKGBUILD file or a pkg.tar.gz package as the argument:

$ namcap [PKGBUILD or pkgname.tar.gz]

Although namcap can help detect dependencies, it is not always correct. Verify dependencies by looking at the documentation of the software.

Submitting Packages to the AUR

Follow these steps to submit a package into the AUR:

  1. Register a new account if you do not already have one.
  2. Check all of the official repositories ([core], [extra], [unstable], and [community]) and see if the package already exists. If it is inside any of those repositories, DO NOT submit the package. If the package is broken, file a bug report.
  3. Check the [unsupported] repository for the package. If it is currently maintained, changes can be submitted in a comment for the maintaner's attention. If it is unmaintained, the package can be adopted and updated.
  4. To upload the package to the AUR, the directory containing the PKGBUILD file and any other required files must be compressed as a tarball. The archive name should contain the name of the package, e.g. foo.tar.gz.. You can easily build a tarball by using makepkg --source in the directory. This makes a tarball named pkgname-pkver-pkrel.src.tar.gz, which you can then upload to the AUR. The tarball cannot contain the binary tarball created by makepkg or the source tarball of the software. Packages that contain binaries or that are very poorly written will be deleted without warning.
  5. Click the Submit link by the menu in the AUR. Choose the appropriate category for your package and upload.

Summary

  1. Download the source tarball of the software you want to package.
  2. Try compiling the package and installing it into an arbitrary directory.
  3. Copy over the prototype /usr/share/pacman/PKGBUILD.proto and rename it to PKGBUILD in a temporary working directory - preferably /var/abs/local.
  4. Edit the PKGBUILD according to the needs of your package.
  5. Run makepkg and see whether the resulting package is built correctly.
  6. If not, repeat the last two steps.

Notoriety

Delicious

Recent Entries

11/24/08 - Backup and Synchronization with Git

11/4/08 - Configuring Special Keys on a ThinkPad T60 (Arch Linux)

9/16/08 - Building Packages in Arch Linux

9/6/08 - Setting Up Wireless on a ThinkPad T60 (Arch Linux)

7/28/08 - Customized Logging in Apache 2.2

Recent Comments

1/2/09 - Backup and Synchronization with Git

12/29/08 - Building Packages in Arch Linux

12/26/08 - Building Packages in Arch Linux

11/26/08 - Backup and Synchronization with Git

11/25/08 - Backup and Synchronization with Git