Some tmux cheats

There's a lot of stuff you can do with tmux. Here are some nice to know things:

• C-b C-o: cycle contents of current windows (or swap if there are only two windows)
• C-b C-SPC: switch between vertical and horizontal split
• C-b n, p: next or previous screen
• C-b [: copy mode
• Use Emacs bindings to copy and paste:
• C-SPC: begin selection
• C-w or M-w to copy
• C-b ] to yank (paste)

New StackExchange: Astronomy

Just so that you know, there is a new StackExchange beta site: Astronomy. You can ask astronomy-related questions there, or search for questions other people have asked. Hopefully, you will get some good answers.

How to copy Mails from iCloud to GMail

You can use the fabulous imapsync tool to copy mails between IMAP servers. For example you can copy a certain folder from Apple's iCloud to Google's gmail:

imapsync \
   --noauthmd5 --ssl1 --ssl2 \
   --host1 mail.me.com --user1 'your.icloud.name' \
   --host2 imap.gmail.com --user2 'your.gmail.name@googlemail.com' \
   --folder 'your/folder/to/be/copied' --sep1 '/' \
   --prefix1 '' --prefix2 '[Google Mail]' --sep2 '/'

The important parts here are the user names for the IMAP servers. Note that you need to generate an application specific password, if you are using Google two factor authentication! Also important is the "[Google Mail]" IMAP prefix.

Edit: It seems gmail has a weird interpretation of all the IMAP folders and stuff. Since they are using labels, the above script might create a weird label for the copied emails, but they will be there nevertheless!

How to get a directory recursively with lftp

Using the command mget, you can fetch files with the program lftp. With mget -c you can continue an aborted download. But how do you download a directory recursively? No, mget -c -r does not work.

It is so simple:

mirror somedirectory

Don't forget to lcd into your download directory before.

Working around connection problems with Emacs Tramp

From time to time I have to edit files on a SunOS 5.10 server. I use Emacs with tramp for this. However, after some time I get this error message from tramp:

File error: Couldn't find exit status of `test -e ...

It seems that the ssh connection goes bad for some reason. After this you won't be able to save the file anymore. You can work around this by running M-x tramp-cleanup-all-connections and then saving again.

Anonymous (lambda) functions in Clojure

I am learning a bit of Clojure from time to time, and today I learned about shorthand notation for anonymous functions:

> (apply (fn [x y] (+ x y)) '(1 1))
2

Which is equivalent to:

> (apply #(+ %1 %2) '(1 1))
2

There are approximately N/ln(N) primes between N and 2N

Just saw this very nice video by @numberphile, and thought I whip up a small Python program to demonstrate the prime number theorem:

#!/usr/bin/env python
#
# "Chebyshev said it, and I say it again: There's always a prime between n and 2n."
#

import sys
import math

class PrimeFinder:

    def __init__( self, n ):
        self.n = n
    
    def isNPrime( self, N ):
        for x in range( 2, int( math.sqrt( N ) ) + 1 ):
            if N % x == 0:
                return False
        return True

    def computeAllPrimesBetweenNAndTwoN( self ):
        result = []
        for N in range( self.n, 2 * self.n + 1 ):
            if self.isNPrime( N ):
                result = result + [ N ]
        return result

def main():
    if len( sys.argv ) != 2:
        print "Prints all prime numbers between N and 2N"
        print "Usage: %s N" % sys.argv[ 0 ]
        print "Where N is some positive, natural number."
        sys.exit( 0 )

    N = int( sys.argv[ 1 ] )
    primeFinder = PrimeFinder( N )
    allPrimes = primeFinder.computeAllPrimesBetweenNAndTwoN()
    print "There are %u primes between %u and %u: %s" % ( 
        len( allPrimes ), N, 2 * N, str( allPrimes )[ 1 : -1 ] 
    )

if __name__ == "__main__":
    main()

And it seems to work, but check WolframAlpha if you don't trust me :)

$ ./myprimes.py 100000
There are 8392 primes between 100000 and 200000: 100003, 100019, 100043 ...

How to read input from a bunch of commands in bash

You can use the <( ... ) subshell syntax to read from a bunch of commands, that you whip up on the fly:

cat < <( echo "foo"; sleep 2; echo "bar" )

You can use this for example for accessing IMAP servers, which need a short delay during login.

How to get Ctrl+Arrow working for programs running in tmux?

The key combination of Ctrl+arrow key is often used for skipping forward or backward whole words. This can be used in the bash command line, Emacs and many other programs. However, when I am using tmux, this will not work. You can fix this, by adding the following to your ~/.tmux.conf:

set-window-option -g xterm-keys on

This was explained in a nice superuser Q&A.

You can interactively try out tmux commands by hitting C-b : -- this will enter the command mode. You can use tab to complete commands.

How to use SciPy Least Squares to minimize multiple functions at once

SciPy comes with a least squares Levenberg-Marquardt implementation. This allows you to minimize functions. By defining your function as the difference between some measurements and your model function, you can fit a model to those measurements.

Sometimes your model contains multiple functions. You can also minimize for all functions using this approach:

• Define your functions that you like to minimize A(p0), B(P1), ...
  their cumulative paramaters will be a tuple (p0, p1, ...).
• Define your function to be minimized as f(x0), where x0 is expanded to the parameter tuple.
• The function f returns a vector of differences between discrete measured sample and the individual functions A, B etc.
• Let SciPy minimize this function, starting with a reasonably selected initial parameter vector.

This is an example implementation:

import math
import scipy.optimize

measured = { 
    1: [ 0, 0.02735, 0.47265 ],
    6: [ 0.0041, 0.09335, 0.40255 ],
    10: [ 0.0133, 0.14555, 0.34115 ],
    20: [ 0.0361, 0.205, 0.2589 ],
    30: [ 0.06345, 0.23425, 0.20225 ],
    60: [ 0.132, 0.25395, 0.114 ],
    90: [ 0.2046, 0.23445, 0.06095 ],
    120: [ 0.2429, 0.20815, 0.04895 ],
    180: [ 0.31755, 0.1618, 0.02065 ],
    240: [ 0.3648, 0.121, 0.0142 ],
    315: [ 0.3992, 0.0989, 0.00195 ]
}

def A( x, a, k ):
    return a * math.exp( -x * k )

def B( x, a, k, l ):
    return k * a / ( l - k ) * ( math.exp( -k * x ) - math.exp( -l * x ) )

def C( x, a, k, l ):
    return a * ( 1 - l / ( l - k ) * math.exp( -x * k ) + k / ( l - k ) * math.exp( -x * l ) )

def f( x0 ):
    a, k, l = x0
    error = []
    for x in measured:
        error += [ C( x, a, k, l ) - measured[ x ][ 0 ],
                   B( x, a, k, l ) - measured[ x ][ 1 ],
                   A( x, a, k ) - measured[ x ][ 2 ]
               ]
    return error

def main():
    x0 = ( 0.46, 0.01, 0.001 ) # initial parameters for a, k and l
    x, cov, infodict, mesg, ier = scipy.optimize.leastsq( f, x0, full_output = True, epsfcn = 1.0e-2 )
    print x

if __name__ == "__main__":
    main()

SciPy returns a lot more information, not only the final parameters. See their documentation for details. You also may want to tweak epsfcn for a better fit. This depends on your functions shape and properties.

Avoiding page allocation failures on the Pi

Since I've been using my 256MB Model B Pi as a server, I had been getting regular page allocation failures of the following kind:

sshd: page allocation failure: order:0, mode:0x20

This is rather annoying and seems to affect stability as well, like disrupted ssh or smb connections. It seems that the kernel has a setting to affect the minimum free memory that it keeps for allocations, which may be too low. So in the /etc/sysctl.conf file you can edit the last lines to bump this value from 8 Mbegabytes to 16 Megabytes:

# rpi tweaks
vm.swappiness=1
vm.min_free_kbytes = 16184

Settings are effective after a reboot. For me, this seems to fix the problem. But to be extra sure, I also changed the memory split from 192/64 to 224/32. Since my Pi runs headless, 64 MBytes seemed too much simply for a framebuffer console that is not even used. So after these tweaks my Pi shows 216 MByte of available memory and is running for some days now without page allocation errors.

Delay warnings when using USB audio on the Raspberry Pi

My kernel.log and debug log files had grown to over 2 GBytes each. This was due to log messages of this kind:

delay: estimated 0, actual 133

As another blog stated, this can be fixed by adding a parameter to the sound-usb-audio module of Alsa. So I did the same thing, creating a file /etc/modprobe.d/snd_usb_audio.conf with this content:

options snd-usb-audio nrpacks=1

After restarting Alsa or a reboot, the problems were gone, the logfiles kept small.

The Raspberry Pi and Ralink rt2800 based WiFi dongles

I strongly discourage the usage of rt2800 based WiFi dongles with the Raspberry Pi. For some weeks I have been debugging why my Raspberry Pi freezes and crashes all the time. And it turns out that it is the WiFi dongle. I was using an Edimax EW-7711UTn, but the chip is also being used in other devices. There seems to be a bug report for the Raspbian kernel, and I have spent quite some time on the Raspberry forum at Stackexchange. Currently, I hooked up the Raspberry Pi via Ethernet directly to my router. This works fine, but the router is upstairs, while the Raspberry is supposed to be downstairs, connected to my stereo. So this is only a stopgap measure until I have found a WiFi dongle that does not crash the Pi. Hints are welcome.

Update: I now switched to a TP-Link TL-WN725N with a RTL8188CUS chip. This adapter is much smaller (no external antenna) and surprisingly also quite a bit faster. I achieve ~1.9 MByte/sec sustained point to point datarates via SMB. So far no crashes or other problems. I will continue testing, but it looks much better than with the Ralink chip.
Update 2: Almost four months into testing the RTL8188CUS based WiFi dongle and it runs absolutely fine. I have uptimes of almost a month without any problems. After that the Pi degrades due to some of the daemons leaking or memory fragmentation. But I think this is bearable!

How to make the mpdas run as a daemon

The other day I installed the mpdas, which is the audio scrobbler for the music player daemon. Since there's no debian package for the Raspberry Pi, I compiled mpdas from scratch and installed it. Now I don't want to run it manually each time the Raspberry Pi boots up. So I found a nice template for writing your own debian-style init-script. I changed it a little and also installed the daemon tool, to turn the interactive mpdas program into a daemon. Just run apt-get install daemon to install it. Then put the following file under /etc/init.d/mpdas and run update-rc.d mpdas defaults. Then mpdas will be run automatically upon boot. Oh, one more thing: put your mpdas configuration under /usr/local/etc/mpdasrc or adjust the DAEMONOPTS in the init script accordingly.

#!/bin/bash
### BEGIN INIT INFO
# Provides:          mpdas
# Required-Start:    $remote_fs $syslog $mpd
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# X-Interactive:     true
# Short-Description: Audio Scrobbler for mpd
# Description:       Starts the Audio Scrobbler for the mpd music player daemon.
### END INIT INFO

DAEMON_PATH="/usr/bin/"

DAEMON=daemon
DAEMONOPTS="-u pi -r -X /usr/local/bin/mpdas"

NAME=mpdas
DESC="The mpdas audio scrobbler for mpd"
PIDFILE=/var/run/$NAME.pid
SCRIPTNAME=/etc/init.d/$NAME

case "$1" in
start)
printf "%-50s" "Starting $NAME..."
cd $DAEMON_PATH
PID=`$DAEMON $DAEMONOPTS > /dev/null 2>&1 & echo $!`
#echo "Saving PID" $PID " to " $PIDFILE
 if [ -z $PID ]; then
     printf "%s\n" "Fail"
 else
     echo $PID > $PIDFILE
     printf "%s\n" "Ok"
 fi
;;
status)
 printf "%-50s" "Checking $NAME..."
 if [ -f $PIDFILE ]; then
     PID=`cat $PIDFILE`
     if [ -z "`ps axf | grep ${PID} | grep -v grep`" ]; then
  printf "%s\n" "Process dead but pidfile exists"
     else
  echo "Running"
     fi
 else
     printf "%s\n" "Service not running"
 fi
;;
stop)
 printf "%-50s" "Stopping $NAME"
     PID=`cat $PIDFILE`
     cd $DAEMON_PATH
 if [ -f $PIDFILE ]; then
     kill -HUP $PID
     printf "%s\n" "Ok"
     rm -f $PIDFILE
 else
     printf "%s\n" "pidfile not found"
 fi
;;

restart)
$0 stop
$0 start
;;

*)
 echo "Usage: $0 {status|start|stop|restart}"
 exit 1
esac

How to make the Raspberry Pi automatically restart the WiFi interface

My WiFi router sometimes goes haywire and the Pi won't notice when the WiFi connection is up again. So I wrote this little script:

#!/bin/bash                                   
                                              
TESTIP=192.168.1.1                            
                                              
ping -c4 ${TESTIP} > /dev/null                
                                              
if [ $? != 0 ]                                
then                                          
    logger -t $0 "WiFi seems down, restarting"
    ifdown --force wlan0                      
    ifup wlan0                                
else                                         
    logger -t $0 "WiFi seems up."            
fi                                                                    

You can put this script under /usr/local/bin and add the following line to the system wide /etc/crontab:

*/5 * * * * root /usr/local/bin/testwifi.sh

This will check every five minutes if the connection is still up, and restart it, if the router cannot be pinged. If you dislike all the syslog messages, you can comment them out in the script.
My corresponding /etc/network/interfaces looks like this (I uninstalled all the network managers):

auto lo                                                                                  
                                                                                         
iface lo inet loopback                                                                   
iface eth0 inet static                                                                   
        address 192.168.3.42                                                             
        netmask 255.255.255.0                                                            
                                                                                         
auto wlan0                                                                               
iface wlan0 inet dhcp                                                                    
      pre-up wpa_supplicant -Dwext -i wlan0 -c /etc/wpa_supplicant/wpa_supplicant.conf -B
      post-down killall wpa_supplicant ; rmmod 8192cu ; modprobe 8192cu                  
                                                                                         
iface default inet dhcp                                                                  

The wpa_supplicant.conf should be easy to generate, there are lots of guides on the web for this.

Good console fonts for OS X terminal applications

Today I was looking for a better console font for the Apple terminal application. Up until now I was using a variant of the Terminus font, but it did not have the line drawing characters. So several programs looked quite borked. But there is a nice page with a collection of the X11 terminal fonts in Apple's dfont format, which you know and like from all kinds of Linux distributions.
You have to tweak the spacings, for example for the 7x14 font you need to pick 0.98 and 0.93 for the horizontal and vertical spacing. Also make sure to use the fonts at their specified size. I.e. the 7x14 font at 14 points only.

Attaching a USB sound card to the Raspberry Pi

Since my Raspberry Pi runs headless, and the analog audio output is not that great, I decided to add a USB sound card to my little machine. I took a Roland UA-1G, which I was using before on an OpenWRT machine. The device was immediately recognized:

Bus 001 Device 007: ID 0582:00ea Roland Corp.

However, ALSA will prohibit the card from becoming sound device #0, thus being the default. For that you have to comment out the following line in /etc/modprobe.d/alsa-base.conf:

# Keep snd-usb-audio from beeing loaded as first soundcard
#options snd-usb-audio index=-2

After rebooting or restarting ALSA, the Roland will become the default sound device:

$ aplay -l
**** List of PLAYBACK Hardware Devices ****
card 0: UA1G [UA-1G], device 0: USB Audio [USB Audio]
  Subdevices: 0/1
  Subdevice #0: subdevice #0
card 1: ALSA [bcm2835 ALSA], device 0: bcm2835 ALSA [bcm2835 ALSA]
  Subdevices: 7/8
  Subdevice #0: subdevice #0
  Subdevice #1: subdevice #1
  Subdevice #2: subdevice #2
  Subdevice #3: subdevice #3
  Subdevice #4: subdevice #4
  Subdevice #5: subdevice #5
  Subdevice #6: subdevice #6
  Subdevice #7: subdevice #7

Playing with REST: SilverShield USB power web frontend

Yesterday I started to code a small tool that lets me turn on and off power outlets on my USB controlled  power strip. You can find the humble beginnings over at github. It uses the sispmctl tool to turn on and off the sockets. It is implemented in Python using the Tornado web server. The rendering is simple, not styled and not yet HTML compliant. But it already scans for attached devices, and lets you toggle single outlets. However, today the power strip sort of died on me, and now I have to look for another project to practice REST with...

How to convert a Python list into a string in a strange way

Given a list in Python, suppose you wanted a string representation of that list. Easy enough:

str( [ 1, 2, 3 ] )

However, suppose you did not want the standard notation, but rather apply some function to each string element or simply do away with the brackets and commas, you can use list comprehension, the join function and an empty string literal:

''.join( str( i ) for i in [ 1, 2, 3 ] )

I already knew list comprehension, but using it in this scenario and with a string literal as an object was new to me. Anyway, using such code is probably a bad idea, since it might be hard to read! At the very least, one should stow it away in a function with a fitting name, such as jointStringRepresentationOfListItems( list ). But really, I am not even sure what I would use that for...

Update: even better is this:

','.join( map( str, [ 1, 2, 3 ] ) )

How to turn the Raspberry Pi into a music server

I have set up my Raspberry Pi as a music and file server. My requirements were:

• Big local storage (HD), shared over WiFi
• Local music playing capability, remotely controllable
• AirPlay speaker capability

The means by which I fulfilled the requirements were:

• Platinum 1TB MyDrive and Edimax EW-7711UTn USB wireless adapter
• mpd Music Player Daemon
• shairport AirPort emulator

For the first part, I bought a WiFi adapter, the Edimax EW-7711UTn. This one works out of the box with Raspbian, using WPA encryption (Note: I switched to a RTL8188 based dongle by now). It identifies itself with lsusb as:

Bus 001 Device 004: ID 7392:7711 Edimax Technology Co., Ltd EW-7711UTn nLite Wireless Adapter [Ralink RT2870]

This can be easily configured using the wpa_gui tool that comes with the default Raspbian installation. Settings will be saved and restored upon reboot.

Second, I installed samba, samba-common and samba-common-bin for sharing my USB drive. The latter one is a Platinum MyDrive, which is attached to a powered Belkin 7-port USB hub, so that I only need two power supplies. One for the Raspberry Py, and one for the Hub and its attached devices. The MyDrive has been formatted with NTFS, so as to be easily mountable under Linux, OS X and Windows. I mount it using the standard /etc/fstab mechanism. Just added one line to the file:

/dev/sda1       /media/MyDrive  ntfs-3g defaults          0       0

The /etc/samba/smb.conf gets in its first iteration only one additional share, a write-for-all public share, as a big file dump: 

[BigDump]

    comment = Free for all

    read only = no

    path = /media/MyDrive/FreeForAll

    guest ok = yes

Note that literally everybody in your LAN can access this and write to it! You may want to fine tune this...

Now to the mpd. It is easily installed by doing apt-get install mpd. When configuring it via /etc/mpd.conf make sure to change the following lines:

music_directory         "/media/MyDrive/Music/"

password                "MyVeryOwnPassword@read,add,control,admin"
bind_to_address         "any"

Change the directory and password to your liking. Then restart the service or reboot you device. You can control the mpd using a magnitude of clients. For example Theremin for OS X or Mpod for iOS.

Finally, I would like to be able to use the RasPi as an AirPlay target for my Mac and my iOS devices. This can be done via shairport. There are already a lot of good howtos for shairport on the Raspberry Pi. So I refer you to one of those. Two things come to my mind, though:

1. The Net::SDP library, required by shairport, is not available on Raspbian by default. It is best to clone the github mirror, and go by its installation instructions. Installation via CPAN fails, plus CPAN needs huge amounts of RAM.
2. The configuration needs to be tweaked a bit. The /etc/init.d/shairport script should be tuned to a sensible name for your RasPi. 

This way, you will be able to see the RasPi in your AirPlay speakers list and it will be happily churning along.

How to determine if current buffer is tramp buffer in Emacs

Tramp is Emacs' way to access remote files, e.g. via SSH. To determine if the current buffer is accessing a tramp file (e.g. to not try to run gtags or similar), try this:

(not (tramp-tramp-file-p (buffer-file-name (current-buffer))))

Quoting a set of parameters for program arguments using sed

I have a script which launches another program. That program takes some command line arguments, which in turn can have parameters. Those parameters may include ";", which has a special meaning in the Bash. To quote these, you can use the following sed command:

$ echo "foo --bar=1 --baz=2" | sed -e 's/\(--[^[:space:]]*=\)\([^[:space:]]*\)/\1"\2"/g'
foo --bar="1" --baz="2"