How to get a directory recursively with lftp

Using the command mget, you can fetch files with the program lftp. With mget -c you can continue an aborted download. But how do you download a directory recursively? No, mget -c -r does not work.

It is so simple:

mirror somedirectory

Don't forget to lcd into your download directory before.

Good console fonts for OS X terminal applications

Today I was looking for a better console font for the Apple terminal application. Up until now I was using a variant of the Terminus font, but it did not have the line drawing characters. So several programs looked quite borked. But there is a nice page with a collection of the X11 terminal fonts in Apple's dfont format, which you know and like from all kinds of Linux distributions.
You have to tweak the spacings, for example for the 7x14 font you need to pick 0.98 and 0.93 for the horizontal and vertical spacing. Also make sure to use the fonts at their specified size. I.e. the 7x14 font at 14 points only.

How to turn the Raspberry Pi into a music server

I have set up my Raspberry Pi as a music and file server. My requirements were:

• Big local storage (HD), shared over WiFi
• Local music playing capability, remotely controllable
• AirPlay speaker capability

The means by which I fulfilled the requirements were:

• Platinum 1TB MyDrive and Edimax EW-7711UTn USB wireless adapter
• mpd Music Player Daemon
• shairport AirPort emulator

For the first part, I bought a WiFi adapter, the Edimax EW-7711UTn. This one works out of the box with Raspbian, using WPA encryption (Note: I switched to a RTL8188 based dongle by now). It identifies itself with lsusb as:

Bus 001 Device 004: ID 7392:7711 Edimax Technology Co., Ltd EW-7711UTn nLite Wireless Adapter [Ralink RT2870]

This can be easily configured using the wpa_gui tool that comes with the default Raspbian installation. Settings will be saved and restored upon reboot.

Second, I installed samba, samba-common and samba-common-bin for sharing my USB drive. The latter one is a Platinum MyDrive, which is attached to a powered Belkin 7-port USB hub, so that I only need two power supplies. One for the Raspberry Py, and one for the Hub and its attached devices. The MyDrive has been formatted with NTFS, so as to be easily mountable under Linux, OS X and Windows. I mount it using the standard /etc/fstab mechanism. Just added one line to the file:

/dev/sda1       /media/MyDrive  ntfs-3g defaults          0       0

The /etc/samba/smb.conf gets in its first iteration only one additional share, a write-for-all public share, as a big file dump: 

[BigDump]

    comment = Free for all

    read only = no

    path = /media/MyDrive/FreeForAll

    guest ok = yes

Note that literally everybody in your LAN can access this and write to it! You may want to fine tune this...

Now to the mpd. It is easily installed by doing apt-get install mpd. When configuring it via /etc/mpd.conf make sure to change the following lines:

music_directory         "/media/MyDrive/Music/"

password                "MyVeryOwnPassword@read,add,control,admin"
bind_to_address         "any"

Change the directory and password to your liking. Then restart the service or reboot you device. You can control the mpd using a magnitude of clients. For example Theremin for OS X or Mpod for iOS.

Finally, I would like to be able to use the RasPi as an AirPlay target for my Mac and my iOS devices. This can be done via shairport. There are already a lot of good howtos for shairport on the Raspberry Pi. So I refer you to one of those. Two things come to my mind, though:

1. The Net::SDP library, required by shairport, is not available on Raspbian by default. It is best to clone the github mirror, and go by its installation instructions. Installation via CPAN fails, plus CPAN needs huge amounts of RAM.
2. The configuration needs to be tweaked a bit. The /etc/init.d/shairport script should be tuned to a sensible name for your RasPi. 

This way, you will be able to see the RasPi in your AirPlay speakers list and it will be happily churning along.

Using Ctrl+Arrow keys in the OS X terminal

The key combination Ctrl + left or right arrow key can be used to go a word left or right in many programs such as GNU Emacs or the bash. In the default configuration of the OS X terminal, this is not the case. That's because Terminal.app sends the wrong key codes for bash's default configuration. So you just need to got to the preferences (cmd+,) and set the key codes for Ctrl+left and right to \033b and \033f respectively:

What about C++11 in Xcode?

This started out as a very small problem: I am using C++, Objective C++ and Objective C mixed in one project. This usually works pretty well. However, when including MapKit, the compiler choked on an Objective C++ file. It seems the MapKit.h header wants to use isinf(). This function has been introduced with ISO C 99, but it it is not included in ISO C++ 98. Hence the failure of Xcode's gcc and clang compilers.

To work around this problem I was thinking of trying to set the project settings to use ISO C++ 11, or C++ 0x, since Xcode's compilers are not that up to date. This is pretty easy:

And I guess this would work for most people. However, I am also using boost in this project. And I guess it is rather up to date, concerning the new language specification. However this also means that there is a problem with clang's support of the language. I get a ton of errors in boost when switching to C++ 0x in clang:

In short, I think the C++ 11 support is not quite there yet with Xcode 4.2 and clang 3.0. I will try to post updates when future Xcode releases are coming out. If, alternatively, someone has hints on how to compile boost with clang 3.0, I am also glad to try that out.

Update: There is a fork on gitorious of the boost on iPhone build script, which already uses the clang compiler. I hacked the script even more to use --std=c++0x as a flag and it builds fine. Now I will try to link this new framework to an iOS project compiled with clang using C++ 0x.

Update 2: I found the root of the problem. The fantastic people over at #boost on freenode helped me figuring it out. In short: boost::fusion defines nil, which is an Objective C keyword. Thus we get lots of problems. Objective C defines nil and Nil in objc/objc.h. A minimal program that compiles successfully:

#include <boost/phoenix/core/reference.hpp>
#import <CoreFoundation/CoreFoundation.h>

int main()
{
    return 1;
}

Using this command line:

/Developer/Platforms/iPhoneOS.platform/Developer/usr/bin/clang++ \
-isysroot /Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS5.0.sdk \
-framework Foundation -Fpath/to/boost/framework --std=c++0x -arch armv7 \
-stdlib=libc++ -mthumb -miphoneos-version-min=5.0 foo.mm -o foo 

If you swap the to include / import statements, the program will not compile, since now the ObjC nil makes boost break. There is a boost ticket tracking this issue, with a proposed patch attached, which renames the boost nil to nil_.

Update 3: The patch works fine, but now I get internal compiler errors from clang. The compile triggers a segmentation violation at some point during the compilation of my project. I guess clang 3.0 is still a bit experimental. I submitted a bug report and will revert to using the default settings of Xcode for the time being.

Using gpsbabel and a USB to serial adapter on OS X to read out a Garmin GPS

For a couple of years I have been owning a Garmin eTrex GPS receiver. It's a nice little tool, pretty ruggedized and reliable. I use it to track for example hiking tracks and also sometimes for Geocaching. It has a serial interface for downloading the data. However, my MacBook Pro does not have a serial port. So I bought one of the ubiquitous Digitus USB to Serial adapters. With this and gpsbabel (sudo port install gpsbabel, if you have MacPorts), it is really easy to download the track data for example to a KML file to be viewed in Google Earth:

gpsbabel -t -i garmin -f /dev/cu.usbserial-XXXXXX -o kml -F mytracks.kml

The XXXXXX in the device filename varies from device to device, you probably can also use /dev/cu.usbserial-* as the device filename, if you have only one of these things attached to your computer. It is important not to use the tty device file, since it seems not to work. I do not know yet what the exact difference is, but I will come back when I found out.

Clementine Player

A couple of years back, I was a big fan of Amarok, the music player. Then came along a rewrite and version 2.0, and I also switched to OS X. The version 2.0 was not very nice, stable or useful. The OS X version was very hard to install, due to the KDE dependencies. So I ditched Amarok. I replaced it with CogX for a while, and now I am using the horrid iTunes and the wonderful mpd (together with MPoD and Theremin). But today I read about the Clementine Player. It's a cross platform Amarok 1.4 fork. And after 10 minutes of testing, I think it's wonderful!

Another OS X git UI: SourceTree

Besides gitx, I now was shown another git user interface for OS X. It is called SourceTree:

It is more mature and feature-rich, compared to gitx. But it is closed source. However, it is free (as in beer) for the time being in the App Store. So far, the program seems really nice. Let's see how long I will continue using it, and if I'll go back to gitx at some point.

XCode 4 is incredibly slow

XCode 4 has always been incredibly slow for me. The first release, 4.0, was especially bad. But that was just a .0 version. The next release 4.1 is much better, but it has also severe drawbacks, concerning performance. Everytime I start it, and not even do much with it, my system gets incredibly slow. That is on both a C2D 2.8 GHz MBP and also on a quad-core i7 MBP. Both machines come with 4 GB of RAM, and after firing XCode up and loading a large project, still at least 500 MB of it remains free. However, speed is abysmal. I just found the tool vmmap in OS X, and it gives me this output:

==== Summary for process 32136
ReadOnly portion of Libraries: Total=265.8M resident=114.4M(43%) swapped_out_or_unallocated=151.5M(57%)
Writable regions: Total=16.2G written=149.6M(1%) resident=360.9M(2%) swapped_out=6156K(0%) unallocated=15.9G(98%)
REGION TYPE                      VIRTUAL
===========                      =======
CG backing stores                  19.4M
CG image                            268K
CG raster data                     2840K
CG shared images                   3472K
CoreAnimation                       180K
CoreGraphics                         16K
CoreImage                           108K
CoreServices                       1704K
IOKit                              61.2M
MALLOC                            337.4M        see MALLOC ZONE table below
MALLOC (reserved)                  15.6G        reserved VM address space (unallocated)
MALLOC freed, no zone              30.5M
MALLOC guard page                    64K
MALLOC metadata                   128.8M
Memory tag=240                        4K
Memory tag=242                       12K
Memory tag=243                        4K
Memory tag=249                      156K
Memory tag=251                       64K
OpenCL                               60K
OpenGL GLSL                        1372K
OpenGL GLSL (reserved)              128K        reserved VM address space (unallocated)
SQLite page cache                  14.6M
STACK GUARD                        56.1M
Stack                              19.7M
VM_ALLOCATE                        16.1M
__CI_BITMAP                          80K
__DATA                             33.9M
__IMAGE                            1256K
__LINKEDIT                         59.5M
__TEXT                            206.4M
__UNICODE                           544K
mapped file                        72.9M
shared memory                      13.6M
===========                      =======
TOTAL                              16.7G
TOTAL, minus reserved VM space      1.1G

So the virtual memory space that XCode takes is more than 16 GB! The actual memory taken is "only" 1.1 GB, which is still huge, but my Emacs also takes 500 MB with tons of C++, Python and LaTeX buffers open.
The question is: can the unallocated, but reserved 16 GB address space degrade the performance? I have too little knowledge of the workings of virtual memory on Intel CPUs under OS X. But this value seems incredibly huge.

Update: I have asked a question on Stackoverflow, and have gotten some useful answers. What did help was removing my build/ folder from the git. Accidentally, a colleague checked in four files in the build/ folder. This made Xcode very slow, since it was checking the git status during compilation all the time. Still, Xcode 4 is much slower than Xcode 3 after this. So I also upgraded our machines to have at least 8 GB of RAM. This was definitely much of an improvement. It seems that development machines using Xcode 4 should have 8 GB RAM minimum. The more, the better...

SSL Everywhere for Safari

The EFF has published a Firefox extension to force HTTPS on as many websites as possible. This is a good idea(tm). But on OS X I use Safari, not Firefox. So I was wondering if there is a similar extension for Safari. It turns out there is. However, this extension is available in source only. The reason for this is given in a blog post by the developer. In short: The Safari extension API is limited, and the extension cannot guarantee that all your session cookies are transferred via HTTPS, making you vulnerable to stuff like Firesheep. And that is a bad thing(tm). But it is still useful, since it will redirect you to secure versions of the websites. One thing to note when following the build instructions: You need to enroll in the free Safari Dev program, and get a developer certificate. This is very well described over at Apple's developer center. After that, you can enable the developer menu in Safari's settings and just install the extension from the cloned git repository.

What are the best Web Galleries?

I am one of the few avid MobileMe users. I use the gallery and like it. Nice upload features, integration into iPhoto. Not perfect, but it does the work, plus I have lots of disk space. However, Apple makes the transition from MobileMe to iCloud. During this transition, several features of MobileMe will be dropped. Besides the very useful syncing of keychains between computers, the gallery will cease to function. Some strange photo stream service will be introduced, which does not seem to be a real gallery replacement, but more like a large cache of you last 1,000 pictures taken. Useful, but only to a limited extent. So what are my alternatives? I need (in order of importance):

• Share with other people (who are not signed in, a la Facebook)
• 5-10 GB space for pictures or more
• Able to make photo albums, user gets a nice web browser based viewer
• Option for users to download the photos
• Optional high resolution versions of photos (2048 pixels or more)
• Good integration with iPhone, iPhoto et al.

I would pay for that service, if it includes 10 GB space or more. 

I already found out that Flickr and Picasa Web offer unlimited space, but in the free version there are quite the restrictions (limited image resolution, image usage by hoster, limited number of photos that can be controlled, advertisements, ...). Plus for example the Picasa Web Uploader for the Mac is horribly outdated (iPhoto '08...?).

So what other options are there? Which one is the best?

Even better gitx with push/pull support

Many moons ago, I posted a link to brotherbard's fork of the GitX GUI for git. There is an even better fork, which is based partially on brotherbard's work, and which is currently actively maintained. You can find it over at German Laullon's website.

Accessing the keychain in OS X from the command line

There is a very useful utility called security(1) in OS X, which lets you manipulate your keychain from the command line. You can easily im- and export keys and certificates using this. This is especially useful for AppStore developers, who code on multiple Macs. Having the signing keys in sync is kind of a challenging solution, if you don't use keychain syncing via MobileMe. For example you can import a key like this into your login keychain:

$ security list-keychains
   "/Users/yourguy/Library/Keychains/login.keychain"
   "/Library/Keychains/System.keychain"
$ security import -k /Users/yourguy/Library/Keychains/login.keychain somekey.pem
1 key imported.

Hope this helps.

Note to self: codesigning for OS X and iOS on the command line

As a reminder for myself, here is how you can codesign an OS X application for the AppStore on the command line:

codesign -f -s "3rd Party Mac Developer Application: Your Company" -v YourApp.app
productbuild --component YourApp.app /Applications --sign "3rd Party Mac Developer Installer: Your Company" YourApp.pkg

There is a lot more to do, of course, like having the correct bundle ID set, but this speeds up codesigning, if you do not use XCode to build your application.

For iOS it is pretty similar, except you don't need the productbuild:

codesign -f -s "iPhone Distribution: Your Company" -v YourApp.app

Fixing broken GNU texinfo dir file on MacPorts

Somehow, MacPorts does not seem to update the GNU texinfo dir file contained in /opt/local/share/info. But there is a nice script available in the ctan, which generates a new dir file. You can download it here.

Direct Dependencies in iOS projects

I found a nice blog entry that explains how to build static libraries and dependent projects with XCode. This is quite nice for iOS development, and relatively elegant. The only thing that bothers me is that the search path for headers has to be augmented manually. I wonder if there is a nice solution...

Valgrind checking of iOS programs

Well, this is awesome news. First, valgrind has been available for OS X for some time now. And second, you can use it to check your iOS programs on the simulator with it.

The idea here is to let your program spawn valgrind itself. Because you cannot tell the simulator to run the program through valgrind. Well, maybe you could build a funky bundle, but I think this works just fine. So here is the  code, taken from the above link:

#define VALGRIND "/usr/local/valgrind/bin/valgrind"
 
int main(int argc, char *argv[]) {
#ifdef VALGRIND_REXEC
    /* Using the valgrind build config, rexec ourself
     * in valgrind */
    if (argc < 2 || (argc >= 2 && strcmp(argv[1], "-valgrind") != 0)) {
        execl(VALGRIND, VALGRIND, "--leak-check=full", "--dsymutil=yes", argv[0], "-valgrind",
              NULL);
    }
#endif
  
    NSAutoreleasePool * pool = [[NSAutoreleasePool alloc] init];
    int retVal = UIApplicationMain(argc, argv, nil, @"PeepsAppDelegate");
    [pool release];
    return retVal;
}

You will obviously want to define the VALGRIND_REXEC macro, if you need valgrind output. You can also pass different command line arguments to valgrind. E.g. you can switch to different valgrind tools this way, or pipe everything to a log-file.
Update: I finally got around to trying out this method. One problem here is that valgrind will fail to run, since it tries to open /dev/random, which I guess is not allowed for sandboxed applications. But one can fix this by patching and recompiling valgrind, which is not too hard. Especially when using MacPorts. Furthermore, I needed to add --dsymutil=yes to the valgrind options, or else the program would just crash.

Cleaning up MacPorts

I like to use MacPorts for installing certain libraries and programs. When running an update old versions are still kept on your disk. I never did care for those old versions, so you can simply let MacPorts remove them while updating:

 $ sudo port -u upgrade outdated

This is taken from another blog post.

Pimping MobileMe #1: Expiration dates on folders

Well, MobileMe does not yet support the expiration of mails in folders. Especially for mailing lists, I want this feature. After let's say 90 days the mails should be automatically deleted. I do not need to store mailing list contents for longer. So I took the nice tool imapfilter and wrote a config file that expires old mails from a selection of folders. I installed a cronjob on my web server that now automatically calls imapfilter every night. The config file (~/.imapfilter/config.lua) looks like this:


Myaccount = IMAP {
   server = 'mail.me.com',
   username = 'your.name@me.com',
   password = 'yourpassword',
   ssl = 'tls1'
}


folders = {'folder1', 'folder2', 'folder3'}


for k,folder in pairs(folders) do
   foldername = 'Mailinglists/' .. folder
   results = myaccount[foldername]:is_older(90)
   myaccount[foldername]:delete_messages(results)
end

My mailing lists are hosted under a sub-folder called "Mailinglists", as you can notice in the config file above. You can tweak this to your liking. It might also be useful to set different expiration dates. It's easy to augment the lua script to do so.

Migrating from GMX IMAP to MobileMe IMAP

I am currently migrating my >4GiB of mail data from GMX to MobileMe. I have been using offlineimap regularly to backup my mail data. So first step was to update the backup. Now the next step is to run imapsync, which can migrate whole IMAP accounts. The full command I am using for this is:

imapsync --sep1 '/' --prefix1 ''  --ssl1 --ssl2 --authmech1 PLAIN --authmech2 PLAIN --host1 imap.gmx.net --user1 "yourname@gmx.de" --host2 mail.me.com --user2 "yourname@me.com"

I will report back once the sync has finished, this might take a while.

Update: Yup, it worked. My mail is now hosted on MobileMe. More disk space. What is missing at MobileMe is that it does not support automatically expiring mail folders. I.e. for mailing lists, I sort the mails into subfolders for each list. The lists are often high volume traffic, and not very important. So I'd like the mails to expire after, say, 30 days. With GMX this was not a problem, but MobileMe does not yet have such a feature. However, I found a nice tool called imapfilter. It allows you to do all kinds of stuff, besides other things it allows you to filter mails according to date and to delete them. This is what I will do. I'll write another post, when I found out how to do that exactly.