Friday, September 09, 2011

iChat, Growl, and Lion

UPDATE: This script has been improved; the new version is now on github.
UPDATE 2: Growl 1.3 was just released, so I've updated the script to work with 1.3.  If you were using this script and it broke with 1.3, then just download the updated version from the same place.  If you're still using Growl 1.2.2, you'll need to use the old version of this script.  All the rest of the instructions are the same regardless of which version of Growl you're using.

Lion is the latest version of OS X.  iChat is the chat program that's part of OS X.  Growl is a system that lets all sorts of programs post little notification windows on your screen.

For example, if you get a message in iChat while you're in another window, it's nice to have the message pop up on your screen so you can decide whether or not to switch windows to read it.  Or if a buddy logs in, it's nice to know about that (which can be done in iChat with an audio notification), and see who it is without having to check your buddy list.

I used to use Chax for this.  Chax is a pretty good program, but when a new version of OS X comes out, it takes a little while to catch up.  The main reasons I use Chax are:
  1. Accept text chat invitations without being prompted.
  2. Put all my buddies in the same window.
  3. Give me Growl notifications.
Everything but the Growl feature is now built into iChat in Lion.  (The auto-accept is an AppleScript that ships with Lion; you can turn that on in Preferences:Alerts.  The option for "Show all my accounts in one list" is in Preferences:General.)

Chax isn't yet compatible with Lion, so I searched online and found a little AppleScript program to do a bit of what I need.  Using the ideas from that, I expanded it a bit.  Goodbye Chax; I'll miss you!

To use this:
  1. Install Growl, if you don't have it already.
  2. Visit in your web browser.  (You can use the formatted page instead.)
  3. Open AppleScript Editor (in Applications:Utilities).
  4. Copy and paste the code from the web page into a new script.
  5. Save it to your home directory's Library:Scripts:iChat folder.  (I used the filename "Growl".)  The iChat folder might not exist; you need to create it.  By default, your Library folder is hidden.  (Personally, I solved that by saving to the desktop, and then I ran "open ~/Library/Scripts" in Terminal so that the Scripts window opened in Finder, and dragged my script there.)
  6. In iChat, open the Preferences.
  7. Under Alerts, select the "Message Received" event.
  8. Turn on "Run an AppleScript script:", and select the script you saved earlier.
  9. Repeat the last two steps for each event you're interested in.

Note that the "Message Received in Chat Room" (which is called "addressed message received" in the AppleScript) and "Invitation to Share Buddy's Screen" (a.k.a. "received remote screen sharing invitation") aren't currently enabled, since it looks like there may be a bug in iChat that will cause them to be confused with each other.

If you use the Growl script for chat invitations, you can't also use the Auto-Accept script, since you can only select one script to be run.  Because of this, I've added an auto-accept capability to this Growl script.  By default, it's turned off, but you can turn it on.  Near the top of the program, there's a line that says:

        property autoAccept : {""}

Change this to:

        property autoAccept : {"text"}

Suggestions for improvements are welcome.  Share and enjoy!

(If you expected the source code to be here in my blog, I've moved it.  See for my latest version.)

Tuesday, July 05, 2011

Copying a filesystem on Linux

In the last two days, I've had to copy a filesystem on my Linux development VM four times (to grow it).  I eventually made a shell script for it, since I kept forgetting details (like the arguments to cpio).  Since I started this blog to have a place to publish random bits of information for future Google users, I'm putting the script here.  The script is below; you pass in the old device name (e.g., sda) as the first argument, and the new one (e.g., sdc) as the second.

There's some significant assumptions in this script, and they may not be warranted for your system:
  • It assumes that you have only one filesystem, and that's on the first partition of the disk.  (It's also easy to add a swap partition using mkswap.)
  • You're expected to use fdisk to set up the new filesystem yourself.  (Linux is fs type 83; Linux swap is 82.)
  • This builds an ext3 filesystem.
  • It assumes that once you're finished, you'll be putting the new device into the old device's place; in other words, it doesn't try to replace sdc with sda in /etc files.  (If you're only using UUIDs in your fstab, this probably doesn't matter.)
  • It uses a default grub install, more or less.  If you're using LILO, or have got something funky going on, it might not carry over.  If you've customized your grub.cfg, for example, it will get overwritten.  (However, changes in grub.d are fine and will be reflected.)
  • As used here, cpio will make sparse files out of non-sparse files.  Normally, this is a good thing, but some programs (such as BitTorrent clients) will intentionally build non-sparse files.  (It's ok to turn them sparse after the torrent completes, and it's only a fragmentation hit if they get turned into sparse before the torrent is done.)
  • This uses a 64k I/O block size.  I'm doing that because that's the block size of the default QEMU backing store under qcow2.  (Remember, in my case, I'm doing this for a VM.)  I'm not sure if that helps anything or not (since the I/O goes through many layers between cpio and qcow), but it's not going to hurt anything.  However, you may want to replace the -C65536 with -B if you're using a normal disk.

This script is better to learn from than to use directly.  But if you need to copy a filesystem (to grow it, move off a failing disk, defragment it (rarely necessary on Linux), or whatever), this may give you a good starting point.
#! /bin/sh

set -xe

fdisk /dev/${out}
mke2fs -j /dev/${out}1
mount /dev/${out}1 /mnt
find / -xdev -depth -print0 | cpio -pdmV0a --sparse -C65536 /mnt
grub-install --root-directory=/mnt /dev/${out}
perl -pi.bak -e s/$(blkid -o value -s UUID /dev/${in}1)/(blkid -o value -s UUID /dev/${out}1)/g /mnt/etc/fstab /mnt/boot/grub/grub.cfg
umount /mnt

A few implementation comments:

  • This makes a lot of assumptions (listed above) for simplicity.  Feel free to improve it.
  • The -depth is so that find will print the directories after their contents, which will cause cpio to fix the modification time.  Otherwise, cpio will create the directory, change its mtime to match the original, and then add contents, thereby changing the directory's mtime to the current time.
  • The Perl can be done by hand: it's changing the old filesystem's UUID to the new filesystem's UUID in fstab and grub.cfg.  You can find out the IDs by running sudo blkid.
  • Note that the Perl line is long; be careful when copying it.  It spans from perl to grub.cfg.

Share and enjoy!

PS: If you're wanting to grow a filesystem on the same disk (i.e., you've deleted the next partition and want to use its free space), there's utilities to do that.  This script is about moving to a different disk.

(Disclaimer: I release the above code to the public domain, as if the public actually wants it.  It's for education, and should be evaluated and customized before running it in any particular environment.  I provide no warranty, expressed or implied; if it breaks, you get to keep both pieces.)

Tuesday, April 26, 2011

iPhone location log to Google Earth, all with shell commands

If you're a privacy geek, then you've no doubt heard people talking about a file on the iPhone called consolidated.db.  This file stores all the places your phone has been.

I don't know why everybody is making such a fuss about it.  I expect my own computers (including my phone) to store logs of what I'm doing.  It's good to be able to reset this information before I give a device to somebody else, and hey, the iPhone has a comprehensive reset function!

I think the concern is that the phone is sending this to Apple periodically.  That would be a very serious concern.  But until somebody gives any sort of evidence to that effect, I think it's just a badly-managed log file.


Yesterday, I saw that I had apparently bought a song on my iPhone that I didn't recognize.  I listened to it, and it didn't sound familiar.  It also didn't sound like something I'd buy.  After a couple of blind alleys, I checked the purchase date in iTunes.  I then correlated that to Google Latitude's history.  Ah, it was a mall in Santa Cruz.  Now I remembered: the reason I went into the mall was to get a lemonade at Starbucks, and while I was there my friend gave me one of the vouchers that Starbucks has for a free iTunes song.

While I was relaying this story to another friend, he asked if I'd used consolidated.db to read it.  Nope, but not a bad idea.  I started wondering what was actually in there.  Most stuff I saw to read that requires me to download some stranger's code, and I wasn't too thrilled about that.

OS X and iOS tend to use property lists and SQLite for all kinds of storage, so I'd guessed it was SQLite.  That meant that it could be read from the command line using the sqlite3 utility, which is part of OS X.

So, without further ado, here's a quick rundown on how to look at all this stuff straight from the command line.  There's one bit of Perl to convert it to Google Earth's KML format, but the Perl really isn't necessary to view the data in lat/long format.  Note that making use of the following requires a certain degree of technical savvy; knowing SQL is going to be very useful to fiddle with the data.  However, you don't need to know anything about OS X (except how to start the terminal), iPhone development, etc.

    # Get the current time in the format used by the database (Apple's CFAbsoluteTime, which is the number of seconds since Jan 1, 2001 at midnight GMT; we use UTC instead of GMT here, but it's close enough for these purposes):
$ perl -MTime::Local -e 'print time() + timegm(0,0,0,1,0,70) - timegm(0,0,0,1,0,101), "\n"'
   # Go to the backups
$ cd ~/Library/Application\ Support/MobileSync/Backup/
   # Go to the most recent backup
$ cd $( ls -t | head -1 )
   # The files in here are named (almost) randomly.
   # Get the database's filename by looking at table lists of all of the SQLite databases and finding one that has CellLocation
   # (This prints the filename and leaves it in $db)
$ for db in $( file * | sed -e '/SQLite/ !d' -e 's/:.*//' ) ; do sqlite3 $db .tables | grep -q CellLocation && echo $db && break ; done
   # Your filename will be different; it's based on a hash of the data.
   # View the file
$ sqlite3 $db
   -- Take a look at the schema of CellLocation
sqlite> schema CellLocation
CREATE TABLE CellLocation (MCC INTEGER, MNC INTEGER, LAC INTEGER, CI INTEGER, Timestamp FLOAT, Latitude FLOAT, Longitude FLOAT, HorizontalAccuracy FLOAT, Altitude FLOAT, VerticalAccuracy FLOAT, Speed FLOAT, Course FLOAT, Confidence INTEGER, PRIMARY KEY (MCC, MNC, LAC, CI));
   -- (That will also print a few other tables that you can ignore.)
   -- View the contents of the past week (this uses the time computed at the beginning of this walkthrough; 604800 is the number of seconds in a typical week)
sqlite> select latitude,longitude,timestamp from CellLocation where timestamp > 325464777 - 604800 order by timestamp;
   -- Back to the shell
sqlite> .exit
   # Put the data through Perl to convert to KML
$ sqlite3 $db "select longitude,latitude,timestamp from CellLocation where timestamp > 325464777 - 1209600 order by timestamp, horizontalAccuracy" | perl -nw -MPOSIX -MTime::Local -e 'BEGIN { our $LTS=0; print "\n";} next if /^[0.]+\|[0.]+\|/; my ($long, $lat, $tstamp) = split /\|/; next if $tstamp==$LTS; $LTS=$tstamp; my @time = gmtime($tstamp - timegm(0,0,0,1,0,70) + timegm(0,0,0,1,0,101)); my $tstr = strftime("%Y-%m-%dT%H:%M:%SZ", @time); print "$tstr$long $lat 0\n"; END { print ""; } ' > ~/Desktop/locations.kml
   # Open it in Google Earth
$ open ~/Desktop/locations.kml

If you're trying to make sense of the Perl (particularly the $LTS bit), note that each timestamp tends to have several locations with varying precisions.  I'm not sure whether that's a triangulation of the user's position, or of the cells the user was able to communicating with, but I suspect it's increasingly precise triangulations of the position that are recorded with the same timestamp.  This code will use the first recorded position for each timestamp, which (because of the ORDER BY clause) is going to be the one with the smallest horizontalAccuracy.  I'm currently supposing that the horizontalAccuracy column is actually the precision radius.

I also played with using the WifiLocation table (which is similar), but it seems to be pretty unreliable data; it seems to include data based on one database that's about 10 miles too far north.  (At least, in my case, it kept showing me jumping abruptly between Santa Clara / Sunnyvale and Fremont.  I haven't been on the Fremont side of the bay in over a month.)  I read online that Verizon customers need to use CdmaCellLocation instead of CellLocation.

If you're a potential employer, please note that the Perl above was thrown together for my own experimenting.  I just put it here on the blog in case somebody else wanted to fiddle with it, and it's not at all what I'd put into production.

Well, that's about all the time I felt like putting into decoding that file.  Feel free to post improvements in the comments section below.  Share and enjoy!

(Disclaimer: I release the above code to the public domain, if you care.  It's for education, and is not intended to run in any particular environment.  I provide no warranty, expressed or implied; if it breaks, you get to keep both pieces.)