Friday, September 09, 2011

iChat, Growl, and Lion

UPDATE: This script has been improved; the new version is now on github.
UPDATE 2: Growl 1.3 was just released, so I've updated the script to work with 1.3.  If you were using this script and it broke with 1.3, then just download the updated version from the same place.  If you're still using Growl 1.2.2, you'll need to use the old version of this script.  All the rest of the instructions are the same regardless of which version of Growl you're using.

Lion is the latest version of OS X.  iChat is the chat program that's part of OS X.  Growl is a system that lets all sorts of programs post little notification windows on your screen.

For example, if you get a message in iChat while you're in another window, it's nice to have the message pop up on your screen so you can decide whether or not to switch windows to read it.  Or if a buddy logs in, it's nice to know about that (which can be done in iChat with an audio notification), and see who it is without having to check your buddy list.

I used to use Chax for this.  Chax is a pretty good program, but when a new version of OS X comes out, it takes a little while to catch up.  The main reasons I use Chax are:
  1. Accept text chat invitations without being prompted.
  2. Put all my buddies in the same window.
  3. Give me Growl notifications.
Everything but the Growl feature is now built into iChat in Lion.  (The auto-accept is an AppleScript that ships with Lion; you can turn that on in Preferences:Alerts.  The option for "Show all my accounts in one list" is in Preferences:General.)

Chax isn't yet compatible with Lion, so I searched online and found a little AppleScript program to do a bit of what I need.  Using the ideas from that, I expanded it a bit.  Goodbye Chax; I'll miss you!

To use this:
  1. Install Growl, if you don't have it already.
  2. Visit in your web browser.  (You can use the formatted page instead.)
  3. Open AppleScript Editor (in Applications:Utilities).
  4. Copy and paste the code from the web page into a new script.
  5. Save it to your home directory's Library:Scripts:iChat folder.  (I used the filename "Growl".)  The iChat folder might not exist; you need to create it.  By default, your Library folder is hidden.  (Personally, I solved that by saving to the desktop, and then I ran "open ~/Library/Scripts" in Terminal so that the Scripts window opened in Finder, and dragged my script there.)
  6. In iChat, open the Preferences.
  7. Under Alerts, select the "Message Received" event.
  8. Turn on "Run an AppleScript script:", and select the script you saved earlier.
  9. Repeat the last two steps for each event you're interested in.

Note that the "Message Received in Chat Room" (which is called "addressed message received" in the AppleScript) and "Invitation to Share Buddy's Screen" (a.k.a. "received remote screen sharing invitation") aren't currently enabled, since it looks like there may be a bug in iChat that will cause them to be confused with each other.

If you use the Growl script for chat invitations, you can't also use the Auto-Accept script, since you can only select one script to be run.  Because of this, I've added an auto-accept capability to this Growl script.  By default, it's turned off, but you can turn it on.  Near the top of the program, there's a line that says:

        property autoAccept : {""}

Change this to:

        property autoAccept : {"text"}

Suggestions for improvements are welcome.  Share and enjoy!

(If you expected the source code to be here in my blog, I've moved it.  See for my latest version.)

Tuesday, July 05, 2011

Copying a filesystem on Linux

In the last two days, I've had to copy a filesystem on my Linux development VM four times (to grow it).  I eventually made a shell script for it, since I kept forgetting details (like the arguments to cpio).  Since I started this blog to have a place to publish random bits of information for future Google users, I'm putting the script here.  The script is below; you pass in the old device name (e.g., sda) as the first argument, and the new one (e.g., sdc) as the second.

There's some significant assumptions in this script, and they may not be warranted for your system:
  • It assumes that you have only one filesystem, and that's on the first partition of the disk.  (It's also easy to add a swap partition using mkswap.)
  • You're expected to use fdisk to set up the new filesystem yourself.  (Linux is fs type 83; Linux swap is 82.)
  • This builds an ext3 filesystem.
  • It assumes that once you're finished, you'll be putting the new device into the old device's place; in other words, it doesn't try to replace sdc with sda in /etc files.  (If you're only using UUIDs in your fstab, this probably doesn't matter.)
  • It uses a default grub install, more or less.  If you're using LILO, or have got something funky going on, it might not carry over.  If you've customized your grub.cfg, for example, it will get overwritten.  (However, changes in grub.d are fine and will be reflected.)
  • As used here, cpio will make sparse files out of non-sparse files.  Normally, this is a good thing, but some programs (such as BitTorrent clients) will intentionally build non-sparse files.  (It's ok to turn them sparse after the torrent completes, and it's only a fragmentation hit if they get turned into sparse before the torrent is done.)
  • This uses a 64k I/O block size.  I'm doing that because that's the block size of the default QEMU backing store under qcow2.  (Remember, in my case, I'm doing this for a VM.)  I'm not sure if that helps anything or not (since the I/O goes through many layers between cpio and qcow), but it's not going to hurt anything.  However, you may want to replace the -C65536 with -B if you're using a normal disk.

This script is better to learn from than to use directly.  But if you need to copy a filesystem (to grow it, move off a failing disk, defragment it (rarely necessary on Linux), or whatever), this may give you a good starting point.
#! /bin/sh

set -xe

fdisk /dev/${out}
mke2fs -j /dev/${out}1
mount /dev/${out}1 /mnt
find / -xdev -depth -print0 | cpio -pdmV0a --sparse -C65536 /mnt
grub-install --root-directory=/mnt /dev/${out}
perl -pi.bak -e s/$(blkid -o value -s UUID /dev/${in}1)/(blkid -o value -s UUID /dev/${out}1)/g /mnt/etc/fstab /mnt/boot/grub/grub.cfg
umount /mnt

A few implementation comments:

  • This makes a lot of assumptions (listed above) for simplicity.  Feel free to improve it.
  • The -depth is so that find will print the directories after their contents, which will cause cpio to fix the modification time.  Otherwise, cpio will create the directory, change its mtime to match the original, and then add contents, thereby changing the directory's mtime to the current time.
  • The Perl can be done by hand: it's changing the old filesystem's UUID to the new filesystem's UUID in fstab and grub.cfg.  You can find out the IDs by running sudo blkid.
  • Note that the Perl line is long; be careful when copying it.  It spans from perl to grub.cfg.

Share and enjoy!

PS: If you're wanting to grow a filesystem on the same disk (i.e., you've deleted the next partition and want to use its free space), there's utilities to do that.  This script is about moving to a different disk.

(Disclaimer: I release the above code to the public domain, as if the public actually wants it.  It's for education, and should be evaluated and customized before running it in any particular environment.  I provide no warranty, expressed or implied; if it breaks, you get to keep both pieces.)

Tuesday, April 26, 2011

iPhone location log to Google Earth, all with shell commands

If you're a privacy geek, then you've no doubt heard people talking about a file on the iPhone called consolidated.db.  This file stores all the places your phone has been.

I don't know why everybody is making such a fuss about it.  I expect my own computers (including my phone) to store logs of what I'm doing.  It's good to be able to reset this information before I give a device to somebody else, and hey, the iPhone has a comprehensive reset function!

I think the concern is that the phone is sending this to Apple periodically.  That would be a very serious concern.  But until somebody gives any sort of evidence to that effect, I think it's just a badly-managed log file.


Yesterday, I saw that I had apparently bought a song on my iPhone that I didn't recognize.  I listened to it, and it didn't sound familiar.  It also didn't sound like something I'd buy.  After a couple of blind alleys, I checked the purchase date in iTunes.  I then correlated that to Google Latitude's history.  Ah, it was a mall in Santa Cruz.  Now I remembered: the reason I went into the mall was to get a lemonade at Starbucks, and while I was there my friend gave me one of the vouchers that Starbucks has for a free iTunes song.

While I was relaying this story to another friend, he asked if I'd used consolidated.db to read it.  Nope, but not a bad idea.  I started wondering what was actually in there.  Most stuff I saw to read that requires me to download some stranger's code, and I wasn't too thrilled about that.

OS X and iOS tend to use property lists and SQLite for all kinds of storage, so I'd guessed it was SQLite.  That meant that it could be read from the command line using the sqlite3 utility, which is part of OS X.

So, without further ado, here's a quick rundown on how to look at all this stuff straight from the command line.  There's one bit of Perl to convert it to Google Earth's KML format, but the Perl really isn't necessary to view the data in lat/long format.  Note that making use of the following requires a certain degree of technical savvy; knowing SQL is going to be very useful to fiddle with the data.  However, you don't need to know anything about OS X (except how to start the terminal), iPhone development, etc.

    # Get the current time in the format used by the database (Apple's CFAbsoluteTime, which is the number of seconds since Jan 1, 2001 at midnight GMT; we use UTC instead of GMT here, but it's close enough for these purposes):
$ perl -MTime::Local -e 'print time() + timegm(0,0,0,1,0,70) - timegm(0,0,0,1,0,101), "\n"'
   # Go to the backups
$ cd ~/Library/Application\ Support/MobileSync/Backup/
   # Go to the most recent backup
$ cd $( ls -t | head -1 )
   # The files in here are named (almost) randomly.
   # Get the database's filename by looking at table lists of all of the SQLite databases and finding one that has CellLocation
   # (This prints the filename and leaves it in $db)
$ for db in $( file * | sed -e '/SQLite/ !d' -e 's/:.*//' ) ; do sqlite3 $db .tables | grep -q CellLocation && echo $db && break ; done
   # Your filename will be different; it's based on a hash of the data.
   # View the file
$ sqlite3 $db
   -- Take a look at the schema of CellLocation
sqlite> schema CellLocation
CREATE TABLE CellLocation (MCC INTEGER, MNC INTEGER, LAC INTEGER, CI INTEGER, Timestamp FLOAT, Latitude FLOAT, Longitude FLOAT, HorizontalAccuracy FLOAT, Altitude FLOAT, VerticalAccuracy FLOAT, Speed FLOAT, Course FLOAT, Confidence INTEGER, PRIMARY KEY (MCC, MNC, LAC, CI));
   -- (That will also print a few other tables that you can ignore.)
   -- View the contents of the past week (this uses the time computed at the beginning of this walkthrough; 604800 is the number of seconds in a typical week)
sqlite> select latitude,longitude,timestamp from CellLocation where timestamp > 325464777 - 604800 order by timestamp;
   -- Back to the shell
sqlite> .exit
   # Put the data through Perl to convert to KML
$ sqlite3 $db "select longitude,latitude,timestamp from CellLocation where timestamp > 325464777 - 1209600 order by timestamp, horizontalAccuracy" | perl -nw -MPOSIX -MTime::Local -e 'BEGIN { our $LTS=0; print "\n";} next if /^[0.]+\|[0.]+\|/; my ($long, $lat, $tstamp) = split /\|/; next if $tstamp==$LTS; $LTS=$tstamp; my @time = gmtime($tstamp - timegm(0,0,0,1,0,70) + timegm(0,0,0,1,0,101)); my $tstr = strftime("%Y-%m-%dT%H:%M:%SZ", @time); print "$tstr$long $lat 0\n"; END { print ""; } ' > ~/Desktop/locations.kml
   # Open it in Google Earth
$ open ~/Desktop/locations.kml

If you're trying to make sense of the Perl (particularly the $LTS bit), note that each timestamp tends to have several locations with varying precisions.  I'm not sure whether that's a triangulation of the user's position, or of the cells the user was able to communicating with, but I suspect it's increasingly precise triangulations of the position that are recorded with the same timestamp.  This code will use the first recorded position for each timestamp, which (because of the ORDER BY clause) is going to be the one with the smallest horizontalAccuracy.  I'm currently supposing that the horizontalAccuracy column is actually the precision radius.

I also played with using the WifiLocation table (which is similar), but it seems to be pretty unreliable data; it seems to include data based on one database that's about 10 miles too far north.  (At least, in my case, it kept showing me jumping abruptly between Santa Clara / Sunnyvale and Fremont.  I haven't been on the Fremont side of the bay in over a month.)  I read online that Verizon customers need to use CdmaCellLocation instead of CellLocation.

If you're a potential employer, please note that the Perl above was thrown together for my own experimenting.  I just put it here on the blog in case somebody else wanted to fiddle with it, and it's not at all what I'd put into production.

Well, that's about all the time I felt like putting into decoding that file.  Feel free to post improvements in the comments section below.  Share and enjoy!

(Disclaimer: I release the above code to the public domain, if you care.  It's for education, and is not intended to run in any particular environment.  I provide no warranty, expressed or implied; if it breaks, you get to keep both pieces.)

Wednesday, March 16, 2011

Magic GarageBand Chords

Magic GarageBand Chords

I sat down to practice my guitar by improvising along with GarageBand's "Magic GarageBand" feature. This handy feature lets you put together a five-piece band from a wide selection of instruments and styles, which will play a song in a genre of your choosing. You can accompany them (using a mic, MIDI instrument, or DI from a guitar or other instrument), and record the result. By choosing the instruments and styles, you can get very different-sounding songs.

This is great if you want to jam but don't have a band with you at the moment. But, there's one bit of information that's missing.

In order to accomplish this mix-and-match, each genre (of the nine possibilities) plays the same basic chord changes. The individual parts riff all over those, but since they're following the same basic pattern, you get a good harmony. This is handy information to know if you're trying to play along! All of these have pretty straightforward patterns, but it's a lot easier to know the changes ahead of time.

Regrettably, I wasn't able to keep up with the chord changes when I sat down to play the other day. I went online to find them, but couldn't find anybody who had written them down for the world to see.All Google brought me is other people looking for the changes.

Hence, today's post: the chord changes for all nine genres in Magic GarageBand. If you have any suggestions for improvements, let me know! (I mean, something that makes this list closer to what GarageBand actually plays. I'm not looking for improvements to the songs; talk to Apple about that!)

Blues (E, 90bpm, 4/4)

When you create the GarageBand project, it says the song is in B,but it's really in E. The loops are also in the wrong key, which can lead to surprises if you try to use them in your own project!

Other than that, there's really no surprises here. This is a moderate tempo 12-bar blues, which is probably what you were expecting here. There's not even a real intro or outro.

Intro1EPickup; can use A as well.
Verse 12EEEE
Verse 214EEEE
Verse 326EEEE

Rock (D, 140bpm, 4/4)

Intro 1D Pickup
2D D D D
Verse 1 6D D D D
10F F D D
14D D D D
18F F A A
Chorus 22BmBmE E Remodulating in the chorus. Can use G♯° instead of E.
26G G A A
Verse 2 30D D D D
34F F D D
38D D D D
42F F D D
Ending 46D D D D
50F F D D
53D D D D Sustained fade

Jazz (F, 164bpm, 4/4)

This is similar to the 12-bar blues progression, but with a more jazzy twist to the chords actually being used.

The basic idea is still the 12 bar pattern F / F / F / F / B♭ / B♭/ F / F / B♭ / C / F / F. However, there's some changes that are often used:

  • The B♭ (IV) is often played as D° (vi°)
  • The F is often played as an F7, particularly in long stretches.
  • Bars 7–8 and 11–12 (of the 12-bar pattern) are usually played to walk into the next phrase (changing chords every two beats) instead of being played straight. In particular, bar 8 is very rarely played as an F, but if you're improvising over an F you'll still be fine.

First, here's the basic pattern:

1F F F F
5B♭B♭F F
9B♭C F F

Now, here's the pattern as I've interpreted it. As I said, there's a lot of possibilities here.

Intro 1- Pickup
Head 2F7 F7 F7 F7 May substitute F for F7.
6F7 C; DNot sure about this phrase. May substitute B♭ or even Fm6 for D°.
10Gm C F7; FB♭; C
Chorus 114F7 F F
18F7 C; DSome of the accompaniments can get a bit muddied in the chords here.
22B♭ Gm; CF; F7B♭; C
Chorus 2Too tired to transcribe these; it's jazz, so just get the idea from the pattern I've done so far.
Chorus 3
Head Out

Country (G, 100bpm, 4/4)

Another 12-bar blues progression, played pretty much straight this time. (In case you haven't gotten the picture, 12-bar blues is great for improvising over, so it's used a lot in Magic GarageBand.)

Intro 1- Pickup
Verse 1 2GGGG
10DCGDIf you want to keep up with the lead-in for the last bar, it's D / C / Bm / Am
Verse 214GGGG
Verse 326GGGG
34DCGDSame lead-in as the end of verse 1
Ending 38GGG Sustained fade

Reggae (C min, 75bpm, 4/4)

I don't really know reggae, so this may be totally off. I've marked it as C minor, even though when you create the project GarageBand lists it as C major.

Intro 1- Pickup
Verse 1 2CmCmA♭; Gm7Cm
6CmCmA♭; Gm7Cm
Chorus 10B♭m; A♭A♭; E♭B♭m; A♭A♭; E♭
14B♭m; A♭A♭; E♭B♭m; A♭A♭; E♭
Verse 218CmCmA♭; Gm7Cm
22CmCmA♭; Gm7Cm
26-Sustain from last note

Funk (C, 120bpm, 4/4)

Also really not sure how accurate this one is.

Intro 1CmCmCmE♭
Verse 1 5C C C CIf it's too monotonous, you can make the first and third bars F; C (two beats each)
9C C C C
13F F C C
17G G C Em7You can use a G for the Em7
Bridge 21F F F F
25F F G G
29G Fermata from bar 28
Verse 230C C C C
34C C C C
Ending 38G7G7C C
42G7G7C C
46C E♭

Latin (Fm, 98bpm, 4/4)

This is pretty simple. There's just one pattern: Fm, E♭, D♭, C,for two beats each.

Intro 1Fm Pickup
Verse 1 2Fm,E♭D♭,CFm,E♭D♭,C
Verse 210Fm,E♭D♭,CFm,E♭D♭,C
Bridge 18Fm FmFmFm
22Fm FmFmFm
26Fm FmFmFm
Verse 332Fm,E♭D♭,CFm,E♭D♭,C

Roots Rock (G, 76bpm, 4/4)

Not really sure what distinguishes the chorus from the verse, but it could just be the instruments I picked don't really do much differently. This is a simple pattern, but has a lot of room for improvisation.

Intro 1G Pickup
Verse 1 2G AmC D
6G AmC D
Chorus 10G AmC D
14G AmC D
Bridge 18EmBmAmG
Verse 226G AmC D
30G AmC D
Outro 34G G

Slow Blues (A♭, 88bpm, 12/8)

Wow, there IS something other than 4/4 in here! Unsurprisingly, it is still 12-bar blues, although with a IV in the second bar to keep it interesting.

Intro 1E♭ Pickup
Verse 1 2A♭D♭A♭A♭
Verse 214A♭D♭A♭A♭
Ending 24A♭A♭A♭The measure breaks are hinky

Saturday, January 08, 2011

Imminent Death of the Net Predicted

I didn't know that it was coming this soon, but in mid-February, the Internet will run out of addresses.

My blog post's title is a bit tongue-in-cheek. People have been talking about the net not being able to handle the growth rate for decades. The address space exhaustion is going to take some work to deal with, but it's not going to cause things to grind to a halt.

There's already some great resources to explain this to people who don't know about the problem:
What I couldn't find is an answer to my friend's question, "What will happen when we run out?" Nobody knows for sure, but here's a basic rundown.

Internet addresses are globally administered by IANA. They give blocks of addresses to the regional groups, RIRs. These cover continent-sized areas, like Europe, Asia/Pacific, North America, etc. The RIRs give blocks to the local registries (LIRs), which are usually large ISPs, but may also be large organizations (like Stanford, although that's mostly because they were one of the first places on the Internet).

Once IANA runs out (which is the event in February), then the individual regions can't get more addresses. The first to run out will be APNIC (Asia/Pacific), in about eight months or so (we're guessing; nobody can be sure). Then APNIC won't be able to give out new addresses to its LIRs. When an LIR can't get new addresses, it can't attach new customers. Its customers are mostly small ISPs or companies.

So there's a trickle in the "supply chain" before things get to the end users. You won't see problems until it gets down to this level, although you may see some changes as ISPs hurriedly try to prepare (with IPv6; I'll get to that in a minute).

Once a small ISP runs out, then it can't put new servers on the Internet: that means no new web servers, no new mail servers, nothing. (To a limited degree, it's possible to piggyback these on existing servers in some cases.) The ISP can attach new home customers (who don't use servers) for a little while, but if they attach too many, then people won't be able to connect. (A home ISP tends to hand out addresses for two hours at a time, as long as the customer's router or computer is on.)

At that point, to deal with that, they'll probably start putting large blocks of home customers behind NATs. NATs let several computers use the same IP address. However, you can't have a server behind a NAT, it can be unreliable, and a lot of games, video or voice chat programs, file-sharing programs, etc. don't behave well with NAT. Because APNIC is running out of addresses, much of Russia and China are already behind regional NATs.

There's also a possibility that organizations will start trading small blocks of IP addresses. (Nobody "owns" them; they're allocated by IANA.) However, routers will have a hard time keeping up with that (a router is designed to think in large blocks, and lots of tiny blocks can be a problem); this might make the Internet more flaky.

I don't really know how the situation is going to be for end users. There'll be some hiccups, but it'll probably come down to just a little bit of reconfiguration; your ISP will tell you to do it, and you'll have some notice. If you've got a particularly old router, you may end up having to replace it.

Ultimately, the solution is to go to IPv6. This has a lot more addresses: 3.4x10^28; that's enough for every gram of matter on Earth to have 10 addresses. (The current mechanism, IPv4, has about 4 billion addresses.) However, IPv6 adoption has been slow, because customers aren't asking for it. Most devices support IPv6, but some don't. Not many websites are currently equipped for IPv6; you have to have IPv4.

I migrated my home network to IPv6 yesterday, and it was a breeze. Right now it's just to get a dancing turtle on Kame's website, but I recommend that my techie friends with a home network do the migration. Even though your ISP probably doesn't provide IPv6 connectivity, there's still peers all over the place that'll give you tunnels.

IPv6 Certification Badge for piquan
For the technically-minded among us,I actually did the migration twice: once using 6to4 for connectivity with the Anycast2002:c058:6301:: peer for 6to4. Then, I did it again using a tunnel (using FreeBSD's gif interface, which also works on OS X) that I got for free from Hurricane Electric's website. The tunnel is faster (23ms ping time vs the 6to4 peer's 96ms), and their website walks you through the process; it's got easy instructions for Windows, OS X, Linux, BSD, or whatever else you've got.