tag:blogger.com,1999:blog-133223522024-03-13T00:14:56.654-07:00StopwordsThe rantings, ravings, musings, wisdom, and silliness of PiquanPiquanhttp://www.blogger.com/profile/06983189170367989592noreply@blogger.comBlogger46125tag:blogger.com,1999:blog-13322352.post-20497803049476988232020-02-15T01:13:00.000-08:002020-02-22T15:52:46.019-08:00LEGO International Space StationYay! My <a href="https://www.lego.com/en-us/product/international-space-station-21321" target="_blank">LEGO International Space Station model</a> finally arrived!<br />
<br />
As I eagerly assembled it, I realized that I really don't know much about the ISS today, or what the different parts of the model were. Fortunately, with the help of NASA's excellent pamphlet <a href="https://www.nasa.gov/sites/default/files/atoms/files/np-2015-05-022-jsc-iss-guide-2015-update-111015-508c.pdf" target="_blank">Reference Guide to the International Space Station</a> and <a href="https://en.wikipedia.org/wiki/International_Space_Station" target="_blank">Wikipedia</a>, I was able to learn a whole lot!<br />
<h2>
Pressurized Modules</h2>
The ISS is designed and assembled from the inside out, so that's a good way to look at it. The most interesting parts are right in the middle; these are the parts of the model that are mostly grey and black. I printed out this diagram for reference; these are the major habitable modules, and are really the best landmarks for getting around the ISS.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgbVi2eZlDUxHWC-7lUiC08krWv-YC89mw4qa5XhjmUQlZsJhB_FMdourcPaBjPBbDCREp95ORDrwa9TIBFcct0rn-p1ipzPmKXcAPhtpcERjJXefi4_rUWIuBlKFoInmJoOlwy/s1600/iss-pressurized-diagram.png" imageanchor="1"><img alt="" border="0" data-original-height="535" data-original-width="1449" height="235" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgbVi2eZlDUxHWC-7lUiC08krWv-YC89mw4qa5XhjmUQlZsJhB_FMdourcPaBjPBbDCREp95ORDrwa9TIBFcct0rn-p1ipzPmKXcAPhtpcERjJXefi4_rUWIuBlKFoInmJoOlwy/s640/iss-pressurized-diagram.png" title="Diagram of habitable modules" width="640" /></a></div>
<br />
This photo shows how these are arranged on the model. I've also included a compass of sorts: forward is the front of the ISS, the direction that most of the LEGO box art shows as the front. Like on a ship, aft is the back, port is left, and starboard is right. The two new directions are the nadir, which points towards Earth; and the zenith, which points towards space. In the finished model, the zenith is up.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjxPqC45IqRRMc7g04UQ7c2aQ23WVKwXGr-bsDqDdqC4BPEyfnaRaDoS3nmPKwfHpZpfMG2vM-3zF3_U9GLNyX5eArr_jrGARPLsbBi3pyvvirSd5UDMNdFi-Fl7OV_ANg8v8l-/s1600/iss-pressurized.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img alt="" border="0" data-original-height="1106" data-original-width="1439" height="489" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjxPqC45IqRRMc7g04UQ7c2aQ23WVKwXGr-bsDqDdqC4BPEyfnaRaDoS3nmPKwfHpZpfMG2vM-3zF3_U9GLNyX5eArr_jrGARPLsbBi3pyvvirSd5UDMNdFi-Fl7OV_ANg8v8l-/s640/iss-pressurized.jpg" title="Photo of LEGO model's habitable modules" width="640" /></a></div>
<br />
In this picture, I've labeled a couple of modules that aren't in the earlier diagram.<br />
<br />
Node 3, aka <i>Tranquility</i>, has a couple of smaller modules attached: the Cupola is a little viewing area, like a bay window, that's attached to the nadir (Earth-facing) side of <i>Tranquility</i>. The <i>Leonardo</i> Permanent Multipurpose Module (PMM) is a storage module. The Bigelow Expandable Activity Module (BEAM) is an inflatable module; it's being used to test the concept of inflatable space modules.<br />
<br />
The Japanese Experiment Module (JEM), often referred to as <i>Kibō</i> (きぼう), has several modules. The main one is a pressurized module (PM) where people work (that's the only one that's in my outline of the habitable sections). There's also a pressurized storage area called the Experiment Logistics Module - Pressurized Section (ELM-PS), and a part for experiments that are exposed to space called the Exposed Facility (EF). JEM also has a robot arm called the Remote Manipulator System (RMS) that's used to fiddle with experiments on the outside of the JEM module.<br />
<br />
I labeled something that I think is the <i>Nauka</i> (Нау́ка) Multipurpose Laboratory Module (MLM). At the time the LEGO model was being designed, the MLM was scheduled to launch before the model hit the store shelves. But the MLM is perpetually “almost ready to launch,” having been postponed several times since the original launch date of 2007. However, I think that part of the model might actually be the <i>Pirs</i> (Пирс), a docking unit that's been part of the ISS since its early days. The <i>Nauka</i> is supposed to replace the <i>Pirs</i> when it launches, and that part of the model (a single round 1x1 open stud, <a href="https://brickset.com/parts/design-3062" target="_blank">3062</a>) looks a bit more like the <i>Pirs</i> than the <i>Nauka</i> MLM, so I might change my diagram.<br />
<br />
The black and white parts of the <i>Destiny</i> module in that photo aren't part of the final build: they're placeholders for where the model connects to the truss. In reality, the truss is only connected at the zenith of Node 1, but the model needs more support because it has to deal with gravity.<br />
<h2>
Resupply Craft</h2>
If you build the model according to the instructions, it has five resupply vehicles docked: four Russian Progress vehicles, plus one Orbital Science Cygnus vehicle.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjX9fx5zg6vH4mnA8OYFN5uA6czqZSkw4bweldtXDhyphenhyphenxvSB9p4IdEy63HxKelMCdTcoHNBxi2FuscfRPWMgegdKyhFwMZXqalGN2vi4eTVicplKgzl8mVLaHhb2FHxSz9_YMKyN/s1600/iss-docked.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img alt="" border="0" data-original-height="1022" data-original-width="1545" height="420" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjX9fx5zg6vH4mnA8OYFN5uA6czqZSkw4bweldtXDhyphenhyphenxvSB9p4IdEy63HxKelMCdTcoHNBxi2FuscfRPWMgegdKyhFwMZXqalGN2vi4eTVicplKgzl8mVLaHhb2FHxSz9_YMKyN/s640/iss-docked.jpg" title="Photo of LEGO model's docked resupply vessels" width="640" /></a></div>
<br />
In this photo, I've also labeled the two docking ports that aren't occupied, PMA2 and PMA3. PMA1 is permanently connected between the FGB and Node 1.<br />
<br />
I've also added the Z1 truss module (the bit with the two antennae), which is where the truss attaches to Node 1 (at the zenith port). This was the first part of the truss that was launched. In the model, Z1 also gets extra structural support from <i>Destiny</i>, and the truss also gets more support at the S0 attachment (the white stripe on <i>Destiny</i> in this picture).<br />
<br />
In reality, there have never been four Progress vehicles docked; there are usually one or two. But that's okay, because it's fun! You can pretend that the Progress vehicles are Soyuz capsules instead (they're almost identical) and say that the ISS invited everybody over for a huge pizza party!<br />
<h2>
Integrated Truss Structure</h2>
Most of the ISS's size is the truss structure. The truss's supports the solar panels, as well as many important experiment and support modules.<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9i4EBUE01uGDHlgSV2PQelQOcFLkl8bQVlKiKAp91LV3bJvBTTDWOWA7bbck7a0bm2i7TmVkInOXKtnECM_saN3eWU8D-tnNjAf72-fl0yPQ2akrwYgWT3jGjz8hObhHmmBGE/s1600/iss-truss.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img alt="" border="0" data-original-height="927" data-original-width="1476" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9i4EBUE01uGDHlgSV2PQelQOcFLkl8bQVlKiKAp91LV3bJvBTTDWOWA7bbck7a0bm2i7TmVkInOXKtnECM_saN3eWU8D-tnNjAf72-fl0yPQ2akrwYgWT3jGjz8hObhHmmBGE/s640/iss-truss.jpg" title="Photo of integrated truss structure, stand, astronauts, and undocked resupply vessels" width="640" /></a></div>
<br />
That's a lot of new things here, so let's start by orienting what we've seen already.<br />
<br />
The pressurized parts of the ISS — what the previous photos show — are near the center of the model. You can't see much of them in this photo, but you can see the forward part — <i>Columbus</i>, Node 2, and the JEM — in the middle there as the grey part. The black docking adapter in the middle of Node 2 is PMA2. You can also see the two dish antennae from the Z1 module, not labeled in this picture.<br />
<br />
In an earlier picture, I noted the Z1 module (mounted on the zenith port of Node 1). This is the only point where the pressurized parts of the ISS are connected to the truss. The Z1 truss module connects to S0, S0 connects to S1 and P1, and the connections go outwards from there. (In the model, the truss is connected not just at Z1, but also by two plates from <i>Destiny</i>, just forward and aft of S0. The model has to hold its weight more rigidly than the ISS!)<br />
<br />
One of the small but utterly cool things in this picture is the <a href="https://ams.nasa.gov/" target="_blank">Alpha Magnetic Spectrometer (AMS)</a>. This is something that's about as important to cosmology as the <a href="https://www.nasa.gov/mission_pages/hubble/main/index.html" target="_blank">Hubble Space Telescope</a>. The AMS measures and categorizes cosmic rays, which is how we're currently working to understand the nature of dark matter.<br />
<br />
There are four ExPRESS Logistics Carrier (ELC) modules, which mostly hold experiments that don't need the astronauts to pay close attention to them. There's also one external stowage platform (ESP) module, which is pretty much a place to stow spare parts.<br />
<br />
There are white panels extending aft from the solar panels, and two white wavy modules near the center. These are radiators, which are very important in space; without air, the ISS can't get rid of its heat by other means. (There's a good discussion of this in the book <i><a href="https://smile.amazon.com/dp/0544272994/" target="_blank">What If?</a></i>, in the chapter “How long could a nuclear submarine last in orbit?”) The ISS can't just use air vents to get rid of its excess heat, so it needs to have giant radiators.<br />
<h2>
Base</h2>
The model nominally comes with two astronauts (<a href="https://brickset.com/parts/design-34959" target="_blank">34959</a>, the same ones as came with the Saturn V model), which are pretty close to scale with the model. Mine included a third astronaut as a spare piece. I arbitrarily decided these were <a href="https://chrishadfield.ca/" target="_blank">Chris Hadfield</a> (who does more to teach people about space than anybody else), <a href="https://www.nasa.gov/astronauts/biographies/peggy-a-whitson" target="_blank">Peggy Whitson</a> (US record holder for time in space), and <a href="http://www.scottkelly.com/" target="_blank">Scott Kelly</a> (who commanded three ISS expeditions and was the subject of twin studies to figure out what effects space has).<br />
<br />
There's a <a href="https://en.wikipedia.org/wiki/Soyuz_(spacecraft)" target="_blank">Soyuz</a> spacecraft included, so you can send personnel to and from your LEGO ISS. The Soyuz manned spacecraft and the Progress unmanned resupply craft are nearly indistinguishable, which isn't surprising: the Progress is pretty much a Soyuz that's reworked for cargo instead of cosmonauts. In the model, the only difference is that the Soyuz has a conical top, while the Progress has a round top. In real life, they're not even that distinct from the outside.<br />
<br />
Of course, there's a Space Shuttle orbital vehicle model included. The Space Shuttle has been retired for some time now, but it delivered much of the ISS to orbit. This was actually the original plan for the Shuttle: Nixon expected it to be a delivery vehicle for a space station. That was eventually designed as the <a href="https://en.wikipedia.org/wiki/Space_Station_Freedom" target="_blank"><i>Freedom</i> space station</a>, a Reagan-era plan. After the end of the Cold War, it was much more beneficial for all of Earth to work together to build the ISS. The ISS was built on the designs of <i>Freedom</i> and <i>Mir</i>, so much of the work was delivered from the Space Shuttle and Soyuz craft.<br />
<br />
There aren't any models of the <a href="https://en.wikipedia.org/wiki/H-II_Transfer_Vehicle" target="_blank">Japanese H-II Transfer Vehicle</a> or the <a href="https://en.wikipedia.org/wiki/Automated_Transfer_Vehicle" target="_blank">ESA's Automated Transfer Vehicle</a> included. That's okay, though; they're easy enough to build: the HTV is pretty much just a cylinder, so you can make it from 2x2 round bricks (like <a href="https://brickset.com/parts/design-6143" target="_blank">6143</a>, or add in some <a href="https://brickset.com/parts/design-4032" target="_blank">4032</a> plates for color). For the ATV, I'd probably use just a <a href="https://brickset.com/parts/design-30361" target="_blank">30361</a> (don't be fooled by the picture in that link: it's the one with the pointed axle holder, like what's used in the Service Module in this model) by itself. If you want to deploy the ATV's solar panels, you'll have to steal a couple of the solar panels from your Progress vehicles, since that part (6309239) isn't around anywhere else yet; you'll also need to dig through your Technic collection to decide what you want to connect them with, but I might see if a <a href="https://brickset.com/parts/design-15100" target="_blank">15100</a> can hold a <a href="https://brickset.com/parts/design-17715" target="_blank">17715</a> in place.<br />
<h2>
Scale</h2>
The whole model is 53.5 cm (21") long (you can get fit it in 49.5 cm by turning the outer solar panels). The <a href="https://www.nasa.gov/image-feature/integrated-truss-structure-diagram" target="_blank">real ISS truss</a> is 109 m (358') long. That comes to about a 1:200 scale. Happily, that means that a person is the same height as a LEGO brick, so the astronauts that come with the model are approximately to scale.<br />
<br />
The modules are a little smaller scale than the truss length implies: the <i>Destiny</i> module, at scale, should be 20 mm in diameter; the modules in the module are 16 mm. Your LEGO astronauts may be at a little bit cramped if you ask them to go into the modules, but not overly so.<br />
<br />
I briefly considered mounting the model high enough so that, to scale, it was the right height above the Earth. The ISS orbits closer to the Earth than you might think; it's <a href="https://spotthestation.nasa.gov/" target="_blank">close enough to see with the naked eye</a>. If the Earth was the size of a Frisbee, the ISS would orbit just a little away, the height of a LEGO brick. But a quick bit of math tells us that won't work out.<br />
<br />
The ISS orbits at about 400 km (250 mi), so for me to mount it at scale height, it would need to be about 2 km (just over a mile) above sea level. Maybe somebody in Denver can pull that off; the top of Republic Plaza is about 1800 m above sea level, and I probably have some rounding errors in those numbers.<br />
<h2>
Advice for Doing This Research Yourself</h2>
A lot of this involved identifying one part of the model, then working out the stuff that's connected to it, and so forth going outward. Fortunately, the Reference Guide has great diagrams (especially pp. 43, 53, and 77), the <a href="https://en.wikipedia.org/wiki/International_Space_Station#Structure" target="_blank">ISS Wikipedia article's “Structure” section</a> has a diagram for all the connections, and each module's Wikipedia page has links for each of the connected modules.<br />
<br />
The Reference Guide's picture of the exploded structure shows both the modules and their connections, but is low-res in the PDF. NASA has a <a href="https://www.nasa.gov/sites/default/files/thumbnails/image/iss_config_exploded_view_page_0.jpg" target="_blank">better one online</a>, which I've reproduced here.<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhEBGZohONxO-knyspFQz1H52MK9mReFkJ1WKfCfgyEI-4pYRPTHw3GbzYA1UqskcYmCdT0xAhuh54TQDFc5TZkrDyshBu95gRmOnAvBa1StICgeCMGX2SZBEIbST-SYOVsVoWo/s1600/iss_config_exploded_view_page_0.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="874" data-original-width="1500" height="372" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhEBGZohONxO-knyspFQz1H52MK9mReFkJ1WKfCfgyEI-4pYRPTHw3GbzYA1UqskcYmCdT0xAhuh54TQDFc5TZkrDyshBu95gRmOnAvBa1StICgeCMGX2SZBEIbST-SYOVsVoWo/s640/iss_config_exploded_view_page_0.jpg" width="640" /></a></div>
<br />
<br />
If you take this on yourself, note that a lot of the connected bits are Progress resupply ships. I discussed them earlier, but briefly: all the small vehicles connected with <a href="https://brickset.com/parts/4286267" target="_blank">spheres</a> are Progress ships, and aren't usually shown in most diagrams of the ISS.<br />
<br />
Be careful about out-of-date information; the ISS is constantly being added onto and rearranged, and some places can't keep up. The model seems to be based on the expected configuration as of mid-2020. For instance, the <a href="https://en.wikipedia.org/wiki/Nauka_(ISS_module)" target="_blank">MLM</a> is in the model, although as of this writing, hasn't launched. (That part of the model might actually be <i>Pirs</i>, though.) There's also some bits in the model for more fun: the FGB's solar panels are deployed, there are more resupply vehicles than have ever been docked at the same time, and there's a Space Shuttle orbiter. Because it's fun!Piquanhttp://www.blogger.com/profile/06983189170367989592noreply@blogger.com0tag:blogger.com,1999:blog-13322352.post-15954842773704949822018-07-12T21:54:00.001-07:002018-07-12T21:54:19.381-07:00On Choosing a MicrocontrollerI haven't posted in <i>ages</i>, I know!<br />
<br />
A friend recently asked me to describe the differences between microcontrollers. It's a wide field, but there are a few things that I tend to look for.<br />
<br />
It came up shortly after I deployed a temperature logger. The initial revision was just an Arduino, a <a href="https://www.adafruit.com/product/165">TMP36</a> sensor, with a <a href="https://www.sparkfun.com/products/13820">ProtoShield</a> with a <a href="https://www.sparkfun.com/products/12044">tiny breadboard</a> to tie everything together. I also wrote a Python program to log the temperature from the probe, and also log the local <a href="https://aviationweather.gov/metar">METAR reports</a> (so we had the outdoor weather), and serve this up with nice plots; all the logging and graphing is through <a href="https://oss.oetiker.ch/rrdtool/index.en.html">RRDtool</a>.<br />
<br />
That got me a quick deployment, but has its drawbacks. The biggest drawback is that the TMP36 only has ±2°C accuracy; I wanted something better. I could use the <a href="https://www.adafruit.com/product/1782">MCP9808 high-accuracy sensor</a> (which I had planned to do), but human comfort — the biggest reason for this particular project — depends heavily on humidity. Instead, I chose to build this around the <a href="https://www.adafruit.com/product/2857">SHT31-D</a>, a high-accuracy temperature and humidity probe. This brought my BOM cost above my initial $20 design point, but I decided it was worth it for the project in question.<br />
<br />
In the process, I decided to change my microcontroller; I had only chosen the Arduino because I had lots of them on hand, and it was dead-easy to code for. My original plan was for an <a href="https://www.adafruit.com/product/3500">Adafruit Trinket M0</a> with a <a href="https://www.adafruit.com/product/2857">SHT31-D breakout board</a> on a <a href="https://www.adafruit.com/product/65">tiny breadboard</a>, but I changed that. Instead, I decided to change my microcontroller to an <a href="https://www.adafruit.com/product/3403">Adafruit Feather M0 Express</a>, on which I could directly mount the <a href="https://www.adafruit.com/product/2857">SHT31-D board</a> with no need for a breadboard. For the remote probes, I settled on an <a href="https://www.adafruit.com/product/3213">ESP3266 Feather board</a>, which I would need to mate with the SHD31-D board.<br />
<br />
When I talked about changing the microcontroller for the next design, my friend asked about the differences between different microcontrollers; here's what I told him.<br />
<br />
A <a href="https://en.wikipedia.org/wiki/Microcontroller">microcontroller</a> is pretty much a scaled-down <a href="https://en.wikipedia.org/wiki/System_on_a_chip">SoC</a>; the boundary is moving. The main differences that distinguish one microcontroller from another are in the CPU, the onboard peripherals, and — for the microcontrollers that have them — the circuit board.<br />
<br />
An <a href="https://www.arduino.cc/">Arduino</a> is a pretty classic microcontroller among DIYers these days. The CPU is from the <a href="https://en.wikipedia.org/wiki/Atmel_AVR">Atmel AVR</a> line, which is an architecture designed for microcontrollers. The peripherals on the chip are pretty standard: there’s a <a href="https://www.arduino.cc/reference/en/language/functions/communication/serial/">serial port</a> (which, on an Arduino, is hooked to an FTDI USB-Serial chip), an <a href="https://learn.sparkfun.com/tutorials/i2c">I2C interface</a> (used to talk to a LOT of peripherals; what USB is to the PC world, so I2C is to the microcontroller world), an <a href="https://learn.sparkfun.com/tutorials/serial-peripheral-interface-spi">SPI interface</a> (more complex than I2C, and less common, but more powerful; if I2C is like USB, then SPI is like FireWire), a few <a href="https://learn.sparkfun.com/tutorials/analog-to-digital-conversion">ADCs</a>, and a <a href="https://learn.sparkfun.com/tutorials/pulse-width-modulation">PWM controller</a> (which can usually take the place of a <a href="https://www.sparkfun.com/tutorials/160">DAC</a> for simple tasks like <a href="https://www.arduino.cc/en/Tutorial/SecretsOfArduinoPWM">adjusting a motor or LED</a>). Some versions of it have more or fewer I/O pins, or serial ports, or what have you. The Arduino design has an easy-to-use board that includes a reasonably robust power supply design and clocking circuitry. It’s dead simple to use for things like the temperature probe.<br />
<br />
More powerful microcontrollers are usually <a href="https://www.arm.com/products/development-tools/microcontrollers-embedded">ARM-based</a>, often using a <a href="https://en.wikipedia.org/wiki/ARM_Cortex-M#Cortex-M0">Cortex M0</a>. The peripherals vary a lot, as does the board.<br />
<br />
For the temperature probe, I had picked out the <a href="https://www.adafruit.com/product/3500">Adafruit Trinket M0</a>. It’s an ARM based chip that’s tiny, <a href="https://cdn-shop.adafruit.com/970x728/3500-02.jpg">about the size of a quarter</a>. Its board has pins at is <a href="https://learn.sparkfun.com/tutorials/integrated-circuits/ic-packages">standard DIP pitch</a>, so I can put it on a <a href="https://learn.sparkfun.com/tutorials/how-to-use-a-breadboard">breadboard</a>. At that size, you can’t fit in a lot of I/O pins in DIP pitch, but I don’t need many: the <a href="https://www.adafruit.com/product/2857">new version of the temperature probe</a> uses I2C, which only uses two I/O pins (plus ground). I chose the <a href="https://www.adafruit.com/product/3500">M0</a> version instead of the <a href="https://www.adafruit.com/product/1500">regular Trinket</a> because the latter’s Atmel chip doesn’t have any built-in serial ports, so talking to the computer would have been annoying.<br />
<br />
Some microcontrollers are much simpler. For instance, the <a href="https://www.microchip.com/design-centers/microcontrollers">PIC microcontrollers</a> are low-cost microcontrollers designed for controlling things like alarm clocks, remote controls, car ignitions, etc. The <a href="https://www.microchip.com/PIC10F320">PIC10F320</a>, for instance, costs 36¢ (in bulk). (The microcontroller I'm using for the project I'm working on now is a <a href="https://www.microchip.com/pic16f1619">PIC16F1619</a>, and it costs $1.51 for individual units). Unlike the <a href="https://www.arduino.cc/">Arduino</a> or <a href="https://www.adafruit.com/product/1500">Trinket</a>, these aren't usually on carrier boards: instead, I would program the chip using <a href="https://www.microchip.com/pickit3">a separate device</a>, and then put it in the circuit where it's supposed to be used. I happen to be prototyping mine in <a href="https://www.microchip.com/DevelopmentTools/ProductDetails/PartNO/DM164137">a board that has a socket, a built-in programmer, four LEDs, one potentiometer, and one switch</a>, so that I can easily test changes without taking it out of the circuit. But once it's done, I'm going to just put the chip on a <a href="https://learn.sparkfun.com/tutorials/how-to-use-a-breadboard">breadboard</a> or <a href="https://learn.sparkfun.com/tutorials/pcb-basics">PCB</a>. (Actually, in this case, I'm hoping to put it in a ballpoint pen.) The PIC has a <a href="http://ww1.microchip.com/downloads/en/DeviceDoc/40001770D.pdf">very strange architecture</a>: for instance, there's only one general-purpose register, the program is in 14-bit words, there are no multiply or divide instructions, using lookup tables is a royal pain, there's essentially no support for recursion, etc. All this is in pursuit of having a cheap, low-power processor; this thing could run for 29 months if it was active from a pair of <a href="https://d2ei442zrkqy2u.cloudfront.net/wp-content/uploads/2018/06/13163411/76ALR44V1_062018.pdf">LR44s</a> (the tiny ⌀11.25mm x 5mm batteries in laser pointers)... and that's just if it's actively running; for standby, its 50nA sleep current will outlive any power supply. And it has a lot of <a href="http://ww1.microchip.com/downloads/en/DeviceDoc/40001770D.pdf">onboard peripherals</a> (I'll just list them for brevity, but can explain these if you want): four configurable logic cells, a complementary waveform generator, two capture/compare/PWM modules, two PWM generators, two signal measurement timers, an angular timer, seven timers with different capabilities, a proportional-integral-derivative controller, a CRC generator, UARTs for serial/I2C/SPI/etc, a 12-channel 10-bit ADC, two voltage comparators, an 8-bit DAC, a zero-crossing detector, 18 GPIO pins, and a multiplexer that can reroute these signals internally or externally to the different pins based on your needs.<br />
<br />
The opposite end has devices that are barely on the "microcontroller" end of the microcontroller/SoC boundary. The <a href="https://www.raspberrypi.org/">Raspberry Pi</a> is very much board-bound, but runs <a href="https://www.raspberrypi.org/documentation/raspbian/">Debian</a> on an ARM, which means you can actually write shell scripts to manipulate the <a href="https://www.raspberrypi.org/documentation/usage/gpio/">GPIOs</a>, or use the <a href="https://rpiplayground.wordpress.com/2014/05/03/hacking-the-gpu-for-fun-and-profit-pt-1/">GPU</a> to run the computer-vision library <a href="https://opencv.org/">OpenCV</a>. It ships with <a href="https://www.wolfram.com/mathematica/">Mathematica</a>, and Wolfram has added <a href="http://www.wolfram.com/raspberry-pi/">a bunch of functions to make it easier to write microcontroller-based programs in the Wolfram Language</a>, such as functions to manage GPIOs. It's a breeze even for beginner programmers, since you can just plug in a mouse, keyboard, and monitor, and start writing <a href="https://www.raspberrypi.org/documentation/usage/python/">Python</a> - or even use <a href="https://minecraft.net/en-us/edition/pi/">Minecraft extensions</a> to use the GPIOs!<br />
<br />
For my interests today, my current pick of the litter is the <a href="https://learn.sparkfun.com/tutorials/esp8266-thing-development-board-hookup-guide">ESP8266</a>. This is is the descendant of the previous IoT darling, the ESP32. It's one of the few microcontrollers with solid WiFi and <a href="https://en.wikipedia.org/wiki/Bluetooth_Low_Energy">Bluetooth LE</a> support. But first, I'll talk about its foundations: the ESP8266 runs <a href="https://www.freertos.org/">FreeRTOS</a>, which is (by design) barely an OS: it handles memory management, process scheduling, and interprocess communication, and nothing else. But FreeRTOS is as solid as you can get if you need hard realtime guarantees: when you do anything, you can calculate the upper bound of how long it will take, which is something that very few OSs can guarantee. If I were building a satellite, I would base it on FreeRTOS. On top of this, the ESP8266 has built-in Bluetooth LE and WiFi radios, and has <a href="https://esp-idf.readthedocs.io/en/latest/api-reference/index.html">libraries for Bluetooth LE, WiFi, TCP/IP, NTP, DNS, HTTP, and even over-the-air updates</a> (where it pulls new version of the firmware online, like phones do), which makes it ideal for <a href="https://en.wikipedia.org/wiki/Internet_of_things">IoT</a> devices. It also has <a href="https://esp-idf.readthedocs.io/en/latest/api-reference/storage/spiffs.html">wear-leveling filesystems</a>, and has <a href="https://esp-idf.readthedocs.io/en/latest/api-reference/storage/fatfs.html">libraries for FAT</a> if you want to use it with a commodity SD card that you can plug into a computer. It also has a super-simple <a href="https://esp-idf.readthedocs.io/en/latest/api-guides/ulp.html">ultra low-power processor</a> that can run while the main CPU is in sleep mode, so it can do things like take readings from the environment and wake up the main CPU if something "interesting" happens. In addition to the WiFi and BTLE, it also has onboard peripherals for ADCs, DACs, GPIOs, I2C, I2S, SPI, timers, touch sensors, a few different PWM modules, LED control, SDMMC (for SD cards and similar buses), and UARTs. My current drawn-out project is monitoring and watering orchids based on an ESP8266. The original orchids have died, but that's my fault rather than that of the chip (or the LEGOs that support it). There are a few different boards based around the ESP8266; Sparkfun has a good one that's pretty much the dev design (that's what I use), and Adafruit has one that fits in better with their microcontroller ecosystem (which was my first introduction to it, and I'll get back to that).<br />
<br />
Now, let's go just to a bit beyond microcontrollers, through the grey area of the microcontroller/SoC boundary, to the most heavyweight end. At that end, we have the <a href="https://blogs.nvidia.com/blog/2016/09/28/xavier/">Nvidia T194 "Xavier" SoC</a>. This uses eight ARM-compatible Carmel CPUs, so can run ARM Linux, Android, etc. There are 512 GPU cores with the <a href="https://www.nvidia.com/en-us/data-center/volta-gpu-architecture/">new Volta architecture</a>; you can use those to play games at high frame rates, but they're mostly designed for managing neural nets. In that pursuit, Xavier also has the open-sourced <a href="http://nvdla.org/">Deep Learning Accelerator</a>, which is designed for running neural nets at high speeds and low power consumption. I use the Xavier's predecessor, Parker, at home as my lab computer; that's even what I used to do my taxes. The Parker is sold as the center of the <a href="https://developer.nvidia.com/embedded/buy/jetson-tx2">Jetson TX-2</a>, which is a Parker chip surrounded by lots of the support circuitry. The Xavier is going to have <a href="https://developer.nvidia.com/jetson-xavier-devkit">a similar kit</a>. The <a href="https://developer.nvidia.com/embedded/buy/jetson-tx2-devkit">Jetson TX-2 dev kit's</a> carrier board is pretty big, but the <a href="https://developer.nvidia.com/embedded/buy/jetson-tx2">module at its core</a> is about the size of a stack of 10 playing cards. I've seen Parkers <a href="https://www.jetsonhacks.com/">used at the core of autonomous drones and rovers</a> (because of how much computer vision processing they need to do), and such, but it's closer to a full-blown computer than a microcontroller.<br />
<br />
(Disclaimer: I work for Nvidia. The opinions in this blog are all my own, and are not official Nvidia communications.)<br />
<br />
Some vendors, such as <a href="https://www.adafruit.com/">Adafruit</a>, design microcontroller boards for particular purposes. For instance, the <a href="https://www.adafruit.com/feather">Feather</a> line of microcontrollers all have a common pinout, so they can share add-on peripherals. Some of these ecosystems make it easy to use prebuilt common add-on boards: these are called <a href="https://learn.sparkfun.com/tutorials/arduino-shields">shields</a> in Arduino culture, <a href="https://www.raspberrypi.org/blog/introducing-raspberry-pi-hats/">HATs</a> for Raspberry Pi, wings for the Feather line, etc. A lot of companies make microcontroller boards that are pinout-compatible with the Arduino so they can work with existing shields.<br />
<br />
Often, the availability of libraries is important. Again, a lot of boards have some degree of Arduino software compatibility so they can be used with the Arduino SDK and libraries. Adafruit also has a system called <a href="https://learn.adafruit.com/welcome-to-circuitpython/overview">Circuit Python</a> that lets you write programs in Python, and the board-specific libraries have all the bits to connect you to the GPIOs etc. When you plug in a Circuit Python board, it shows up as a flash drive and (if the board supports it) a serial port, so you can just put your program on the flash drive and connect to the serial console. If using Python on a microcontroller sounds exciting, you should also check out <a href="https://micropython.org/">MicroPython</a>.<br />
<br />
There are a LOT of variations within this; I've only given you a few of the "landmarks". For instance, if I was choosing a wearable microcontroller, just off of the top of my head, I can think of two that are Arduino-compatible, two that are Circuit Python compatible, and one that is the size of a quarter, and I'm not even into wearables.<br />
<br />
When I choose a microcontroller, here's the things I'm looking at:<br />
<ul></ul>
<br />
<li>Does it have enough pins to support the devices I'm trying to work with?</li>
<br />
<li>Does it have the built-in peripherals I want?</li>
<br />
<li>Does it have a form factor that I need?</li>
<br />
<li>If I'm working on a battery-powered device, is the power consumption appropriate?</li>
<br />
<li>If I still have multiple choices (and I usually do), which one is easiest to program?</li>
<br />
<br />
<br />
For the first revision of the thermometer probe, I chose an Arduino (specifically, an Arduino UNO clone with a Sparkfun Protoshield that I previously mounted a micro-sized breadboard on), based on some basic requirements:<br />
<ul></ul>
<br />
<li>I needed one ADC to read the analog temperature probe (a TMP36), and no other GPIOs (most uCs can meet this)</li>
<br />
<li>I needed a serial-USB port (leaves out the Gemma M0 and a few others)</li>
<br />
<li>I wanted something fairly small</li>
<br />
<li>I didn't care about power consumption</li>
<br />
<li>I have lots of Arduinos lying around</li>
<br />
<li>Arduinos are really easy to connect devices to, particularly with the Protoshield breadboard I had lying around</li>
<br />
<li>It's dead-simple to program simple stuff on an Arduino (although it's hard to write complex stuff on)</li>
<br />
<br />
<br />
For the second revision (in progress), I wanted to design for size. I'm switching from the TMP36 (analog output) to a SHT31-D (I2C interface), because the latter has a superior accuracy. (I also picked it for the humidity sensor, and other aspects that I can discuss separately.) For the ones plugged into computers, for the microcontroller, I settled on the Adafruit Feather M0 Express, because:<br />
<ul></ul>
<br />
<li>I only need one I2C I/O channel (two pins), and no other GPIO,</li>
<br />
<li>I need a serial console (which excludes the Adafruit Trinket)</li>
<br />
<li>It can be powered by USB, so I don't need an external power supply,</li>
<br />
<li>The prototyping space lets me mount the SHT31-D breakout board without needing a breadboard,</li>
<br />
<li>It has a UART that can show up as a regular USB serial port (which, sadly, lets out the Adafruit Trinket),</li>
<br />
<li>It is in stock this week (which, much more sadly, lets out the Adafruit Trinket M0; I originally spec'd this for the Trinket M0, but before I ordered it, they ran out of stock) (update: it's back in stock)</li>
<br />
<li>It's cheap</li>
<br />
<br />
<br />
For the WiFi remote probes, my overwhelming concern was easy-to-use WiFi, which pretty much left me with the ESP8266 (or ESP32). Among these, I chose the Adafruit Feather ESP8266, for easy compatibility with the Adafruit Feather ecosystem. Since I had already chosen a Feather for the teathered (USB-connected) probes, I figured that sticking with a Feather would be best for the remotes. (The ESP8266 boards I've used previously weren't Feathers, but there's not a huge difference for the purposes in front of me.)<br />
<br />
There were lots of choices for these probes, so a lot of this was "what can I get working in the least amount of time". Of course, the aspect of "time" is asymmetrically weighed between developer time and wall time. As it happened, I had the parts at home to build the version with an Arduino and TMP36, which is also the easiest one to code, so I could quickly deploy it and start getting readings. But for higher accuracy, smaller size, and placement flexibility (which needs to take into account WiFi capability and power consumption), there were a few other choices, so I'm drastically switching microcontroller architectures for the rev2 deployment.Piquanhttp://www.blogger.com/profile/06983189170367989592noreply@blogger.com0tag:blogger.com,1999:blog-13322352.post-27787579044619589382011-09-09T15:08:00.000-07:002011-10-03T22:08:01.052-07:00iChat, Growl, and LionUPDATE: This script has been improved; the new version is now on <a href="https://github.com/Piquan/iChat-Growl">github</a>.<br />
UPDATE 2: Growl 1.3 was just released, so I've updated the script to work with 1.3. If you were using this script and it broke with 1.3, then just download the updated version from the same place. If you're still using Growl 1.2.2, you'll need to use <a href="https://github.com/Piquan/iChat-Growl/blob/033f9a864390fee3d9d653ffc1da8159db7a66e7/Growl.applescript">the old version of this script</a>. All the rest of the instructions are the same regardless of which version of Growl you're using.<br />
<br />
<a href="http://www.apple.com/macosx/">Lion</a> is the latest version of OS X. <a href="http://www.apple.com/macosx/apps/all.html#ichat">iChat</a> is the chat program that's part of OS X. <a href="http://growl.info/">Growl</a> is a system that lets all sorts of programs post little notification windows on your screen.<br />
<br />
For example, if you get a message in iChat while you're in another window, it's nice to have the message pop up on your screen so you can decide whether or not to switch windows to read it. Or if a buddy logs in, it's nice to know about that (which can be done in iChat with an audio notification), and see who it is without having to check your buddy list.<br />
<br />
I used to use <a href="http://www.ksuther.com/chax/">Chax</a> for this. Chax is a pretty good program, but when a new version of OS X comes out, it takes a little while to catch up. The main reasons I use Chax are:<br />
<ol><li>Accept text chat invitations without being prompted.</li>
<li>Put all my buddies in the same window.</li>
<li>Give me Growl notifications.</li>
</ol><div>Everything but the Growl feature is now built into iChat in Lion. (The auto-accept is an AppleScript that ships with Lion; you can turn that on in Preferences:Alerts. The option for "Show all my accounts in one list" is in Preferences:General.)</div><div><br />
</div><div>Chax isn't yet compatible with Lion, so I searched online and found <a href="http://amccloud.com/lion-ichat-growl-notifications-50108">a little AppleScript program</a> to do a bit of what I need. Using the ideas from that, I expanded it a bit. Goodbye Chax; I'll miss you!<br />
<br />
To use this:<br />
<ol><li>Install <a href="http://growl.info/">Growl</a>, if you don't have it already.</li>
<li>Visit <a href="https://github.com/Piquan/iChat-Growl/raw/master/Growl.applescript">https://github.com/Piquan/iChat-Growl/raw/master/Growl.applescript</a> in your web browser. (You can use the <a href="https://github.com/Piquan/iChat-Growl/blob/master/Growl.applescript">formatted</a> page instead.)</li>
<li>Open AppleScript Editor (in Applications:Utilities).</li>
<li>Copy and paste the code from the web page into a new script.</li>
<li>Save it to your home directory's Library:Scripts:iChat folder. (I used the filename "Growl".) The iChat folder might not exist; you need to create it. By default, your Library folder is hidden. (Personally, I solved that by saving to the desktop, and then I ran "open ~/Library/Scripts" in Terminal so that the Scripts window opened in Finder, and dragged my script there.)</li>
<li>In iChat, open the Preferences.</li>
<li>Under Alerts, select the "Message Received" event.</li>
<li>Turn on "Run an AppleScript script:", and select the script you saved earlier.</li>
<li>Repeat the last two steps for each event you're interested in.</li>
</ol><div><br />
</div></div><div>Note that the "Message Received in Chat Room" (which is called "addressed message received" in the AppleScript) and "Invitation to Share Buddy's Screen" (a.k.a. "received remote screen sharing invitation") aren't currently enabled, since it looks like there may be a bug in iChat that will cause them to be confused with each other.<br />
<br />
If you use the Growl script for chat invitations, you can't also use the Auto-Accept script, since you can only select one script to be run. Because of this, I've added an auto-accept capability to this Growl script. By default, it's turned off, but you can turn it on. Near the top of the program, there's a line that says:<br />
<br />
<div style="font: 12.0px Verdana; margin: 0.0px 0.0px 0.0px 39.2px; text-indent: -39.3px;"><b> property</b> <span style="color: #279800;">autoAccept</span> : {""}</div><br />
Change this to:<br />
<br />
<div style="font: 12.0px Verdana; margin: 0.0px 0.0px 0.0px 39.2px; text-indent: -39.3px;"><b> property</b> <span style="color: #279800;">autoAccept</span> : {"text"}</div></div><div><br />
</div><div>Suggestions for improvements are welcome. Share and enjoy!<br />
<br />
(If you expected the source code to be here in my blog, I've moved it. See <a href="https://github.com/Piquan/iChat-Growl/raw/master/Growl.applescript">https://github.com/Piquan/iChat-Growl/raw/master/Growl.applescript</a> for my latest version.)</div><div></div>Piquanhttp://www.blogger.com/profile/06983189170367989592noreply@blogger.com34tag:blogger.com,1999:blog-13322352.post-77947919524435219862011-07-05T02:52:00.000-07:002011-07-05T02:52:18.553-07:00Copying a filesystem on LinuxIn the last two days, I've had to copy a filesystem on my Linux development VM four times (to grow it). I eventually made a shell script for it, since I kept forgetting details (like the arguments to cpio). Since I started this blog to have a place to publish random bits of information for future Google users, I'm putting the script here. The script is below; you pass in the old device name (e.g., <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">sda</span>) as the first argument, and the new one (e.g., <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">sdc</span>) as the second.<br />
<br />
There's some significant assumptions in this script, and they may not be warranted for your system:<br />
<ul><li>It assumes that you have only one filesystem, and that's on the first partition of the disk. (It's also easy to add a swap partition using <a href="http://manpages.debian.net/cgi-bin/man.cgi?query=mkswap&sektion=8">mkswap</a>.)</li>
<li>You're expected to use <a href="http://tldp.org/HOWTO/Partition/fdisk_partitioning.html">fdisk</a> to set up the new filesystem yourself. (Linux is fs type 83; Linux swap is 82.)</li>
<li>This builds an ext3 filesystem.</li>
<li>It assumes that once you're finished, you'll be putting the new device into the old device's place; in other words, it doesn't try to replace <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">sdc</span> with <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">sda</span> in <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">/etc</span> files. (If you're only using UUIDs in your <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">fstab</span>, this probably doesn't matter.)</li>
<li>It uses a default grub install, more or less. If you're using LILO, or have got something funky going on, it might not carry over. If you've customized your <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">grub.cfg</span>, for example, it will get overwritten. (However, changes in <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">grub.d</span> are fine and will be reflected.)</li>
<li>As used here, cpio will make sparse files out of non-sparse files. Normally, this is a good thing, but some programs (such as BitTorrent clients) will intentionally build non-sparse files. (It's ok to turn them sparse after the torrent completes, and it's only a fragmentation hit if they get turned into sparse before the torrent is done.)</li>
<li>This uses a 64k I/O block size. I'm doing that because that's the block size of the default QEMU backing store under qcow2. (Remember, in my case, I'm doing this for a VM.) I'm not sure if that helps anything or not (since the I/O goes through many layers between cpio and qcow), but it's not going to hurt anything. However, you may want to replace the <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">-C65536</span> with <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">-B</span> if you're using a normal disk.</li>
</ul><div><br />
This script is better to learn from than to use directly. But if you need to copy a filesystem (to grow it, move off a failing disk, defragment it (<a href="http://www.tldp.org/LDP/sag/html/filesystems.html">rarely necessary on Linux</a>), or whatever), this may give you a good starting point.</div><blockquote><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">#! /bin/sh</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">in=$1</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">out=$2</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">set -xe</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">fdisk /dev/${out}</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">mke2fs -j /dev/${out}1</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">mount /dev/${out}1 /mnt</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">find / -xdev -depth -print0 | cpio -pdmV0a --sparse -C65536 /mnt</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">grub-install --root-directory=/mnt /dev/${out}</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">perl -pi.bak -e s/$(blkid -o value -s UUID /dev/${in}1)/(blkid -o value -s UUID /dev/${out}1)/g /mnt/etc/fstab /mnt/boot/grub/grub.cfg</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">umount /mnt</span></blockquote><br />
A few implementation comments:<br />
<br />
<ul><li>This makes a lot of assumptions (listed above) for simplicity. Feel free to improve it.</li>
<li>The <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">-depth</span> is so that find will print the directories after their contents, which will cause cpio to fix the modification time. Otherwise, cpio will create the directory, change its mtime to match the original, and then add contents, thereby changing the directory's mtime to the current time.</li>
<li>The Perl can be done by hand: it's changing the old filesystem's UUID to the new filesystem's UUID in <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">fstab</span> and <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">grub.cfg</span>. You can find out the IDs by running <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">sudo blkid</span>.</li>
<li>Note that the Perl line is long; be careful when copying it. It spans from <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">perl</span> to <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">grub.cfg</span>.</li>
</ul><br />
Share and enjoy!<br />
<br />
PS: If you're wanting to grow a filesystem on the same disk (i.e., you've deleted the next partition and want to use its free space), there's <a href="http://www.google.com/search?q=ext3+grow+filesystem">utilities to do that.</a> This script is about moving to a different disk.<br />
<br />
<br />
<div class="post-body entry-content" style="color: #333333; font-family: Verdana, sans-serif; font-size: 13px; line-height: 19px;"><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-size: xx-small;">(Disclaimer: I release the above code to the public domain, as if the public actually wants it. It's for education, and should be evaluated and customized before running it in any particular environment. I provide no warranty, expressed or implied; if it breaks, you get to keep both pieces.)</span></div><div style="clear: both;"></div></div><div class="post-footer" style="background-attachment: initial; background-clip: initial; background-color: initial; background-image: url(http://www.blogblog.com/tictac_blue/tictac_grey.gif); background-origin: initial; background-position: 0% 8px; background-repeat: no-repeat no-repeat; color: #999999; font-family: Verdana, sans-serif; font-size: 11px; line-height: 19px; margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px; padding-bottom: 0px; padding-left: 14px; padding-right: 0px; padding-top: 0px;"></div>Piquanhttp://www.blogger.com/profile/06983189170367989592noreply@blogger.com0tag:blogger.com,1999:blog-13322352.post-22031960948886722462011-04-26T00:46:00.000-07:002011-04-26T01:23:14.235-07:00iPhone location log to Google Earth, all with shell commandsIf you're a privacy geek, then you've no doubt heard people talking about a file on the iPhone called <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><a href="http://www.google.com/search?q=consolidated.db">consolidated.db</a></span>. This file stores all the places your phone has been.<br />
<br />
I don't know why everybody is making such a fuss about it. I expect my own computers (including my phone) to store logs of what I'm doing. It's good to be able to reset this information before I give a device to somebody else, and hey, the iPhone has a comprehensive reset function!<br />
<br />
I think the concern is that the phone is sending this to Apple periodically. That would be a very serious concern. But until somebody gives any sort of evidence to that effect, I think it's just a badly-managed log file.<br />
<br />
Anyhow.<br />
<br />
Yesterday, I saw that I had apparently bought a song on my iPhone that I didn't recognize. I listened to it, and it didn't sound familiar. It also didn't sound like something I'd buy. After a couple of blind alleys, I checked the purchase date in iTunes. I then correlated that to <a href="https://www.google.com/latitude/b/0/history/manage">Google Latitude's history</a>. Ah, it was a mall in Santa Cruz. Now I remembered: the reason I went into the mall was to get a lemonade at Starbucks, and while I was there my friend gave me one of the vouchers that Starbucks has for a free iTunes song.<br />
<br />
While I was relaying this story to another friend, he asked if I'd used consolidated.db to read it. Nope, but not a bad idea. I started wondering what was actually in there. Most stuff I saw to read that requires me to download some stranger's code, and I wasn't too thrilled about that.<br />
<br />
OS X and iOS tend to use <a href="http://developer.apple.com/library/mac/#documentation/Cocoa/Conceptual/PropertyLists/AboutPropertyLists/AboutPropertyLists.html%23//apple_ref/doc/uid/10000048i-CH3-SW2">property lists</a> and <a href="http://www.sqlite.org/">SQLite</a> for all kinds of storage, so I'd guessed it was SQLite. That meant that it could be read from the command line using the <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">sqlite3</span> utility, which is part of OS X.<br />
<br />
So, without further ado, here's a quick rundown on how to look at all this stuff straight from the command line. There's one bit of Perl to convert it to Google Earth's <a href="http://code.google.com/apis/kml/documentation/kmlreference.html">KML</a> format, but the Perl really isn't necessary to view the data in lat/long format. Note that making use of the following requires a certain degree of technical savvy; knowing SQL is going to be very useful to fiddle with the data. However, you don't need to know anything about OS X (except how to start the terminal), iPhone development, etc.<br />
<br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> # Get the current time in the format used by the database (Apple's CFAbsoluteTime, which is the number of seconds since Jan 1, 2001 at midnight GMT; we use UTC instead of GMT here, but it's close enough for these purposes):</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">$ perl -MTime::Local -e 'print time() + timegm(0,0,0,1,0,70) - timegm(0,0,0,1,0,101), "\n"'</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">325464777</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> # Go to the backups</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">$ cd ~/Library/Application\ Support/MobileSync/Backup/</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> # Go to the most recent backup</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">$ cd $( ls -t | head -1 )</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> # The files in here are named (almost) randomly.</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> # Get the database's filename by looking at table lists of all of the SQLite databases and finding one that has CellLocation</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> # (This prints the filename and leaves it in $db)</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">$ for db in $( file * | sed -e '/SQLite/ !d' -e 's/:.*//' ) ; do sqlite3 $db .tables | grep -q CellLocation && echo $db && break ; done</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">4096c9ec676f2847dc283405900e284a7c815836</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> # Your filename will be different; it's based on a hash of the data.</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> # View the file</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">$ sqlite3 $db</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> -- Take a look at the schema of CellLocation</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">sqlite> schema CellLocation</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">CREATE TABLE CellLocation (MCC INTEGER, MNC INTEGER, LAC INTEGER, CI INTEGER, Timestamp FLOAT, Latitude FLOAT, Longitude FLOAT, HorizontalAccuracy FLOAT, Altitude FLOAT, VerticalAccuracy FLOAT, Speed FLOAT, Course FLOAT, Confidence INTEGER, PRIMARY KEY (MCC, MNC, LAC, CI));</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> -- (That will also print a few other tables that you can ignore.)</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> -- View the contents of the past week (this uses the time computed at the beginning of this walkthrough; 604800 is the number of seconds in a typical week)</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">sqlite> select latitude,longitude,timestamp from CellLocation where timestamp > 325464777 - 604800 order by timestamp;</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> -- Back to the shell</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">sqlite> .exit</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> # Put the data through Perl to convert to KML</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">$ sqlite3 $db "select longitude,latitude,timestamp from CellLocation where timestamp > 325464777 - 1209600 order by timestamp, horizontalAccuracy" | perl -nw -MPOSIX -MTime::Local -e 'BEGIN { our $LTS=0; print "<kml xmlns:gx="\"http://www.google.com/kml/ext/2.2\"" xmlns="\"http://www.opengis.net/kml/2.2\""><document><placemark><gx:track>\n";} next if /^[0.]+\|[0.]+\|/; my ($long, $lat, $tstamp) = split /\|/; next if $tstamp==$LTS; $LTS=$tstamp; my @time = gmtime($tstamp - timegm(0,0,0,1,0,70) + timegm(0,0,0,1,0,101)); my $tstr = strftime("%Y-%m-%dT%H:%M:%SZ", @time); print "<when>$tstr</when><gx:coord>$long $lat 0</gx:coord>\n"; END { print "</gx:track></placemark></document></kml>"; } ' > ~/Desktop/locations.kml</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"> # Open it in Google Earth</span><br />
<span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">$ open ~/Desktop/locations.kml</span><br />
<div><br />
</div><br />
If you're trying to make sense of the Perl (particularly the <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">$LTS</span> bit), note that each timestamp tends to have several locations with varying precisions. I'm not sure whether that's a triangulation of the user's position, or of the cells the user was able to communicating with, but I suspect it's increasingly precise triangulations of the position that are recorded with the same timestamp. This code will use the first recorded position for each timestamp, which (because of the <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">ORDER BY</span> clause) is going to be the one with the smallest <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">horizontalAccuracy</span>. I'm currently supposing that the <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">horizontalAccuracy</span> column is actually the precision radius.<br />
<br />
I also played with using the <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">WifiLocation</span> table (which is similar), but it seems to be pretty unreliable data; it seems to include data based on one database that's about 10 miles too far north. (At least, in my case, it kept showing me jumping abruptly between Santa Clara / Sunnyvale and Fremont. I haven't been on the Fremont side of the bay in over a month.) I read online that Verizon customers need to use <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">CdmaCellLocation</span> instead of <span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">CellLocation</span>.<br />
<br />
<br />
<div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">If you're a potential employer, please note that the Perl above was thrown together for my own experimenting. I just put it here on the blog in case somebody else wanted to fiddle with it, and it's not at all what I'd put into production.</div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><br />
</div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;">Well, that's about all the time I felt like putting into decoding that file. Feel free to post improvements in the comments section below. Share and enjoy!</div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><br />
</div><div style="margin-bottom: 0px; margin-left: 0px; margin-right: 0px; margin-top: 0px;"><span class="Apple-style-span" style="font-size: xx-small;">(Disclaimer: I release the above code to the public domain, if you care. It's for education, and is not intended to run in any particular environment. I provide no warranty, expressed or implied; if it breaks, you get to keep both pieces.)</span></div>Piquanhttp://www.blogger.com/profile/06983189170367989592noreply@blogger.com10tag:blogger.com,1999:blog-13322352.post-6483471599391760902011-03-16T19:53:00.000-07:002011-03-16T19:53:31.629-07:00Magic GarageBand Chords<style type="text/css">/*<![CDATA[*/
/* The comments are there to allow this to be parsed as HTML or XHTML. */
table.chords {
table-layout: auto;
border-spacing: 0px;
border: 1px solid black;
empty-cells: hide;
background: #ddd;
}
table.chords th, table.chords td {
/* General */
border-left: 1px solid #ddd;
border-right: 1px solid #ddd;
border-top: none;
border-bottom: 1px solid #ddd;
margin: 0px;
padding-top: 0.2em;
padding-bottom: 0.2em;
background: white;
vertical-align: baseline;
}
table.chords > tbody > tr > th:first-child {
/* Verse */
width: auto;
min-width: 5em;
padding-left: 0.3em;
padding-right: 0.3em;
}
table.chords > tbody > tr > th:first-child + th {
/* Measure */
width: 2em;
text-align: right;
padding-left: 0.3em;
padding-right: 0.3em;
}
table.chords > tbody > tr > td {
/* Chords (and comments, if not overridden) */
border-left: none;
border-right: none;
border-top: none;
border-bottom: 1px solid #ddd;
padding-left: 0.5em;
padding-right: 0.5em;
width: auto;
min-width: 2em;
}
table.chords > tbody > tr > th:first-child + th + td + td + td + td + td,
table.chords > tbody > tr > .chord-comment {
/* Comments */
border-left: 1px solid #ddd;
border-right: 1px solid #ddd;
padding-left: 1em;
padding-right: 1em;
width: auto;
min-width: 3em;
empty-cells: show;
vertical-align: middle;
}
span.chordadd {
vertical-align: super;
font-size: .83em;
}
span.chordsub {
vertical-align: sub;
font-size: .83em;
}
span.timesig-num {
vertical-align: super;
font-size: .83em;
}
span.timesig-denom {
vertical-align: sub;
font-size: .83em;
}
/*]]>*/</style><br />
<h1>Magic GarageBand Chords</h1><p>I sat down to practice my guitar by improvising along with GarageBand's "Magic GarageBand" feature. This handy feature lets you put together a five-piece band from a wide selection of instruments and styles, which will play a song in a genre of your choosing. You can accompany them (using a mic, MIDI instrument, or DI from a guitar or other instrument), and record the result. By choosing the instruments and styles, you can get very different-sounding songs.</p><p>This is great if you want to jam but don't have a band with you at the moment. But, there's one bit of information that's missing.</p><p>In order to accomplish this mix-and-match, each genre (of the nine possibilities) plays the same basic chord changes. The individual parts riff all over those, but since they're following the same basic pattern, you get a good harmony. This is handy information to know if you're trying to play along! All of these have pretty straightforward patterns, but it's a lot easier to know the changes ahead of time.</p><p>Regrettably, I wasn't able to keep up with the chord changes when I sat down to play the other day. I went online to find them, but couldn't find anybody who had written them down for the world to see.All Google brought me is other people looking for the changes.</p><p>Hence, today's post: the chord changes for all nine genres in Magic GarageBand. If you have any suggestions for improvements, let me know! (I mean, something that makes this list closer to what GarageBand actually plays. I'm not looking for improvements to the songs; talk to Apple about that!)</p><h2>Blues (E, 90bpm, <span class="timesig-num">4</span>/<span class="timesig-denom">4</span>)</h2><p>When you create the GarageBand project, it says the song is in B,but it's really in E. The loops are also in the wrong key, which can lead to surprises if you try to use them in your own project!</p><p>Other than that, there's really no surprises here. This is a moderate tempo 12-bar blues, which is probably what you were expecting here. There's not even a real intro or outro.</p><table class="chords"><tbody>
<tr><th>Intro</th><th>1</th><td>E</td><td/><td/><td/><td>Pickup; can use A as well.</td></tr> <tr><th>Verse 1</th><th>2</th><td>E</td><td>E</td><td>E</td><td>E</td><td/></tr> <tr><th/><th>6</th><td>A</td><td>A</td><td>E</td><td>E</td><td/></tr> <tr><th/><th>10</th><td>B</td><td>A</td><td>E</td><td>E</td><td/></tr> <tr><th>Verse 2</th><th>14</th><td>E</td><td>E</td><td>E</td><td>E</td><td/></tr> <tr><th/><th>18</th><td>A</td><td>A</td><td>E</td><td>E</td><td/></tr> <tr><th/><th>22</th><td>B</td><td>A</td><td>E</td><td>E</td><td/></tr> <tr><th>Verse 3</th><th>26</th><td>E</td><td>E</td><td>E</td><td>E</td><td/></tr> <tr><th/><th>30</th><td>A</td><td>A</td><td>E</td><td>E</td><td/></tr> <tr><th/><th>34</th><td>B</td><td>A</td><td>E</td><td>E</td><td/></tr> </tbody></table><!--INTRO [ 1] E Pickup; can use A as well.VERSE 1 [ 2] E E E E [ 6] A A E E [10] B A E EVERSE 2 [14] E E E E [18] A A E E [22] B A E EVERSE 3 [26] E E E E [30] A A E E [34] B A E E--><br />
<h2>Rock (D, 140bpm, <span class="timesig-num">4</span>/<span class="timesig-denom">4</span>)</h2><table class="chords"><tbody> <tr><th>Intro </th><th> 1</th><td>D </td><td> </td><td> </td><td> </td><td>Pickup</td></tr> <tr><th> </th><th> 2</th><td>D </td><td>D </td><td>D </td><td>D </td><td></td></tr> <tr><th>Verse 1 </th><th> 6</th><td>D </td><td>D </td><td>D </td><td>D </td><td></td></tr> <tr><th> </th><th>10</th><td>F </td><td>F </td><td>D </td><td>D </td><td></td></tr> <tr><th> </th><th>14</th><td>D </td><td>D </td><td>D </td><td>D </td><td></td></tr> <tr><th> </th><th>18</th><td>F </td><td>F </td><td>A </td><td>A </td><td></td></tr> <tr><th>Chorus </th><th>22</th><td>Bm</td><td>Bm</td><td>E </td><td>E </td><td>Remodulating in the chorus. Can use G♯° instead of E.</td></tr> <tr><th> </th><th>26</th><td>G </td><td>G </td><td>A </td><td>A </td><td></td></tr> <tr><th>Verse 2 </th><th>30</th><td>D </td><td>D </td><td>D </td><td>D </td><td></td></tr> <tr><th> </th><th>34</th><td>F </td><td>F </td><td>D </td><td>D </td><td></td></tr> <tr><th> </th><th>38</th><td>D </td><td>D </td><td>D </td><td>D </td><td></td></tr> <tr><th> </th><th>42</th><td>F </td><td>F </td><td>D </td><td>D </td><td></td></tr> <tr><th>Ending </th><th>46</th><td>D </td><td>D </td><td>D </td><td>D </td><td></td></tr> <tr><th> </th><th>50</th><td>F </td><td>F </td><td>D </td><td>D </td><td></td></tr> <tr><th> </th><th>53</th><td>D </td><td>D </td><td>D </td><td>D </td><td>Sustained fade</td></tr> </tbody></table><h2>Jazz (F, 164bpm, <span class="timesig-num">4</span>/<span class="timesig-denom">4</span>)</h2><p>This is similar to the 12-bar blues progression, but with a more jazzy twist to the chords actually being used.</p><p>The basic idea is still the 12 bar pattern F / F / F / F / B♭ / B♭/ F / F / B♭ / C / F / F. However, there's some changes that are often used:</p><ul><li>The B♭ (IV) is often played as D° (vi°)</li> <li>The F is often played as an F<span class="chordadd">7</span>, particularly in long stretches.</li> <li>Bars 7–8 and 11–12 (of the 12-bar pattern) are usually played to walk into the next phrase (changing chords every two beats) instead of being played straight. In particular, bar 8 is <em>very</em> rarely played as an F, but if you're improvising over an F you'll still be fine.</li></ul><p>First, here's the basic pattern:</p><table class="chords"><tbody><tr><th></th><th> 1</th><td>F </td><td>F </td><td>F </td><td>F </td><td></td></tr><tr><th></th><th> 5</th><td>B♭</td><td>B♭</td><td>F </td><td>F </td><td></td></tr><tr><th></th><th> 9</th><td>B♭</td><td>C </td><td>F </td><td>F </td><td></td></tr></tbody></table><p>Now, here's the pattern as I've interpreted it. As I said, there's a lot of possibilities here.</p><table class="chords"><tbody><tr><th>Intro </th><th> 1</th><td>- </td><td> </td><td> </td><td> </td><td>Pickup</td></tr><tr><th>Head </th><th> 2</th><td>F<span class="chordadd">7</span> </td><td>F<span class="chordadd">7</span> </td><td>F<span class="chordadd">7</span> </td><td>F<span class="chordadd">7</span> </td><td>May substitute F for F<span class="chordadd">7</span>.</td></tr><tr><th> </th><th> 6</th><td>D°</td><td>D°</td><td>F<span class="chordadd">7</span> </td><td>C; D</td><td>Not sure about this phrase. May substitute B♭ or even Fm<span class="chordadd">6</span> for D°.</td></tr><tr><th> </th><th>10</th><td>Gm </td><td>C </td><td>F<span class="chordadd">7</span>; F</td><td>B♭; C</td><td></td></tr><tr><th>Chorus 1</th><th>14</th><td>F<span class="chordadd">7</span> </td><td>D°</td><td>F </td><td>F </td><td></td></tr><tr><th> </th><th>18</th><td>D°</td><td>D°</td><td>F<span class="chordadd">7</span> </td><td>C; D</td><td>Some of the accompaniments can get a bit muddied in the chords here.</td></tr><tr><th> </th><th>22</th><td>B♭ </td><td>Gm; C</td><td>F; F<span class="chordadd">7</span></td><td>B♭; C</td><td></td></tr><tr><th>Chorus 2</th><td class="chord-comment" colspan="6" rowspan="3">Too tired to transcribe these; it's jazz, so just get the idea from the pattern I've done so far.</td></tr><tr><th>Chorus 3</th></tr><tr><th>Head Out</th></tr></tbody></table><h2>Country (G, 100bpm, <span class="timesig-num">4</span>/<span class="timesig-denom">4</span>)</h2><p>Another 12-bar blues progression, played pretty much straight this time. (In case you haven't gotten the picture, 12-bar blues is great for improvising over, so it's used a lot in Magic GarageBand.)</p><table class="chords"><tbody><tr><th>Intro </th><th> 1</th><td>-</td><td> </td><td> </td><td> </td><td>Pickup</td></tr><tr><th>Verse 1</th><th> 2</th><td>G</td><td>G</td><td>G</td><td>G</td><td></td></tr><tr><th> </th><th> 6</th><td>C</td><td>C</td><td>G</td><td>G</td><td></td></tr><tr><th> </th><th>10</th><td>D</td><td>C</td><td>G</td><td>D</td><td>If you want to keep up with the lead-in for the last bar, it's D / C / Bm / Am</td></tr><tr><th>Verse 2</th><th>14</th><td>G</td><td>G</td><td>G</td><td>G</td><td></td></tr><tr><th> </th><th>18</th><td>C</td><td>C</td><td>G</td><td>G</td><td></td></tr><tr><th> </th><th>22</th><td>D</td><td>C</td><td>G</td><td>G</td><td></td></tr><tr><th>Verse 3</th><th>26</th><td>G</td><td>G</td><td>G</td><td>G</td><td></td></tr><tr><th> </th><th>30</th><td>C</td><td>C</td><td>G</td><td>G</td><td></td></tr><tr><th> </th><th>34</th><td>D</td><td>C</td><td>G</td><td>D</td><td>Same lead-in as the end of verse 1</td></tr><tr><th>Ending </th><th>38</th><td>G</td><td>G</td><td>G</td><td> </td><td>Sustained fade</td></tr></tbody></table><h2>Reggae (C min, 75bpm, <span class="timesig-num">4</span>/<span class="timesig-denom">4</span>)</h2><p>I don't really know reggae, so this may be totally off. I've marked it as C minor, even though when you create the project GarageBand lists it as C major.</p><table class="chords"><tbody><tr><th>Intro </th><th> 1</th><td>- </td><td></td><td></td><td></td><td>Pickup</td></tr><tr><th>Verse 1</th><th> 2</th><td>Cm</td><td>Cm</td><td>A♭; Gm<span class="chordadd">7</span></td><td>Cm</td><td></td></tr><tr><th> </th><th> 6</th><td>Cm</td><td>Cm</td><td>A♭; Gm<span class="chordadd">7</span></td><td>Cm</td><td></td></tr><tr><th>Chorus </th><th>10</th><td>B♭m; A♭</td><td>A♭; E♭</td><td>B♭m; A♭</td><td>A♭; E♭</td><td></td></tr><tr><th> </th><th>14</th><td>B♭m; A♭</td><td>A♭; E♭</td><td>B♭m; A♭</td><td>A♭; E♭</td><td></td></tr><tr><th>Verse 2</th><th>18</th><td>Cm</td><td>Cm</td><td>A♭; Gm<span class="chordadd">7</span></td><td>Cm</td><td></td></tr><tr><th> </th><th>22</th><td>Cm</td><td>Cm</td><td>A♭; Gm<span class="chordadd">7</span></td><td>Cm</td><td></td></tr><tr><th> </th><th>26</th><td>-</td><td></td><td></td><td></td><td>Sustain from last note</td></tr></tbody></table><h2>Funk (C, 120bpm, <span class="timesig-num">4</span>/<span class="timesig-denom">4</span>)</h2><p>Also really not sure how accurate this one is.</p><table class="chords"><tbody><tr><th>Intro </th><th> 1</th><td>Cm</td><td>Cm</td><td>Cm</td><td>E♭</td><td></td></tr><tr><th>Verse 1</th><th> 5</th><td>C </td><td>C </td><td>C </td><td>C</td><td rowspan="2">If it's too monotonous, you can make the first and third bars F; C (two beats each)</td></tr><tr><th> </th><th> 9</th><td>C </td><td>C </td><td>C </td><td>C</td></tr><tr><th> </th><th>13</th><td>F </td><td>F </td><td>C </td><td>C</td><td></td></tr><tr><th> </th><th>17</th><td>G </td><td>G </td><td>C </td><td>Em<span class="chordadd">7</span></td><td>You can use a G for the Em<span class="chordadd">7</span></td></tr><tr><th>Bridge </th><th>21</th><td>F </td><td>F </td><td>F </td><td>F</td><td></td></tr><tr><th> </th><th>25</th><td>F </td><td>F </td><td>G </td><td>G</td><td></td></tr><tr><th> </th><th>29</th><td>G </td><td> </td><td> </td><td> </td><td>Fermata from bar 28</td></tr><tr><th>Verse 2</th><th>30</th><td>C </td><td>C </td><td>C </td><td>C</td><td></td></tr><tr><th> </th><th>34</th><td>C </td><td>C </td><td>C </td><td>C</td><td></td></tr><tr><th>Ending </th><th>38</th><td>G<span class="chordadd">7</span></td><td>G<span class="chordadd">7</span></td><td>C </td><td>C</td><td></td></tr><tr><th> </th><th>42</th><td>G<span class="chordadd">7</span></td><td>G<span class="chordadd">7</span></td><td>C </td><td>C</td><td></td></tr><tr><th> </th><th>46</th><td>C </td><td>E♭</td><td> </td><td> </td><td></td></tr></tbody></table><h2>Latin (Fm, 98bpm, <span class="timesig-num">4</span>/<span class="timesig-denom">4</span>)</h2><p>This is pretty simple. There's just one pattern: Fm, E♭, D♭, C,for two beats each.</p><table class="chords"><tbody><tr><th>Intro </th><th> 1</th><td>Fm </td><td></td><td></td><td></td><td>Pickup</td></tr><tr><th>Verse 1</th><th> 2</th><td>Fm,E♭</td><td>D♭,C</td><td>Fm,E♭</td><td>D♭,C</td><td></td></tr><tr><th> </th><th> 6</th><td>Fm,E♭</td><td>D♭,C</td><td>Fm,E♭</td><td>D♭,C</td><td></td></tr><tr><th>Verse 2</th><th>10</th><td>Fm,E♭</td><td>D♭,C</td><td>Fm,E♭</td><td>D♭,C</td><td></td></tr><tr><th> </th><th>14</th><td>Fm,E♭</td><td>D♭,C</td><td>Fm,E♭</td><td>D♭,C</td><td></td></tr><tr><th>Bridge </th><th>18</th><td>Fm </td><td>Fm</td><td>Fm</td><td>Fm</td><td></td></tr><tr><th> </th><th>22</th><td>Fm </td><td>Fm</td><td>Fm</td><td>Fm</td><td></td></tr><tr><th> </th><th>26</th><td>Fm </td><td>Fm</td><td>Fm</td><td>Fm</td><td></td></tr><tr><th> </th><th>30</th><td>Fm,E♭</td><td>D♭,C</td><td></td><td></td><td></td></tr><tr><th>Verse 3</th><th>32</th><td>Fm,E♭</td><td>D♭,C</td><td>Fm,E♭</td><td>D♭,C</td><td></td></tr><tr><th> </th><th>36</th><td>Fm,E♭</td><td>D♭,C</td><td>Fm,E♭</td><td>D♭,C</td><td></td></tr><tr><th> </th><th>40</th><td>Fm </td><td></td><td></td><td></td><td></td></tr></tbody></table><h2>Roots Rock (G, 76bpm, <span class="timesig-num">4</span>/<span class="timesig-denom">4</span>)</h2><p>Not really sure what distinguishes the chorus from the verse, but it could just be the instruments I picked don't really do much differently. This is a simple pattern, but has a lot of room for improvisation.</p><table class="chords"><tbody><tr><th>Intro </th><th> 1</th><td>G </td><td> </td><td> </td><td> </td><td>Pickup</td></tr><tr><th>Verse 1</th><th> 2</th><td>G </td><td>Am</td><td>C </td><td>D</td><td></td></tr><tr><th> </th><th> 6</th><td>G </td><td>Am</td><td>C </td><td>D</td><td></td></tr><tr><th>Chorus </th><th>10</th><td>G </td><td>Am</td><td>C </td><td>D</td><td></td></tr><tr><th> </th><th>14</th><td>G </td><td>Am</td><td>C </td><td>D</td><td></td></tr><tr><th>Bridge </th><th>18</th><td>Em</td><td>Bm</td><td>Am</td><td>G</td><td></td></tr><tr><th> </th><th>22</th><td>Em</td><td>Bm</td><td>Am</td><td>D</td><td></td></tr><tr><th>Verse 2</th><th>26</th><td>G </td><td>Am</td><td>C </td><td>D</td><td></td></tr><tr><th> </th><th>30</th><td>G </td><td>Am</td><td>C </td><td>D</td><td></td></tr><tr><th>Outro </th><th>34</th><td>G </td><td>G </td><td> </td><td> </td><td></td></tr></tbody></table><h2>Slow Blues (A♭, 88bpm, <span class="timesig-num">12</span>/<span class="timesig-denom">8</span>)</h2><p>Wow, there IS something other than <span class="timesig-num">4</span>/<span class="timesig-denom">4</span> in here! Unsurprisingly, it <em>is</em> still 12-bar blues, although with a IV in the second bar to keep it interesting.</p><table class="chords"><tbody><tr><th>Intro </th><th> 1</th><td>E♭</td><td> </td><td> </td><td></td><td>Pickup</td></tr><tr><th>Verse 1</th><th> 2</th><td>A♭</td><td>D♭</td><td>A♭</td><td>A♭</td><td></td></tr><tr><th> </th><th> 6</th><td>D♭</td><td>D♭</td><td>A♭</td><td>A♭</td><td></td></tr><tr><th> </th><th>10</th><td>E♭</td><td>D♭</td><td>A♭</td><td>E♭</td><td></td></tr><tr><th>Verse 2</th><th>14</th><td>A♭</td><td>D♭</td><td>A♭</td><td>A♭</td><td></td></tr><tr><th> </th><th>18</th><td>D♭</td><td>D♭</td><td>A♭</td><td>A♭</td><td></td></tr><tr><th> </th><th>22</th><td>E♭</td><td>D♭</td><td> </td><td></td><td></td></tr><tr><th>Ending </th><th>24</th><td>A♭</td><td>A♭</td><td>A♭</td><td></td><td>The measure breaks are hinky<!--; it's a <i>rallentando</i> (or <i>tempo rubato</i> in its purest form)--></td></tr></tbody></table>Piquanhttp://www.blogger.com/profile/06983189170367989592noreply@blogger.com6tag:blogger.com,1999:blog-13322352.post-17371082211406550392011-01-08T19:43:00.000-08:002011-01-09T01:04:25.817-08:00Imminent Death of the Net Predicted<div style="float: right;"><script src="http://ipv6.he.net/v4ex/sidebar.js" type="text/javascript">
</script></div>I didn't know that it was coming this soon, but in mid-February, the Internet will run out of addresses.<br />
<br />
My blog post's title is a bit tongue-in-cheek. People have been talking about the net not being able to handle the growth rate for decades. The address space exhaustion is going to take some work to deal with, but it's not going to cause things to grind to a halt.<br />
<br />
There's already some great resources to explain this to people who don't know about the problem:<br />
<ul><li>You can get an app for your smartphone, blog, etc that shows the countdown from <a href="http://ipv6.he.net/statistics/">Hurricane Electric's IPv6 page</a>. (Hurricane Electric is a very IPv6-friendly ISP. If you're a technical person, they can set you up with a free tunnel.)</li>
<li>ABC wrote a story, <a href="http://abcnews.go.com/Technology/story?id=3445885">"IP address shortage to limit Internet access"</a></li>
<li>A slightly more technical, but still pretty accessible, FAQ is on <a href="http://www.potaroo.net/ispcol/2010-09/exhaustguide.html">Geoff Huston's blog</a>.</li>
<li>Of course, Wikipedia has a page on <a href="http://en.wikipedia.org/wiki/IPv4_address_exhaustion">IPv4 address exhaustion</a> for all the backgroundy stuff that Wikipedia tends to have.</li>
</ul>What I couldn't find is an answer to my friend's question, "What will happen when we run out?" Nobody knows for sure, but here's a basic rundown.<br />
<br />
Internet addresses are globally administered by IANA. They give blocks of addresses to the regional groups, RIRs. These cover continent-sized areas, like Europe, Asia/Pacific, North America, etc. The RIRs give blocks to the local registries (LIRs), which are usually large ISPs, but may also be large organizations (like Stanford, although that's mostly because they were one of the first places on the Internet).<br />
<br />
Once IANA runs out (which is the event in February), then the individual regions can't get more addresses. The first to run out will be APNIC (Asia/Pacific), in about eight months or so (we're guessing; nobody can be sure). Then APNIC won't be able to give out new addresses to its LIRs. When an LIR can't get new addresses, it can't attach new customers. Its customers are mostly small ISPs or companies.<br />
<br />
So there's a trickle in the "supply chain" before things get to the end users. You won't see problems until it gets down to this level, although you may see some changes as ISPs hurriedly try to prepare (with IPv6; I'll get to that in a minute).<br />
<br />
Once a small ISP runs out, then it can't put new servers on the Internet: that means no new web servers, no new mail servers, nothing. (To a limited degree, it's possible to piggyback these on existing servers in some cases.) The ISP can attach new home customers (who don't use servers) for a little while, but if they attach too many, then people won't be able to connect. (A home ISP tends to hand out addresses for two hours at a time, as long as the customer's router or computer is on.)<br />
<br />
At that point, to deal with that, they'll probably start putting large blocks of home customers behind NATs. NATs let several computers use the same IP address. However, you can't have a server behind a NAT, it can be unreliable, and a lot of games, video or voice chat programs, file-sharing programs, etc. don't behave well with NAT. Because APNIC is running out of addresses, much of Russia and China are already behind regional NATs.<br />
<br />
There's also a possibility that organizations will start trading small blocks of IP addresses. (Nobody "owns" them; they're allocated by IANA.) However, routers will have a hard time keeping up with that (a router is designed to think in large blocks, and lots of tiny blocks can be a problem); this might make the Internet more flaky.<br />
<br />
I don't really know how the situation is going to be for end users. There'll be some hiccups, but it'll probably come down to just a little bit of reconfiguration; your ISP will tell you to do it, and you'll have some notice. If you've got a particularly old router, you may end up having to replace it.<br />
<br />
Ultimately, the solution is to go to IPv6. This has a lot more addresses: 3.4x10^28; that's enough for every gram of matter on Earth to have 10 addresses. (The current mechanism, IPv4, has about 4 billion addresses.) However, IPv6 adoption has been slow, because customers aren't asking for it. Most devices support IPv6, but some don't. Not many websites are currently equipped for IPv6; you have to have IPv4.<br />
<br />
I migrated my home network to IPv6 yesterday, and it was a breeze. Right now it's just to get a dancing turtle on <a href="http://www.kame.net/">Kame's website</a>, but I recommend that my techie friends with a home network do the migration. Even though your ISP probably doesn't provide IPv6 connectivity, there's still peers all over the place that'll give you tunnels.<br />
<br />
<div style="float: right;"><a href="http://ipv6.he.net/certification/scoresheet.php?pass_name=piquan" target="_blank"><img alt="IPv6 Certification Badge for piquan" border="0" height="194" src="http://ipv6.he.net/certification/create_badge.php?pass_name=piquan&badge=2" width="250" /></a></div>For the technically-minded among us,I actually did the migration twice: once using 6to4 for connectivity with the Anycast2002:c058:6301:: peer for 6to4. Then, I did it again using a tunnel (using FreeBSD's gif interface, which also works on OS X) that I got for free from <a href="http://tunnelbroker.net/">Hurricane Electric's website</a>. The tunnel is faster (23ms ping time vs the 6to4 peer's 96ms), and their website walks you through the process; it's got easy instructions for Windows, OS X, Linux, BSD, or whatever else you've got.Piquanhttp://www.blogger.com/profile/06983189170367989592noreply@blogger.com0tag:blogger.com,1999:blog-13322352.post-1074701336840420312010-08-17T20:06:00.000-07:002010-09-18T20:41:32.265-07:00Dealing With Dashcode, Part 4: Dashcode and DebuggingUPDATE: As Todd points out in a comment, the bug described here has been fixed in 3.2.4. That's now out of beta, and registered developers can <a href="http://developer.apple.com/technologies/xcode.html">download it from Apple</a>.<br />
<br />
(This is a continuation of a <a href="http://stopwords.piquan.org/2010/07/dealing-with-dashcode-part-3-dashcode.html%22">previous post</a>. You may want to read the <a href="http://stopwords.piquan.org/2010/07/dealing-with-dashcode-part-1.html">first post</a> in this series for an overview of the architecture.)<br />
<br />
The last problem I'm blogging about is actually the first one that I encountered. It's also what first inspired me to write this "Dealing With Dashcode" series of posts.<br />
<br />
Early on in my development process, I made sure that I had the latest XCode installed (3.2.3), which is the iOS 4 development environment; my app is targeting iOS 4 on iPhone 3GS and 4. The iOS SDK includes the iPhone Simulator, with environments for iOS 3.2 on the iPad, and iOS 4 on the iPhone 3GS and iPhone 4.<br />
<br />
Dashcode comes with a quite handy JavaScript debugger. Since Mobile Safari's built-in developer support is fairly scant (you can globally turn on and off console logging), having a JavaScript debugger is really handy. The JavaScript debugger works against Mobile Safari in the simulator: when you're running your program in the iPhone Simulator, you can use breakpoints, inspection, single-stepping, etc. from within Dashcode's debugger. Nice, huh? Too bad it doesn't work.<br />
<br />
Here's what I've been able to work out. Dashcode implements this debugger by loading an extra framework into Mobile Safari when it launches. When you click the "Run" button, Dashcode launches Mobile Safari with some extra arguments to load a framework called CayenneClientDebugger (which is within the Dashcode executable). It looks like Mobile Safari (as well as the Browser Simulator that's part of Dashcode's Safari targeting) has hooks to load this framework for some debugging usage. The debugger UI that's built into Dashcode communicates with the debugger backend loaded in Mobile Safari.<br />
<br />
(By the way: I looked online to see if I could find any references to this Cayenne. There's an Apache project named Cayenne, but it seems unrelated. I suspect that Cayenne was the internal codename for Dashcode; it looks like the bulk of Dashcode is implemented in a framework called CYKit.)<br />
<br />
Now, here's the catch. iOS 4 uses a different ABI than 3.x did. It looks like the CayenneClientDebugger is built for the old version. Mobile Safari refuses to load it, so instead of a debugger, all you get is a message in system.log.<br />
<br />
I tried to cherry-pick a few bits of XCode from a backup I had from before I upgraded it. I tried copying iPhoneSimulator3.0.sdk into the current installation, and although the iPhone Simulator would display the 3.0 version in its Version menu, if I tried to actually select it, then the Simulator would crash. I tried a few other combinations of replacing the Simulator, CayenneClientDebugger, and even all of Dashcode, but couldn't get anything really going. I probably could back out the entire XCode installation, but that's a bit far.<br />
<br />
As it was, I never did solve this problem. I just made a lot of use of console.log for debugging purposes. If I needed something more than I could get that way, then I'd use Safari (the OS X one); I could load my webpage there and use Safari's built-in development tools. The most signifiant bit of UI I was using for this app (StackLayout; it's essentially a nested list that you navigate one level at a time) doesn't actually work in Safari — only Mobile Safari — but I could still test little bits in some ways. It was enough for me to finish coding.<br />
<br />
I haven't looked into the 4.1 SDK yet (as of this writing it's in beta and has limited availability); I hope that Apple's fixed this problem. Not having access to a debugger makes Mobile Safari development a bit tricky.<br />
<br />
By the way: there are a few options out there. The best of these seems to be "<a href="http://www.joehewitt.com/blog/firebug_for_iph.php">Firebug for iPhone</a>". Don't get your hopes up, though: it's not <a href="http://getfirebug.com/">Firebug</a> (the must-have development tool for Firefox). It's not even close. Firebug for iPhone mostly just relays console messages to your server, and lets you send commands to the browser. While this is very useful, it's still not a substitute for a full JavaScript debugger. (I considered writing something similar, using AJAX to relay messages between my code and a program running on my server.)<br />
<br />
IBM has an article on using <a href="http://www.aptana.com/">Aptana</a> (a webapp development system for <a href="http://www.eclipse.org/">Eclipse</a>) while developing iPhone web apps. It's entitled "<a href="http://www.ibm.com/developerworks/opensource/tutorials/os-eclipse-iphone-debug/">Debug iPhone Web applications with Eclipse</a>", but in truth, Aptana doesn't support the iPhone for debugging. Instead, the article recommends you use Firefox with Aptana to debug your application. Uh... no thanks. WebKit and Gecko are different enough that, if I'm developing an iPhone-specific application, I'll want to do my development with Mobile Safari.<br />
<br />
Neither of these was really worth the effort of learning; the app in question is simple enough that console.log does everything I needed. But I do hope that Apple has fixed this problem in the newest Dashcode.Piquanhttp://www.blogger.com/profile/06983189170367989592noreply@blogger.com2tag:blogger.com,1999:blog-13322352.post-57382629746683202212010-07-25T16:29:00.000-07:002010-07-25T16:56:15.563-07:00Dealing With Dashcode, Part 3: Dashcode, WebDAV, and Trailing Slashes(This is a continuation of a <a href="http://stopwords.piquan.org/2010/07/dealing-with-dashcode-part-2-apache.html">previous post</a>. You may want to read the <a href="http://stopwords.piquan.org/2010/07/dealing-with-dashcode-part-1.html">first post</a> in this series for an overview of the architecture.)<br />
<br />
At this point, I had a WebDAV-accessible directory that I had configured Dashcode to deploy to. It worked to deploy once, but after that, future deploys would fail. A dialog box would appear with the title, "An error occured while deploying the project to the server." and the text, "The folder “myapp” cannot be created on the server." After that, the status at the top of the Run & Share page would show "Last deploy attempt failed: The folder “myapp” cannot be created on the server."<br />
<br />
The problem is that Apple's WebDAVFS will send a WebDAV PROPFIND request for "serv/myapp" (which is a folder), get a 301 redirect to "serv/myapp/" (with a trailing slash this time), and then send a PROPFIND for "serv/myapp/", but using credentials which are only valid for "serv/myapp". Apache doesn't like that, sends a 401 Forbidden response, and WebDAVFS gives up and sends an error to Dashcode. (I'm not sure whether this particular problem of retaining credentials across redirects is specific to WebDAVFS, or if it's actually in CFNetwork.)<br />
<br />
The solution is to not have Apache send the 301 redirect. Adding this to the Location section for my DAV URL does the trick:<br />
<blockquote style="font-family: 'Courier New', Courier, monospace;"><Location /myapp-devel/dav/><br />
<div style="margin-left: 4ex;"><div># In addition to the DAV-related directives</div><div># I talked about in Part 1, add:</div><div>DirectorySlash Off</div></div></Location></blockquote>This is probably a good thing to do on all DAV directories, since redirection isn't exactly a great thing to do within a filesystem.<br />
<br />
After this, I could deploy repeatedly from Dashcode with no problems.<br />
<br />
This is as much as I needed to deploy my app. Debugging it, though, was still of an issue. See the next post!Piquanhttp://www.blogger.com/profile/06983189170367989592noreply@blogger.com0tag:blogger.com,1999:blog-13322352.post-83166708007646529312010-07-25T15:11:00.000-07:002010-07-25T17:03:17.581-07:00Dealing With Dashcode, Part 2: Apache, WebDAV, and umasks(This is a continuation of a <a href="http://stopwords.piquan.org/2010/07/dealing-with-dashcode-part-1.html">previous post</a>.)<br />
<br />
At this point, I had set up a directory that I could publish to using WebDAV and also edit as my regular user id. Since WebDAV works as user www, my normal user id (piquan) wouldn't have access to edit them. So I needed to set up a directory using FreeBSD's ACLs to give write access to both me and www.<br />
<blockquote style="font-family: 'Courier New', Courier, monospace;">$ mkdir serv<br />
$ setfacl -m u:piquan:rwx,u:www:rwx piquan<br />
$ setfacl -d -m u:piquan:rwx,u:www:rwx,u::rwx,g::rx,o::rx,mask::rwx serv</blockquote><div>The problem is, when WebDAV created a directory, it created it with mode 755 (rwxr-xr-x). The problem is, if a program that's not aware of ACLs creates (or chmods) a file or directory, the ACLs mask bits get taken from the group access bits. In other words, the permissions that the program gives for the group restrict the permissions of any u:… or g:… entries in the ACLs. The r-x mask that WebDAV created my directory with masked my u:piquan:rwx ACL entry to an effective u:piquan:r-x. (Since www was the directory's owner, its u::rwx entry didn't get masked.)<br />
<br />
In other words, I could no longer add, rename, or delete files from the directories that WebDAV created. I also couldn't edit the files, since they were created with mode 644.<br />
<br />
After some investigation, I found that WebDAV creates files and directories based on the Apache process's <a href="http://en.wikipedia.org/wiki/Umask">umask</a>.<br />
<br />
On one hand, I could edit /usr/local/sbin/apachectl and /usr/local/etc/rc.d/apache22 to set the umask before launching Apache. However, I didn't like that idea, since then any Apache upgrades would blow away my changes. I wanted something a little more lasting.<br />
<br />
Instead, I used – or rather abused – a mechanism in rc.d/apache22 that's designed for setting <a href="http://www.freebsd.org/cgi/man.cgi?query=ulimit&sektion=3">process limits</a>. It includes a bit that runs some shell code and evaluates the output. I added to /etc/rc.conf:<br />
<blockquote style="font-family: 'Courier New', Courier, monospace;">apache22limits_enable=YES<br />
apache22limits_args="echo umask 0002"</blockquote><div>After that, I ran "sudo /usr/local/etc/rc.d/apache22 restart", sudo rm'd the files that WebDAV had created already, and published it again. This time, WebDAV created files I could edit. Mind you, after that, Dashcode couldn't delete the directory it had created to republish it, but that's for my next post.</div></div>Piquanhttp://www.blogger.com/profile/06983189170367989592noreply@blogger.com0tag:blogger.com,1999:blog-13322352.post-46787094697734534092010-07-25T01:54:00.000-07:002010-08-12T15:27:34.341-07:00Dealing With Dashcode, Part 1: Architecture and Basic ConfigurationI've been writing a web app for the iPhone recently. The main tool for this type of web app is Dashcode, which is part of Xcode. There's a couple of problems with my setup: one of them I've solved, and the other I haven't. In the hopes that somebody else on the Internets might find this information useful, here ya go.<br />
<br />
The solved problem is how to publish this on my own server. Most of this revolved around my Apache configuration. Most of this should be pretty obvious to anybody who's dealt with WebDAV, but one problem with authentication wasn't obvious; that's at the end, but first I'll talk about my setup.<br />
<br />
My app involves a combination of three major elements: static HTML documents, dynamically generated data, and the web app that's generated by Dashcode. (The web app is served as a static page by Apache, of course – it's the client that does all the computation there – but it has its own considerations in the Apache configuration.) My web site, therefore, has three sections. (All the project names, domain names, etc. in this are replaced with more generic versions. These are not real URLs.)<br />
<ul><li>http://www.piquan.org/myapp/gen/ : Dynamic content, generated by Python scripts. This is being read by Ajax, so there's some files in here that generate XML, some that generate JSON, and for my own convenience it's nice to have a few static files in here as well.</li>
<li>http://www.piquan.org/myapp/htdocs/ : Static content, just a bunch of HTML files. The web app will sometimes send the user to these files (and out of the app entirely) by setting window.location.</li>
<li>http://www.piquan.org/myapp/serv/ : This holds the app generated by Dashcode. I want Dashcode to be able to blow everything under here away, and replace it.</li>
</ul><div>All of the files are currently in ~/src/myapp, and while I'm just working on the initial version, I want it to be served straight out of there. While I'm doing development, all of this is under http://www.piquan.org/myapp-devel/ , and I'll move to /myapp/ once I've got a releasable version.</div><div><br />
</div><div>To get the static files to be served is pretty easy. I just put this in my Apache config (within the VirtualHost section for www.piquan.org):</div><blockquote style="font-family: 'Courier New', Courier, monospace;">Alias /myapp-devel/ /home/piquan/src/myapp/<br />
<Directory /home/piquan/src/myapp><br />
<div style="margin-left: 4ex;">AllowOverride All<br />
Order allow,deny<br />
Allow from 192.168.42.0/24</div></Directory></blockquote><br />
(The "AllowOverride All" is to make it easier for me to experiment using .htaccess instead of needing to change my Apache config and restart the server. Of course, .htaccess is somewhat limited in some ways, as we'll discuss later.)<br />
<br />
Ok, so what next? Well, I need to serve the dynamic content. Remember that there's both static and dynamic files in there. Ideally, it should be transparent whether the content is coming from a static file or being dynamically generated.<br />
<br />
To generate the dynamic content, I chose to use Python with raw WSGI. This is a very simple way to write simple dynamic content. For complex projects, then a web application framework like Django would be more appropriate, but here I'm looking at about 300 lines of code, so Django would be overkill. Also, remember that here I'm only sending either simple XML or JSON. Now, WSGI is actually an interface standard (like CGI), not an implementation; the Apache WSGI implementation is <a href="http://code.google.com/p/modwsgi/">mod_wsgi</a>. I installed this (using FreeBSD's Ports mechanism).<br />
<br />
The default for mod_wsgi is to serve data from within the Apache process. On one hand, this is very efficient. On the other hand, it's very inconvenient for development, because if you change your code, you have to restart Apache. There's an easy solution: put the WSGI handlers in a separate group of processes that Apache automatically manages. When you change your code, mod_wsgi will automatically kill and restart those. If you don't understand that, then don't worry: the config file additions are quite simple. I just added to the VirtualHost section:<br />
<blockquote style="font-family: 'Courier New', Courier, monospace;">WSGIDaemonProcess piquan.org display-name=%{GROUP}<br />
WSGIProcessGroup piquan.org</blockquote>This sets up a simple set of processes to manage all WSGI requests. In this configuration, the same set manages everything across my entire domain. (That's not because of "piquan.org" there; that's just an identifier for the process group. It applies to my entire domain because it's within the VirtualHost section, not within a Directory section.) Once it's time to release, I'll change the name from "piquan.org" to "myapp" or something, tune the WSGIDaemonProcess line to use the appropriate amount of resources (in terms of processes, timeouts, etc), and move the WSGIProcessGroup to within a Directory section so that it only applies to my program.<br />
<br />
The mod_wsgi <a href="http://code.google.com/p/modwsgi/wiki/QuickConfigurationGuide">Quick Configuration Guide</a> says to use WSGIScriptAlias to tell mod_wsgi what directory should be considered WSGI programs. However, remember that I have a mix of dynamic and static files, so I took a different tack.<br />
<br />
To deal with this, I put the static files in files named things like staticdata.js, staticdata.xml. The dynamic files are named things like dynamicdata.js.wsgi and dynamicdata.js.wsgi. Then, a bit of .htaccess magic, along with the miracle of MultiViews (which I don't know WHY it's not more widespread) lets me tell Apache to send things the right place.<br />
<blockquote style="font-family: 'Courier New', Courier, monospace;">Options All MultiViews<br />
MultiviewsMatch Handlers Filters <br />
AddType application/json .js<br />
AddType text/xml .xml<br />
AddHandler wsgi-script .wsgi</blockquote><br />
This is actually a bit of overkill. I specified a very widespread <a href="http://httpd.apache.org/docs/2.2/mod/core.html#options">Options</a> line. The only option I really needed for this was MultiViews. [Edit: I also need ExecCGI. Thanks, Graham!] It's just more convenient during development to have things like Indexes and FollowSymLinks on. Also, I set the <a href="http://httpd.apache.org/docs/2.2/mod/mod_mime.html#multiviewsmatch">MultiviewsMatch</a> to include Filters, when really I only need it to deal with Handlers. (I'll probably turn on the mod_deflate filter later, since this is an iPhone web app that will sometimes be sent over 3G and even EDGE networks, but that's not something that MultiviewsMatch needs to be involved in.) I generally tend to put in pretty broad web capabilities during dev, and then tighten it when I deploy.<br />
<br />
The two AddType directives are because I thought one of the libraries I was using was being a bit picky about the Content-Type it gets back. (I could have named my JSON files with .json instead of .js, since Apache already associates application/json with the .json extension, but the .json extension irks me a bit.) As it turns out, the library wasn't as picky as I thought (for instance, it would be happy with Apache's default of application/xml, which is arguably more appropriate for this purpose; note that both are valid MIME types for XML), but I left them in anyway. Note that the AddType directives only apply to the static data; they don't apply to WSGI scripts, since those send their own Content-Type header.<br />
<br />
(By the way: when you're writing apps like this, you can handle REST-style URLs pretty easily. For example, http://www.piquan.org/myapp/gen/person/piquan could be handled by a script named gen/person.wsgi, which can look at environ['PATH_INFO'] to read the "/mary" bit. Note that this is the environ passed to application, not sys.environ. If you really wanted to get fancy, you can probably add some MultiViews magic along with <limit> sections to separate this into person.GET.wsgi, person.PUT.wsgi, person.POST.wsgi, etc. Hmmm... maybe I'll write a filter module to let that happen easily.)</limit><br />
<br />
Finally, we have the Dashcode-generated content. This is created on my <a href="http://www.amazon.com/Apple-MacBook-MC371LL-15-4-Inch-Laptop/dp/B003G2ZJTG?ie=UTF8&tag=stopwords-20&link_code=btl&camp=213689&creative=392969" target="_blank">Mac</a><img alt="" border="0" height="1" src="http://www.assoc-amazon.com/e/ir?t=stopwords-20&l=btl&camp=213689&creative=392969&o=1&a=B003G2ZJTG" style="border: none !important; margin: 0px !important; padding: 0px !important;" width="1" />, and I need to send it to my web server. Hello, WebDAV! WebDAV is pretty nice to have on your web server if you use a Mac as your <a href="http://www.amazon.com/Apple-Mac-Mini-MC270LL-Desktop/dp/B0013FK9U2?ie=UTF8&tag=stopwords-20&link_code=btl&camp=213689&creative=392969" target="_blank">desktop</a><img alt="" border="0" height="1" src="http://www.assoc-amazon.com/e/ir?t=stopwords-20&l=btl&camp=213689&creative=392969&o=1&a=B0013FK9U2" style="border: none !important; margin: 0px !important; padding: 0px !important;" width="1" />: it lets you keep an iCal calendar shared on a website (without paying for <a href="http://www.amazon.com/Apple-MC660Z-A-Mobileme-Individual/dp/B003TLJUJE?ie=UTF8&tag=stopwords-20&link_code=btl&camp=213689&creative=392969" target="_blank">MobileMe</a><img alt="" border="0" height="1" src="http://www.assoc-amazon.com/e/ir?t=stopwords-20&l=btl&camp=213689&creative=392969&o=1&a=B003TLJUJE" style="border: none !important; margin: 0px !important; padding: 0px !important;" width="1" />!), gives you a pretty convenient and WAN-accessible file storage from the Finder, you can publish from <a href="http://www.amazon.com/Apple-MB966Z-A-iLife-09/dp/B0014X5XEK?ie=UTF8&tag=stopwords-20&link_code=btl&camp=213689&creative=392969" target="_blank">iWeb</a><img alt="" border="0" height="1" src="http://www.assoc-amazon.com/e/ir?t=stopwords-20&l=btl&camp=213689&creative=392969&o=1&a=B0014X5XEK" style="border: none !important; margin: 0px !important; padding: 0px !important;" width="1" />, and so on. (Ok, blatant advertising done.)<br />
<br />
<b><span class="Apple-style-span" style="color: red;">WARNING</span></b>: Don't configure WebDAV until you have secured your web server! WebDAV lets people write to your disk. That's its point. If you don't have a secure server, then you may find people you don't want writing to your disk.<br />
<br />
In fact, don't trust my configs on this. Read over the docs for <a href="http://httpd.apache.org/docs/2.2/mod/mod_dav.html">mod_dav</a>, and you should have a decent understanding of web server security. At a bare minimum, read Apache's docs on <a href="http://httpd.apache.org/docs/2.2/misc/security_tips.html">Security Tips</a>, and also the docs on <a href="http://httpd.apache.org/docs/2.2/howto/auth.html">Authentication, Authorization and Access Control</a>. You should also know why the latter is insecure. (Hint: Don't use Basic auth; use Digest instead!)<br />
<br />
Ok, now that I've made it clear that I don't want you to open your disk to the whole world, here's a bit of my configuration to allow Dashcode to publish using WebDAV. I actually opened up a lot more than I needed to: instead of just the area where Dashcode puts its static files, I have it set up to allow WebDAV access to my entire program. This lets me fiddle with stuff in the Finder if I need to.<br />
<br />
Now, here's why that's a really bad idea from a security perspective. WebDAV lets people write files. WSGI lets people run files. That means that with the two together, an attacker could write to, and then execute, a file on MY computer. Bad news. Don't do this unless you're satisfied with your security: in particular, and at least put a <a href="http://httpd.apache.org/docs/2.2/howto/access.html">reasonable Allow clause</a> in place.<br />
<br />
Here's the configuration I used. Again, this is back in the Apache config file, in the VirtualHost section.<br />
<blockquote style="font-family: 'Courier New', Courier, monospace;">Alias /myapp-devel/dav/ /home/piquan/src/myapp/<br />
<Location /myapp-devel/dav/><br />
<div style="margin-left: 4ex;">Dav On<br />
Options Indexes<br />
Require user piquan</div></Location></blockquote>And then in .htaccess:<br />
<blockquote style="font-family: 'Courier New', Courier, monospace;">AuthType Digest<br />
AuthName "myapp DAV area"<br />
AuthUserFile /home/piquan/.htdigest</blockquote>(By the way: everything I put in my .htaccess, you can put in the regular Apache config. I just prefer to use .htaccess for everything I can, since it lets me change options without restarting Apache.)<br />
<br />
Finally, I set up my password file by using the <a href="http://httpd.apache.org/docs/2.2/programs/htdigest.html">htdigest</a> shell command on my web server:<br />
<blockquote style="font-family: 'Courier New', Courier, monospace;">$ htdigest ~/.htdigest "myapp DAV area" piquan</blockquote>(I already had a ~/.htdigest, and just was adding a new realm+username+password tuple. If I didn't have an .htdigest, I would need to specify the -c option.)<br />
<br />
Now, the <a href="http://httpd.apache.org/docs/2.2/mod/mod_dav.html">mod_dav</a> docs recommend using completely different URLs for the WebDAV URL and the "regular" serving URL. That's probably a little bit easier to configure. As for me, I kinda wanted to keep my entire project under the same top-level directory (myapp-devel), so I made the DAV-enabled section a subtree.<br />
<br />
Ok, there's one element left: permissions. I needed the web server to be able to write to the serv directory (where Dashcode was to write its files), but I also wanted to be able to – as my normal user on the web server – edit those files (to edit the manifest, or for whatever other post-deploy changes I wanted to make). Since WebDAV accesses files as user www, and I access them as user piquan, I needed to make serv writable by both. But I didn't want to make it globally writable, either.<br />
<br />
For this, I turned to <a href="http://www.freebsd.org/doc/en/books/handbook/fs-acl.html">FreeBSD's ACL support</a>. I'm not going to go into details on how to configure a filesystem or kernel for ACL support; there's guides for that online already. (If you don't like the Handbook's dry style, try the <a href="http://onlamp.com/pub/a/bsd/2005/09/22/FreeBSD_Basics.html">O'Reilly ONLamp</a> article instead.)<br />
<br />
I needed the serv directory itself to be writable by www and piquan, and also I needed any files within that either I or WebDAV create to be the same (unless explicitly changed). This means configuring both an access ACL and a default ACL.<br />
<blockquote style="font-family: 'Courier New', Courier, monospace;"><div>$ mkdir serv</div><div>$ setfacl -m u:piquan:rwx,u:www:rwx piquan</div><div># The next command is all on one line.</div><div>$ setfacl -d -m u:piquan:rwx,u:www:rwx,u::rwx,g::rx,o::rx,mask::rwx serv</div></blockquote>After all this was done, I went back to Dashcode's "Run & Share" section, and added my web server as a new destination. Finally, it was time to publish!<br />
<br />
At this point, everything I've talked about so far worked. But soon after that, I started having problems. I'll tackle each of these in a separate post.Piquanhttp://www.blogger.com/profile/06983189170367989592noreply@blogger.com2tag:blogger.com,1999:blog-13322352.post-37607044626344600982009-04-18T15:36:00.000-07:002010-05-17T17:20:44.453-07:00Good BusinessesHere's some businesses I dealt with over the last week that have given me excellent customer service, and I'd like to recognize publicly.<br />
<div><ul><li><a href="http://speakeasy.net/">Speakeasy</a>, my DSL provider<a href="http://www.starvingmusician.com/"></a></li>
<li><a href="http://www.starvingmusician.com/">The Starving Musician</a>, where I get instruments and equipment. (I don't deal with them online; I go to their Santa Clara location.)</li>
<li><a href="http://www.statefarm.com/bank/bank.asp">State Farm Bank</a>, my bank (natch)</li>
<li><a href="http://www.audible.com/">Audible</a>, an audiobook website</li>
</ul><div>Shouts out to these companies for taking care of their customers!</div></div>Piquanhttp://www.blogger.com/profile/06983189170367989592noreply@blogger.com0tag:blogger.com,1999:blog-13322352.post-24678704153355603412009-02-19T01:30:00.001-08:002009-02-19T01:50:57.966-08:00A few non-secretsA month and a half without a post, yow. Ok, so here's the thing. I've had stuff going on in my life, but a lot of it involves friends, and their privacy. I can't talk about the friend getting surgery, because that's medical. I can't talk about the acquaintance who I suspect may be trying to manipulate me, because it's just a hunch right now. There's a lot of stuff I can't talk about going on.<div><br /></div><div>Here's some things I <span class="Apple-style-span" style="font-style: italic;">can</span> talk about.</div><div><br /></div><div>Two of my best friends are getting married in two weeks. Congratulations, Scott and Mili! Scott and I have been close friends since fourth grade, and he and Mili have been crazy about each other for years.</div><div><br /></div><div>My work has been progressing well. I've still been working on the project I talked about in my last post, but the pilot has been running more slowly than we had expected. There's been some minor technical changes from the description I made (most notably changing from using a magic uid/gid as a marker to using UFS2 extended attributes), but all the details will be out when we open-source it.</div><div><br /></div><div>As you know, I've occasionally been writing <a href="http://www.soundclick.com/piquan">some music</a>. Mostly funny stuff, not good stuff, although I do kinda like "<a href="http://soundclick.com/share?songid=7176212">Baby Wombats (House Remix)</a>". I've also been working on a metal song about puppies and rainbows and stuff. I've got it written, but need to re-record it since what I've recorded so far is just draft quality.</div><div><br /></div><div>I hope I'll have something more interesting to post soon!</div>Piquanhttp://www.blogger.com/profile/06983189170367989592noreply@blogger.com0tag:blogger.com,1999:blog-13322352.post-51485765576315990972008-12-31T13:18:00.000-08:002010-05-17T17:23:46.658-07:00What I'm working on at JuniperI haven't had much time to update my blog for a while. It's mostly been work, so I think I'll tell you all about it.<div><br />
</div><div><div>I'm loving my job at Juniper, although I must say it's keeping me very busy lately. The project I'm on is pretty close to the wire. In the weeks approaching the Christmas holidays, I was spending every waking minute working on my code to get it ready. That left me able to go to Christmas and enjoy it without worrying about work (more about my vacation in another post, but the basic summary is, it was <span class="Apple-style-span" style="font-style: italic;">terrific</span>!). Even though my part is working great, there's some other parts to the project that are a bit tense, so I'm helping where I can. Since I got back from vacation, though, things have been a bit more relaxed for me.</div><div><br />
</div><div>Anyway, I can't go into all the background on the project, but I can tell you about my part. (This gets a bit technical from here on in, so you can skip it if you're not a computer geek.) The goal is to make it possible to compile with source code on NFS servers and object code on local disk. The source then can be edited remotely on workstations, and easily backed up; the objects don't take up expensive NetApp space, and the compile proceeds at local speeds. This mechanism speeds up compiles by a factor of 2-3.</div><div><br />
</div><div>The core of this is a new filesystem layer that I wrote, lcachefs. This is a type of loopback filesystem, similar to nullfs or unionfs. lcachefs does some magic mirroring involving three directories. As with any filesystem, one of these is the mountpoint: where lcachefs shows its results. Unlike most filesystems, though, it involves two sources: one is known as the source directory (which holds the NFS-mounted sources), and the other is a storage directory (local disk).</div><div><br />
</div><div>Ok, let's pretend that you first copy your sources from a source directory to the storage directory. Then you <a href="http://www.freebsd.org/cgi/man.cgi?query=mount_nullfs">loopback-mount</a> the storage directory over your original source directory. Then, you compile there. (You mount it over your original source directory so that the filenames in the debugging information point to the original sources, instead of to a temporary storage directory.) This looks, to the compiler, just like a regular compile. However, instead of doing the compile on NFS, you're doing it on local disk.</div><div><br />
</div><div>This has two advantages. The one that's easy to explain is that you're storing your objects and products on local disk (which is <a href="http://www.pricewatch.com/hard_removable_drives/">cheap</a>), instead of on NetApps (which, despite all their greatness, are <a href="http://www.google.com/products?q=netapp%20nas&brand=netapp">expensive</a>). Objects can easily be recreated, and take up most of the space in a build tree.</div><div><br />
</div><div>The other part is that you're working entirely on local disk. This is much, MUCH faster than using NFS. It's not because of the bulk data transfer (i.e., reads and writes), so higher bandwidth doesn't help. The problem is with the per-request overhead. There are a LOT of requests going across the wire in a compile.</div><div><br />
</div><div>Let's consider what happens when you build a file. Specifically, we'll talk about if you're just building hello.o. Make will look for Makefile, makefile, BSDmakefile, and .depend. Then it will look at hello.c and hello.o. (If it needs to, it may look for hello.s, hello.S, hello.f, and whatever other potential sources you may have.) It'll look up hello.c.gch (if you have <a href="http://gcc.gnu.org/onlinedocs/gcc/Precompiled-Headers.html">precompiled header support</a>), hello.gcda (if you have <a href="http://gcc.gnu.org/onlinedocs/gcc/Gcov-Data-Files.html">coverage support</a>). Then it has to look for stdio.h. Well, let's suppose that you have two directories in your -I path; let's call them foo and bar. Now it needs to check for foo/stdio.h.gch, foo/stdio.h, bar/stdio.h.gch, and bar/stdio.h, all of which are in your source repository (hence usually on NFS) before it can go on to look at /usr/include/stdio.h. Well, the first thing that stdio.h does (on FreeBSD; your OS may vary) is to include sys/cdefs.h. You got it... another four lookups in the source repository to find sys/cdefs.h! Repeat for sys/_null.h, sys/_types.h, and machine/_types.h. In the end, just to compile hello.o, you've done 39 file lookups into your source directory. Of those lookups, 21 are for .h files... and hello.c only includes one file, stdio.h! The average .c file (using the FreeBSD source base as a sample) includes 9 files directly.</div><div><br />
</div><div>That's a lot of work. These lookups are all blocking I/O, so it's one round trip each; from watching network traces, I'm actually very impressed with how fast NetApps can respond, but it's still a non-zero time. Moreover, though, you've got to look at the CPU usage. NFS uses RPC, and the overhead isn't cheap. In my testing, using NFS to compile something requires four times as much kernel processing-- which works out to twice as much CPU overall-- as using local disk! (That's all in the kernel, too, and that means that preemption isn't as easy, so scheduling takes a little hit... and don't forget the new network interrupts!)</div><div><br />
</div><div>Finally, NFS is designed around concurrent use. You can't assume that the contents of an NFS-mounted directory will be the same now as they were 90 seconds ago. While local disk can use metadata caches very, very effectively (and FreeBSD's filesystems do), when you're using NFS, you have to expire caches a lot. (There are also some things that FreeBSD could do better when it comes to its NFS client, but I fixed those and only got a 5% boost.)</div><div><br />
</div><div>Ok, now you should be convinced that the speed hit for NFS is bad, and that building on local disk is much, much, faster. Let's go back to talking about lcachefs.</div><div><br />
</div><div>Earlier, I asked you to think about copying your sources to local disk, and doing the compile there. That's exactly what lcachefs does, but with a twist: it does this lazily, by which I mean it won't copy files until they're needed.</div><div><br />
</div><div>When you first mount an lcachefs directory, it scans just the top-level of your source directory. It then creates stub files for each entry in the local storage. This is just an empty file with a special uid/gid pair that marks it as a stub. If you ls -l that file, then the filesystem will report the correct uid/gid, size, link count, etc, but in reality the locally-stored file is empty.</div><div><br />
</div><div>When you actually try to read (or otherwise use) a file or directory that's a stub, then it will copy it over to local storage before the system call returns to userland. In the case of a file, this means it copies the contents and sets the uid/gid to its real, non-stub values. In the case of a directory, it scans the directory and populates the stubs just like I described in the previous paragraph.</div><div><br />
</div><div>That's the basic overview. The reality is quite complicated because of hard links, the need to watch out for vnode lock order reversal, and other icky stuff like that. You can see all the gory details, though, once we open-source lcachefs. I'm currently awaiting clearance from legal, but I hope to either contribute it to FreeBSD, or release it as an independent open-source project.</div></div>Piquanhttp://www.blogger.com/profile/06983189170367989592noreply@blogger.com1tag:blogger.com,1999:blog-13322352.post-47443394000492611012008-12-07T15:59:00.001-08:002008-12-07T16:11:38.305-08:00Good FriendsI was thinking that I should make a post today, and was wondering what. I was trying to think of cheery topics, and looked at my blog to get ideas. Then the Google ads were one for losing fat, and one for help with alcohol & depression. Boy, how cheery can you get?<div><br /></div><div>Actually, quite so. In the last several weeks, I've gotten in touch with a <span class="Apple-style-span" style="font-style: italic; ">lot</span> of friends who are very dear to me. People I haven't heard from in years and years. Some of them contacted me, I contacted some of them. Most of this has been over Facebook or MySpace, one was on this blog which led to some emailing (and there'll be more of that, I promise!), one I got to visit in person while I was in Texas (rock on!), others I've only spoken with on the phone. But it really seems like the stars have aligned for reopening old friendships!</div><div><br /></div><div>Really, think of everybody who's ever been your life who has their own big mansion in your heart. Then imagine getting to visit with each of them, over the span of a couple of weeks. It's been an incredibly special time for me lately.</div><div><br /></div><div>I'm gonna be grinning like a fool for a long time now.</div>Piquanhttp://www.blogger.com/profile/06983189170367989592noreply@blogger.com0tag:blogger.com,1999:blog-13322352.post-38630835044218555182008-12-04T01:53:00.001-08:002010-05-17T17:40:03.507-07:00Not what I'm doing at JuniperI don't often make technical blog posts (ok, lately I don't often make posts), because two of the three people who read my blog don't really care. But sometimes it's nice to be able to brag about what I'm doing at work. Besides, last post I promised to talk about what I'm doing.<br />
<div><br />
</div><div>Since then, I haven't made any posts, because I didn't have the time to say what I was doing at work. Which is stupid, really. You three people don't read my blog because it's got juicy technical details; that's for the <a href="http://stopwords.piquan.org/2007/08/ive-been-found.html">rest of the masses</a>, and they don't care what I might have said I was going to post about until I post it. I've had other things to write about, but since I said I'd write about work, I didn't want to write about anything else. I several times thought, "Ooh, I should blog about this!" and then thought, "No, I said my next post would be about my work at Juniper."</div><div><br />
</div><div>Well, I am hereby renouncing that commitment. What's more, I am advising you three (tonight's loyal fans) to take anything I might tease about as merely speculation on future topics, and not be sure I won't write about something completely different first.</div><div><br />
</div><div>So there. I've just freed myself to write about other stuff. I don't really know why I felt it necessary to maintain an obligation that nobody cares about; why would anybody care if I put some posts in between the previous post (when I said I'd talk about my Juniper work) and the future post when I actually do, is totally beyond me. I just did feel that it's necessary, so now I can go on with more regular posts.</div><div><br />
</div><div>Except it's late and I'm going to bed instead of subjecting you to more of my incredibly convoluted late-night grammar.</div>Piquanhttp://www.blogger.com/profile/06983189170367989592noreply@blogger.com1tag:blogger.com,1999:blog-13322352.post-38274310262205934312008-11-17T17:18:00.000-08:002008-11-17T17:31:01.596-08:00Gasp! It's a post!So here's the thing.<div><br /></div><div>I was miserable at Google. It's a great company, and I think it's a wonderful place to work. But the position I was in just wasn't right for me. I didn't want to blog at the time, because I didn't want people to get the wrong impression about Google. Seriously, it's a great company; I just wasn't where I should have been.</div><div><br /></div><div>So at the end of July, I left Google and went back to Juniper. I was happy at Juniper, and wanted to be happy again. Besides that, Juniper offered me a promotion, a raise, my choice of projects, and the opportunity to work under a manager I know I work well with. I'll talk about the work I'm doing for Juniper in my next blog post.</div><div><br /></div><div>After I went back to Juniper, I've been concentrating on getting into the swing of things, so I still haven't been blogging. Now I'm doing well, and can start posting again.</div><div><br /></div><div>Now, here's one last note to the blogosphere: you may note that I'm still using a Google's blogger.com, storing the photos using Google's Picasa Web, and displaying Google AdSense ads. If I were upset with the company, I wouldn't be doing these things. I'm strenuously emphasizing that I like the company, because things get taken the wrong way in the blogosphere, and I don't want anybody to get the wrong idea. I'm also free to badmouth the company if I wanted to, since they're no longer my employer and have no control over my actions. But I'm not; I'm talking about how great they are.</div><div><br /></div><div>Anyway, my point is, hello world, it's nice to see you again!</div>Piquanhttp://www.blogger.com/profile/06983189170367989592noreply@blogger.com1tag:blogger.com,1999:blog-13322352.post-56274116563309488322007-09-04T21:58:00.000-07:002007-09-04T22:28:32.936-07:00An Oakland ShadeWow, that was a whiny post last weekend! Fortunately, this weekend was much better... after all, Labor Day, right?<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://picasaweb.google.com/Piquan/OaklandQuickshade/photo#5106581045665193138"><img style="margin: 0pt 10px 10px 0pt; float: left; cursor: pointer; width: 200px;" src="http://lh3.google.com/Piquan/Rt44Z7ud3LI/AAAAAAAAAr0/dmjADqqePgc/s200/IMG_0241.JPG" alt="" border="0" /></a>My friend Jenny threw a party this weekend, a holiday barbecue. She had a problem: there's been a heat wave lately, and her house was going to be positively sweltering. She came to me, a friendly neighborhood engineer, to build a shade.<br /><br />The party was Sunday, so on Saturday I went over there, scoped out the project, took some measurements, and made a list of supplies. Then, off we went to the hardware store to get supplies, which naturally deviated considerably from my list. (Anybody who likes going to hardware stores understands what I mean. Computer stores too.) Three canvas sheets, some rope, twine, dowels, and various fasteners. For once, I needed exactly what I got! (Well, I had a small pack of garden stakes left over.) A couple of supports, stitching together the canvas, some guy lines, and the job was done.<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://picasaweb.google.com/Piquan/OaklandQuickshade/photo#5106580809441991778"><img style="margin: 0pt 10px 10px 0pt; float: right; cursor: pointer; width: 200px;" src="http://lh4.google.com/Piquan/Rt44MLud3GI/AAAAAAAAArM/mYbieA1ETvU/s200/IMG_0236.JPG" alt="" border="0" /></a>It's not the best shade in the world, but for an afternoon job, I liked it. I also had a lot of fun building it; I may have had more fun building the shade than going to the party! It reminds me of something I used to do when I was a kid called Christmas in April. A lot of people would get together and rebuild the house of somebody who really needed it, and couldn't afford it; usually, this was an elderly or severely disabled person on a very minimal income. We'd have people who knew what they were doing well enough to lead our efforts, plus <span style="font-style: italic;">lots</span> of hands helping get the work done.<br /><br />I like building stuff; I think I get that from my Dad. I think I was first attracted to computers because I could make my creations come to life, from my mind to the computer. Even so, there's something satisfying about building something in the physical world.Piquanhttp://www.blogger.com/profile/06983189170367989592noreply@blogger.com1tag:blogger.com,1999:blog-13322352.post-40856834299185129312007-08-27T00:00:00.000-07:002010-05-17T17:43:40.664-07:00Why This Post Isn't About The Javascript DOMMy original title for this post was "Why The DOM Is The Best Lousy Thing We've Got". I guess I'm not feeling terribly creative with my titles tonight. To be honest, I'm not feeling terribly creative at all tonight. It was a terrific week, but a pretty lousy weekend. I had some unexpected work take up all my time, and it put me in a sour mood for most of the weekend.<br />
<br />
It's strange how creativity is so dependent on mood. I'm not just talking about blogging, here; I'm talking about coding, too. When I'm doing well, I can code anything, start to finish, piece of cake. When I'm doing poorly, then even writing hello.c is a huge effort. The work I talked about earlier involved some coding at the end of it— nothing big, just a few dozen lines to shove some data around— but if it weren't for that, I probably would have had a <span style="font-style: italic;">really</span> lousy weekend, instead of the merely lousy one I had.<br />
<br />
Ok, now I'm just sounding needlessly whiny. If you start seeing emo bands show up on my weekly top ten chart on the sidebar, send help. Like a box of kittens or something.<br />
<br />
Anyway, where was I. Oh, that's right, having a lousy weekend because of work. I really started thinking that maybe I made a mistake taking this job; maybe I was more cut out for the Juniper job than the Google one. Now, here's the irony. When I left Friday, I was writing a document that includes advice to people who feel that way... pretty much saying, "No, it's normal, everybody feels that way, you'll do fine". (This is advice I had <a href="http://stopwords.piquan.org/2007/07/noogler-day-18-geekiest-social-scene.html">previously given Cheryl</a>, and also <a href="http://stopwords.piquan.org/2007/07/noogler-day-19-tgif.html">talked about more in my blog</a>.) And then this weekend I get hit with this wave. Wasn't expecting it.<br />
<br />
<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://picasaweb.google.com/Piquan/ThorAndMe/photo#5103277610224180290"><img style="margin: 0pt 10px 10px 0pt; float: right; cursor: pointer; width: 200px;" src="http://lh6.google.com/Piquan/RtJ78rud3EI/AAAAAAAAApA/-3gzznkyRmU/s200/desk.jpg" alt="" border="0" /></a>I did get some useful stuff done for myself this weekend, though. The office is in much better shape, with the bookshelf moved, lighting hung, and desk accessories connected. Even the living room looks cleaner for some reason. At the end of it, I wanted to relax by writing some code. I was going to learn some SVG. Still, as I sat down to work on it, nothing came to mind. I hate hacker's block.<br />
<br />
If you haven't read Jhonen Vasquez's "I Feel Sick", it's about an artist who is suffering from artistic withdrawl once she takes a job at a big company. I hadn't read it, and didn't know what it was about, but picked it up tonight to read and forget about my bad mood. (Hint: the "Alice" soundtrack works well with Vasquez.) Y'know, that just didn't help me forget as well as I had expected.<br />
<br />
Finally, I decided to write about a blog post I've been wanting to write for a while... I even alluded to it in <a href="http://stopwords.piquan.org/2007/08/restraint_13.html">a post from two weeks ago</a> (when I refer to the "later, more technically-oriented post"). And that's what I've sat down to do here. Hmmm... I seem to have done a great job, haven't I?Piquanhttp://www.blogger.com/profile/06983189170367989592noreply@blogger.com1tag:blogger.com,1999:blog-13322352.post-66568609076455189782007-08-17T19:01:00.000-07:002007-08-17T19:08:31.969-07:00MovingWe moved from upstairs to downstairs in the same building. I like the new crib, although now I'm back in a cube. Still, it's the best cube I've worked in, and there's plenty of opportunity for cube dressing. I'm going to go by the surplus store, hardware store, and toy store to get some supplies for a few ideas I have, and cube decorations from past jobs (which are still in the box I took home from Juniper) will also play a heavy part.<br /><br />There was some discussion about airspace issues. There's 18' ceilings in this area, so the question of how to handle the airspace becomes relevant. Naturally, the discussion mostly involved us playing with Karl's RC helicopters, and trajectory arcs of Nerf guns.<br /><br />Anybody have cube decoration ideas they want to throw into the ring?Piquanhttp://www.blogger.com/profile/06983189170367989592noreply@blogger.com0tag:blogger.com,1999:blog-13322352.post-58157166173685379562007-08-17T16:10:00.000-07:002007-08-17T16:27:46.920-07:00SRE R00lz!<a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://picasaweb.google.com/Piquan/Google/photo#5099788593541274674"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer; width: 400px;" src="http://lh4.google.com/Piquan/RsYWtLud3DI/AAAAAAAAAog/dsavxW8R4qg/s288/IMG_0173.jpg" alt="" border="0" /></a><br /><p>This was taken at a company picnic yesterday. We had a tropical theme going... steel drums, flower leis, etc. There was a climbing wall, human foosball, and, of course, volleyball. Since my knee has been giving me problems (I actually had to leave early to go to physical therapy), I stayed away from the volleyball pits and instead played frisbee.</p>Piquanhttp://www.blogger.com/profile/06983189170367989592noreply@blogger.com0tag:blogger.com,1999:blog-13322352.post-4257555364605641052007-08-15T16:02:00.000-07:002007-08-15T16:08:54.891-07:00Yesterday's BeansOkay, yesterday's post was evidently more tantalizing than I meant for it to be. I started by saying that I wasn't talking about my new friend. Well, that seems to have been raising questions among my other friends, so I've been getting bugged to talk about it.<br /><br />Well, not from all of my friends. Some of my friends were there, some of them are too polite to pry, some are so busy that we haven't talked since then, but this is not an exclusive partition of my friends. So <a href="http://matthewfletcher.com/blog/">m3tus</a>, this post is for you.<br /><br />The thing is, it's not like there's anything scandalous that would make Matt Drudge drool. I just talked to my new friend about some problems she's been going through. I thought those problems might make for an interesting post, but her problems are hers to tell the world, not mine. That's all.Piquanhttp://www.blogger.com/profile/06983189170367989592noreply@blogger.com0tag:blogger.com,1999:blog-13322352.post-60343812483747992292007-08-13T21:38:00.000-07:002007-08-14T15:58:38.889-07:00RestraintAll week now, I've been mulling over a blog post. It's about a new friend I made last weekend, and the interesting circumstances under which we met. But there's a problem: that would be a violation of her privacy. I spent a fair bit of the week thinking about this. For a while, I was considering writing a version that left out some parts. Finally, I decided that I should let her tell her own story to the people she wants to; it's not my place to do so.<br /><br />This is the sort of thing I have to think about when I'm writing to a public audience. (Okay, that public audience is-- in the last week-- only 56, but ethics is not a numbers game.) I have to do this all the time when I write about work. Obviously, this is partly because I don't want to end up like <a href="http://www.blogherald.com/2005/02/17/is-mark-jen-the-blogospheres-dumbest-blogger/">Mark Jen</a> or something. But moreover, it's because I have a responsibility to my employer, and don't want to disclose information when it might do harm.<br /><br />This is a sort of self-restraint, which is a good thing. It suppose that in this context, it's similar to the legal concept of <a href="http://en.wikipedia.org/wiki/Prior_restraint">prior restraint</a>, which is generally considered a bad thing under most circumstance. However, there's a big difference. Here, I'm avoiding talking about things on my own volition; in the case of prior restraint, it's from a governmental gag order.<br /><br />This weekend was my friend's birthday. His girlfriend and I had planned out a birthday party for him, culminating with us seeing a performance of The Compleat Wks Of Wm Shakespeare (Abridged). But it started off with us playing Guitar Hero Encore: Rocks the 80s. I've got Guitar Hero and Guitar Hero II, and we have a blast playing them multiplayer. Unfortunately, we realized when we started playing that most tracks in multiplayer aren't available until you unlock them in single player mode. That meant that, for the time being, we only had eight songs available. Luckily, by the time we finished those eight, it was time for us to head up to the city.<br /><br />Now, that all happened Saturday, so Sunday I had to myself. I took care of some personal errands, like laundry and groceries, but also played Guitar Hero for a while... y'know, just to unlock the songs. I normally play GH on Expert, but this time, I played on Hard. That way, my friend and I would be able to play the Expert level for the first time together. (I did unlock all the songs, by the way. If you want the track list, and have iTunes, I put together <a href="http://phobos.apple.com/WebObjects/MZStore.woa/wa/viewIMix?id=262018582">an iMix</a> with the songs.)<br /><br />I thought about some of the stuff I had going on at work. There's plenty that I'd enjoy doing, but I was keeping myself from doing work. I was bound and determined to make the weekend about myself, and NOT the shiny new job that's been occupying me for the last few weeks.<br /><br />I got pretty bored playing Guitar Hero on Hard, but wanted to unlock the songs. So I kept at it. I played on Hard, and not on Expert. I unlocked songs, instead of doing some interesting work.<br /><br />Why the restraint there? I have to wonder if at some times, I'm holding back where I shouldn't.<br /><br />I feel like that at work too. I keep feeling like I'm marking time, slowly learning, waiting until I get to the point where I get to take off the kid gloves and really get to it. Thursday and Friday, I sat down and did some coding. I started on a program to help me maintain my calendar, and another one to give me a cheat sheet for a weekly meeting. The former is pretty mundane, but the latter was certainly fun. I'll talk about why in a later, more technically-oriented post.<br /><br />For now, I'll have to think about when's the right time to hold back, and when's the right time to let loose.Piquanhttp://www.blogger.com/profile/06983189170367989592noreply@blogger.com0tag:blogger.com,1999:blog-13322352.post-80789931365229202672007-08-03T21:51:00.000-07:002010-05-17T17:58:20.495-07:00I've been found!<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh13Gsepv2JOlbQ97Lf1w2dOakZBCaeUZt-hdKCwyzNq0Ss3PMWMNDRBh5jR2iTr4vRq3tquOzdQxYtulpGk8ot41qtSRXnBUOYK_aZlYkX7ymDwQNTB61ixThMDBfiaFZlUcUW/s1600/FirefoxScrShot001.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="172" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh13Gsepv2JOlbQ97Lf1w2dOakZBCaeUZt-hdKCwyzNq0Ss3PMWMNDRBh5jR2iTr4vRq3tquOzdQxYtulpGk8ot41qtSRXnBUOYK_aZlYkX7ymDwQNTB61ixThMDBfiaFZlUcUW/s320/FirefoxScrShot001.png" width="320" /></a></div>Okay, this is a surprise. I've been writing this under the idea that I'd have some small portion of my friends and family reading. And maybe the occasional Google search. I figured that's where the <a href="http://www.piquan.org/blog/2007/07/noogler-day-11-work-and-play-at-google.html">one comment I've had</a> came from. Then, today, I did a little log analysis.<br />
<br />
Since I opened the site a couple of weeks ago, I've had 2,490 visits from all over the world, from Malaysia to Tanzania. The visitors are pretty telling of the type of audience: 49% have been using Windows, 29% using Macs, and 21% using Linux. The browser numbers are more optimistic: 69% Firefox, 12% Safari, and only 7.5% using Internet Explorer.<br />
<br />
I will point out that these numbers may well be skewed. The statistics were only collected for about 2/3 or less of the page views. In other words, I know that there's been <span style="font-style: italic;">at least</span> 6,000 views, but only about 4,000 of those had statistics logged.<br />
<br />
How did all this come about? Well, somebody found my <a href="http://www.piquan.org/blog/2007/07/noogler-day-11-work-and-play-at-google.html">post about Sawzall</a>, and posted it to <a href="http://reddit.com/help/faq">reddit</a>. There was a little bit of activity before that, but I haven't yet looked at what led them to find it.<br />
<br />
Then again, I never did publicize <a href="http://vigor.sourceforge.net/">Vigor</a> at all (I just sent Illiad an email), and enough people downloaded it that I was getting dozens of emails a day. I suppose even the most trivial writings are still going to be found by the zillions of eyes on the Internet.Piquanhttp://www.blogger.com/profile/06983189170367989592noreply@blogger.com1tag:blogger.com,1999:blog-13322352.post-87480242871553256242007-08-03T20:47:00.000-07:002010-05-17T18:01:23.822-07:00Carving out timeYou haven't heard from me all week, and you're starting to wonder if I vanished into thin air, lost my password, or what. Well, I've just been busy.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiOy8nBOnJEaVjYbBqQn0cAScuD14cy4VxJZ_occfvy3fv1i-lxEX-_FmkbATm5FgKMJGP39JtbfurNnkjuBfk7W6ZTMgZmoKsNTiUSS60GKHaVkOrwIJmxg0ADVgVq67E7XDvv/s1600/iCalScrShot001.png" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="371" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiOy8nBOnJEaVjYbBqQn0cAScuD14cy4VxJZ_occfvy3fv1i-lxEX-_FmkbATm5FgKMJGP39JtbfurNnkjuBfk7W6ZTMgZmoKsNTiUSS60GKHaVkOrwIJmxg0ADVgVq67E7XDvv/s400/iCalScrShot001.png" width="400" /></a></div>All last week, I was wondering why I was having such a hard time getting anything done. I had so much stuff that I wanted to get done, but never seemed to be making any traction on it. Then I looked at my calendar, and it all became clear.<br />
<br />
Okay, admittedly, that screen shot is from two weeks ago, when I still had a full schedule of training. But nevertheless, I've been going to a lot of meetings. Some of them are necessary, some aren't. I realized that I'd have to defend my schedule against meetings.<br />
<br />
That's what I've been doing the past week: I was declining meetings that I didn't need to go to, and making time to work instead. This involved studying, coding, the usual stuff. This week, I was getting stuff done— but I was staying there until 11:30 PM at the earliest, 1:30 AM at the latest. I wasn't getting <span style="font-style: italic;">nearly</span> enough sleep. On the upside, spending all this time in the office gave me a chance to learn about the laundry facilities at work, but overall, I think it was more of a bad thing.<br />
<br />
Next week, I'm going to be defending my personal schedule against my work schedule. I need to make sure that I give myself time to relax, to take care of personal errands, to visit with friends, sleep, and do all of the rest of the things I have to do that don't involve work.<br />
<br />
You may have noticed that my "Recent News I Found" sidebar has had a fairly high number of life organization articles. (Remember that it's a separate feed from my blog. To subscribe to that, click the "Read More..." in the sidebar, and subscribe to the feed there.) I suppose that, as I'm finding my feet in my new job, I've been thinking a lot about getting my life better organized.Piquanhttp://www.blogger.com/profile/06983189170367989592noreply@blogger.com0