Noogler, Day 17: The Light Dawns

This is going to be a short one, because I'm not getting started until pretty late.

I previously talked about the fact that I don't have a lot of experience with some programming architectures. One of these is something called cluster computing. This is when you have a lot of computers together all working on jobs.

Now, as you know, Google has a lot of data to process. We use cluster computing to deal with that. That means that there's a cluster infrastructure managing things. If you type a query into Google, the job of searching the web for what you typed (and correcting spelling, and looking for ads) gets spread out among many, many computers.

In a cluster system, there's some sort of cluster management infrastructure between the program and the machine. How thick this layer is depends a lot on your system. In an nCube system (which is more akin to a supercomputer than a cluster, but there's similarities), it's pretty obvious to the program that it's part of a cluster. In a Beowulf cluster, the entire cluster acts more like one computer than many. In a Cell cluster, the reality is somewhere in between.

Despite this range, in any cluster system, there's a layer between the program and the hardware. Now, as a programmer, I have to deal with abstractions all the time. It's a big part of my job. Nevertheless, on a typical system, I can talk about the computer at all levels. At the top level might be a user deciding to click on a hyperlink. What makes him want to do this? Why does he know what effect this will have? How did he decide that this was a useful action to take? All these are questions of user interface theory, which is an area that has gotten very little research until recently. At the lowest level, we have transistors. These are taking advantage of field effects to allow electrons to flow from one point to another, depending on the behavior of electrons at a third point. This is the fundamental component on which computers are generally built.

I've got a pretty good understanding of the layers between those two points. It's a bit sketchy near the extremes— my subatomic physics isn't really up to snuff— but in the middle, particularly in the lower level of the software, I've got it covered.

Now, a cluster system puts an entire abstraction layer between the software and the hardware. Instead of saying that program X is running on machine Y, you would say that program X is running on this cluster over there, that consists of a lot of machines. Depending on the cluster system, it might be running evenly spread across each machine, or parts of it are on some machine and parts on another, etc.

This has made me a bit uneasy. I've been having to learn about a lot of programs running, but I haven't really had a good feel for HOW they run. When we were talking about different programs, or subprograms communicating, I couldn't tell how they were communicating... whether it involved sharing some RAM, sending data over a network, or somewhere in between. All of these are possibilities in a cluster architecture, and I didn't know anything about ours.

Today, we went over the cluster architecture in general, so I'm quite happy about that. Now I have a basic groundwork for the computing infrastructure, and yesterday I got the basic groundwork about the data flow I'm supporting. Tomorrow, I've got training on a number of other job-specific aspects, such as what's expected when you're on call, and more detail about our specific systems and how they work.

I've also been talking to some people around here about some ideas I have. One involves using some AI techniques to make our lives easier. These are ideas I'd been thinking about at Juniper, but wasn't able to get the buy-in to start the project. Here, people seem enthusiastic about the idea, so I think I'm going to start looking in that direction pretty soon.

I've also got some ideas about ways to organize the communication between parts of our computers, but I'm less likely to be able to see that out; it may be too radical to be implemented with today's state of the art, even here. Nevertheless, I'm going to throw the idea around some and see if it sparks some good, more practical ideas.

As a brief note on another topic, I've been playing with a lot of the Google products I hadn't looked at closely before. I've got a couple of ads accounts now to play with, for example; that only makes sense, since I'm in the ads team! I'm also exploring some of the other properties, such as Google Reader, Co-Op, etc. Expect to hear more about that later, but for now, it's time for bed.

Oh, I almost forgot. I got my ergonomic evaluation today. They're backordered on the Kinesis keyboards, but I'm going to go by Fry's at some point and get one for home so I can learn the layout while I wait. Anybody care to play two-player typing speed games while I'm learning it?

One final note: in the picture, you can't tell clearly, but that balloon between the desk and the door is maintaining its position unsupported. It doesn't have much buoyancy, but it's still got a little.