Get some code

I spent a made two months really pouring lots of time and energy into code to produce the visualizaiton I have shared in this blog. In the process I learned a bunch about Ruby, KML, and Census data - and had great fun.

That was almost two years ago and I've consistently made promises to myself to get back on the wagon and keep working on the project. However, these just have not materialized.

Because of this, I've decided to open up the code, in its raw state, for public consumption. It is full of bad programming practices, from too many files to hard coded strings it would get an F in almost any code review. This prevented me from sharing it for a long time, but as the months kept ticking by and I didn't make any progress, I realized that either I was going to get it out there or it would simply die as a small blog.

Over the coming posts I'll share the structure in lots more detail, but to start, I wanted to share what the most important files are.

Let me give you a small tour around the code:

• GenPlacesKml.rb : This is the main file. This is what you "run". I've created many different "main" routines to create the outputs. This file also contains all of the code to create the actual KML document. I never created an object model for the KML components, I simply created functions that generated the components as rexml objects. This then gets dumped into the KML output file.
• Polygon.rb : This contains the definition of the Polygon object, which is the core object that holds the Census shape date. This also contains the routine to load Census shape data from a file.
• BlockGroups.rb, County.rb, etc. : These contain the objects to hold the Census meta data for each of the different types of geographies the Census creates. It also contains the method to load them.
• Data.rb : This is a generic data loading set of functions.
• Colors.rb : This generates the color palettes used in the files
• FIPS.rb : This contains some data definitions to do lookups on Census abbreviations.

If you want to start playing around with the code, the easiest "main" routine to look at is "genWABlockGroups()".

Now, where is the code? I've decided to use Google Code to share it & I've put it out under the GPL. This uses Subversion, which I am completely unfamiliar with, to store and share the code. It does seem like a powerful system, but I'm just learning how to use it appropriately. I am very open to any and all feedback on its proper setup and use.

I'm hoping the community is interested in breathing life back into the code. In the coming posts I'll walk through the structure (to the extent that it exists).

You can find the source at http://code.google.com/p/rubykml/

Federal Election Commission Campaign Contribution Data

I am stretching my wings beyond Census data - and what could be more interesting then campaign contribution data? I have an interest in politics and thought it would be fascinating to see if there were ways to make the data about the amount spent on elections available. Using the same basic framework I created to read in Census data, I decided that the Federal Election Commission (FEC) would be a good source of interesting data about elections.

The Census provides boundary files for the last eight Congresses. The FEC provides a wealth of data and I have just started to explore the full extent of what is possible. I started by using the Candidate Financial Summary Without PAC Breakdown data. This data, of course, is in a rather complex format, so writing a Ruby module to read it out was the first order of business.

The Census boundary files for the 110th and 109th are in a relatively similar format (the 108th and before begin to deviate substantially) so I used these as my polygon files. I used the Candidate Financial Summary Without PAC Breakdown (CFS) files from the FEC. These files are the most current but do have some potential accounting issues. As the FEC states:

The cost of this timelines, though, is that some of the information available here is less precise than for "cansum". For example, in "cansum" you can see how much a campaign received from Corporate PACs or Labor PACs, while here there is only one value for the total received from "other political committees." This includes all PAC contributions, but it may also contain contributions from other candidates, and some other types of committees we don't typically think of as PACs. We can't do the full breakdowns until all the information about specific contributions has been entered into the database.

When using these summary files you need to be aware of some possible double counting of activity. Some candidates have more then one committee authorized to raise and spend funds on their behalf. The activity reflected in this file represents the sum of those committees. If they transfer funds back and forth among each other, this activity would be counted twice. Information about "transfers from authorized committees" and "transfers to authorized committees" is included in the file and if there are values in both of these fields it is necessary to subtract these from total receipts and total disbursements to obtain a more accurate value for actual activity.

In creating a data structure to store the CFS data, I created a somewhat flexible way to aggregate the data using some of the interesting features of Ruby. In particular the eval statement make it quite easy to pass in free text to allow the caller of the aggregation function to specify which field to aggregate quite easily.

I encountered two main challenges mapping the CFS data back to the Census Congressional District polygons. First, the FEC uses the two-letter acronym to identify the state and the Census uses FIPS codes to identify states. Using regular expressions in TextMate, I converted the list from the Census website to a couple of different Ruby hash tables so that I could convert back and forth from two-letter acronym to two-digit code.

Second, the FEC files are somewhat inconsistent (at least based on how I am reading it) about how they handle states with only one Congressional District. The Census bureau is pretty clear that it uses "00" to identify districts that are the sole district for a given state. The FEC seems to follow this convention for the most part, except in a couple of states. For example, in Wyoming there are candidates for the House of Representatives listed in Congressional Districts "00" and "01". Wyoming has only one seat in the 110th congress. Here is a snippet from the webl06.zip file which contains the CFS data for the 2005-2006 election cycle (the ellipses represent where I have cut from the line for the sake of readability):

  H4WY00055CUBIN, BARBARA L                      I2REP...WY01 W W48...
  H6WY01025TRAUNER, GARY S                        1DEM...WY01 W L47...
  H6WY00118WINNEY, JUSTIN WILLIAM JR              2REP...WY00     0...


According to the documentation for the file, the two digits following the state acronym represent the district. In this case, it would appear to suggest that there are candidates for House in district "00" and "01", when in fact there is only one district in Wyoming. To handle this, I simply rolled up all of the House candidates, regardless of the district in the FEC file for one district states. This may be the wrong thing to do, so I've got an e-mail into the FEC to find out the actual answer.

Alright, enough with that background, let's see some pictures. In the following screenshots, I am mapping the total amount received by House of Representative candidates in each Congressional District for a given election cycle. For each $1,000 received, the district gets one meter in height. The districts with higher amounts are more green, those with lower amounts are more red. Missouri is missing from the 109th Congress because the polygon metadata file is missing from the Census website.

109th Congress from above:


110th Congress from above:


109th looking North:


110th looking North:


109th looking West:


110th looking West:


109th looking East:


110th looking East:


109th looking South:


110th looking South:


That is it for now. I'm close to sharing the KMZs for these, so be on the look out for a post soon.

"Dynamic" Labels & StyleMaps

I've been unhappy with how Google Earth handles labeling of shapes - the Placemark KML object is pretty good at containing a single shape, but breaks down when handling multiple shapes. Placemark elements contain the name and description elements, but only Point elements pick up on these. As I have talked about before, you can't label a Polygon without using a Point. I started to use the MultiGeometry to group together Polygon elements that belong to one geography (i.e. when a County has a couple of noncontiguous shapes). The annoying part is that I needed to add a Point for labels to show up.

When dealing with complex geographies, like Block Groups, having the labels show up the entire time doesn't work very well. For example in the last map I shared, there are ~4,300 Block Groups in WA and having the labels all show up doesn't work very well because they overlap and make it quite confusing. You can play with breaking out the labels into a different Placemark and perhaps a folder structure at the County/County Subdivision level might help, but it still isn't perfect because you would have to hunt and peck for what you were looking. In looking through the KML spec, I was excited to find the StyleMap element. The StyleMap element provides a mechanism to have a Placemark respond to mouseover/click/highlight events. You can define a Style element for both normal and highlight classes. I thought this would be a great way to provide a label: when a user moves their mouse over a geography (e.g. Block Group, County Subdivision, County) the label could show up - the normal style would have it transparent and the normal would have it opaque.

Well, turns out it doesn't quite work that way. Unfortunately, the only thing that sparks the transformation is the user moving their mouse over the icon of a Point: the style doesn't change if they mouseover the label or any of the Polygon elements in the Placemark. What is odd, however, is that all of the elements of the Placemark do respond to the new style when you mouseover the icon.

What you'll see below are maps of Median Household Value (variable H85 MEDIAN VALUE (DOLLARS) FOR ALL OWNER-OCCUPIED HOUSING UNITS [1] from Summary File 3) by County Subdivision. I'm not entirely happy with my new labeling, but it works such that when you mouseover the icon at the center of the polygon, the name of the County Subdivision shows up and the border is highlighted in white. I like the effect, but I am not happy with how I have to have an icon show up. In the maps below, the more red an area/shape, the higher the median household value - the greener, the lower the median household value. In the 3D maps, each $1,000 of value adds one meter of height.

Movie:


Screenshot (notice how Leavenworth-Lake Wenatchee is highlighted):

Blogger ate my post on new features/code improvements

Today's 500 word post was just eaten by the Blogger spell-checker. I'm going to attempt to retrieve it, but it isn't looking good. The topic of the post was some major improvements I've made to the code base that make it much more generic and support two new features:

• Exclusions are now handled - if a polygon has an exclusion listed in the Census shape file, this is now handled. This turns out to be quite important for some areas. I'm working on a map of Washington state at the Block Group level and it turns out that in the rural parts of the state there are a couple of small towns that have a single Block Group surrounding the town and then separate, smaller Block Groups for the town itself. Before, these smaller Block Groups might have been covered up.


• A geography made up of multiple polygons now acts as one object in the KMZ output file.

In addition, I've made a number of enhancements under the hood. While I had been using some object-orientation before, I finally made the classes much more complete and generic so that it is much easier to add new geographies. For example, I created the Polygon object which encapsulates all of the information for a given polygon in one container:

class Polygon
  attr_reader :id, :centerLon, :centerLat, :mainCoords, :exCoords
  attr_writer :id, :centerLon, :centerLat
  
  def initialize(id, centerLon, centerLat)
    @id = id
    @centerLon = centerLon
    @centerLat = centerLat
    @mainCoords = Array.new()
    @exCoords = Array.new()
  end
end


With these improvements, I hope to have some maps to share soon. When I am once again inspired, I will share a lot more detail about the process and recent improvements.

How hard can it be to generate KMZ files?

One of the items I have had on my to do list for the past few days is to change the output of my program to KMZs from KMLs. At high-levels of geographic complexity (such as Block Groups or County Subdivisions) the KML outputs can be very large (5 to 10 megabytes per state). KMZs are simply KMLs, but zipped. Everything I've tried to do in Ruby has turned out to be so easy, I figured this would be as well. I was wrong!

I started by trying to use the built in zlib module that appears to be a part of the standard Ruby install. I first tried the following bit of code:

outputFile = File.new(outputFilename,"w+")
zipWriter = GzipWriter.new(outputFile)
xml.write( zipWriter )
zipWriter.close

I was thinking to myself that this was just too easy, and it was. The output file came out as a KMZ file which Google Earth promptly refused to open. I kept getting some sort of bizarre error like "unexpected token at line 1, column 0". What got interesting is that if I just changed the file's extension to .zip and unzipped it using the built in Archive tool (on Mac OS X), the resulting KML file worked just fine - no problems!

Next, I tried writing the XML to a string which I then wrote to a zipped file, in case the streams weren't playing well together.

xmlString = StringIO.open("", "w+")
xml.write( xmlString, 0 )
zipWriter = GzipWriter.new(outputFile)
xmlString.rewind
zipWriter.write( xmlString.read )
xmlString.close
zipWriter.close

This code produced the exact same output and the exact same error in Google Earth.

Getting frustrated, I did some online searching. I found out that the KMZ spec says that the main KML in the KMZ should be named doc.xml. So, I tried writing out the KML file and then zipping that into another file:

xml.write( File.new("doc.kml","w+"), 0 )

GzipWriter.open("doc.kmz") do |gz|
  gz.orig_name = "doc.kml"
  gz.write(File.read("doc.kml"))
  gz.close
end

Same error! At each iteration I could change KMZ to ZIP, unarchive the file and open the resulting KML. I could even take a KML file, zip it, change the extension to .KMZ and open it up in Google Earth, regardless of the original filename (no "doc.kml" requirement seemed needed)! What was going on?

Next, I tried using the system gzip command, thinking maybe there was a problem with the Zlib module:

xml.write( File.new("doc.kml","w+"), 0 )
system("gzip", "doc.xml". "-S .kmz")

Again, same error!

At this point, I figured that all ZIP algorithms must not have been created equal, so I went searching for another ZIP Ruby module. I found one called rubyzip. I was encouraged by this new module because it had a lot more functionality than zlib and a lot better documentation. After giving up on using RubyGems to install the darned thing, I simple downloaded the code and installed it myself. A few sweet moments later, I held my breadth and ran the following code:

ZipFile.open( outputFilename + ".kmz", ZipFile::CREATE) {|zipfile|
  zipfile.get_output_stream("doc.kml") {|file| xml.write(file, 0)}
  }

It worked! I wish I had a better conclusion as to exactly what was going wrong with zlib, but the moral of the story is if you are trying to create KMZ files in Ruby, use rubyzip!

Finally got 3D colors working!

Thanks to the very first (and still only) comment left so far on CensusKML I have finally gotten good (well, maybe just bright) colors working when extruding the Census data out on Google Earth. I admit that I am terrible at picking color schemes, so please forgive the white-to-red one I'm using here. The trick was relatively simple: the coordinates must be listed in counter-clockwise order in order for Google Earth to properly draw the colors. I don't understand why this is the case, but it is. The Census boundary files list the coordinates for each polygon in clockwise fashion. Ruby made it very simple to reverse the coordinates. The coordinates are initially stored in an array in the order they are read from the Census files - when it comes time to read them out and put them into the KML file, all you have to do is call reverse! on the coordinate array and the array is reversed in place.

The following is a screen shot of the population data from the 2000 Census, broken out at the County Subdivision level. The height of each subdivision is equal to 1 meter for every 10 people. The colors also represent population, but use the state's max county subdivision population as the denominator, meaning they are only relevant within the state, not comparable across states. The largest subdivision will be bright red and the small ones will be white. I'm experimenting with this to try and get data on two levels - national and state. Among all of the data, note Boston on the upper-left, Florida on the upper-far-right in the distance, and Chicago on the bottom-right. Here you go:


You can compare the previous picture to this one to see how big a difference it makes to list the coordinates in a counter-clockwise fashion:

Ruby has a big standard library

One of the aspects of Ruby that I've been most impressed by is the size and sophistication of its standard libraries. There are complete, easy to use classes for almost all of the functionality I've needed to date - and importantly they have the functions one really needs to be productive built right in. A good example of this is code to loop through a directory and operate on certain types of files. After some fishing around the web, I found some bits and pieces of very simple code that does exactly what I need it to do. Now, I am not expert, so I would appreciate critiques and better solutions, as I know that I'm missing ways to do this better, but I've already found a pretty simple way to do this. You'll also probably note that I like long variable names.

The following code is looking through a directory, all the way down to the leafs (file in sub-directories) and returning files that end in "a.dat". This is the extension of the polygon metadata files from the Census website. It then creates the file name for the actual polygon files (they have the same name, without an "a" before the extension) and the metadata files. From here, it passes those into the functions which read, parse these files to generate the KML outpus, which I will share when they are more robust and mature.


Find.find("/Users/xxx/Desktop/Census 2000 CSDs") do |path|
  if path.match("a.dat")

# This returns the file name, minus the second parameter
    filename = File.basename(path, "a.dat")
    path = File.dirname(path)
    polyShapeFilename = path + "/" + filename + ".dat"
    polyDataFilename = path + "/" + filename + "a.dat"
    outputFilename = Dir.getwd + "/batch_outputs/" + filename + ".kml"

    puts "Starting #{filename}"

# I'll publish this method when it is more complete
    genOutput(...)

    puts "Done #{filename}"

  end
end


An example of productivity I am talking about is the Find class. This built in class recursively searches through directories - how helpful!

One of the most powerful concepts in Ruby that I am just getting comfortable with is the block, like Find.find("/Users/xxx/Desktop/Census 2000 CSDs") do |path| that lets you run loops over collections of things. In this case, the code is going to loop every time the find method returns a path. I'm sure I'm butchering the Ruby language, but if a novice can figure these parts out, then the language, and its authors, have done a number of things right.