Federal Election Commission Campaign Contribution Data

I am stretching my wings beyond Census data - and what could be more interesting then campaign contribution data? I have an interest in politics and thought it would be fascinating to see if there were ways to make the data about the amount spent on elections available. Using the same basic framework I created to read in Census data, I decided that the Federal Election Commission (FEC) would be a good source of interesting data about elections.

The Census provides boundary files for the last eight Congresses. The FEC provides a wealth of data and I have just started to explore the full extent of what is possible. I started by using the Candidate Financial Summary Without PAC Breakdown data. This data, of course, is in a rather complex format, so writing a Ruby module to read it out was the first order of business.

The Census boundary files for the 110th and 109th are in a relatively similar format (the 108th and before begin to deviate substantially) so I used these as my polygon files. I used the Candidate Financial Summary Without PAC Breakdown (CFS) files from the FEC. These files are the most current but do have some potential accounting issues. As the FEC states:

The cost of this timelines, though, is that some of the information available here is less precise than for "cansum". For example, in "cansum" you can see how much a campaign received from Corporate PACs or Labor PACs, while here there is only one value for the total received from "other political committees." This includes all PAC contributions, but it may also contain contributions from other candidates, and some other types of committees we don't typically think of as PACs. We can't do the full breakdowns until all the information about specific contributions has been entered into the database.

When using these summary files you need to be aware of some possible double counting of activity. Some candidates have more then one committee authorized to raise and spend funds on their behalf. The activity reflected in this file represents the sum of those committees. If they transfer funds back and forth among each other, this activity would be counted twice. Information about "transfers from authorized committees" and "transfers to authorized committees" is included in the file and if there are values in both of these fields it is necessary to subtract these from total receipts and total disbursements to obtain a more accurate value for actual activity.

In creating a data structure to store the CFS data, I created a somewhat flexible way to aggregate the data using some of the interesting features of Ruby. In particular the eval statement make it quite easy to pass in free text to allow the caller of the aggregation function to specify which field to aggregate quite easily.

I encountered two main challenges mapping the CFS data back to the Census Congressional District polygons. First, the FEC uses the two-letter acronym to identify the state and the Census uses FIPS codes to identify states. Using regular expressions in TextMate, I converted the list from the Census website to a couple of different Ruby hash tables so that I could convert back and forth from two-letter acronym to two-digit code.

Second, the FEC files are somewhat inconsistent (at least based on how I am reading it) about how they handle states with only one Congressional District. The Census bureau is pretty clear that it uses "00" to identify districts that are the sole district for a given state. The FEC seems to follow this convention for the most part, except in a couple of states. For example, in Wyoming there are candidates for the House of Representatives listed in Congressional Districts "00" and "01". Wyoming has only one seat in the 110th congress. Here is a snippet from the webl06.zip file which contains the CFS data for the 2005-2006 election cycle (the ellipses represent where I have cut from the line for the sake of readability):

  H4WY00055CUBIN, BARBARA L                      I2REP...WY01 W W48...
  H6WY01025TRAUNER, GARY S                        1DEM...WY01 W L47...
  H6WY00118WINNEY, JUSTIN WILLIAM JR              2REP...WY00     0...


According to the documentation for the file, the two digits following the state acronym represent the district. In this case, it would appear to suggest that there are candidates for House in district "00" and "01", when in fact there is only one district in Wyoming. To handle this, I simply rolled up all of the House candidates, regardless of the district in the FEC file for one district states. This may be the wrong thing to do, so I've got an e-mail into the FEC to find out the actual answer.

Alright, enough with that background, let's see some pictures. In the following screenshots, I am mapping the total amount received by House of Representative candidates in each Congressional District for a given election cycle. For each $1,000 received, the district gets one meter in height. The districts with higher amounts are more green, those with lower amounts are more red. Missouri is missing from the 109th Congress because the polygon metadata file is missing from the Census website.

109th Congress from above:


110th Congress from above:


109th looking North:


110th looking North:


109th looking West:


110th looking West:


109th looking East:


110th looking East:


109th looking South:


110th looking South:


That is it for now. I'm close to sharing the KMZs for these, so be on the look out for a post soon.

Median Household Value for Washington state by Block Group

Using the new functionality I've discussed in the last two posts, I am pushing forward in creating new maps. Today I'm going to share a few maps of median household value (variable H76 - MEDIAN VALUE (DOLLARS) FOR SPECIFIED OWNER-OCCUPIED HOUSING UNITS [1] from Summary File 3). There is another variable, H85, which might be better for what I want, but I'm going to go ahead and share these maps before going back. These new maps show the median household value by block group for Washington state. There are ~4,300 block groups in Washington and a wide array of values in the data, so this data provides a good test of the new functionality (labels, excluded polygons, logarithmic color scales). Given the large number of block groups, I've found that it takes several different map formats to fully explore the data. I've created maps that have the 3D views I've shared before and maps that are flat with some transparency so that you can see the underlying geography.

One of the challenges I found in creating these maps was gathering the Census data at the block group for the entire state. NHGIS doesn't provide many variables beyond the basic population and economic ones and the Census FactFinder website doesn't make it easy to download all of the block groups for a given state at once - you have to download each county separately. Given these challenges, I looked into download the raw data from Summary File 3. There raw files are available by FTP but, of course, are in a very complex format. The Census Bureau provides an Access database template that contains empty versions of each of the ~80 tables needed to work with the data (including import specs which is quite helpful). Feeling intrepid I downloaded all of the data for Washington state (FTP site) and loaded the tables I needed to into the Access template. This worked pretty well, but is somewhat confusing, particularly because joining the geographic identifiers for each record is not quite as straight forward as the Census documentation would lead you to believe. I finally got it to work and this provided the data for the maps provided below - perhaps you can now understand why I haven't gone back and re-run the maps using the H85 variable yet.

In the maps below, the more red an area/shape, the higher the median household value - the greener, the lower the median household value. In the 3D maps, each $1,000 of value adds one meter of height. You'll note that some areas show up as white, this is because the data provided by the Census for these block groups is 0. This makes sense for places like Mt. Rainier, but not for a region of downtown Seattle that shows up as white. I'm going to look into this next.

Now, some maps (don't forget you can click on each picture to get a larger version)!

Entire state (flat from overhead):

Entire state (flat from overhead, borders around each block group, some transparency):

Entire state (3D from overhead):

Entire State (3D looking North):

Entire State (3D looking East):

Seattle (3D looking Southeast):

Seattle (flat from overhead, borders around each block group, some transparency):

Hawaii Population Data & New Features

As promised, my break gave me a new burst of energy. I spent my week off in Hawaii and figured it would be a good region to experiment with a few new map features. I tackled two items that were on my to-do list: labels & logarithmic color scales. I have generated a new map of population by County Subdivsion for Hawaii to demonstrate these.

First, on labels. I've found that Google Earth is quite powerful except when it comes to labeling polygons. You can provide "names" for polygons, but these only appear in the "Places" panel on the left of the map view. I suppose this may have to do with the difficulty in figuring out where to put these names on the map display (given how oddly shaped a polygon can be), but I would have thought there was some good default behavior for this (if I am missing a feature of KML, please let me know!). To place a label on the map display, you have to create a "Point" placemark. Out of a desire to keep moving, I pushed forward without labels. The Census Bureau's shape files do actually include a center point for each polygon, so I have now gone back and updated my code to generate "Point" placemarks for each of these center points. With these points, I can now have labels appear on the map. This is very helpful, particularly when dealing with geographies below County (County Subdivision, Block Group, etc.).

Second, on logarithmic color scales. One of the challenges in mapping any sort of data that has a very wide range and is not very evenly distributed across the range is that in can be hard to find a color scheme that provides clarity at either extreme. I have talked about this in a couple of previous posts, but finally gotten around to implementing a logarithmic scheme that more evenly distributes the data across the range. I'm not entirely happy with what I've implemented, so I plan to work on it further.

On to some screenshots. Below you'll find 3 perspectives of population, by County Subdivision, from the 2000 Census for Hawaii. Every meter in height represents 5 people; Greener represents lower population, Red higher population. The labels come in handy because I'm not that familiar with the islands. The new logarithmic color scheme comes in handy because Honolulu has a much higher population that all of the other County Subdivisions. I've used the same Green to Red color scheme I've used before, but with the logarithmic scaling it now does a much better job of helping one to distinguish between the Subdivisions on the lower end of the population range. Without this new scheme, Honolulu would be red and everything else green.

From directly overhead:


From an angle, looking North:


From an angle, looking South (so you can see the northern side of Oahu):


One other note of interest: I was perplexed for a few minutes because the Midway Islands and the other islands west of Kauai all showed up with tall, red polygons - meaning they have a high population (I excluded them from the screenshots for this reason). It turns out that the Honolulu County Subdivision includes all of these islands, hence they get the data for the entire Subdivision. I suppose this demonstrates one of the perils of geographic aggregation when working with an island chain.

New KMZ!

Today I'm going to share a KMZ of a new variable that I've been working on. I wanted to experiment with more complicated variables, beyond Median Household Income, to really push the flexibility of the code I'm writing. The migration variables of summary file 3 from the Census are fascinating and fit the bill. They allow one to understand how people are moving about the country and include quite a bit of granularity. I'm going to share 7 states worth of data: Connecticut, Maine, Massachusetts, New Hampshire, New York, Rhode Island, and Vermont. Before I share links to the actual file, I want to be sure to share some important definitions, data sources, and notes.

What is mapped?

• There are many migration variables, but for this example I've chosen to use two. I used the Population 5 years and over: Total - P024001 variable which represents the number of people in 2000 over the age of 5 and used it to divide the Population 5 years and over: Different house in 1995 - P024003 variable. The resulting percent should represent the number of people, over the age of 5, who didn't live in the house they lived in during the 2000 Census in 1995. Put it simply, the percent of people who moved in the last 5 years. The migration data provides a much more detailed breakout of where the people that moved came from, which I hope to work further with.
  
• The data is presented broken out by county: the taller, the more blue a county is in the file the higher a percent that moved - the shorter, the more green, the smaller percent that moved. The actual percent can be found in the description for each polygon, however it is multiplied by 1,000 there so the actual value is what is found in the description divided by 1,000.
• In New England, the region in the provided KMZ, Tompkins County, New York has the max value at ~ 58%. This means that ~ 58% of people in Tompkins County moved since 1995. There are many counties at the low end: Hamilton County, New York is quite low at around ~30%, but so are Aroostook County, Maine and Orange County, Vermont, both around ~33%.
• You'll also note how the counties are organized into folders within the KMZ file. This is a recent improvement to the KMZ generation process. I've also modified the code to follow the best practice of referencing repeated styles by ID (each polygon references a base style and only overrides what it needs to) - I thought this would save a lot on file size, but it didn't because ZIP was quite efficient at compressing the bits that were repeated again and again.
  

How about a quick tour of the map?
[YouTube seems to like to cut down the length of videos, so this may feel a bit choppy - not sure why it is doing this]


Where is the data from?

• I downloaded the data from the FactFinder Download Center.
• The County boundary files came from a Census website: 2000 County Cartographic Boundary Files.
• The Census boundary files include data the denotes cut outs when a polygon should not cover an area. These are denoted with a -99999 ID in the Census boundary files. While I read these in, I have not decided the best way to handle them so that data is not represented here: in other words, some of the polygons may inappropriately cover an area.

Notes/Disclaimers:

• This is a preliminary release.
• Turn off Terrain for best viewing.
• Rotate, fly around, change the viewing angle to get a real sense of the visualization!
• Commercial use of this file is prohibited. If you are interested in using this file commercially, please drop an e-mail to censuskml [at] [Gmail].
• I do not warrant in any way, the accuracy of these maps. Use at your own risk.

Files:

• 2000 Census New England Migration Data.kmz

Please pass along any feedback/thoughts/inquires via comments!

Get your real, live examples!

After only posting screenshots for the past few days, it is time to share some real, live KMZ files. I'm going to share 3 states worth of data: California, Massachusetts, and Wyoming. Before I share links to the actual files, I want to be sure to share some important definitions, data sources, and notes.

What is mapped?

• I wanted to start simple: the following KMZs map the "Total Population" variable from the 2000 Census, broken out at the County Subdivision level. The height of each subdivision is equal to 1 meter for every 10 people. The colors also represent population, but use the state's max county subdivision population as the denominator, meaning they are only relevant within the state, not comparable across states. The largest subdivision will be bright red and the small ones will be white. I'm experimenting with this to try and get data on two levels - national and state.

Where is the data from?

• While I could have gotten the data from the FactFinder, I got the actual Total Population variable data from National Historical Geographic Information System (NHGIS). This system is powerful because, among other things, it lets you download the data for the entire US at once. The citation for this data is as follows:
• John S. Adams, William C. Block, Mark Lindberg, Robert McMaster, Steven Ruggles, and Wendy Thomas, National Historical Geographic Information System: Pre-release Version 0.1 Minneapolis: Minnesota Population Center University of Minnesota, 2004.
• The County Subdivsion boundary files came from a Census website: 2000 County Subdivisions Cartographic Boundary Files.
• The Census boundary files include data the denotes cut outs when a polygon should not cover an area. These are denoted with a -99999 ID in the Census boundary files. While I read these in, I have not decided the best way to handle them so that data is not represented here: in other words, some of the polygons may inappropriately cover an area.

Notes/Disclaimers:

• The actual population (in people) is listed for each County Subdivision in the name for each polygon. The number listed here is the actually population divided by 10, which is the height of the polygon in meters.
• This is a preliminary release.
• I license the use of these files under Creative Commons and Commercial use of the data is prohibited by NHGIS. The data can be gotten for other ways, so Commercial use is not out of the question in the future.
  
• Turn off Terrain for best viewing.
• Rotate, fly around, change the viewing angle to get a real sense of the visualization!
  
• I do not warrant in any way, the accuracy of these maps. Use at your own risk.

Maps of Total Population (people) from the 2000 Census by County Subdivsion:

• KMZs:
• California
• Massachusetts
• Wyoming
• Google Maps (viewing the KMZ files in the Google Maps interface - may only work in some browsers):
• California - won't work - too large
• Massachusetts
• Wyoming

Please pass along any feedback/thoughts/inquires via comments!

Some data resources & Google Maps

I've been travelling this past weekend so my development efforts were on hiatus. I plan to share some more nation-wide outputs tomorrow, but I wanted to share a fantastic website that someone passed along to me for Census data. The National Historical Geographic Information System (NHGIS) website is an extremely powerful data source for Census data. The website describes it self as follows:

The National Historical Geographic Information System (NHGIS) is a project
to create and freely disseminate a database incorporating all available
aggregate census information for the United States between 1790 and 2000.

The website is made by the Minnesota Population Center at the University of Minnesota. One of the most helpful aspects of the site is that it is slightly more intuitive and more powerful than the Census' FactFinder. For example, when looking at data at the County Subdivision level, you can download data for all of the states at once - on FactFinder it seems that the only way to get the data for all 50 states is to download each one individually which is a lot of wasted work. If I am missing this facility on FactFinder - please let me know!

The other aspect of the NHGIS website is that it seems built to provide the data in time series - something that the FactFinder website doesn't really provide. The Census website is admirably powerful, but lacks a couple of features that would make it much more so.

Just to keep up with the screenshots - I've got one more to share. One of the nifty things you can do with Google Maps (the website as opposed to the desktop app Google Earth) is displays KML files. In the screenshot below I simply put the KML file I had been displaying in Google Earth on a public HTTP site and loaded up the URL in the search box on Google Maps. My KML file had a bit too much data, so Google Maps can't draw all of the polygons, but with the right parsing down of data, this would be a very effective way to share KML files with a broad audience. In fact, this is a bit of a cleaner interface than Google Earth desktop app. I think you loose the 3D, but the you do get transparency and good colors: