tag:blogger.com,1999:blog-42917158321990912422024-03-13T15:01:14.671-05:00Census KML Data VisualizationA blog to track experiments in geographic data visualization using Census data, Ruby, KML, and Google EarthUnknownnoreply@blogger.comBlogger31125tag:blogger.com,1999:blog-4291715832199091242.post-85106793450638555522009-03-28T15:33:00.001-05:002009-03-28T15:33:18.240-05:00Decomposing genWABlockGroups() – Part 1<p>In my last post I mentioned that a place to start is to check out the <strong>genWABlockGroups()</strong> function. Today I am going to walk through first couple of lines to explain what it does to begin to unwind the mystery. </p> <p>This function was written to create maps based on Census Block Groups. Specifically, I built it to map the median home values in Washington state from the 2000 Census. Today I’m going to review the first couple of lines and input files:</p> <blockquote><code>baseDir = "/Users/aidan/Desktop/" <br />projectDir = "Census 2000 WA BGs/" <br />polygonShapeFile = baseDir + projectDir + "bg53_d00.dat" <br />polygonMetaDataFile = baseDir + projectDir + "bg53_d00a.dat" <br />dataFile = baseDir + projectDir + "Median HH Value.txt" <br />outputFile = baseDir + projectDir + "output" <br /></code></blockquote> <p>The first set of lines are all about setting up the input files and output files. In general, there are three input files required.</p> <ol> <li>You need a “shape” file. I talk quite a bit about the types of data sources I use in this post: <a title="http://censuskml.blogspot.com/2007/03/get-your-real-live-examples.html" href="http://censuskml.blogspot.com/2007/03/get-your-real-live-examples.html">http://censuskml.blogspot.com/2007/03/get-your-real-live-examples.html</a>. The files I use come straight from the Census Cartographic Boundary files website: <a title="http://www.census.gov/geo/www/cob/bdy_files.html" href="http://www.census.gov/geo/www/cob/bdy_files.html">http://www.census.gov/geo/www/cob/bdy_files.html</a>. Here is a sample of what a shape file looks like: <br /> <br /><code>1 -0.122233088267882E+03 0.489843885617284E+02 <br />-0.122285651792493E+03 0.490024370686774E+02 <br />-0.122251693540713E+03 0.490024929621633E+02 <br />-0.122251621999248E+03 0.490024930799168E+02 <br />... <br />-0.122285651792493E+03 0.490024370686774E+02 <br />END</code> <br /> <br />The structure is pretty simple. The shape's unique identifier within the file comes first, "1" in the above example. The coordinates that immediately follow this are the center point for the shape. Then the points that dictate the boundary points are separated by a line break after the first one, ending with the word "END". You’ll note that the first of the boundary points matches the last one exactly. The shapes are always 100% complete. A single Census cartographic component came be made up of multiple shapes (for example, an chain of islands that are in one county will be made up of distinct shapes). The “metadata” file is crucial to piecing all of this together. </li> <li>You need a shape “metadata” file. Again, this comes straight from the same Census website. Here is what the first few lines of the file look like: <br /> <br /><code>0 <br />" " <br />" " <br />" " <br />" " <br />" " <br />" " <br />" " <br /> <br />1 <br />"53" <br />"073" <br />"0102" <br />"2" <br />"2" <br />"BG" <br />" " <br />... <br /></code> <br />It can get hard to decipher these files. At a basic level this maps a shape to a specific Census area, in this case a Block Group. The metadata starts with the unique shape ID. Then there are a series of identifiers in quotation marks: (a) “53” = the State ID of Washington, (b) “073” = the county identifier within WA, (c) “0102” = tract within the county, (d) “2” = “the block group within the tract”, (e) “2” = the LSAD identifier with is kind of a numeric identifier that marks this as a Block Group, (f) “BG” = the translation of the previous LSAD code, and finally (g) the last blank is an area to put miscellaneous data. As I mention in the description of the previous file, there can be a one-to-many relationship between the shapes and actual Census area – multiple shapes can have the exact same metadata. Check out the <strong>BlockGroup.rb</strong> file to see how to programmatically read this data. </li> <li>You need a data file. The data file needs a way to map back to the “shapes” that are loaded. The “key” must match the shapes unique identifier. Again, I try and use files that are as straight from the Census website as possible. Typically I pull the data down from Tiger or FactFinder. Again, another sample: <br /> <br /><code>150,53,001,950100,1,530019501001,68800,88800,115900 <br />150,53,001,950100,2,530019501002,63300,78800,97800 <br />150,53,001,950100,3,530019501003,41300,58600,81600 <br /></code> <br />This is a CSV where the 6th column (if are are counting 1-based) contains the unique identifier for the Census geographic area and the three columns after it contain data about that specific area. You'll note how the identifier is a combination of the data from the metadata file. It took me a while to figure out how to construct the unique identifier, but I cracked the code. In the code, each shape object file has a function to build up it’s own unique ID that will map to the way Census spits out data with respect to that area.</li> </ol> <p>Next time I will go through the actual function calls that load in these files. </p> Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4291715832199091242.post-54817529421300380422009-03-24T20:57:00.002-05:002009-03-24T21:24:33.129-05:00Get some codeI spent a made two months really pouring lots of time and energy into code to produce the visualizaiton I have shared in this blog. In the process I learned a bunch about Ruby, KML, and Census data - and had great fun.<br /><br />That was almost two years ago and I've consistently made promises to myself to get back on the wagon and keep working on the project. However, these just have not materialized.<br /><br />Because of this, I've decided to open up the code, in its raw state, for public consumption. It is full of bad programming practices, from too many files to hard coded strings it would get an F in almost any code review. This prevented me from sharing it for a long time, but as the months kept ticking by and I didn't make any progress, I realized that either I was going to get it out there or it would simply die as a small blog.<br /><br />Over the coming posts I'll share the structure in lots more detail, but to start, I wanted to share what the most important files are.<br /><br />Let me give you a small tour around the code:<br /><ul><li><strong>GenPlacesKml.rb</strong> : This is the main file. This is what you "run". I've created many different "main" routines to create the outputs. This file also contains all of the code to create the actual KML document. I never created an object model for the KML components, I simply created functions that generated the components as rexml objects. This then gets dumped into the KML output file.</li><li><strong>Polygon.rb</strong> : This contains the definition of the Polygon object, which is the core object that holds the Census shape date. This also contains the routine to load Census shape data from a file.</li><li><strong>BlockGroups.rb, County.rb, etc.</strong> : These contain the objects to hold the Census meta data for each of the different types of geographies the Census creates. It also contains the method to load them.</li><li><strong>Data.rb </strong>: This is a generic data loading set of functions.</li><li><strong>Colors.rb</strong> : This generates the color palettes used in the files</li><li><strong>FIPS.rb </strong>: This contains some data definitions to do lookups on Census abbreviations.</li></ul><br /><br />If you want to start playing around with the code, the easiest "main" routine to look at is "genWABlockGroups()".<br /><br />Now, where is the code? I've decided to use Google Code to share it & I've put it out under the GPL. This uses Subversion, which I am completely unfamiliar with, to store and share the code. It does seem like a powerful system, but I'm just learning how to use it appropriately. I am very open to any and all feedback on its proper setup and use.<br /><br />I'm hoping the community is interested in breathing life back into the code. In the coming posts I'll walk through the structure (to the extent that it exists).<br /><br />You can find the source at <a href="http://code.google.com/p/rubykml/">http://code.google.com/p/rubykml/</a>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4291715832199091242.post-58385165992034745302009-03-15T22:35:00.000-05:002009-03-15T22:36:37.765-05:00Hmm... it has been a whileIt has been far too long since I've actively driven this project. As such, I'm seeking ways to share the source code I've generated. I'm exploring Google Code and will report back once that is good a setup.<br /><br />Thanks for staying tuned and I hope opening up the code will spark new life into the work.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4291715832199091242.post-2417507872261308852007-07-25T01:30:00.000-05:002008-11-13T07:44:52.489-05:00Real Progress - 5 digit ZCTAsIt has been 4 months since real progress was made - but tonight represented a real break through. I got the code up and running on my new laptop and transitioned over to using Eclipse, a great IDE that has integrated debugging of Ruby. But talking about how I am developing is not the purpose of this post. The purpose is to breath life into the project once more. <br /><br />I've gotten many e-mails over the past few months asking for help in creating maps. One person in particular had a very specific ask that I thought would both expand the functionality of the code & test a new data source. For this set of maps, I needed to start mapping 5-digit zip codes (Zip Code Tabulation Areas to the <a href="http://www.census.gov/">Census</a>), my first foray into zip codes in this work.<br /><br />As proof of life, enjoy the screenshot below which shows all of the zip codes for the state of Maine, colored with random colors. <br /><br /><a href="http://2.bp.blogspot.com/_zk-KTrrAVtI/Rqbv8jTXK-I/AAAAAAAAAFc/9Yka9wMT0N8/s1600-h/ME+5+Digit+ZCTAs.jpg"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;" src="http://2.bp.blogspot.com/_zk-KTrrAVtI/Rqbv8jTXK-I/AAAAAAAAAFc/9Yka9wMT0N8/s400/ME+5+Digit+ZCTAs.jpg" border="0" alt=""id="BLOGGER_PHOTO_ID_5091020252336565218" /></a><br /><br />Now that the code is running again and I've got some specific projects to get going again, I look forward to getting back to a much more regular schedule. Please keep up the contact.Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-4291715832199091242.post-33333253582908083262007-05-23T21:50:00.000-05:002007-05-23T22:23:51.503-05:00Congressional District maps for every state!It has been almost six weeks since I've had something substantive to add. I wanted to share the complete outputs from the <a href="http://censuskml.blogspot.com/2007/04/federal-election-commission-campaign.html">campaign contribution work I spoke about earlier</a>. Please read that post for a full explanation of what the data is and how I put the files together.<br /><br />The following two ZIP files contain a KMZ for every single state. I used the <code>TimeSpan</code> element, all of the states can be loaded at once. <br /><br />Without further ado:<br /><ul><br /><li><a href="http://amarcuss.bingodisk.com/bingo/public/CensusKML/109th_Congresional_Districts_Contributions.zip">109th Congressional Districts</a></li><br /><li><a href="http://amarcuss.bingodisk.com/bingo/public/CensusKML/110th_Congresional_Districts_Contributions.zip">110th Congressional Districts</a></li><br /></ul><br />If you have any problems with these files, please let me know.<br /><br />I hope to be back up and running soon, so please stay tuned.Unknownnoreply@blogger.com4tag:blogger.com,1999:blog-4291715832199091242.post-22194088197548256602007-05-10T10:55:00.001-05:002007-05-10T10:55:36.664-05:00New blog on the block<span xmlns=''><p>There is a new blog at <a href='http://www.google.com/'>Google</a> published by the Google Earth and Maps team: <a href='http://google-latlong.blogspot.com/'>Lat Long Blog</a>. I've begun spinning up my efforts to produce more maps and code and hope to share some new outputs soon. Thanks for staying connected during this intermission.<br /></p></span>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4291715832199091242.post-45490869226805997322007-04-22T21:48:00.000-05:002007-04-22T21:52:14.641-05:00A bit of a break...I apologize for not updating the blog for the last week - I've been silent because I have just started a new job and haven't had the time to come back to the work here yet. I do plan to come back to it, however at a slower pace. In the mean time, do not hesitate to reach out for help with similar efforts by either using the comment system or dropping me an e-mail at <em>censuskml at gmail</em>.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4291715832199091242.post-88894463354670936052007-04-13T13:08:00.000-05:002007-04-13T13:13:24.942-05:00Fixed KMZ DownloadsAn alert reader pointed out that the <a href="http://censuskml.blogspot.com/2007/04/fec-data-kmzs.html">KMZs I posted yesterday</a> were not properly downloading as KMZ files - the ended up as ZIP files on the desktop. This was because I was using <a href="http://www.amazon.com/S3-AWS-home-page-Money/b/ref=sc_fe_l_2/104-6757461-3035902?ie=UTF8&node=16427261&no=3435361&me=A36L942TSJ2AJA">Amazon's S3 service</a> to host the files which apparently doesn't have the KMZ mime type set. I've moved the KMZs to <a href="http://www.bingodisk.com/">BingoDisk</a> which fixes the problem in my testing. Please let me know if you have difficulty getting these new files. <br /><br />As a side note, the Google Maps links don't work with the <a href="http://www.bingodisk.com/">BingoDisk</a> files, but do with S3, so I've left a copy there as well.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4291715832199091242.post-69807807738530728802007-04-11T17:13:00.000-05:002007-04-13T13:08:05.462-05:00FEC Data KMZsTo follow up on my last <a href="http://censuskml.blogspot.com/2007/04/federal-election-commission-campaign.html">post</a> I wanted to share some of the actual KMZ files I generated to create the screenshots I showed. Please read that <a href="http://censuskml.blogspot.com/2007/04/federal-election-commission-campaign.html">post</a> for a full explanation of what these KMZs contain. The KMZ document descriptions also contain a brief explanation. <br /><br />Please note that I do not warrant in any way, the accuracy of these maps. Use at your own risk. That being said, I have done spot checking for 3 random districts in each of the attached KMZs using the FEC query tool, which can be found <a href="http://www.fec.gov/finance/disclosure/srssea.shtml">here</a>. The numbers I calculated are pretty close, and sometimes match exactly, to what the FEC tool returns. In the cases they do not, I suspect is is because of the data file I choose to use, with the FEC explicitly states might contain some inaccuracies given their attempt to make them as up-to-date as possible. <br /><br />I've also used the <code>TimeSpan</code> element, so you can load both the 109th and 110th Congress files into Google Earth at one time. Using the time slider you can move between the two Congresses - the effect is pretty cool. I've also included my <code>StyleMap</code> trick, so that when you mouseover the point near the center of each district a label with the district's name and the total receipts in dollars appears.<br /><br />I'm posting California, New York, and Texas - all states that have many districts. If you are interested in others, please feel free to e-mail me at <span style="font-style:italic;">censuskml [at gmail]</span>.<br /><br /><span style="font-weight:bold;">Total Receipts for House of Representative Candidates by Congressional District KMZs:</span><br /><ul><br /><li><a href="http://amarcuss.bingodisk.com/bingo/public/CensusKML/California_109th_Congress.kmz">California 109th Congress</a></li><br /><li><a href="http://amarcuss.bingodisk.com/bingo/public/CensusKML/California_110th_Congress.kmz">California 110th Congress</a></li><br /><li><a href="http://amarcuss.bingodisk.com/bingo/public/CensusKML/New_York_109th_Congress.kmz">New York 109th Congress</a></li><br /><li><a href="http://amarcuss.bingodisk.com/bingo/public/CensusKML/New_York_110th_Congress.kmz">New York 110th Congress</a></li><br /><li><a href="http://amarcuss.bingodisk.com/bingo/public/CensusKML/Texas_109th_Congress.kmz">Texas 109th Congress</a></li><br /><li><a href="http://amarcuss.bingodisk.com/bingo/public/CensusKML/Texas_110th_Congress.kmz">Texas 110th Congress</a></li><br /></ul><br /><span style="font-weight:bold;"><br />California and Texas are too large to load into Google Maps, but New York works:</span><br /><ul><br /><li><a href="http://maps.google.com/maps?f=q&hl=en&q=http://s3.amazonaws.com/G9CMqlwkvDJEyFnvEEeQ-CensusKML/New_York_109th_Congress.kmz&layer=&ie=UTF8&z=7&ll=42.714732,-75.267334&spn=4.875328,8.833008&om=1">New York 109th Congress in Google Maps</a></li><br /><li><a href="http://maps.google.com/maps?f=q&hl=en&q=http://s3.amazonaws.com/G9CMqlwkvDJEyFnvEEeQ-CensusKML/New_York_110th_Congress.kmz&layer=&ie=UTF8&z=7&ll=42.714732,-75.267334&spn=4.875328,8.833008&om=1">New York 110th Congress in Google Maps</a></li><br /></ul><br /><br />There are many interesting insights that can be gleaned from the data, in particular when contrasting the 109th and 110th election cycles. I'll save interpretation for another post. For now, enjoy.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4291715832199091242.post-17802258268470363652007-04-10T20:35:00.000-05:002008-11-13T07:44:54.187-05:00Federal Election Commission Campaign Contribution DataI am stretching my wings beyond Census data - and what could be more interesting then campaign contribution data? I have an interest in politics and thought it would be fascinating to see if there were ways to make the data about the amount spent on elections available. Using the same basic framework I created to read in Census data, I decided that the <a href="http://www.fec.gov/">Federal Election Commission (FEC)</a> would be a good source of interesting data about elections. <br /><br />The Census provides <a href="http://www.census.gov/geo/www/cob/bdy_files.html">boundary files for the last eight Congresses</a>. The <a href="http://www.fec.gov/">FEC</a> provides a wealth of data and I have just started to explore the full extent of what is possible. I started by using the <a href="http://www.fec.gov/finance/disclosure/ftpsum.shtml">Candidate Financial Summary Without PAC Breakdown</a> data. This data, of course, is in a rather complex format, so writing a Ruby module to read it out was the first order of business. <br /><br />The Census boundary files for the <a href="http://www.census.gov/geo/www/cob/cd110.html">110th</a> and <a href="http://www.census.gov/geo/www/cob/cd109.html">109th</a> are in a relatively similar format (the 108th and before begin to deviate substantially) so I used these as my polygon files. I used the <a href="http://www.fec.gov/finance/disclosure/ftpsum.shtml">Candidate Financial Summary Without PAC Breakdown (CFS)</a> files from the <a href="http://www.fec.gov/">FEC</a>. These files are the most current but do have some potential accounting issues. As the <a href="http://www.fec.gov/">FEC</a> states:<br /><blockquote><span style="font-style:italic;">The cost of this timelines, though, is that some of the information available here is less precise than for "cansum". For example, in "cansum" you can see how much a campaign received from Corporate PACs or Labor PACs, while here there is only one value for the total received from "other political committees." This includes all PAC contributions, but it may also contain contributions from other candidates, and some other types of committees we don't typically think of as PACs. We can't do the full breakdowns until all the information about specific contributions has been entered into the database.<br /><br />When using these summary files you need to be aware of some possible double counting of activity. Some candidates have more then one committee authorized to raise and spend funds on their behalf. The activity reflected in this file represents the sum of those committees. If they transfer funds back and forth among each other, this activity would be counted twice. Information about "transfers from authorized committees" and "transfers to authorized committees" is included in the file and if there are values in both of these fields it is necessary to subtract these from total receipts and total disbursements to obtain a more accurate value for actual activity.</span></blockquote><br /><br />In creating a data structure to store the CFS data, I created a somewhat flexible way to aggregate the data using some of the interesting features of Ruby. In particular the <code>eval</code> statement make it quite easy to pass in free text to allow the caller of the aggregation function to specify which field to aggregate quite easily. <br /><br />I encountered two main challenges mapping the CFS data back to the Census Congressional District polygons. First, the <a href="http://www.fec.gov/">FEC</a> uses the two-letter acronym to identify the state and the Census uses <a href="http://www.itl.nist.gov/fipspubs/fip5-2.htm">FIPS</a> codes to identify states. Using regular expressions in <a href="http://www.macromates.com/">TextMate</a>, I converted the list from the Census website to a couple of different Ruby hash tables so that I could convert back and forth from two-letter acronym to two-digit code. <br /><br />Second, the FEC files are somewhat inconsistent (at least based on how I am reading it) about how they handle states with only one Congressional District. The <a href="http://www.census.gov/">Census bureau</a> is <a href="http://www.census.gov/geo/www/cob/cd_metadata.html">pretty clear</a> that it uses "00" to identify districts that are the sole district for a given state. The <a href="http://www.fec.gov/">FEC</a> seems to follow this convention for the most part, except in a couple of states. For example, in Wyoming there are candidates for the House of Representatives listed in Congressional Districts "00" and "01". Wyoming has only one seat in the 110th congress. Here is a snippet from the <a href="ftp://ftp.fec.gov/FEC/webl06.zip">webl06.zip</a> file which contains the CFS data for the 2005-2006 election cycle (the ellipses represent where I have cut from the line for the sake of readability):<br /><code><br /> H4WY00055CUBIN, BARBARA L I2REP...WY01 W W48...<br> H6WY01025TRAUNER, GARY S 1DEM...WY01 W L47...<br> H6WY00118WINNEY, JUSTIN WILLIAM JR 2REP...WY00 0...<br><br /></code><br />According to the <a href="ftp://ftp.fec.gov/FEC/webl.txt">documentation for the file</a>, the two digits following the state acronym represent the district. In this case, it would appear to suggest that there are candidates for House in district "00" and "01", when in fact there is only one district in Wyoming. To handle this, I simply rolled up all of the House candidates, regardless of the district in the FEC file for one district states. This may be the wrong thing to do, so I've got an e-mail into the FEC to find out the actual answer. <br /><br />Alright, enough with that background, let's see some pictures. In the following screenshots, I am mapping the total amount received by House of Representative candidates in each Congressional District for a given election cycle. For each $1,000 received, the district gets one meter in height. The districts with higher amounts are more green, those with lower amounts are more red. Missouri is missing from the 109th Congress because the polygon metadata file is missing from the Census website.<br /><br />109th Congress from above:<br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://3.bp.blogspot.com/_zk-KTrrAVtI/Rh0QHFz7uHI/AAAAAAAAAEI/ox4-T66n3WA/s1600-h/109+Overhead.jpg"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;" src="http://3.bp.blogspot.com/_zk-KTrrAVtI/Rh0QHFz7uHI/AAAAAAAAAEI/ox4-T66n3WA/s400/109+Overhead.jpg" border="0" alt=""id="BLOGGER_PHOTO_ID_5052212070984693874" /></a><br /><br />110th Congress from above:<br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://1.bp.blogspot.com/_zk-KTrrAVtI/Rh0P4lz7uGI/AAAAAAAAAEA/rldW4TbMpWs/s1600-h/110+Overhead.jpg"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;" src="http://1.bp.blogspot.com/_zk-KTrrAVtI/Rh0P4lz7uGI/AAAAAAAAAEA/rldW4TbMpWs/s400/110+Overhead.jpg" border="0" alt=""id="BLOGGER_PHOTO_ID_5052211821876590690" /></a><br /><br />109th looking North:<br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://3.bp.blogspot.com/_zk-KTrrAVtI/Rh0QSFz7uII/AAAAAAAAAEQ/P9Kr09sicIk/s1600-h/109+looking+North.jpg"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;" src="http://3.bp.blogspot.com/_zk-KTrrAVtI/Rh0QSFz7uII/AAAAAAAAAEQ/P9Kr09sicIk/s400/109+looking+North.jpg" border="0" alt=""id="BLOGGER_PHOTO_ID_5052212259963254914" /></a><br /><br />110th looking North:<br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://4.bp.blogspot.com/_zk-KTrrAVtI/Rh0QcVz7uJI/AAAAAAAAAEY/7Y1mrCjNRp0/s1600-h/110+looking+North.jpg"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;" src="http://4.bp.blogspot.com/_zk-KTrrAVtI/Rh0QcVz7uJI/AAAAAAAAAEY/7Y1mrCjNRp0/s400/110+looking+North.jpg" border="0" alt=""id="BLOGGER_PHOTO_ID_5052212436056914066" /></a><br /><br />109th looking West:<br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://3.bp.blogspot.com/_zk-KTrrAVtI/Rh0QpFz7uKI/AAAAAAAAAEg/JbiYEm9In7A/s1600-h/109+looking+West.jpg"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;" src="http://3.bp.blogspot.com/_zk-KTrrAVtI/Rh0QpFz7uKI/AAAAAAAAAEg/JbiYEm9In7A/s400/109+looking+West.jpg" border="0" alt=""id="BLOGGER_PHOTO_ID_5052212655100246178" /></a><br /><br />110th looking West:<br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://4.bp.blogspot.com/_zk-KTrrAVtI/Rh0SHVz7uNI/AAAAAAAAAEo/_o9l2CH0Phg/s1600-h/110+looking+West.jpg"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;" src="http://4.bp.blogspot.com/_zk-KTrrAVtI/Rh0SHVz7uNI/AAAAAAAAAEo/_o9l2CH0Phg/s400/110+looking+West.jpg" border="0" alt=""id="BLOGGER_PHOTO_ID_5052214274302916818" /></a><br /><br />109th looking East:<br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://2.bp.blogspot.com/_zk-KTrrAVtI/Rh0SX1z7uOI/AAAAAAAAAEw/7RXXfy9LIiA/s1600-h/109+looking+East.jpg"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;" src="http://2.bp.blogspot.com/_zk-KTrrAVtI/Rh0SX1z7uOI/AAAAAAAAAEw/7RXXfy9LIiA/s400/109+looking+East.jpg" border="0" alt=""id="BLOGGER_PHOTO_ID_5052214557770758370" /></a><br /><br />110th looking East:<br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://4.bp.blogspot.com/_zk-KTrrAVtI/Rh0TCVz7uPI/AAAAAAAAAE4/95Y_em04mZU/s1600-h/110+looking+East.jpg"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;" src="http://4.bp.blogspot.com/_zk-KTrrAVtI/Rh0TCVz7uPI/AAAAAAAAAE4/95Y_em04mZU/s400/110+looking+East.jpg" border="0" alt=""id="BLOGGER_PHOTO_ID_5052215287915198706" /></a><br /><br />109th looking South:<br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://3.bp.blogspot.com/_zk-KTrrAVtI/Rh0TPFz7uQI/AAAAAAAAAFA/ejlENnTpNFk/s1600-h/109+looking+South.jpg"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;" src="http://3.bp.blogspot.com/_zk-KTrrAVtI/Rh0TPFz7uQI/AAAAAAAAAFA/ejlENnTpNFk/s400/109+looking+South.jpg" border="0" alt=""id="BLOGGER_PHOTO_ID_5052215506958530818" /></a><br /><br />110th looking South:<br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://3.bp.blogspot.com/_zk-KTrrAVtI/Rh0TZFz7uRI/AAAAAAAAAFI/B8lv3r6piRw/s1600-h/110+looking+South.jpg"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;" src="http://3.bp.blogspot.com/_zk-KTrrAVtI/Rh0TZFz7uRI/AAAAAAAAAFI/B8lv3r6piRw/s400/110+looking+South.jpg" border="0" alt=""id="BLOGGER_PHOTO_ID_5052215678757222674" /></a><br /><br />That is it for now. I'm close to sharing the KMZs for these, so be on the look out for a post soon.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4291715832199091242.post-38353176667163149082007-04-09T10:34:00.000-05:002008-11-13T07:44:54.351-05:00"Dynamic" Labels & StyleMapsI've been unhappy with how Google Earth handles labeling of shapes - the <code>Placemark</code> KML object is pretty good at containing a single shape, but breaks down when handling multiple shapes. <code>Placemark</code> elements contain the <code>name</code> and <code>description</code> elements, but only <code>Point</code> elements pick up on these. As I have talked about before, you can't label a <code>Polygon</code> without using a <code>Point</code>. I started to use the <code>MultiGeometry</code> to group together <code>Polygon</code> elements that belong to one geography (i.e. when a County has a couple of noncontiguous shapes). The annoying part is that I needed to add a <code>Point</code> for labels to show up. <br /><br />When dealing with complex geographies, like Block Groups, having the labels show up the entire time doesn't work very well. For example in the last map I shared, there are ~4,300 Block Groups in WA and having the labels all show up doesn't work very well because they overlap and make it quite confusing. You can play with breaking out the labels into a different <code>Placemark</code> and perhaps a folder structure at the County/County Subdivision level might help, but it still isn't perfect because you would have to hunt and peck for what you were looking. In looking through the KML spec, I was excited to find the <code>StyleMap</code> element. The <code>StyleMap</code> element provides a mechanism to have a <code>Placemark</code> respond to mouseover/click/highlight events. You can define a <code>Style</code> element for both <span style="font-style:italic;">normal</span> and <span style="font-style:italic;">highlight</span> classes. I thought this would be a great way to provide a label: when a user moves their mouse over a geography (e.g. Block Group, County Subdivision, County) the label could show up - the normal style would have it transparent and the normal would have it opaque.<br /><br />Well, turns out it doesn't quite work that way. Unfortunately, the only thing that sparks the transformation is the user moving their mouse over the icon of a <code>Point</code>: the style doesn't change if they mouseover the label or any of the <code>Polygon</code> elements in the <code>Placemark</code>. What is odd, however, is that all of the elements of the <code>Placemark</code> do respond to the new style when you mouseover the icon.<br /><br />What you'll see below are maps of Median Household Value (variable H85 MEDIAN VALUE (DOLLARS) FOR ALL OWNER-OCCUPIED HOUSING UNITS [1] from <a href="http://factfinder.census.gov/jsp/saff/SAFFInfo.jsp?_pageId=sp4_decennial_sf3&_submenuId=&ds_name=&_ci_nbr=&qr_name=&_industry=">Summary File 3</a>) by County Subdivision. I'm not entirely happy with my new labeling, but it works such that when you mouseover the icon at the center of the polygon, the name of the County Subdivision shows up and the border is highlighted in white. I like the effect, but I am not happy with how I have to have an icon show up. In the maps below, the more red an area/shape, the higher the median household value - the greener, the lower the median household value. In the 3D maps, each $1,000 of value adds one meter of height.<br /><br />Movie:<br /><object width="425" height="350"><param name="movie" value="http://www.youtube.com/v/NisIFzbn3Sk"></param><param name="wmode" value="transparent"></param><embed src="http://www.youtube.com/v/NisIFzbn3Sk" type="application/x-shockwave-flash" wmode="transparent" width="425" height="350"></embed></object><br /><br />Screenshot (notice how <span style="font-style:italic;">Leavenworth-Lake Wenatchee</span> is highlighted):<br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://4.bp.blogspot.com/_zk-KTrrAVtI/Rhpm4lddp7I/AAAAAAAAAD4/3llOlh4NPFI/s1600-h/WA_CSDs_Median_HH_Value_Highlight.jpg"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;" src="http://4.bp.blogspot.com/_zk-KTrrAVtI/Rhpm4lddp7I/AAAAAAAAAD4/3llOlh4NPFI/s400/WA_CSDs_Median_HH_Value_Highlight.jpg" border="0" alt=""id="BLOGGER_PHOTO_ID_5051463054363830194" /></a>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4291715832199091242.post-18249915355415338532007-04-07T12:45:00.000-05:002007-04-07T12:56:20.667-05:00Change your feed readers!Good morning! For those of you that enjoy getting updates via the feed, I kindly request that you change your readers to use my new feed:<br /><br /><div style="text-align: center;"><a href="http://feeds.feedburner.com/CensusKmlDataVisualization">http://feeds.feedburner.com/CensusKmlDataVisualization</a><br /></div><br /><br />Blogger is good at many things, but getting a sense of how your feed is used is not one of them. Thank you!Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4291715832199091242.post-75281079071714753462007-04-06T14:29:00.000-05:002007-04-06T14:36:37.801-05:00Follow up to Median Household Value for WashingtonI just recently discovered the <code>TimeSpan</code> element in KML and thought it might be an interesting way to animate the maps. To try this out, I add this to the maps I discussed in my <a href="http://censuskml.blogspot.com/2007/04/median-household-value-for-washington.html">previous post</a>. The dates are arbitrary, but I gave each block group a date starting from the one with the lowest median household value and ending with the block with the highest. You can see the effect in the following video which animates the drawing of each of the block groups: building all of them up in order and then removing them in reverse order.<br /><br /><object width="425" height="350"><param name="movie" value="http://www.youtube.com/v/aakR0bVU-3Y"></param><param name="wmode" value="transparent"></param><embed src="http://www.youtube.com/v/aakR0bVU-3Y" type="application/x-shockwave-flash" wmode="transparent" width="425" height="350"></embed></object><br /><br />In Google Earth you can control many aspects of the animation, including the speed at which it moves through dates. I think this may be an interesting way to make the maps come alive a bit more.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4291715832199091242.post-46576182784070084592007-04-06T10:59:00.000-05:002008-11-13T07:44:55.338-05:00Median Household Value for Washington state by Block GroupUsing the new functionality I've discussed in the last two posts, I am pushing forward in creating new maps. Today I'm going to share a few maps of median household value (variable H76 - MEDIAN VALUE (DOLLARS) FOR SPECIFIED OWNER-OCCUPIED HOUSING UNITS [1] from <a href="http://factfinder.census.gov/jsp/saff/SAFFInfo.jsp?_pageId=sp4_decennial_sf3&_submenuId=&ds_name=&_ci_nbr=&qr_name=&_industry=">Summary File 3</a>). There is another variable, H85, which might be better for what I want, but I'm going to go ahead and share these maps before going back. These new maps show the median household value by block group for Washington state. There are ~4,300 block groups in Washington and a wide array of values in the data, so this data provides a good test of the new functionality (labels, excluded polygons, logarithmic color scales). Given the large number of block groups, I've found that it takes several different map formats to fully explore the data. I've created maps that have the 3D views I've shared before and maps that are flat with some transparency so that you can see the underlying geography. <br /><br />One of the challenges I found in creating these maps was gathering the Census data at the block group for the entire state. <a href="http://www.nhgis.org/">NHGIS</a> doesn't provide many variables beyond the basic population and economic ones and the <a href="http://factfinder.census.gov/servlet/DownloadDatasetServlet?_lang=en">Census FactFinder website</a> doesn't make it easy to download all of the block groups for a given state at once - you have to download each county separately. Given these challenges, I looked into download the raw data from <a href="http://factfinder.census.gov/jsp/saff/SAFFInfo.jsp?_pageId=sp4_decennial_sf3&_submenuId=&ds_name=&_ci_nbr=&qr_name=&_industry=">Summary File 3</a>. There raw files are available by FTP but, of course, are in a very complex format. The Census Bureau provides an Access database template that contains empty versions of each of the ~80 tables needed to work with the data (including import specs which is quite helpful). Feeling intrepid I downloaded all of the data for Washington state (<a href="ftp://ftp.census.gov/census_2000/datasets/Summary_File_3/Washington">FTP site</a>) and loaded the tables I needed to into the Access template. This worked pretty well, but is somewhat confusing, particularly because joining the geographic identifiers for each record is not quite as straight forward as the Census documentation would lead you to believe. I finally got it to work and this provided the data for the maps provided below - perhaps you can now understand why I haven't gone back and re-run the maps using the H85 variable yet. <br /><br />In the maps below, the more red an area/shape, the higher the median household value - the greener, the lower the median household value. In the 3D maps, each $1,000 of value adds one meter of height. You'll note that some areas show up as white, this is because the data provided by the Census for these block groups is 0. This makes sense for places like Mt. Rainier, but not for a region of downtown Seattle that shows up as white. I'm going to look into this next. <br /><br />Now, some maps (don't forget you can click on each picture to get a larger version)!<br /><br />Entire state (flat from overhead):<br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://1.bp.blogspot.com/_zk-KTrrAVtI/RhZuq1ddp0I/AAAAAAAAADA/ILPJ3nX4IHQ/s1600-h/WA_Median_HH_Value_Flat_Entire_State.jpg"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;" src="http://1.bp.blogspot.com/_zk-KTrrAVtI/RhZuq1ddp0I/AAAAAAAAADA/ILPJ3nX4IHQ/s400/WA_Median_HH_Value_Flat_Entire_State.jpg" border="0" alt=""id="BLOGGER_PHOTO_ID_5050345714326808386" /></a><br />Entire state (flat from overhead, borders around each block group, some transparency):<br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://3.bp.blogspot.com/_zk-KTrrAVtI/RhZu4Vddp1I/AAAAAAAAADI/oOjSNajQs1c/s1600-h/WA_Median_HH_Value_Flat_Entire_State_Transparent_Borders.jpg"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;" src="http://3.bp.blogspot.com/_zk-KTrrAVtI/RhZu4Vddp1I/AAAAAAAAADI/oOjSNajQs1c/s400/WA_Median_HH_Value_Flat_Entire_State_Transparent_Borders.jpg" border="0" alt=""id="BLOGGER_PHOTO_ID_5050345946255042386" /></a><br />Entire state (3D from overhead): <br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://1.bp.blogspot.com/_zk-KTrrAVtI/RhZvK1ddp2I/AAAAAAAAADQ/GES4QXHbyG8/s1600-h/WA_Median_HH_Value_3D_Entire_State.jpg"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;" src="http://1.bp.blogspot.com/_zk-KTrrAVtI/RhZvK1ddp2I/AAAAAAAAADQ/GES4QXHbyG8/s400/WA_Median_HH_Value_3D_Entire_State.jpg" border="0" alt=""id="BLOGGER_PHOTO_ID_5050346264082622306" /></a><br />Entire State (3D looking North):<br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://2.bp.blogspot.com/_zk-KTrrAVtI/RhZvZFddp3I/AAAAAAAAADY/BbJgVkppngE/s1600-h/WA_Median_HH_Value_3D_Entire_State_Looking_North.jpg"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;" src="http://2.bp.blogspot.com/_zk-KTrrAVtI/RhZvZFddp3I/AAAAAAAAADY/BbJgVkppngE/s400/WA_Median_HH_Value_3D_Entire_State_Looking_North.jpg" border="0" alt=""id="BLOGGER_PHOTO_ID_5050346508895758194" /></a><br />Entire State (3D looking East):<br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://1.bp.blogspot.com/_zk-KTrrAVtI/RhZvi1ddp4I/AAAAAAAAADg/MwfqB7xX0uw/s1600-h/WA_Median_HH_Value_3D_Entire_State_Looking_East.jpg"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;" src="http://1.bp.blogspot.com/_zk-KTrrAVtI/RhZvi1ddp4I/AAAAAAAAADg/MwfqB7xX0uw/s400/WA_Median_HH_Value_3D_Entire_State_Looking_East.jpg" border="0" alt=""id="BLOGGER_PHOTO_ID_5050346676399482754" /></a><br />Seattle (3D looking Southeast):<br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://2.bp.blogspot.com/_zk-KTrrAVtI/RhZvuFddp5I/AAAAAAAAADo/P2Z3MdaEmog/s1600-h/WA_Median_HH_Value_3D_Seattle_Zoom.jpg"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;" src="http://2.bp.blogspot.com/_zk-KTrrAVtI/RhZvuFddp5I/AAAAAAAAADo/P2Z3MdaEmog/s400/WA_Median_HH_Value_3D_Seattle_Zoom.jpg" border="0" alt=""id="BLOGGER_PHOTO_ID_5050346869673011090" /></a><br />Seattle (flat from overhead, borders around each block group, some transparency):<br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://4.bp.blogspot.com/_zk-KTrrAVtI/RhZv4lddp6I/AAAAAAAAADw/Yx4o42upoxs/s1600-h/WA_Median_HH_Value_Flat_Seattle_Transparent_Borders.jpg"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;" src="http://4.bp.blogspot.com/_zk-KTrrAVtI/RhZv4lddp6I/AAAAAAAAADw/Yx4o42upoxs/s400/WA_Median_HH_Value_Flat_Seattle_Transparent_Borders.jpg" border="0" alt=""id="BLOGGER_PHOTO_ID_5050347050061637538" /></a>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4291715832199091242.post-74131503867474276882007-04-05T15:28:00.000-05:002007-04-05T15:57:17.613-05:00Blogger ate my post on new features/code improvementsToday's 500 word post was just eaten by the Blogger spell-checker. I'm going to attempt to retrieve it, but it isn't looking good. The topic of the post was some major improvements I've made to the code base that make it much more generic and support two new features:<br /><ul><br /><li>Exclusions are now handled - if a polygon has an exclusion listed in the Census shape file, this is now handled. This turns out to be quite important for some areas. I'm working on a map of Washington state at the Block Group level and it turns out that in the rural parts of the state there are a couple of small towns that have a single Block Group surrounding the town and then separate, smaller Block Groups for the town itself. Before, these smaller Block Groups might have been covered up.</li><br /><li>A geography made up of multiple polygons now acts as one object in the KMZ output file.</li><br /></ul><br />In addition, I've made a number of enhancements under the hood. While I had been using some object-orientation before, I finally made the classes much more complete and generic so that it is much easier to add new geographies. For example, I created the <code>Polygon</code> object which encapsulates all of the information for a given polygon in one container:<br /><code><br />class Polygon<br /> attr_reader :id, :centerLon, :centerLat, :mainCoords, :exCoords<br /> attr_writer :id, :centerLon, :centerLat<br /> <br /> def initialize(id, centerLon, centerLat)<br /> @id = id<br /> @centerLon = centerLon<br /> @centerLat = centerLat<br /> @mainCoords = Array.new()<br /> @exCoords = Array.new()<br /> end<br />end<br /></code><br /><br />With these improvements, I hope to have some maps to share soon. When I am once again inspired, I will share a lot more detail about the process and recent improvements.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4291715832199091242.post-9854075929324605542007-04-02T21:08:00.000-05:002008-11-13T07:44:55.659-05:00Hawaii Population Data & New FeaturesAs promised, my break gave me a new burst of energy. I spent my week off in Hawaii and figured it would be a good region to experiment with a few new map features. I tackled two items that were on my to-do list: labels & logarithmic color scales. I have generated a new map of population by County Subdivsion for Hawaii to demonstrate these.<br /><br />First, on labels. I've found that Google Earth is quite powerful except when it comes to labeling polygons. You can provide "names" for polygons, but these only appear in the "Places" panel on the left of the map view. I suppose this may have to do with the difficulty in figuring out where to put these names on the map display (given how oddly shaped a polygon can be), but I would have thought there was some good default behavior for this (if I am missing a feature of KML, please let me know!). To place a label on the map display, you have to create a "Point" placemark. Out of a desire to keep moving, I pushed forward without labels. The Census Bureau's shape files do actually include a center point for each polygon, so I have now gone back and updated my code to generate "Point" placemarks for each of these center points. With these points, I can now have labels appear on the map. This is very helpful, particularly when dealing with geographies below County (County Subdivision, Block Group, etc.).<br /><br />Second, on logarithmic color scales. One of the challenges in mapping any sort of data that has a very wide range and is not very evenly distributed across the range is that in can be hard to find a color scheme that provides clarity at either extreme. I have talked about this in a couple of previous posts, but finally gotten around to implementing a logarithmic scheme that more evenly distributes the data across the range. I'm not entirely happy with what I've implemented, so I plan to work on it further. <br /><br />On to some screenshots. Below you'll find 3 perspectives of population, by County Subdivision, from the 2000 Census for Hawaii. Every meter in height represents 5 people; Greener represents lower population, Red higher population. The labels come in handy because I'm not that familiar with the islands. The new logarithmic color scheme comes in handy because Honolulu has a much higher population that all of the other County Subdivisions. I've used the same Green to Red color scheme I've used before, but with the logarithmic scaling it now does a much better job of helping one to distinguish between the Subdivisions on the lower end of the population range. Without this new scheme, Honolulu would be red and everything else green. <br /><br />From directly overhead:<br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://3.bp.blogspot.com/_zk-KTrrAVtI/RhG8UHFTuqI/AAAAAAAAACo/WH4Y5q1TSjU/s1600-h/HI_Pop_Overhead.jpg"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;" src="http://3.bp.blogspot.com/_zk-KTrrAVtI/RhG8UHFTuqI/AAAAAAAAACo/WH4Y5q1TSjU/s400/HI_Pop_Overhead.jpg" border="0" alt=""id="BLOGGER_PHOTO_ID_5049023710943754914" /></a><br /><br />From an angle, looking North:<br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://4.bp.blogspot.com/_zk-KTrrAVtI/RhG8fXFTurI/AAAAAAAAACw/etRXuK7TSgA/s1600-h/HI_Pop_North_Angle.jpg"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;" src="http://4.bp.blogspot.com/_zk-KTrrAVtI/RhG8fXFTurI/AAAAAAAAACw/etRXuK7TSgA/s400/HI_Pop_North_Angle.jpg" border="0" alt=""id="BLOGGER_PHOTO_ID_5049023904217283250" /></a><br /><br />From an angle, looking South (so you can see the northern side of Oahu):<br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://4.bp.blogspot.com/_zk-KTrrAVtI/RhG8rXFTusI/AAAAAAAAAC4/4wdABuf-RuY/s1600-h/HI_Pop_South_Angle.jpg"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;" src="http://4.bp.blogspot.com/_zk-KTrrAVtI/RhG8rXFTusI/AAAAAAAAAC4/4wdABuf-RuY/s400/HI_Pop_South_Angle.jpg" border="0" alt=""id="BLOGGER_PHOTO_ID_5049024110375713474" /></a><br /><br />One other note of interest: I was perplexed for a few minutes because the Midway Islands and the other islands west of Kauai all showed up with tall, red polygons - meaning they have a high population (I excluded them from the screenshots for this reason). It turns out that the Honolulu County Subdivision includes all of these islands, hence they get the data for the entire Subdivision. I suppose this demonstrates one of the perils of geographic aggregation when working with an island chain.Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-4291715832199091242.post-86087967938442302922007-04-02T18:01:00.000-05:002008-11-13T07:44:55.912-05:00I haven't disappeared...I apologize for the lack of posts last week. I spent the week on vacation and didn't dream of touching a computer. My trip spurred a bunch of ideas for interesting maps that I am already working on. In the mean time, I will share a quick panorama I put together from pictures overlooking <a href="http://maps.google.com/maps?f=q&hl=en&q=Hana,+HI&layer=&ie=UTF8&z=8&ll=20.75804,-155.990238&spn=3.574607,6.108398&om=1">Hana</a>, on the eastern side of Maui. <br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://1.bp.blogspot.com/_zk-KTrrAVtI/RhGMsnFTupI/AAAAAAAAACg/YvGX9q8MhnM/s1600-h/pano+25%25.jpeg"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;" src="http://1.bp.blogspot.com/_zk-KTrrAVtI/RhGMsnFTupI/AAAAAAAAACg/YvGX9q8MhnM/s400/pano+25%25.jpeg" border="0" alt=""id="BLOGGER_PHOTO_ID_5048971355292416658" /></a>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4291715832199091242.post-10490959318584096712007-03-22T22:52:00.000-05:002008-11-13T07:44:56.919-05:00More Migration AnalysisI'm finding the migration variables fascinating. These questions are a part of the long form and can be found in Summary File 3 on the <a href="http://factfinder.census.gov/servlet/DownloadDatasetServlet?_lang=en">Census website</a>. For the maps in the screenshots below, I used the <code>Population 5 years and over: Different house in 1995; In United States in 1995; Different county; Different state; ...</code> variables. The variables allow the identification of where people are moving from, which is quite interesting. The variables are broken up into 4 regions: Northeast, Midwest, South, and West. What I have mapped is the number of people (over the age of 5) who have moved from a state in a particular region to another state (which may also be in that region). For example, in the Northeast map, if a person lived in Maine and moved to Arizona (the map below will show this appears to be quite a popular destination for New Englanders), they would be counted in the county they moved to in Arizona. If a person lived in Maine and moved to Vermont, they would be counted in the county they moved to in Vermont. <br /><br />This data is broken up by county and the more red and taller a county, the more people that moved there. The heights are quite exaggerated: each person adds 10 meters of height to the county. These maps show how the linear color scale I've been employing to date only really work on datasets that have quite small ranges. I am working on a logarithmic scaling technique that should help on these sorts of datasets, where there may be a smaller number of values that may distort the distribution of values.<br /><br />Also, I realize it might not have been intuitive: you can click on the pictures for a much larger version of the image. This is true for all of the pictures on the blog.<br /><br /><span style="font-weight:bold;">Northeast:</span> New Englanders appear to be moving to Florida, Arizona, and California in droves. Chicago & Seattle get a fair number as well.<br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://1.bp.blogspot.com/_zk-KTrrAVtI/RgQ-RmSg9eI/AAAAAAAAAB8/nKdws-srxqs/s1600-h/2000_NE_Migration.jpg"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;" src="http://1.bp.blogspot.com/_zk-KTrrAVtI/RgQ-RmSg9eI/AAAAAAAAAB8/nKdws-srxqs/s400/2000_NE_Migration.jpg" border="0" alt=""id="BLOGGER_PHOTO_ID_5045225954618701282" /></a><br /><br /><span style="font-weight:bold;">Midwest:</span> Midwesterners are more focused on Arizona than California and quite drawn to Chicago.<br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://3.bp.blogspot.com/_zk-KTrrAVtI/RgQ-wGSg9fI/AAAAAAAAACE/Lb7qmbqYaRY/s1600-h/2000_MW_Migration.jpg"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;" src="http://3.bp.blogspot.com/_zk-KTrrAVtI/RgQ-wGSg9fI/AAAAAAAAACE/Lb7qmbqYaRY/s400/2000_MW_Migration.jpg" border="0" alt=""id="BLOGGER_PHOTO_ID_5045226478604711410" /></a><br /><br /><span style="font-weight:bold;">South:</span> Southeners appear to be moving all over including California, Arizona, Georgia , Texas, North Carolina, and DC. <br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://4.bp.blogspot.com/_zk-KTrrAVtI/RgQ_GWSg9gI/AAAAAAAAACM/kZQcA_6ga6E/s1600-h/2000_South_Migration.jpg"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;" src="http://4.bp.blogspot.com/_zk-KTrrAVtI/RgQ_GWSg9gI/AAAAAAAAACM/kZQcA_6ga6E/s400/2000_South_Migration.jpg" border="0" alt=""id="BLOGGER_PHOTO_ID_5045226860856800770" /></a><br /><br /><span style="font-weight:bold;">West:</span> Westerners shun moving to the Midwest, South, or East, favoring consolidation in Las Vegas (yes, the more northern red spike is Vegas) and Phoenix. You can see some movements to Hawaii & Alaska in the distance.<br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://3.bp.blogspot.com/_zk-KTrrAVtI/RgQ_zGSg9hI/AAAAAAAAACU/EQj1wpWZHbI/s1600-h/2000_West_Migration.jpg"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;" src="http://3.bp.blogspot.com/_zk-KTrrAVtI/RgQ_zGSg9hI/AAAAAAAAACU/EQj1wpWZHbI/s400/2000_West_Migration.jpg" border="0" alt=""id="BLOGGER_PHOTO_ID_5045227629655946770" /></a>Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-4291715832199091242.post-27591823622200702582007-03-22T21:40:00.000-05:002007-03-22T21:43:55.222-05:00New way to reach CensusKMLTo further facilitate the conversation about mapping Census data in Google Earth, I've created an e-mail address: censuskml [at] [gmail] (I hope you can decipher it). Feel free to reach out to me with specific questions or inquiries using either the e-mail address or the comment system.Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4291715832199091242.post-4019895504808729882007-03-22T21:36:00.000-05:002007-03-22T21:40:34.771-05:00KML (& KMZ) Support Added to the Google Maps API<a href="http://googlemapsapi.blogspot.com/2007/03/kml-and-georss-support-added-to-google.html">Google Maps API Official Blog: KML and GeoRSS Support Added to the Google Maps API</a> - this is pretty interesting since the functionality has been available in Google maps for some time [see my <a href="http://censuskml.blogspot.com/2007/03/get-your-real-live-examples.html">previous post</a>]. The post doesn't explicitly state it, but I KMZs seem to work just as well. There appears to be some size limit to between 100 - 200 kb, from simply experimentation. Just to share again, here are two KMZs of population by County Subdivision from the 2000 Census that you can view on Google Maps:<br /><ul><li><a href="http://maps.google.com/maps?f=q&hl=en&q=http://s3.amazonaws.com/G9CMqlwkvDJEyFnvEEeQ-CensusKML/cs25_d00.kmz&layer=&ie=UTF8&z=8&om=1">Massachusetts</a> </li><li><a href="http://maps.google.com/maps?f=q&hl=en&q=http://s3.amazonaws.com/G9CMqlwkvDJEyFnvEEeQ-CensusKML/cs56_d00.kmz&layer=&ie=UTF8&z=7&om=1">Wyoming</a></li></ul>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4291715832199091242.post-13160803161657219612007-03-22T09:39:00.000-05:002007-04-09T09:09:15.631-05:00New KMZ!Today I'm going to share a KMZ of a new variable that I've been working on. I wanted to experiment with more complicated variables, beyond Median Household Income, to really push the flexibility of the code I'm writing. The migration variables of summary file 3 from the Census are fascinating and fit the bill. They allow one to understand how people are moving about the country and include quite a bit of granularity. I'm going to share 7 states worth of data: Connecticut, Maine, Massachusetts, New Hampshire, New York, Rhode Island, and Vermont. Before I share links to the actual file, I want to be sure to share some important definitions, data sources, and notes.<br /><br /><span style="font-weight: bold;">What is mapped?</span><br /><ul><li>There are many migration variables, but for this example I've chosen to use two. I used the <code>Population 5 years and over: Total - P024001</code> variable which represents the number of people in 2000 over the age of 5 and used it to divide the <code>Population 5 years and over: Different house in 1995 - P024003</code> variable. The resulting percent should represent the number of people, over the age of 5, who didn't live in the house they lived in during the 2000 Census in 1995. Put it simply, the percent of people who moved in the last 5 years. The migration data provides a much more detailed breakout of where the people that moved came from, which I hope to work further with.<br /></li><li>The data is presented broken out by county: the taller, the more blue a county is in the file the higher a percent that moved - the shorter, the more green, the smaller percent that moved. The actual percent can be found in the description for each polygon, however it is multiplied by 1,000 there so the actual value is what is found in the description divided by 1,000.</li><li>In New England, the region in the provided KMZ, Tompkins County, New York has the max value at ~ 58%. This means that ~ 58% of people in Tompkins County moved since 1995. There are many counties at the low end: Hamilton County, New York is quite low at around ~30%, but so are Aroostook County, Maine and Orange County, Vermont, both around ~33%.</li><li>You'll also note how the counties are organized into folders within the KMZ file. This is a recent improvement to the KMZ generation process. I've also modified the code to follow the best practice of referencing repeated styles by ID (each polygon references a base style and only overrides what it needs to) - I thought this would save a lot on file size, but it didn't because ZIP was quite efficient at compressing the bits that were repeated again and again.<br /></li></ul><span style="font-weight: bold;">How about a quick tour of the map?</span><br />[<span style="font-style: italic;">YouTube seems to like to cut down the length of videos, so this may feel a bit choppy - not sure why it is doing this</span>]<br /><object width="425" height="350"><param name="movie" value="http://www.youtube.com/v/rn-Buj4WWlg"></param><param name="wmode" value="transparent"></param><embed src="http://www.youtube.com/v/rn-Buj4WWlg" type="application/x-shockwave-flash" wmode="transparent" width="425" height="350"></embed></object><br /><br /><span style="font-weight: bold;">Where is the data from?</span><br /><ul><li>I downloaded the data from the <a href="http://factfinder.census.gov/servlet/DownloadDatasetServlet?_lang=en">FactFinder Download Center</a>.</li><li>The County boundary files came from a Census website: <a href="http://www.census.gov/geo/www/cob/co2000.html">2000 County Cartographic Boundary Files</a>.</li><ul><li>The Census boundary files include data the denotes cut outs when a polygon should not cover an area. These are denoted with a -99999 ID in the Census boundary files. While I read these in, I have not decided the best way to handle them so that data is not represented here: in other words, some of the polygons may inappropriately cover an area.</li></ul></ul><span style="font-weight: bold;">Notes/Disclaimers:</span><br /><ul><li>This is a preliminary release.</li><li>Turn off Terrain for best viewing.</li><li>Rotate, fly around, change the viewing angle to get a real sense of the visualization!</li><li>Commercial use of this file is prohibited. If you are interested in using this file commercially, please drop an e-mail to censuskml [at] [Gmail].</li><li>I do not warrant in any way, the accuracy of these maps. Use at your own risk.</li></ul><span style="font-weight: bold;">Files:</span><br /><ul><li><a href="http://s3.amazonaws.com/G9CMqlwkvDJEyFnvEEeQ-CensusKML/2000%20Census%20New%20England%20Migration%20Data.kmz">2000 Census New England Migration Data.kmz</a></li></ul><br />Please pass along any feedback/thoughts/inquires via comments!Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4291715832199091242.post-12102356199064756762007-03-20T20:07:00.000-05:002008-11-13T07:44:57.441-05:00Getting better color schemes...I'm making good progress in finding color schemes that actually make sense. As a sneak peak, I've posted a video of a tour over the median household income data from the 2000 Census by County. Green is low median income and red is high (the other way around, which may be more logical there is just way too much red on the screen). The main hurdle I am working on now is how to reduce the file size. The file shown in the movie is ~7 mb. <br /><br /><object width="425" height="350"> <param name="movie" value="http://www.youtube.com/v/LAoN_ht2uQ8"> </param> <embed src="http://www.youtube.com/v/LAoN_ht2uQ8" type="application/x-shockwave-flash" width="425" height="350"> </embed> </object><br /><br />Here are some of the color schemes that I have put together by hand (enough of the auto-hex code generation that led me to a random walk across hues & saturations). The movie above uses the scheme third from left.<br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://1.bp.blogspot.com/_zk-KTrrAVtI/RgCHZmSg9dI/AAAAAAAAAB0/DPYei1l6d48/s1600-h/color_schemes.png"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;" src="http://1.bp.blogspot.com/_zk-KTrrAVtI/RgCHZmSg9dI/AAAAAAAAAB0/DPYei1l6d48/s400/color_schemes.png" border="0" alt=""id="BLOGGER_PHOTO_ID_5044180456499639762" /></a>Unknownnoreply@blogger.com1tag:blogger.com,1999:blog-4291715832199091242.post-36118380289161613842007-03-19T22:35:00.000-05:002008-11-13T07:44:57.605-05:00Median Household IncomeI have made progress and am now generating maps at the County level with data. I'm not happy with the color schemes yet, so I will be pushing on this before I share the KMZ file. The orientation of the screenshot below is looking Northeast over New England at an altitude of 900 KM. The data is Median Household Income by County from the 2000 Census (Summary File 3). The brighter & lighter a color the higher the income. The heights of each County represent one meter for every dollar of median household income. If anyone knows of any good resources for color pallets, please drop me a line in the comments!<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://3.bp.blogspot.com/_zk-KTrrAVtI/Rf9X8WSg9cI/AAAAAAAAABs/whH0sSl8vNE/s1600-h/census_2000_counties_median_income_east.jpg"><img style="display:block; margin:0px auto 10px; text-align:center;cursor:pointer; cursor:hand;" src="http://3.bp.blogspot.com/_zk-KTrrAVtI/Rf9X8WSg9cI/AAAAAAAAABs/whH0sSl8vNE/s400/census_2000_counties_median_income_east.jpg" border="0" alt=""id="BLOGGER_PHOTO_ID_5043846801965249986" /></a>Unknownnoreply@blogger.com4tag:blogger.com,1999:blog-4291715832199091242.post-54563794929976429112007-03-19T19:30:00.000-05:002008-11-13T07:44:57.909-05:00CountiesIt is amazing how life can get in the way of making progress! I've been out of town for the past few days and did not have a chance to push the project forward. I finally got some time today and in an effort to generate output I can share broadly, I'm working to generate some interesting KMZs to share. I wanted to put together KMZs with median income by county subdivision, but could not get <a href="http://www.nhgis.org/">NHGIS</a> to generate this data. Given the complication with downloading the data from the <a href="http://factfinder.census.gov/servlet/DownloadDatasetServlet?_lang=en">Download Center on the Census website</a> at the County Subdivision level, I've decided to roll up to the County level. The Census makes it easy to download variables for the entire country at this level and I am working to chart a couple of variables. If there any you an particularly interested in seeing just leave a comment.<br /><br />I've gotten the code ready for Counties and I wanted to share a screenshot of what the 48 contiguous states look like broken out at the County level. Real data soon!<br /><br /><a onblur="try {parent.deselectBloggerImageGracefully();} catch(e) {}" href="http://4.bp.blogspot.com/_zk-KTrrAVtI/Rf8s2mSg9bI/AAAAAAAAABk/Gs-wCzkIPF4/s1600-h/census_2000_counties_random_colors.jpg"><img style="margin: 0px auto 10px; display: block; text-align: center; cursor: pointer;" src="http://4.bp.blogspot.com/_zk-KTrrAVtI/Rf8s2mSg9bI/AAAAAAAAABk/Gs-wCzkIPF4/s400/census_2000_counties_random_colors.jpg" alt="" id="BLOGGER_PHOTO_ID_5043799424181007794" border="0" /></a>Unknownnoreply@blogger.com0tag:blogger.com,1999:blog-4291715832199091242.post-12029453361710750732007-03-14T19:37:00.000-05:002007-03-14T20:16:15.509-05:00Get your real, live examples!After only posting screenshots for the past few days, it is time to share some real, live KMZ files. I'm going to share 3 states worth of data: California, Massachusetts, and Wyoming. Before I share links to the actual files, I want to be sure to share some important definitions, data sources, and notes.<br /><br /><span style="font-weight: bold;">What is mapped?</span><br /><ul><li>I wanted to start simple: the following KMZs map the "Total Population" variable from the 2000 Census, broken out at the County Subdivision level. The height of each subdivision is equal to 1 meter for every 10 people. The colors also represent population, but use the state's max county subdivision population as the denominator, meaning they are only relevant within the state, not comparable across states. The largest subdivision will be bright red and the small ones will be white. I'm experimenting with this to try and get data on two levels - national and state.</li></ul><span style="font-weight: bold;">Where is the data from?</span><br /><ul><li>While I could have gotten the data from the <a href="http://factfinder.census.gov/home/saff/main.html?_lang=en">FactFinder</a>, I got the actual Total Population variable data from <a href="http://www.nhgis.org/">National Historical Geographic Information System (NHGIS)</a>. This system is powerful because, among other things, it lets you download the data for the entire US at once. The citation for this data is as follows:</li><ul><li>John S. Adams, William C. Block, Mark Lindberg, Robert McMaster, Steven Ruggles, and Wendy Thomas, National Historical Geographic Information System: Pre-release Version 0.1 Minneapolis: Minnesota Population Center University of Minnesota, 2004.</li></ul><li>The County Subdivsion boundary files came from a Census website: <a href="http://www.census.gov/geo/www/cob/cs2000.html">2000 County Subdivisions Cartographic Boundary Files</a>.</li><ul><li>The Census boundary files include data the denotes cut outs when a polygon should not cover an area. These are denoted with a -99999 ID in the Census boundary files. While I read these in, I have not decided the best way to handle them so that data is not represented here: in other words, some of the polygons may inappropriately cover an area.</li></ul></ul><span style="font-weight: bold;">Notes/Disclaimers:</span><br /><ul><li>The actual population (in people) is listed for each County Subdivision in the name for each polygon. The number listed here is the actually population divided by 10, which is the height of the polygon in meters. </li><li>This is a preliminary release.</li><li>I license the use of these files under Creative Commons and Commercial use of the data is prohibited by <a href="http://www.nhgis.org/">NHGIS</a>. The data can be gotten for other ways, so Commercial use is not out of the question in the future.<br /></li><li>Turn off Terrain for best viewing.</li><li>Rotate, fly around, change the viewing angle to get a real sense of the visualization!<br /></li><li>I do not warrant in any way, the accuracy of these maps. Use at your own risk.</li></ul><br /><span style="font-weight: bold;">Maps of Total Population (people) from the 2000 Census by County Subdivsion:</span><br /><ul><li>KMZs:</li><ul><li><a href="http://s3.amazonaws.com/G9CMqlwkvDJEyFnvEEeQ-CensusKML/cs06_d00.kmz">California</a></li><li><a href="http://s3.amazonaws.com/G9CMqlwkvDJEyFnvEEeQ-CensusKML/cs25_d00.kmz">Massachusetts</a></li><li><a href="http://s3.amazonaws.com/G9CMqlwkvDJEyFnvEEeQ-CensusKML/cs56_d00.kmz">Wyoming</a></li></ul><li>Google Maps (viewing the KMZ files in the Google Maps interface - may only work in some browsers):</li><ul><li>California - won't work - too large </li><li><a href="http://maps.google.com/maps?f=q&hl=en&q=http://s3.amazonaws.com/G9CMqlwkvDJEyFnvEEeQ-CensusKML/cs25_d00.kmz&layer=&ie=UTF8&z=8&om=1">Massachusetts</a> </li><li><a href="http://maps.google.com/maps?f=q&hl=en&q=http://s3.amazonaws.com/G9CMqlwkvDJEyFnvEEeQ-CensusKML/cs56_d00.kmz&layer=&ie=UTF8&z=7&om=1">Wyoming</a></li></ul></ul> Please pass along any feedback/thoughts/inquires via comments!Unknownnoreply@blogger.com3