Home > Uncategorized > V3 Station mash up

V3 Station mash up

Hmm, lets see how a fusion table works with map maker. The process works like this. We read in the GHCN inventory file into R. ( see code below ) We then clean it up ( the names fields and missing values) and we output it to a  *csv  file. Then open the csv in excell and save it as XLS. ( I can avoid this step in the future by exporting xls direct ) Anyways, when you have an xls, you import that into a Fusion table.

http://tables.googlelabs.com/DataSource?dsrcid=267862

Then we can select or filter for Mountain valleys and export the KLM link:


url_GhcnV3 <- "ftp://ftp.ncdc.noaa.gov/pub/data/ghcn/v3/ghcnm.latest.qcu.tar.gz"

v3Widths <- c(11,9,10,7,31,5,1,5,2,2,2,2,1,2,16,1)

v3Names <- c("Id","Lat","Lon","Altitude","Name","GridEl","Rural","Population",
"Topography","Vegetation","Coastal","DistanceToCoast","Airport",
"DistanceToTown","NDVI","Light_Code")

# then a bunch of utility code to download the  *gz, gunzip it, untar it and grep for the

# inventory file name. or you can unzip it by hand and just copy down the file name. When the whole

# process is automated i'll post complete code. Below find the function to clean the data up. It returns

# a data.frame which you merely write out as a  table or csv file.

setUpV3Inv <- function(filename){

ghcn_inventory=read.fwf(filename,widths=v3Widths,comment.char="",col.names=v3Names)

##################################################
# strip the commas out of names
# strip commas from other feilds
# write in  NA for -9 Values and -999 values
# Use True/False for airports
##################################################
ghcn_inventory$Name<-gsub(",","",ghcn_inventory$Name);
ghcn_inventory$Name<-gsub("/"," ",ghcn_inventory$Name)
ghcn_inventory$Name<-gsub("[(]"," ",ghcn_inventory$Name)
ghcn_inventory$Name<-gsub("[)]"," ",ghcn_inventory$Name)
ghcn_inventory$Name<-gsub("[']"," ",ghcn_inventory$Name)
ghcn_inventory$Name<-gsub("[.]"," ",ghcn_inventory$Name)
ghcn_inventory$Name<-gsub("[&]"," ",ghcn_inventory$Name)
ghcn_inventory$Name<-gsub("  ","",ghcn_inventory$Name)
ghcn_inventory$NDVI<-gsub(",","",ghcn_inventory$NDVI)
ghcn_inventory$NDVI<-gsub("[.]"," ",ghcn_inventory$NDVI)
ghcn_inventory$NDVI<-gsub("/"," ",ghcn_inventory$NDVI)
ghcn_inventory$Altitude<-gsub("-999","NA",ghcn_inventory$Altitude)
ghcn_inventory$Population<-gsub("-9","NA",ghcn_inventory$Population)
ghcn_inventory$Vegetation<-gsub("xx","NA",ghcn_inventory$Vegetation)
ghcn_inventory$DistanceToCoast<-gsub("-9","NA",ghcn_inventory$DistanceToCoast)
ghcn_inventory$Airport<-gsub("A","TRUE",ghcn_inventory$Airport)
ghcn_inventory$Airport<-gsub("x","FALSE",ghcn_inventory$Airport)
ghcn_inventory$DistanceToTown<-gsub("-9","NA",ghcn_inventory$DistanceToTown)

return(ghcn_inventory)

}

About these ads
Categories: Uncategorized
  1. curious
    October 7, 2010 at 5:13 AM

    Looks good. Just wondering if the lat and long coordinates are truncated to 2dp in some export/import step? I quickly tried a couple of stations (started with Funchal) and then zoomed but station location showed up as off the coast. If the lat and longs are available more accurately and this translates to the Google base ok I think it would be worth turning up the dp?

    • Steven Mosher
      October 7, 2010 at 8:43 AM

      this is a known problem with the metadata. The data are exported without any loss of precision. The problem is they are not accurate to begin with. the US data is more accurate, but even there there are issues. There are also GE issues, I’ve corrected one station ( Nuuk in Greenland) placing a pin exactly on the stevenson screen and returned to see it shifted into the sea.

      The coastal stations are the ones where this issue becomes most evident.

  2. curious
    October 7, 2010 at 10:01 AM

    ok – thanks for the extra info.

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 36 other followers

%d bloggers like this: