Here is their home page

The package for now is really simple, but all of the packages  I’m building are getting simpler. In the end ( whenever that is ) I think I’ll end up with a host of packages that manage the downloading of data and the formating of it into “analysis friendly” formats.  The CRN posed an interesting challenge. They have hourly data and  over 30 measurands.  Going forward I’m seeing a collection of packages that looks like this: a series of packages that takes online climate data and reformats it into some standard formats that a few of us have been converging on.  In an OOP design they will become the core objects. Then we have a series of functions for doing basic spatial and time series stats, and we have our spatial tools and Time series tools.  Here is the package line up as of today

1. RghcnV3:   data formation and analysis

2. CHCN ( enviroment canada data )

3. Ghcndaily:  ghcn daily data

4. crn : climate reference network

5. Metadata.

Over time the goal will be to refactor RghcnV3 and strip out the analysis part of it into a separate package. All the data formating packages thus would have a common set of formats and objects and then analysis code would be written as methods on that.. eh well thats the dream

So here is what you can do with the crn package today. There are 3 core functions:  downloadCRN, collate*, and writeDataset().  The download function does all the heavy lifting to download both daily and hourly data from CRN. Data starts in the year 2000 and extend to today. The downloadCRN function uses RCurl to get the directory listings from the ftp and create the download lists. Then the process of downloading the 1000 + files starts.

The function lets you control whether you want daily files or hourly. I just get both.  the data comes in station files. One file for every station for every year.  The  next step  we take is to collate these files into one monolithic file. One file that contains the data for all the stations. For daily data we use collateDaily() and for hourly data we use collateHourly().  These functions have two side effects. They write a consolidated datafile and a metadata file that records station names, lat/lon and Id number. In the case of the hourly data this file is quite large over 1GB. Moreover the file contains all the variables:  T min, tmax, solar radiation, soil temperatures.  The last function turns these monolithic files into what we are used to. Files with one variable for all stations.  That function is  writeDataset(). The function operates on either hourly or daily files and collects  a single variable such as T_MEAN.  the function is defined like so

writeDataset(filename, cnames = colnamesDaily, varname = “T_DAILY_MEAN”)

The first variable filename  is supplied to point to either the monolithic daily data or hourly data. the next variable “cnames”  points to the column names of the dataset. These are predefined as constants for the package. the last variable “varname” is the variable you want to collect. When you run this function the side effect is a file is written contain all the T_DAILY_MEAN data for every station. Effectively the package allows you to download the CRN data and build subsets of data from the huge collection. There are over 30 climate variables available, so I constructed the tool so that you can build datasets from the source data.

Version 1.0 is posted to CRAN, should be up shortly.  In the next installment I’ll probably add support for “zoo” objects and function to create daily from hourly and monthly from daily. At that point it will be fully integrated with the RghcnV3 data structures

So quick demos when it hits

This package has been a long time in the making.  In the end it’s more of a data package than a functional package, but pulling all the pieces together required me to learn some really cool packages: raster ( which I already knew ) rgdal and RCurl.  I’ll provide a littlebit of an overview of what comes together on this package and why I built it, and then maybe some future directions.

In the course of looking at climate stations and the question of UHI I got kinda fascinated with the question of metadata: the variables that describe a stations location and physical features.  The goal of the project was to assemble a somewhat complete set of datasets to describe the geographical conditions of  any climate station.  I set out some requirements, first and foremost I wanted to use data that was open and not behind registration walls. That way I can write code to simply go get the data, download it and unpack it for the package user. I failed. In the end I ended up with a few datasets that require minimal registration headaches. I think with some help from RCurl experts I could tackle most of those issues. we will see.  Lets start by canvasing all the datasets I collected

1. GHRSST distance from coast dataset.  This 1km resolution file gives you the distance from the coast for all bodies of water.

2. NASA distance from Coast. This 1km dataset gives you distance from coast for  sea pixels only.

Dataset 1 and 2 along with some other files can be used to create a 1km land mask. Here I use them to tell if a station is on the coast or not.

3. Nightlights data.  radiance calibrated nightlights from  DSMP. a 1km dataset of radiance calibrated nightlights. This is the latest and greatest nightlights data with a much larger dynamic range than other products.

4. Impervious surface data. another 1km data product of impervious surfaces

5. Airports. 40K plus airport locations are rasterized into a 1km raster.  When I get time I’ll turn this into a distance map ( one call in raster) so that each cell contains distance to the nearest airport.

6. Harmonized Land Use data.  The package build a raster brick from 7 land use files  at 5 minutes resolution: urban land, cultivated land,irrigated land, rain irrigated land, forest, sparse vegetation, grassland.

7. Bluewater irrigation in 5 minute resolution: The amount of bluewater used for irrigation on the croplands in the grid.

8. Modis urban extent. 500 meter data. This file  requires registration and permission from the PI. Its freely given and I really wanted to use the new Moid data

7. Landcover. This is a previous Modis product that also requires registration, but its painless. Also cool because the data is in 72 tiles

8. Hyde population. Historical population density in 5 minute resolution. Also rural population counts and urban population counts. This requires regsitration

9. GPW population. Gridded population of the world in 2.5 minute resolution. Requires registration.

10. Grump urban extent. Urban extent at 1km resolution. requires registration.

With the exception of Modis I’m pretty sure if I were better with RCurl I could get those to download by filling in the registration forms programatically. Hyde is also easy except the server likes to download incomplete files and I should probably work on that a bit. For now, you have to download a few of the files manually.

The package has just a couple core functions that are tied to these files. The functions are primarily for doing complilations of datasets. The first function createRasters() just does the work of reading in files and creating native raster files. So for Hyde, ISA, landcover, airports, and Modis, I found it beneficial to reclass some of the datasets and save them as native rasters. That function takes a while to run but you only run it once. The next function if collateMetadata(). This function takes a dataframe of lat/lons and attaches metadata for every position. Obviously of you know raster you dont need any of this as you can just use the raster “extract” function to pull metadata from any of the assets.

test code just finished and it’s headed to CRAN in the morning.

Must sleep

Here’s another link providing some back ground

And here is a link to a paper on the subject http://www.astrophys-space-sci-trans.net/7/315/2011/astra-7-315-2011.pdf

That paper took a rather indirect approach to the problem looking at  changes to DTR and making two inferences. The first inference of course is that the change in DTR is related to Forbush events. The second inference is that the change in DTR was related to changes in cloud cover.

I’m going to take a more direct approach. The hypothesis is that decreased GCR (coincident with a Forbush event) should lead to a decrease in clouds and hence an increase in solar radiation reaching the surface.

The dataset  I have is collected  is from  the best climate stations in the US:  CRN

The data is solar radiation reaching the surface.  I’ve just started the test code, downloading all the datasets and getting the data into nice formats for plotting an analysis. I’ll give you a taste of it here:  On feb 19th, 2011 there was a Forbush event. Below, see the hourly solar radiation for 1 of 206 stations in the CRN database.

 And below find a sample of 9 stations ( ignore the date– my bad it says jan but its feb ) and an example of 25 charts

So,  while I wait on some bug fixes to be upstreamed for the metadata project, I may play around with the solar data.

readV3Data(filename=”foo”, output = c(“Array”,”Zoo”,”Mts”).  On ingest you decide what format you want to work in. If you change your mind there are a set of functions to handle transformation:  asZoo(), asMts(), asArray() and of course a set of logical functions to determine types. The core analysis functions have also been rewritten to accept ANY of these three object types. So, passesCamZoo() and passesCamMts()  etc have all been replaced with one function passesCam() that function accepts all three types of objects and just works. Some functions such as Roman’s function and Tamino’s function, and rasterizeZoo()  still have limited input: they require, for example, an Mts input or Zoo input. I’ll probably enhance those functions in another release and then it’s done.

With 2.0 thus you have these  kind of paths

readV3Data(); passesCam();anomalize();rasterizeZoo()  and you have stations selected by CAM and area averaged

readV3data(); averageStations() rasterizeCells() and you have stations estimated by Romans regression by grid cell

readV3Data(); inverseDensity();solveTemperature() and you have Nick stokes solution

and then you have Taminos approach as well. What we know faster all this is that the methods of computing averages for temperature stations yield the same global answers. There may however be slightly different answers if you look at smaller regions or have data that is too sparse in the temporal domain for CAM.

Next Steps:

There are several things I want to do and a few things I have to do.

1. Start work on GHCN Daily

2. incorporate Zeke’s paired station approach

3. Do some more work on CHCN

4. OOP. S4 classes and methods

5. Incorporate more of Nicks work

6. metadata package.

7. Demos and studies.

So,  it’s ok to write a scraper and possible to write one.  My goal was to do this in R.  I  really enjoy DrJ’s  code and his other work, but I had to try this on my own. The folks at Environment Canada suggested that scraping was the best option as their SOAP and REST mechanism wasnt entirely supported. That was fine with me as SOAP in R  wasn’t a good option. I won’t go into the reasons. So my plan of attack was to leverage a small piece of DrJs work and do the rest in R.

The Key master file is created by  one of DrJs scrapers and can be downloaded here as a csv.  At some point I will go duplicate that in R, but for now I rely on this file. The file lists the master list of all stations. It contains the two bits of information we need to scrape data: the station webId and the FIRST YEAR of monthly data.  So there is a function to get that csv file, cleverly named downloadMaster().  The next step is to read that file and select only those stations that report monthly data:  writeMonthlyStations().  The next step is to scrape the data: I looked at several ways of doing this. First I tried “on the fly” scraping. That means making a request, and then parsing the file into a  friendly R object. This was some nasty code since the csv file is in an unfriendly format. Metadata in the first 17 lines and then 25 columns of climate data. It took some fiddling but I was able to manage it. It involved doing two reads on the connection. The first read to just get the metadata and the second read to skip the metadata and read the data. That function worked if the server cooperated. Alas, the server had a habit of crapping out and dumping the connection.  The prospect of doing error trapping in R “trycatch()”  wasn’t in the cards.  But I suppose in a future version I will do that. So I oped for the brute force. Download every file. As it turns out that is  7676 csv files. The good new is that with a downloaded CSV file I can work at my leisure and not try to debug things hoping that the connection will time out so I can test the code.

Consequency we have a function scrapeToCsv()  which takes the list of stations and makes a http request for every one. That works pretty slick and takes a long time. Of course, that process is also prone to server timeouts. When it balks we have a function to clean up after that: this function getMissingScrape() looks at your download directory and the files there, looks at the list of stations you wanted to scrape and figures out what is missing. calling scrapeToCvs(get = getMissingScrape())  will restart a scrape and chug along.  When the scrapes are finished you have to do one last check  getEmptyCsv(). There are times when the  connection is made, the local file name is written, but no data is transmitted. So you get zero sized files. No problem, we detect that and rescrape:  scrapeToCsv(get = getEmptyCsv()). Clever folks can just write a while loop that exits on the conditions that all files are there and non empty.

After downloading 7676 files you then create a inventory with metadata:  createInventory() This includes the station name, lat, lon province. and various identifiers ( WMO etc).

      "Id" "Lat" "Lon" "Altitude" "Name" "Province" "ClimateId" "WMO" "TCid"
  99111111 "49.91" "-99.95" "409.40" "BRANDON.A" "MANITOBA" "5010480" "71140" "YBR"
  99111112 "51.10" "-100.05" "304.50" "DAUPHIN.A" "MANITOBA" "5040680" "" "PDH"

Then you create a huge master datafile  createDataset(). This has all the data ( temperatures, rain etc). Next you can just extract the mean temperature  asChcn() which creates a GHCN like data structure of temperature data.

version 1.1 is done and is being tested. Should hit CRAN when some outside users report back.

readCruStations <- function(filename=”stations-data.txt”, output = c(“Array”,”Zoo”,”Mts”)){

if (length(output) > 1) {
warning(“Select One of either V3 or ARRAY or Mts. Using first element by default”)
returnType <- output[1]
} else {
returnType <- output
}
X <- readLines(“station-data.txt”)

# suck out the headers and work on them to make an inventory
dex <- which(nchar(X[]) == 72)
headers <- X[dex]

#  build an inventory that RghcnV3 likes!

Inv <- data.frame(Id = as.numeric(substr(headers,1,6)),
Lat = as.numeric(substr(headers,7,10)),
Lon = as.numeric(substr(headers,11,15)),
Altitude = as.numeric(substr(headers,16,20)),
Name = substr(headers,22,42),
Country = substr(headers,43,56),
StartYear = as.numeric(substr(headers,57,60)),
EndYear = as.numeric(substr(headers,61,64)),
Source = as.numeric(substr(headers,67,68)),
GoodFirst = as.numeric(substr(headers,69,72)))

Inv$Lat[Inv$Lat == -999] <- NA
Inv$Lon[Inv$Lon == -1999] <- NA
Inv$Lat <- Inv$Lat / 10
Inv$Lon <- Inv$Lon / 10

Inv$Altitude[Inv$Altitude == -999] <- NA

begin <- min(Inv$StartYear)
end <- max(Inv$EndYear)
Years <- begin:end
stations <- nrow(Inv)

##  build a data array that RghcnV3 likes !
DATA <- array(NA,dim=c(stations,12,length(Years)))
dimnames(DATA)<-list(Inv$Id,month.abb,Years)

stationCount <- 0
for(line in 1:length(X)){
if (nchar(X[line]) == 72){
# we have a header. increment station counter ( its an array index)
stationCount <- stationCount + 1
} else {
# we have a data record.  use the regular expression  (” +”) to split the string into numbers
thisLine <- as.numeric(unlist(strsplit(X[line],” +”)))
thisLine[thisLine == -999] <-NA
thisYear <- thisLine[1] – begin + 1
DATA[stationCount,,thisYear] <- thisLine[2:13]/10

}
}
## Prepare for output
if (returnType == “Mts” | returnType == “Zoo”){
DATA <- apply(DATA,MARGIN = 1, FUN = c)
DATA <- ts(DATA, start = minYear, frequency = 12)
if (returnType == “Zoo”) DATA <- asZoo(DATA)
}
if (returnType == “Mts”){
return(list(Inventory = Inv,Mts = DATA))

}
if (returnType == “Zoo”){
return(list(Inventory = Inv,Zoo = DATA))
}
if (returnType == “Array”){
return(list(Inventory = Inv,Array = DATA))
}

}

Of course, then it is a simple matter to load that data into a raster and compute the area averaged global temperature:  What’s it show? Given CRU’s source data and given an algorithm that is patterned after their’s, They are cooler. Perhaps, they posted only the source data and not any adjustments they do.  These results match Nick Stokes who is also a bit warmer than CRU.

On with the forking. Nick Stokes and I exchanged a few emails looking at various ways of speeding up readV3Data().  During the course of that we found a small “bug” to report to NCDC. As I thought more about his data structure, a 3D array of stations, months, and years it became obvious that the whole package could be easily rewritten to support it. And I could bury forever the old 14 column v2 format. The way I’ve implemented that is an option on the readV3data() function. You specify which kind of object you want returned: a zoo object, mts object or an Array type object. Zoo and mts are just variants of each other with station data in columns and time in rows. Then I defined a series of functions asZoo(), asMts(), asArray(), which allow you to switch from one representation to another. Then I rewrote all the functions to accept any type of object. This cut down the number of actual functions I need and moved me closer to OOP. A couple of the functions, still accept only one type of data, for example Romans and Taminos, are still “mts/zoo” centric.  Some routines disappeared all together: windowV3() is gone and windowArray() takes it place, although you could do  test <- asArray(window(asZoo(Data),start =1920, end = 1940 +11/12).

so you can now  write  anomalize() and pass it a zoo, mts or Array. And  passesCam() replaces the several variants.  The demos have been rewritten and I added a couple test programs. Building a new package will take time as the manual will require much rewriting.

Then I had another little task. Rewriting Nick’s  weights program. The function is used to calculate a series of weights for the data array. These weights are a function of the monthly count of reports in a grid cell and an area weight for that gridcell. My main goal here was to see if I could rewrite the code using raster. Well, I did, but it wasn’t entirely clear and last bits of it were particularly nasty so I decided instead to stick with Nicks code, add some clearer variable names, and change it so that you can use any size grid cell. Here is what the algorithm does: It takes a all the temperatures series and  gives them a “cell” id,  exactly like the cell value in a raster. Every cell receives a weight that is proportional to its area, basically sin() of Latitude so that cells at the equatoer which are larger have a weight of 1 and cells toward the poles are weighted less. Then a “count” is established for every valid temperature measure in the cell for every month of every year.  So a given cell could have 5 reports one month and no reports the next month or 3 or 2, and so forth. Then the area weight is divided by the counts in that cell. And I refer to this as “inverse density”  the more reports, the less each report is weighted. The smaller the area, the smaller the weight.  As the function stands now, I’ve added a  raster as a calling parameter. I’ll probably change that to simply a cell size, as everything I need can be calculated from a cell size.  Here is the code. This function, of course, doesnt take a zoo or mts as a parameter, but that’s relatively easy to handle with the asArray() function, so I’ll probably change that as well before adding it.

inverseDensity <- function(inv, Data, r = GLOBE5){

rClass <- class(r)[1]
isRL <- rClass == "RasterLayer"
if (isInventory(inv) & isArray(Data) & isRL){
# check that inventory stations and Data stations match
if (!identical(getStations(inv), getStations(Data))) stop("stations must match")
# dimensions of Array
dimensions <- dim(Data)
# array of weights for output that looks like temperatures
weights <- array(NA,dimensions)

lonBin <- ncol(r)
latBin <- nrow(r)
halfLat <- latBin/2
eq <- halfLat * (lonBin +1) + (halfLat+1)
cellSize <- res(r)[1]
cellId <- floor(inv$Lat/cellSize) * lonBin + floor(inv$Lon/cellSize) + eq
#weights for latitude changes
latWeight <- sin((1:latBin-0.5)*pi/latBin)
#weights for every lat/lon
allWeights <- rep(latWeight, each = lonBin)
for(months in 1:dimensions[2]){
      for(years in 1:dimensions[3]){
      # get T/F if temperature is there
      reports <- !is.na(Data[ , months, years])
      # get the cell numbers of those cells with reports
      cells <- cellId[reports]
      # a count of reports per cell per month
      counts <- tabulate(c(cells, lonBin * latBin))
      # Ok is the logical of counts > 0
      ok <- counts > 0
      # make a copy of allweights to modify
      density <- allWeights
      # note only "ok" elements are valid
      density[ok] <- allWeights[ok] / counts[ok]
      #print(density[ok]) those True cells get a density area/count
      weights[reports, months, years] <- density[cells]

    }

  }
return(weights)
}
if ( !isInventory(inv) ) stop(" must be a valid inventory")
if ( !isArray(Data)) stop("must be a valid data array")
if ( !isRL ) stop("must be a valid raster")

}
 

We exchanged code and few mails thrashing through details and I’m now in a position to start the integration work of his approach into Rghcnv3. In addition to providing a new method Nick also contributed some code to make the import of data considerably faster. It’s a good lesson in some of the trade offs between code clarity and maintainability and raw speed. The function in question was readV3data().  My approach with that function was to build something that was maintainable, so the function relies on a set of constants or FILE.PARAMETERS.  While GHCNV3 was in beta I wanted to build something that could be changed easily by changing constants in one place rather than by editing code. So the function relies on read.fwf() and that function is controlled by constants in parameters. With GhcnV3  NCDC has added 34 extra columns of data. That drives the read time up to 8 minutes. By tweaking the “widths’ parameter I was able to get this down to 2.5 minutes. Nick’s approach BLEW THE DOORS off that tweak.  His approach comes at a small price. The price is code clarity and maintainability. After spending some time working back and forth we were able to fix a bug and make the code more clear  and robust. If NCDC decides to change format, then I’ll have to change code and not just constants. I don’t think that is likely. So I choose speed here.  How fast was Nick’s approach? 30-40 seconds.   The approach depends upon  using readLine() and then “manually” editing the lines using substr() to get the file into a format that can be read by  read.table().  That requires writing a temporary file. I think that step may also be unnecessary, but I’ll leave that for another day.

Next up was integrating the code that Nick produced for doing his method. If you’re unfamilar with his method please have a look at the link above . The unique thing about Nick’s approach, from my software perspective, was his data structure. Most people who work at this use a 2D data structure of some sort. You have time series. You have stations. That has to mean you should use a 2D structure ! So all of RghcnV3 looks to get the data into the right 2D structure  so that various functions can use them. Nick does something entirely different and it took me a while to wrap my head around it. Since the problem was bigger than my head that took some time and plodding. But the result has some wonderful properties for data manipulation which I will cover later.

The 3D data structure is an array. The dimensions are Stations, Months, Years. Picture a cube. The face of the cube has stations on the left, months across the top, and is years deep.  That’s a really nice arrangement for some of the things we want to do with station data.  Below find the code that takes a 2D v3 format ( 14 columns,  with a row per station) and creates a 3D structure. Most importantly, this code will insert missing years into the data, a function that v3ToZoo() and v3ToMts() perform. That opens a new possibility that I’ll show at the end.

Nicks Code, adapted and formatted to be more in line with R style guides:

</pre>
v3ToArray <-  function(v3Data){
 # author Nick Stokes
 # Steve Mosher: rewrite for style, interface, error checking and comments
 # check the input data and set up parameters for processing
 if  (!isV3(v3Data))  stop("v3Data must be a v3 object")
 stations <- nrow(v3Data)
 # the first year gets an offset for computing some indices below
 start <- min(v3Data$Year) - 1
 end <- max(v3Data$Year)
 # Inside a v3 format every station takes up a different number
 # of rows. we are going to process those rows in "chunks"
 # station index is a vector of where different stations start and stop.
 # a subset command could be used inside the loop but this quick
 stationIndex <- c(0, which(diff(v3Data$Id) > 0), stations)
 stationCount <- length(stationIndex) - 1
 # the array is stations by months by number of years
 x <- array(NA,c(stationCount,12, end - start))
 # I add dimnames because I like to use them to identify what is what
 # also, I can now easily reconcile a temperature data file with an inventory
 dimnames(x) <- list(unique(v3Data$Id),month.abb,min(v3Data$Year):max(v3Data$Year))
 # the tricky part. You grab a stations work of data and insert it into
 # the 3D array.
 for(i in 1:stationCount){ # loop over stations
 # this gets you the start row and end row for a station
 records <- c(stationIndex[i]+1, stationIndex[i+1])
 iy <- as.matrix(v3Data[records[1]:records[2], ])
 oy <- (iy[ ,2] > start) & (iy[ ,2] <= end)

 if(sum(oy) < 2 ) next
 iz <- iy[oy, ]
 x[i, , iz[ ,2]  -  start] <-  t(iz[ ,3:14])
 }
 return(x)
 }
<pre>

And now for the real treat.  Because we’ve increased the speed of readV3data(), we can actually create Nicks data structure when we read the data in. So  readv3Data()  now has an output format

readV3Data(filename, output = c(“V3″, “ARRAY”), Parameters = FILE.PARAMETERS)

That means you can read the data into the program in about 10 seconds using the “V3″ output format, or select the ARRAY output format and be all set to call Nick’s method with the data output by the red function. Basically, v3ToArray() gets called immediately after the read.

I’ve got one more test to run on this code and then it goes into version 1.6.  version 1.5 is on CRAN.  Version 1.6 will have Roman’s Method, a demo of romans method and Nick Stokes faster readv3Data().  The rest of Nick’s code is slated for 1.7.

PASSED.

I’ve succeeded in getting a version of RomanM and JeffId’s Thermal hammer working with version 1.3 of RghcnV3. This is going to be a long post because there is a lot of ground to cover. First, some errata, the “Globe” demo in V1.3 appears to have a missing line, looks like an editor bug, so I will have to push 1.4 to CRAN. CRAN has not been building packages for a bit, so I’ll post 1.4 here.

Roman’s method. Roman’s method is covered at a variety of posts at his site and at Jeffs. My goal here isn’t to explain his method in detail but rather to show how it will fit into RghcnV3,  Roman’s method works by using all the information in all the time series to come up with an  average of all the time series.  The function is currently called  ”temp.combine(), however that name will change as we move it into RghcnV3. I will also rename the function that uses Tamino’s method.  As it stands temp.combine() takes a “mts” object or multiple time series object and  a series of weights. It outputs a single time series, offsets for all time series, predicted values for all time series and residuals.  For the first iteration I will leave the weights in. In discussions with Roman he indicated that these weights could be used for distance weighting, but we won’t we using that in today’s work.

As I plowed through the code it became apparent to me that the paradigm of holding station data in columns ( the method used by me, roman and tamino) actually has some drawbacks especially when it comes to combining stations with ‘aggregate()’.  I’m not in the mood to change a bunch of code from column based to row based; and, its not simply a matter of doing a few transpositions. Practically what this means is that some of the code used in Roman’s method will use loops. I’m actively working to remove that, but loops work, so working code wins the day.

Plan of attack:

The goal for this exercise is to  use Roman’s method to calculate an average for a  grid cell. In the CAM method we use a simple ‘mean’ to calculate the average for a grid cell. That happens in the function “rasterizeZoo()”. That function takes an inventory of stations and a 2 dimensional Zoo object and “rasterizes” stations into the grid structure of a 3D brick. A brick is lat/lon/time.  Some side notes on rasterizeZoo(). RasterizeZoo() calls the raster function “rasterize”. Rasterize need the following inputs: a matrix of xy points (station lon/lat), a base raster to use for every layer, a matrix of values to rasterize into cells, and a function. The matrix has to have time in columns, stations in rows.  So we feed it a zoo object and we transpose that zoo object  ’under the hood’ inside the function. The lastest version of zoo just added a transpose function. arrg I needed that a couple months back. But again, I’m not rewriting a bunch of code, just yet.

Our plan then will be to call a function like rasterizeZoo().  However, it will be slightly different. RasterizeZoo() takes multiple stations and averages them per cell. When we use Roman’s method we will  do this ‘averaging’  BEFORE we call ‘rasterize’. Essentially we will call “rasterize” with “cells” rather than “points”.  We could also use the raster function “setValues()”  but  that function needs a little tweak to allow this, so I’ll work with rasterize. That’s our goal. Get a zoo data structure and associated CELLS  (not points) to feed to rasterize.  Our first function must then take a mutiple time series, divide the stations into the right CELLS, and then romanize those stations to come up with a cell time series.

Here we go:

averageByCell <- function(r =GLOBE5, inventory, Mts){
  # check classes code goes here
  # check to make sure that all the inputs are the right classes. To do.
  # get the cell number associated with each station. xy maps to a cell in the raster
  myCells <-  cellFromXY(r, getLonLat(inventory))
       # a obscure hack to get the stations associated with a cell!
       # used to get the column numbers for each cell
       # recall that every column is a station. so we will use column numbers to subset
  DF <- data.frame(Id =getStations(inventory), Cell = myCells)

  # now we get a cell count. we are going to loop over the unique cells
  cellcount <- length(unique(myCells))
  # we need the unique cells in a vector for referencing and for NAMING output columns
  uCells <- unique(myCells)
 # loop over cells. IF mts were transposed we could use aggregate
 # but that would entail rewriting all of romans code.. or maybe use mapply
  for( cell in 1:cellcount){
       # dex will hold the positions of stations in DF$Cell. its also the columns we need
       dex <- which(DF$Cell == uCells[cell])
       # call temp.combine NOTE I can also call taminos here or a simple mean
       CellAve <- temp.combine(Mts[ ,dex])
       if (cell == 1){
          out <-  CellAve$temps
        }else {
          out <- cbind(out,CellAve$temps)
          # COPY the cell number to the column name! Critical
          colnames(out)<-uCells[1:cell]
         }
   }
  # make it a zoo object! we want to pass a zoo object to rasterizeZoo()
  Zout <- as.zoo(out)
  # put out the xy for every cell. This will probably be dropped in production
  xy   <- xyFromCell(r,uCells)
  return(list(Points=xy,Zoo=Zout))

}

So that’s it. We have a function that we feed  the following values: a mts object. an inventory and a blank raster. That function will figure out which stations belong in which cells. It will then call “temp.combine()”  with  SUBSETS of the full mts object.  temp.combine(Mts[,dex]) uses “dex” to select those stations that have matching cell numbers. So temp.combine works on a grid cell at a time. Then all those individual time series ( grid series) are collected into a zoo object that has CELL NUMBERS as column names. This is different from zoo objects that have station Ids as column names.

Now that we have a zoo object of averaged grid cells, we will call a function like “rasterizeZoo”  but for clarity I’ll create a different function.  I’ll probably come up with a special class for this and have just once function, but for now we have two. Here is that function.

rasterizeCells <- function( ZooCells, r = GLOBE5){
  require("zoo")
  require("raster")
  # check the various calling parameters
  # check for a good cellsize

  #check Zoo as a zoo
  # stop if they dont match
  # proceeding otherwise
  if (!is.zoo(ZooCells)){
    cat("Zoo is a ", class(ZooCells), "\n" )
    stop("Zoo must be a zoo object")
  }
  # now we should check that the cells in the column names can be found in
  # the raster. What we really need is a tighter coupling so we KNOW the resolution matches: OOP someday
  cells <- 1:ncell(r)
  zoo.cells <- as.integer(colnames(ZooCells))
  common    <- intersect(zoo.cells,cells)

  matching <- identical(intersect(zoo.cells,cells),zoo.cells)

  if (matching == FALSE){
    print(" mismatching cells between Zoocell and raster")
    stop("check your Raster")

  }
  if (!class(r)[1] == "RasterLayer") stop("r must be a raster object")
  resolution <- res(r)
  if (resolution[1] != resolution[2])stop(" must use a raster with square cells")

  ###########  ready to process
  #  get the lon lats of Cells
  #  suck the time out of the zoo object
  #  transform the zoo object into a matrix with right orientation
  #  rasterize with mean. Need to see if we can change rasterize to merely copy.
  # also replace this code with zoo  transform.. requires LATEST zoo library, so change depends

  xy      <- xyFromCell(r,zoo.cells)
  TIME    <- time(ZooCells)
  # core data wacks the row names
  data    <- coredata(ZooCells)
  dimnames(data)[[1]] <- TIME
  tdata   <- t(data)
  gridded <- rasterize(xy, r, field = tdata, fun = mean, na.rm = TRUE)
  gridded <- setZ(gridded, as.yearmon(time(ZooCells)), name = 'time')
  return(gridded)

}

That’s it.  We now will be able to call  readV3data(),  v3ToMts(), averageByCell(), rasterizeCells()  and  the result will be a brick where every grid cell has the average temperature as computed by Roman’s method. Slick.

So I did a little test with Texas.  Here is what a month of texas data looks like: Please note the nasty image bug with R.2.13.1. yuk. Note also that I used a 3×3 grid to do this.

And if you compare  the answer you get with and without grid based averaging

Update: Sources and windows package in the drop box.   CRAN is a bit slow in getting around to things.

Update: AirVent Post source is in the drop box.

For a few hours last night I struggled with a bug, err several bugs. In the end some of the bugs were mine, some of them were “upstream” and some of them were  far “upstream”.  Image  bugs in R 2.13.1. But the math bugs ( or flat out errors on my part) have all been fixed. And the verification of the results is done.  version 1.3 should be available for download here in  bit, I just need to pretty up some code in the demos.

The process of using raster bricks  to create a global average is a bit tricky if you are not used to thinking  in simple operations.

Anyway, I worked my way through my own stupidity and ended up with this