• A colleague sent me the cwt_data_subm_template_2013.xls today
• You can download a copy here coweeta.uga.edu/resources/forms/cwt_data_subm_template_2013.xls
• LTER is The U.S. Long-Term Ecological Research (LTER) network
• I made the following notes, this is not intended to be a nasty critique
• The following is a few Frank and Fearless comments I’ll be using to compare the pros and cons of a variety of data documentation approaches

Critique

• opened first on windows, saw comments on cells with instructions
• opened next on linux with libreOffice and comments are gone
• opened at the last tab (split in two for no reason?)
• noticed recommended name “GCE site” = Site, otherwise “permanent plot” = Plot?
• GCE = Georgia Coastal Ecosystems LTER program
• flip to first tab, point 4 suggests there is some export functionality I cannot see (a VBA script?)
• cell 11 a NOTE: When submitting updated metadata or re-using templates please highlight fields with modified contents in yellow
• and use glitter pen???
• personnell tab OK
• instrumentation, variable measured is free text. ok but for eg “max temp”, “temperature maxima”, “maximum temperature (c)” “maximum temperature in 24 hours after 9am local time in degrees” etc
• too wide, last column was off my wide screen! noticed wasted real estate in column A
• tabular data “– Paste or enter your data values into the ‘Values’ section (white cells), starting with the indicated cell”
• this is an invitation for clerical error! Too many “copy-and-paste” actions will inevtably introduce errors
• I do like the extra metadata Column Name: – Description: – Units: – Data type: – Variable type: – Number type: – Precision: – Code values: – Calculations: – QC: Minimum Valid: – QC: Minimum Expected: – QC: Maximum Expected: – QC: Maximum Valid: – QC: Custom: – Fill in missing values in the table with NaN (not a number), including text fields, and do not skip columns
• but what about missing values imbued with other meanings (NA = not observed, censored etc)?
• ask users to format digit rounding in Excel?? oh no
• old excel users may still be restricted to 65,536 rows by 256 columns.
• non tabular sheet is ok

• The first dataset I downloaded from ASN for playing around with was the Tumba Lidar.
• I had thought it might be better to offer this as a OGC service rather than downloadable geotiff
• just in terms of the size firstly (104MB)
• but I also soon realised it would need some specialised tweaking which non-GIS users might struggle a bit and could avoid if the serverside data is set up by a GIS specialist (although can we assume only GIS specialists will download this kind of data)?
• kudos to http://stackoverflow.com/questions/11966503/how-to-replace-nas-in-a-raster-object

Code:tumbarumba-supersite-dem

setwd("~/data/supersite/tumba-lidar")
require(raster)
fname  <- dir(pattern = "tif$")
fname
r  <- raster(fname)
str(r)
dfr <- as.data.frame(r)
summary(dfr)
# the -1 code looks like it might be missing?  They are around the edge.
rna <- reclassify(r, cbind(-1, 1197))
png("tumba-lidar.png")
plot(rna, col=terrain.colors(100), xlab = "eastings (m)", ylab = "northings (m)")
title("Tumbarumba Supersite Digital Elevation Model")
title(sub = "packageId=lloyd.374.2")
dev.off()

Alternately use a geoserver

• If we use a Geoserver we could set it up so people can view this without downloading the data
• following the instructions http://suite.opengeo.org/opengeo-docs/processing/contour/setup.html
• using the ANU GIS library server

;

• PS the map might take a minute to show up, not sure why, might ask the sysadmin to look at the server performance

“reml now extends R’s data.frame class by introducing the data.set class which includes additional metadata required by EML” https://github.com/ropensci/reml

and “I’d like to define a class that acts just like a data.frame, just like the data.table class does, but contains some additional metadata (e.g. the units associated with the columns) and has some additional methods associated with it (e.g. that might do something with those units) while also working with any function that simply knows how to handle data.frame objects. How might this be done?” http://carlboettiger.info/2013/09/11/extending-data-frame-class.html

Also this guys attempt was interesting (I like TraMineR too!) http://ivanhanigan.github.io/2013/11/handling-survey-data-with-r/

• John Myles White invented the ProjectTemplate R Package
• This is a great application that helps streamline the process of creating a data analysis project
• Recently John posted about some tips for best practices for statistical programming

Best Practices for Statistical Programming

• Write Out a Directed Acyclic Graph (DAG)
• Vectorize Your Operations
• Profile your code and understand where it spends its time
• Generate Data and Fit Models
• Correctness: always ensure that code infers parameters of models given simulated data with known parameters.

Additional suggestions

• Unit Testing (use testthat)
• Create modular code with discrete chunks
• Write functions as much as possible, put these into a personal ‘misc’ package