Welcome to my Open Notebook

This is an Open Notebook with Selected Content - Delayed. All content is licenced with CC-BY. Find out more Here.


Reproducible ecosystem risk assessment as virtual desktop case study

Our paper on a computational environment for reproducible workflows for climate change assessments is available for free access for limited time until Jan 28 via https://authors.elsevier.com/a/1UBJF5c6cKey5~

  • Guru, S., Hanigan, I. C., Nguyen, H. A., Burns, E., Stein, J., Blanchard, W., Lindenmayer, D., Clancy, T. (2016). Development of a cloud-based platform for reproducible science: A case study of an IUCN Red List of Ecosystems Assessment. Ecological Informatics, 36, 221–230. http://doi.org/10.1016/j.ecoinf.2016.08.003.

In this project we re-implemented a previously created project from an ArcGIS and R scripted process into a reproducible workflow in Kepler. That original analysis was for a climate change risk assessment using the IUCN Red list of Ecosystems Assessment framework and was done for Mountain Ash forests located in the Central Highlands of Victoria, Australia. This served as our case study.

Our aim was to demonstrate how this could be implemented as a more reproducible workflow and how it could be disseminated by creating a standalone computation environment for sharing the entire operating system and toolchain along with the work. The IT infrastructure we developed offers analysis tools as a “Platform as a Service” in a virtual desktop offered as a “Desktop as a Service” at https://www.coesra.org.au and new features are under development at https://portal.coesra.org.au. A R package of generic functions was created https://bitbucket.org/coesra/iucnecosystemriskassessment/src, and execution is possible as a pipeline of R scripts https://bitbucket.org/coesra/iucn_ecosystemriskassessment_mountainashforests as well as a Kepler workflow.

I made heavy use of the flowchart R function 'newnode' which I wrote as part of my PhD project (and is distributed in my own misc R package available on Github https://github.com/ivanhanigan/disentangle. This function takes a simple data.frame of steps, inputs and outputs and returns a string of text written in the dot language which can be rendered in R using the DiagrammeR package, or the standalone graphviz package. This creates the graph view shown below.

Alternative text - include a link to the PDF!

Posted in  reproducible research pipelines reproducible research reports reproducible research cloud building

One Health Ecohealth Conference Melbourne Dec 2016

I've just come out of an amazing 5 day conference on One Health (where animal health science meets human health science, and a whole lot of ecological thinking).

It was truly terrifying to hear about the cascading risks to health from environmental change. I was deeply affected by the story of monitoring worm infestation rates in children working in mines in the DRCongo, and saddened by the mass death of coral this year on the barrier reef (among many other tragic stories).

There was good news too:

  • Pepper and cinnamon fed to broiler chickens in India has been found to have antimicrobial properties
  • Bhutan have made progress with rabies
  • Yanks working with Ozzies are making progress on Hendra antibodies/vaccine for horses and humans
  • I met some very inspiring early career researchers and students
  • I met some equally inspiring senior scientists
  • My drought index/Government Drought Declarations poster was viewed by the former head of the Gov. Department who was in charge of the drought declarations (we agreed that the underlying governmental decision rules are vague and multipurpose, and a climate index has great value when looking at human ecology)

I also gathered some AVA CPD points (whatever they are!) as alerted by this certificate they sent me.


Posted in  training

UPDATE re Ripping data from georeferenced images with QGIS

  • Thinking about this again, it should be possible to use the pixel values to get the drought areas based on the colour.


Posted in  drought awap grids data operation

Ripping data from georeferenced images with QGIS


Drought Declarations are made by government in Australia to determine when and where a drought is occuring. The Declarations are linked to agricultural and social support policies. We are working on a climatic drought index that should correlate with these Drought Declarations. The New South Wales Government has provided us with spatial data from 1986 but they also have a graphical visualisation available for earlier times, especially noteworthy is the extreme 1982-1983 drought. This post is a document of the process I will try to derive spatial data from the image.



Download the drought maps. The current maps are at: http://www.dpi.nsw.gov.au/content/agriculture/emergency/drought/planning-archive/climate/advance-retreat

But there is a higher resolution map archived at: http://pandora.nla.gov.au/pan/120345/20120529-0000/advance-retreat-drought-map-april-2011.pdf



Step 1 - load the reference layer

  • Load up the NSW boundaries by selecting "Add PostGIS Layers" > Connect > gislibrary
  • Then select a map layer of NSW

Step 2 - set up the raster data into the Georeferencer

  • Launch the Georeferencer from Raster > Georeferencer > Georeferencer.
  • If you do not see that menu item, you will need to enable the Georeferencer GDAL plugin from Plugins > Manage and install Plugins > Installed.
  • Zoom in on the month of interest
  • Now click on the Add Point button on the toolbar and select an easily identifiable location on the image. Corners, intersections, poles etc. make good control points.
  • Once you click on the image at a control point location, you will see a pop-up asking you to enter map coordinates. Click the button From map canvas.
  • Find the same location in your reference layer and click there. The coordinates are auto-populated from your click on the map canvas. Click Ok. Similarly, choose at least 4 points on the image and add their coordinates from the reference layer.
  • Now go to Settings > Transformation settings (yellow cog wheel).
  • Choose the settings as shown below.
  • Transformation type = Thin Plate Spline
  • output raster = someting
  • target EPS = 4283 (match what your reference layer is)
  • Make sure the Load in QGIS when done button is checked. Click OK.
  • Back in the Georeferencer window, go to File > Start georeferencing. This will start the process of warping the image using the GCPs and creating the target raster.

Step 3 - rip the data to a shapefile

  • create a new shp (Layer > New)
  • type = ploygon
  • Specify CRS = EPSG:4283
  • attribute = drought, text
  • save to a shp like 'drt198610rip' and OK
  • right-click, toggle editing, add new feature
  • save layer edits



This work


Compared to NSW DPI drought declarations data



The process produces data that is very prone to imprecision. We may be able to use some extra data from before 1986 though.

Posted in  drought awap grids data operation



  • The old DROUGHT-BOM-GRIDS data came from a Barnes IDW algorithm, at 25km (actually 0.25 decimal degrees) resolution
  • 1890-2008
  • Going forward the AWAP grids are Preferred
  • this is a quick and dirty approach


  • AWAP grids are 5km (0.05 dd) but this will take longer to process and is probably overkill for monthly total rainfall when focus is on drought
  • For the centroid point of each 0.25 dd extract the data value (might be better to take mean of sub-cells within each, plan for future)
  • compare this with old BOM GRIDS



compare with bom grids for Jan 2000


For a pixel (west wyalong my old home town)


Posted in  drought awap grids