These instructions will assume a Windows environment, but note that these steps can be easily adapted for macOS or Linux.

These instructions assume Docker is installed (including Docker Compose) and that the reader is familiar with basic Docker usage.

Download and save the following file into a new folder: https://raw.githubusercontent.com/nesp-tsr3-1/tsx/master/containers/docker-compose.yml

Run the following command inside the folder:

After loading and starting the necessary containers, you will be presented with a bash prompt which you can use to follow along with the rest of this guide.

Before observation data can be imported, a taxonomic list must first be imported which identifies all valid taxa that will be processed by the workflow. A sample taxonomic list containing Australian birds can be found in sample-data/TaxonList.xlsx.

The Taxonomic list file format is a useful reference if you want to build your own taxonomic lists for use in the workflow.

Import the sample taxonomic list:

If the import is successful, the command will complete without any output.

Type 1 observation data may now be imported into the database. Some sample Type 1 data can be found in sample-data/type_1_sample.csv. Import this data, by running the following command:

The --type 1 part of the command tells the import script that you are importing Type 1 data. This is important because Type 1 data has different requirements and is stored in a separate database table compared to Type 2 data. The -c flag is short for “commit” and causes the imported data to be committed to the database; without this flag, the command only performs a “dry run” and does not modify the database. This feature is also present in most of the data processing scripts, and is a useful way to test whether the data/processing is valid without actually making any change to the database.

The import script will run a range of checks on the imported data, which will generate warnings and/or errors. Warnings are advisory, while errors will prevent the data from being imported until they are fixed. This helps to ensure data quality.

The Type 1 data is now imported into the t1_survey, t1_sighting and t1_site database tables and is ready for data processing. You may choose to skip the rest of this section, which deals with importing Type 2 data, and proceed directly to Data Pre-processing & Filtering.

Type 2 data is imported in much the same way as Type 1 data. Sample Type 2 data can be found in sample-data/type_2_sample.csv. Import this data by running the following command:

This imports data into the t2_survey and t2_sighting tables. (Note that the t2_site table is populated in a separate step because Type 2 data does not have sites defined in the raw observation data).

During data processing, all observations are matched to Interim Biogeographic Regionalisation subregions (SubIBRA regions). The Interim Biogeographic Regionalisation for Australia (IBRA), Version 7 classify Australia’s landscapes into 89 large geographically distinct bioregions based on common climate, geology, landform, native vegetation and species information. Within these, there are 419 subregions which are more localised and homogenous geomorphological units in each bioregion. Observations outside of SubIBRA regions are suppressed from the final output.

Import the SubIBRA regions into the database:

Observation data is aggregated to monthly resolution by grouping all records with the same month, taxon, data source, site, method (search type) and units of measurement.

Each group of records is aggregated by calculating the average value, maximum value or reporting rate (proportion of records with a non-zero value) of the individual records. Which of these three aggregation methods are used for each grouping is determined by the “Processing methods” file which specifies which aggregation method should be used for each taxon/source/method/unit combination.

For further information, see Processing methods file format.

After monthly aggregation, the data is then aggregated to yearly by averaging the monthly aggregated values.

The sample processing methods file is available at sample-data/processing_methods.csv. Import this by running:

To aggregate the Type 1 data by month and year, run the following command:

This will aggregate the data and put the result into the aggregated_by_year and aggregated_by_month tables.

You may now choose to skip to Calculate spatial representativeness (Type 1 & 2 data) if you are not processing Type 2 data at this stage.

Type 2 data typically contains presence-only data, however the LPI analysis requires time series based on data that include absences (or true zeros indicating non-detections). To solve this problem we need to generate (pseudo-)absences for surveys where a taxon was not recorded but is known to be sometimes present at that location. In order to identify areas where the pseudo-absences should be allocated for a species, we first generate an alpha hull based on all known observations of the species. The alpha hulls are drawn from all Type 1 and 2 data, as well as an “Incidental sightings” file which contains observation data that did not meet the Type 1 or Type 2 criteria but is still useful for determining potential presences of a taxon.

After generating these alpha hulls at the species level they are trimmed down to ultrataxon polygons by intersecting with expert-curated polygons of known taxon ranges that are defined in an auxiliary “Range Polygons” input file.

There is a sample incidental sightings input file at sample-data/incidental_sightings.csv, which can be imported by running:

There are sample range polygons at sample-data/spatial/species-range/, which can be imported by running:

The specifications for both the incidental sightings file and the species range polygons can be found in the Appendix.

After importing these files, you can then run the alpha hull processing script:

This will perform the alpha hull calculations, intersect the result with the range polygons, and places the result into the taxon_presence_alpha_hull database table.

Now that alpha hulls have been calculated for each ultrataxon that we are interested in, we can proceed with aggregating the type 2 data.

This step of the workflow makes these assumptions about type 2 data:

The workflow performs the following processing steps for each ultrataxon of interest:

Note that the mean abundance calculation includes pseudo-absences as zeroes, rather than simply being a mean of counts of sightings with abundance units.

The type 2 aggregation we have just described can be performed by running the following command

This will aggregate the data and put the result into the aggregated_by_year and aggregated_by_month tables, alongside any existing aggregated type 1 data.

Spatial representativeness is a measure of how much of the known range of a taxon is covered by a given data source. It is calculated by generating an alpha hull based on the records for each taxon/source combination, and then measuring the proportion of the known species range that is covered by that alpha hull.

This step requires the range polygons file to be imported first. If you skipped to this section from the Type 1 data aggregation step, then you will need to import this now. A set of sample range polygons can be imported by running:

The spatial representativeness processing can now be run with this command:

This will produce alpha hulls, intersect them with the taxon core range, and populate them into the taxon_source_alpha_hull database table. The area of the core range and the alpha hulls is also populated so that the spatial representativeness can be calculated from this.

Note that both Type 1 and Type 2 data (if imported and processed according to all the preceding steps) will be processed by this step.

The final step before exporting the aggregated time series is to filter out time series that do not meet certain criteria.

The time series are not actually removed from the database in this step, instead a flag called include_in_analysis (found in the aggregated_by_year table) is updated to indicate whether or not the series should be exported in the subsequent step.

The filtering criteria applied at the time of writing are:

In order to calculate these filtering criteria, data source metadata must be imported (See Data sources file format).

Sample metadata can be imported by running:

The time series can then be filtered by running:

The data is now fully processed and ready for export into the “wide table” CSV format that the LPI analysis software requires.

To export the data, run:

This will place an output file into sample-data/export/lpi-filtered.csv.

This file is ready for LPI analysis!

It is also possible to export an unfiltered version:

or a version aggregated by month instead of year:

It is possible to run all the data pre-processing & filtering in a single command:

After which you must export the data again, e.g.:

This is useful when you import some updated input files and wish to re-run all the data processing again.

Alternatively, you can run the LPI calculation directly on the command line:

This will generate the result data in a file named infile_infile_Results.txt, but it will not display a plot.

To run the entire workflow, including the LPI calculation, run the following command:

The TSX virtual machine shares its files using network sharing so that you can access them in the same way that you would access files from another computer on your network. Don’t worry if your computer isn’t connected to an actual network, these steps should work regardless.

To access the TSX files, open File Explorer and go to Network. If you see an error message ("Network discovery is turned off…."), you’ll need to turn on Network discovery to see devices on the network that are sharing files. To turn it on, select the Network discovery is turned off banner, then select Turn on network discovery and file sharing.

You may see a TSX icon appear immediately, but if not, try typing \\TSX into the location bar near the top of the window and pressing enter.

You should now be able to browse and edit files in the TSX virtual machine. If not, try the following: - If you have only just started the virtual machine, try waiting for a few minutes before retrying the steps above. - Report an issue at https://github.com/nesp-tsr3-1/tsx/issues - Try the alternative method, Accessing files via SFTP

An alternative to network sharing is to access the files over SFTP.

First you will need to download an SFTP program, such as WinSCP. (Download WinSCP)

Start WinSCP, and in the Login Dialog, enter the following details:

Then click Login to connect.

You should now be able to browse the TSX files. Unlike the networking sharing method, you can’t edit the files directly on the virtual machine. Instead you will have to edit the files in a folder on your computer’s hard drive, and download and upload files from the virtual machine as necessary.

File format: Excel Spreadsheet (xlsx)

Sample file: TaxonList.xlsx

File format: CSV

Sample file: processing_methods.csv

File format: CSV

Sample file: incidental_sightings.csv

File format: Shapefile

Sample files: spatial/species-range/*

Format: Shapefile

Sample file: spatial/t2_site.shp

Format: Shapefile

Sample file: spatial/10min_mainland.shp

No columns required

Citation: Australian Government Department of the Environment and Energy, and State Territory land management agencies. 2012. IBRA version 7. Australian Government Department of the Environment and Energy and State/Territory land management agencies, Australia.

Format: Shapefile

Sample file: spatial/Regions.shp

Format: CSV

Sample file: data_sources.csv

Type 1 data must satisfy the following requirements:

Type 2 data must satisfy the following requirements: