Data stewardship for long-term field experiments

Data management for long-term field experiments is more than storing spreadsheets and laboratory results. It is the disciplined work of preserving context, quality and comparability across seasons, researchers and instrument changes. A trial begun with one treatment design may outlive its original protocol, funding agreement and field team, yet its early observations often become the baseline against which later decisions are judged.

The IASSIST 2017 meeting at the University of Kansas in Lawrence showed why data work benefits from a common language across disciplines. Held from May 23 to 26 under the theme “Data in the Middle: The Common Language of Research,” the event brought together work on big data, deep learning, digital agriculture and global food security. Its preserved program and schedule remain a useful example of making specialist knowledge, session details and shared methods accessible to people working beyond a single discipline.

Australian field research adds particular demands. Trials around Wagga Wagga or Yanco may cross a severe drought, a timely winter rain and a flood in one project lifetime, while sites near Toowoomba can alternate between fragile soils and intense storms. Heat, dust, termites, lightning and unreliable mobile coverage can affect both equipment and data transfer. In Australian farm conversations, the relevant unit is often the “paddock,” a season is understood as a whole run of weather and growth, and a dataset is trusted when it tells growers what happened in the soil rather than merely publishing a number.

Why longitudinal records need active stewardship

The value of a longitudinal record changes with time. A clean baseline may be enough for a short trial, but a sustained record can reveal treatment-by-year interactions, lagged responses and the frequency of rare events. It can also expose whether an apparent trend is biological, managerial or an artefact of changed measurement. That is why a database needs a history of protocols, personnel, equipment and decisions, not just a table of current values.

Common failures are quiet rather than spectacular. A sensor is moved but the location field is not updated; rainfall is entered in millimetres while another column records inches; a missing reading becomes zero; or a researcher replaces an implausible value without preserving the original. Each shortcut reduces confidence and may be invisible to the person analysing the file years later. Version control, controlled vocabularies and explicit quality flags make such actions visible and reversible.

Stewardship also has to survive staffing turnover. The technician who knows why a plot was fenced differently, the agronomist who recognises a suspected sensor fault and the data manager responsible for a repository should not have to be the same person. Written handover notes, training records and a named custodian preserve institutional memory. For grower partnerships, this continuity is commercially important as well as scientifically valuable, because a trial report may inform a district management decision years after the original contract ends.

Designing a resilient Australian data architecture

A resilient architecture begins with identifiers that do not change when a project is renamed. Every dataset, site, plot, treatment block, management event, sample and instrument should have a stable identifier, with relationships recorded in a data dictionary. Australian datasets should also distinguish state, district, latitude and longitude, local time zone and daylight-saving offset. Soil descriptions can use the Australian Soil Classification, while management records capture cultivar, sowing date, grazing, fertiliser, irrigation and harvest.

Storage should combine copies rather than rely on one laptop or one cloud account. A 3-2-1 approach, with an encrypted institutional copy, another geographically separate copy and a restricted working copy, suits institutions that manage public and commercial data. Offline field capture should synchronise opportunistically, and critical records should be reconciled against a checksum or row count. Because commercial confidentiality is common in Australian agronomy trials, access controls and data-use agreements should be planned before collection, not imposed after publication.

A minimum record set should be established before the first observation enters the system:

• A persistent project and dataset identifier, plus a versioned title
• Stable identifiers for sites, plots, treatments, samples and instruments
• A protocol history with dated amendments and reasons for changes
• A data dictionary covering units, codes, spatial references and missing values
• Provenance linking raw observations to cleaning, transformation and analysis steps
• Agreed retention, review, access and decommissioning dates

Metadata, governance and shared meaning

Metadata turns files into evidence. Machine-readable metadata should state the objective, sampling design, spatial coverage, temporal coverage, variable definitions, methods, instruments, calibration status, software versions, processing decisions and known limitations. For ecological observations, Darwin Core or an agreed domain schema can provide a foundation; for operational sensor streams, a documented API and time-series format may be more useful. Whatever the standard, the data management plan should record how records will remain findable, accessible, interoperable and reusable without requiring the original researcher.

Governance must answer practical questions about authority and access. Who owns a dataset? Who may approve a release? Who corrects an error, and how is the correction disclosed? A trial sponsor, host institution, data custodian and data steward may have different responsibilities, while grower committees may expect aggregated results rather than plot-level records that reveal commercial performance. Agreements should cover intellectual property, publication rights, confidentiality, personal information and the return of findings to participants. Reviewing these arrangements every few years prevents them becoming disconnected from current staff and technology.

This need for shared meaning is visible in the archive of the IASSIST meeting. The IASSIST 2017 program sets out plenaries and presentations across big data, deep learning, digital agriculture and global food security, while the wider site preserves the schedule, check-in guidance, transport information and local recommendations. Technical content and practical logistics belong to the same record of an event: one helps participants interpret the science, and the other helps them act on it. The same principle applies to field experiments, where a treatment result cannot be separated from how the trial was run.

Instruments, calibration and defensible measurements

Instrument records need a chain of evidence. Soil-moisture probes, weather stations and temperature loggers can drift, fail or record values under conditions outside their calibration range. Store serial numbers, installation dates, firmware, calibration coefficients, certificates and maintenance actions with the observations. Preserve raw readings as received, then derive corrected or calibrated values through documented code. If a station is relocated, record the new coordinates, elevation, surrounding vegetation and the overlap period rather than treating it as the same site.

Procurement decisions should include local servicing and calibration support, not only purchase price. When a project is shortlisting equipment, the monitoring equipment suppliers under consideration should be checked for compatibility with existing loggers, availability of replacement parts in regional centres and whether calibration procedures can be transferred between field teams. Australian conditions make these issues visible: sealed boxes still age under heat and ultraviolet exposure, cables are vulnerable to machinery and termites, and batteries are less reliable through long hot periods.

Quality assurance should operate continuously rather than wait for final analysis. Automated checks can flag impossible ranges, sudden jumps, frozen sensors, duplicate timestamps and disagreement between nearby instruments; trained staff then review those flags against field notes. Keep uncertainty estimates, detection limits and a reason code for every exclusion. Synchronise clocks, store timestamps in UTC while retaining the local offset, and scan paper notebooks soon after collection. These practices retain the audit trail required for publications, regional recommendations and later machine-learning models, none of which can compensate for undocumented transformations.

Operational checklists for Australian field teams

Two recurring checklists help a project turn good intentions into repeatable practice. The first is designed for mobilisation and annual planning; the second is used during closeout and archiving.

Before the field season begins

• Confirm dataset ownership, access controls, storage locations and the three-copy backup
• Issue stable identifiers for new sites, plots, treatments, samples and instruments
• Record protocol versions, amendments, staff training and planned calibration dates
• Test offline entry, time-zone conversion, validation rules and synchronisation on every device
• Prepare concise field sheets that match the database fields and data dictionary
• Brief growers, contractors and students on documentation, privacy and incident reporting

At project closeout

• Reconcile raw files with validated tables and document every transformation
• Mark missing, estimated, corrected and deleted observations with explicit reason codes
• Deposit the final dataset, metadata, code, protocols and data dictionary in the agreed repository
• Obtain persistent identifiers and cite the archive version used in reports and publications
• Set review dates for embargoed data and preserve the contact details of the custodian

Australian field experiments are strongest when stewardship is designed as an Australian partnership rather than an administrative afterthought. Research organisations, universities, Rural Research and Development Corporations, agronomy consultancies, commercial agronomy firms and local grower groups each hold part of the evidence chain. Their records become most useful when responsibilities are explicit, local knowledge is retained and the archive can be read by the next field team. That approach supports annual reporting today while leaving a trustworthy foundation for cross-site analysis, digital agriculture and global food security tomorrow.