Cleaning Ag Data for More Trustworthy Research

The IASSIST 2017 Workshop on Data Cleaning for Ag Data formed part of IASSIST 2017, “Data in the Middle: The Common Language of Research,” held at the University of Kansas in Lawrence from 23 to 26 May 2017. It focused on a basic research need: observations from farms, laboratories, sensors and supply chains must be converted into evidence that can be compared, reproduced and trusted.

Agricultural information is often called “big data,” yet scale can conceal inconsistency. A yield file may use hectares while a trial report uses acres; weather records may mix local time with UTC; names for districts, crops or treatments can vary between agencies. Duplicate sensor readings, stale records, silent unit changes and implausible values all weaken analysis. Cleaning identifies those problems, preserves defensible observations and documents every intervention.

The workshop’s value also lies in its place within a broader conference on digital agriculture, deep learning and global food security. Machine-learning models received attention, but a model cannot compensate for corrupted inputs or forgotten transformations. The archived sessions treat data quality as shared infrastructure across disciplines, connecting technical cleaning with metadata, governance and clearer communication among researchers, extension teams and industry.

For an Australian audience, these lessons remain practical. Agricultural datasets connect metropolitan markets in Sydney, Melbourne and Brisbane with producers across the wheat belt, northern savannas, pastoral regions and coastal horticulture. The archived programme offers a useful reference point for deciding which quality controls should travel across regions and which must reflect local crops, institutions, commercial arrangements and legal duties.

Why agricultural data needs deliberate cleaning

A farm dataset is assembled across boundaries. Soil tests come from laboratories, machinery reports arrive in proprietary formats, satellite products carry grid references, and field notebooks contain shorthand entered by different people. Each source has its own identifiers, missing-value codes and precision. Joining them without a data dictionary can create false matches, attach rainfall to the wrong paddock or treat a repeated transmission as another harvest event.

Cleaning therefore covers more than deleting errors. It includes data validation against physical, agronomic and commercial expectations; normalisation of units, categories, dates and geospatial references; deduplication; and correction of inconsistent records. A suspicious value may be genuine. A low-yield result can reflect drought, disease or a changed measurement method, so replacement rules should preserve uncertainty instead of masking it with an average.

The goal is fitness for purpose. A dataset suitable for regional rainfall analysis may not suit machine learning across farms, and a public map may require stronger privacy controls than an internal trial. Clear provenance, definitions and transformation logs let another researcher understand what was removed, retained or derived. In this sense, cleaning turns scattered farm records into traceable research evidence rather than merely a tidier spreadsheet.

The workshop’s common-language approach

Data in the Middle placed special emphasis on systems that connect people and information. That framing suits data cleaning, because a defect in one field often becomes a communication problem later. When missingness has several possible meanings, when an agency renames a soil attribute or when a timestamp lacks a timezone, analysts can make different decisions without noticing that their source data disagree.

The surrounding plenaries on big data and deep learning sharpened the stakes. Agricultural algorithms may ingest millions of observations, while deep learning can fit complex relationships that obscure the provenance of a prediction. Clean, representative inputs remain essential, but so do documented training samples, fair comparison groups and checks for leakage from one paddock, season or region into another. Responsible analysis begins before the first model is fitted.

Practical conference information is part of the archive as well. Visitors checked routes, parking and local travel around Lawrence before attending sessions; the archived transport guidance remains a concrete reminder that good data work depends on reliable arrangements for the people doing it. Travel details may be historical, yet the exercise shows how an event programme can pair scholarly exchange with usable operational information.

Turning mixed farm records into research-ready evidence

A dependable workflow begins with profiling. Analysts inspect field names, data types, ranges, frequency distributions and missing-value patterns, then examine relationships among date, location, crop, treatment and equipment. They establish the unit of analysis—a paddock, animal, sample, field trial or season—before joining datasets. This prevents many errors, since a record counted once at farm level may be duplicated when expanded to paddock or trial level.

Next comes reconciliation. Hectres, acres, kilograms, tonnes and laboratory-specific concentrations need explicit conversions. Local dates should be paired with time zones where machines moved across boundaries. Coordinate systems should be transformed into a common reference frame, while crop codes and enterprise identifiers should be mapped through versioned lookup files. Rules should distinguish an absent measurement from zero, a true outlier from a valid extreme and a placeholder from unknown.

Validation continues after cleaning. Plots can reveal suspicious sequences in yields or machinery speed; cross-source totals can expose missing shipments; and repeat measurements can test consistency. Each correction should retain the original value, the replacement, the reason and the responsible person or process. That audit trail supports reproducible analysis and allows a research team to revisit a rule when a new crop, sensor or season changes the context.

Australian realities that shape the cleaning rules

Australian cities show why connected food and data systems matter. Shoppers in Brisbane or Melbourne may check rainfall on a phone, compare supermarket prices, use loyalty cards and expect fresh produce to carry reliable origin information. These ordinary habits generate assumptions about provenance and quality. A cleaning workflow that removes a meaningful regional label or an imperfect but explainable measurement may erase information consumers and researchers actually need.

The local market adds further complexity. Australia combines broadacre wheat and canola, irrigated cotton and horticulture, sugar, beef, sheep and mixed farming. Producers sell through cooperatives, stock exchanges, processors, contractors and export chains, each with different reporting rhythms. Grain grades, carcass weights, chemical residues and payment calculations are governed by commercial terms, so a value that looks odd can reflect a legitimate market rule. Reconciliation must understand the chain before it standardises the field.

Legal duties also belong in the data specification. Under the Privacy Act 1988 and the Australian Privacy Principles, personal information about farm operators, workers, landholders or research participants must be handled according to applicable coverage and consent arrangements. The Spam Act 2003 can matter for electronic marketing by organisations using contact data, while Australian Consumer Law and state fair-trading codes affect commercial claims about products and supporting evidence. Legal review should identify obligations without turning every research dataset into a compliance exercise.

Quality assurance patterns worth retaining

The archive is most useful when its methods are translated into repeatable checks. These raw-data controls reduce ambiguity before any edits occur:

• Preserve an immutable copy of every source file, with receipt dates, checksums and source identifiers.
• Profile field names, types, codes, ranges, missingness and duplicate patterns before combining records.
• Reconcile units, time zones, geospatial references and category labels using documented lookup files.
• Confirm the unit of analysis and verify identifiers before joining farm, weather, laboratory and market sources.

After processing, the evidence needs a second layer of checks:

• Validate plausible ranges and relationships, then investigate anomalies in their agricultural and commercial context.
• Test aggregation totals, longitudinal sequences and cross-file consistency after every major transformation.
• Retain an audit log of corrections, exclusions and derived fields, with reasons and version history.
• Check representativeness, privacy exposure and documentation before releasing a cleaned dataset.

Together, these controls make quality assurance a traceable process rather than a final formatting exercise. They also distinguish a deliberate exclusion from an accidental loss and preserve enough context for another analyst to understand how the research dataset was produced.

Comparing the workshop with an Australian research agenda

The comparison below translates the workshop’s shared concerns into Australian terms. It is not a checklist of universal thresholds; crop zones, enterprise contracts and research questions determine what counts as valid. The value lies in making those decisions visible and testable.

Research concern	Workshop emphasis	Australian application	Evidence to retain
Data heterogeneity	Shared language and interoperability	Machinery, laboratory, weather and market files use different structures	Source map, data dictionary and versioned mappings
Missing or duplicate records	Profiling, validation and repair	Remote sensing, telematics and farm paperwork may repeat or omit observations	Flags, correction log and unresolved-case notes
Values and units	Normalisation with context	Hectares, tonnes, rainfall, grades and laboratory units require reconciliation	Conversion rules, original values and anomaly reviews
Models and scale	Clean inputs for big data and deep learning	Regional or national models must avoid leakage across seasons, farms and regions	Sampling frame, split records and representativeness checks
Governance and use	Communication and shared responsibility	Privacy, commercial claims and research agreements shape access and disclosure	Consent basis, access conditions, provenance and disclosure decisions

Applied in this way, the archive connects conference themes rather than isolating cleaning as an administrative chore. It helps explain why the same harmonised variable can still require local definitions, why an apparently wrong number may be a valid market event, and why traceability is as important as accuracy. For Australian collaborators, that is the bridge between a shared technical standard and defensible evidence about food, land and production.