Making DDI metadata workable at IASSIST 2017

The archived programme for IASSIST 2017 records a four-day meeting at the University of Kansas in Lawrence, Kansas, from 23 to 26 May 2017. Its theme, “Data in the Middle: The Common Language of Research,” suited a gathering concerned with big data, deep learning, digital agriculture and global food security. Within that programme, the DDI workshops addressed a practical question: how can datasets remain understandable after the researchers who created them have moved on?

Data Documentation Initiative, usually shortened to DDI, is a family of standards for describing research data, surveys and metadata. It can document a study’s purpose, sampling, geographic coverage, collection methods, variables, questions, response categories and processing history. This detail helps people interpret a dataset without reverse-engineering filenames, spreadsheets or analysis scripts.

For an Australian audience, these workshops offer a way to connect international metadata standards with local research systems. Australian teams work across universities, CSIRO, government agencies, industry partners and regional communities. They also face long distances, distributed project teams and complex arrangements for sensitive data, making clear documentation essential to reuse.

Documentation as conference infrastructure

Research data across the IASSIST programme varied considerably. A national survey might contain thousands of coded variables, while an agricultural study could combine sensor readings, laboratory measurements and satellite images. Deep-learning projects added another layer through training corpora, software dependencies and model outputs. A common metadata language gives these assets a shared descriptive foundation.

Documentation determines whether data can be found, cited, checked and reused. Sparse descriptions may record an author and a date but omit units, collection periods or geographic boundaries. Rich documentation preserves context, distinguishes observed values from derived variables and explains why records were excluded or transformed.

The DDI workshop theme therefore treated documentation as research infrastructure rather than administrative work completed after a project. A usable record allows a librarian, analyst or partner organisation to understand the study without relying on private knowledge held by its original investigators.

What DDI captures across the lifecycle

DDI can describe a study at several levels, from the overall project to individual variables and questions. Study-level metadata normally covers purpose, population, sampling and data collection. More detailed structures represent data-collection events, processing activities, file organisation, classifications and the relationship between source and analysis variables.

Variable-level documentation is particularly valuable for survey and observational research. A variable label such as “yield” does not reveal the crop, measurement unit, time basis or treatment of missing values. DDD can connect the label with a definition, representation, code list, universe, source and processing statement, reducing ambiguity for users in different countries.

DDI does not replace every form of documentation. Repository descriptions, data dictionaries, codebooks, software readmes, transcripts and model documentation remain important. Its strength lies in linking these materials through structured metadata, preserving relationships that are often lost when supporting files are stored separately.

How workshops turn a standard into practice

Standards become workable when participants apply them to realistic records. A workshop can begin with a published survey, move through its questionnaire and codebook, then ask participants to build or revise the corresponding metadata. This sequence makes abstract elements such as universes, classifications and variable representations visible in context.

The archived conference social page also reflects the value of maintaining a traceable conference record. Connections to discussions, photographs and shared materials help participants reconstruct how ideas developed beyond the formal programme, even when the main sessions dealt with technical or policy questions.

Peer review is equally important during a workshop. Participants can test whether a variable definition is understandable, whether processing steps are reproducible and whether access conditions are visible. The output is not simply a completed form; it is a documented record that another person can interpret and challenge.

Digital agriculture and food security

Digital agriculture illustrates why interoperable metadata matters in an Australian setting. Projects may combine rainfall and soil data from the wheat belts of New South Wales and Victoria, pastoral records from inland Australia, or sensor networks used by farming businesses. Consistent descriptions allow evidence from different regions and instruments to inform food-security research.

Machine learning adds further complications. Training data may combine imagery, weather observations and agricultural statistics, while a model changes as algorithms and parameters are revised. DDI can document the study, source variables, collection context and processing sequence. Model documentation can then add training procedures, evaluation measures, software versions and known limitations.

Global food-security datasets often pass among researchers, policymakers and organisations in different countries. Shared variable definitions reduce the risk that similarly named indicators—such as food expenditure, crop yield or household resilience—carry incompatible meanings. Clear metadata supports comparison while preserving differences in definitions, units and collection contexts.

Carrying DDD into Australian research

The Australian research market operates through competitive grants, institutional projects and partnerships involving universities, CSIRO and industry. National services such as the Australian Research Data Commons and Trove provide discovery pathways, but local repository teams still decide how much contextual detail to record. DDI gives those teams a structured way to enrich catalogue entries without rebuilding every study description.

Sensitive material requires careful treatment. The Privacy Act 1988, ethics approvals and data-use agreements may restrict access to identifiable records. Indigenous data governance also requires respect for community authority, cultural responsibilities and restrictions on commercial reuse. Documentation should record those conditions without exposing information that people or communities have chosen to keep controlled.

Australia’s geography adds a practical metadata requirement. A project involving researchers in Perth, Melbourne, Sydney and regional centres must state time zones, field-site boundaries, coverage gaps and transfer arrangements. Australian English terms such as “programme,” “metadata steward” and “data custodian” may sound familiar, but a “fair go” for reuse still requires precise definitions and durable identifiers.

Discoverability also crosses borders. A workshop can compare a DDI record with the navigation choices on an external subject site; the astronomy resource portal provides a non-social-science example of presenting technical resources to a wider audience. The exercise shows which cues help a first-time visitor locate relevant material.

Choosing methods for different research assets

The appropriate balance between DDI and companion documentation changes with the research asset. Structured metadata is most useful when relationships between a study, its collection process and its variables carry analytical meaning.

Research asset	DDD provides most value through	Complementary material
Survey or questionnaire data	Study purpose, sampling, universes, variables, question wording, response scales and weighting variables	Instrument release, README, codebook and analysis notes
Agricultural and environmental observations	Sites, dates, geographic coverage, collection methods, measured variables, processing steps and provenance	Data dictionary, laboratory references, processing scripts and README
Text, image or audio collections	Study context, access conditions, collection instruments, file-level concepts and processing histories	File inventory, transcription guide, rights statements and access workflow
Code and deep-learning workflows	Study description, source variables, data-collection context, processing steps and documented outputs	Repository citation, dependency record, model card or data sheet, and evaluation protocol

Repository teams can combine a concise DDI record with persistent identifiers, downloadable documentation and links to related publications. Survey data may receive deeper variable-level treatment, while complex computational outputs need equally strong documentation for software, models and evaluation procedures.

A workshop checklist for Australian teams

Preparation determines how much can be learned during a hands-on DDI session. A small, complete study is usually more useful than an unfinished collection of unrelated examples.

• Nominate a study with a clear owner, depositor and contact point.
• Inventory source files, questionnaires, codebooks, scripts and derived data.
• Identify sensitive fields, access restrictions and Indigenous data responsibilities.
• Agree on variable names, units, missing-value codes and version conventions.

Review should test whether the metadata communicates both meaning and provenance without relying on the original researcher.

• Can a new analyst identify the population, geography and time coverage?
• Can principal variables be interpreted without opening the original analysis code?
• Can processing, quality controls and exclusions be traced to the resulting data?
• Are responsibilities assigned for maintenance, access decisions and future corrections?

The lasting value of the IASSIST 2017 DDI topic is a transferable principle: research data become durable when their structure, context and responsibilities are recorded as carefully as the data themselves. DDI supplies that common layer across disciplines, while flexible companion documentation preserves the details that make each resource fit for its intended research use.