Data discovery tools highlighted at IASSIST 2017

The IASSIST 2017 programme highlighted data discovery tools as shared infrastructure that can turn scattered records into research-ready evidence. Held at the University of Kansas in Lawrence from 23 to 26 May, the conference centred on “Data in the Middle: The Common Language of Research” and examined the services connecting people, publications, datasets and analytical methods. Big data, deep learning, digital agriculture and global food security provided practical settings for asking what a useful tool must actually do.

For Australian researchers, that question has two dimensions: can a dataset be found, and can it be trusted and combined? The archived program contains plenaries, session schedules and presentations, offering a map of discovery workflows rather than a static product catalogue. The important pattern is the linkage between repositories, metadata, persistent identifiers, search systems and computational services.

A system may be excellent at locating rainfall records but weak on stating licence terms, lineage or sampling bias. It may index millions of images yet offer no route to the source dataset. IASSIST 2017 is valuable because it places these trade-offs beside the research cycle, from planning a study to publishing reusable outputs. Applying the lessons locally requires attention to Australia’s distributed agencies, farm sectors and metadata habits.

Reading the archived programme

The archive should be read as a snapshot of a rapidly changing data environment. In 2017, organisations were already combining institutional repositories with national catalogues, domain portals, application programming interfaces and cloud-based analytics. The emphasis was not on one glamorous search box. It was on how a researcher moves from a web search to a documented record, then on to analysis that other people can verify.

The plenary and parallel sessions gave that process several scales. General sessions addressed common language and research infrastructure; technical sessions dealt with machine learning and large-scale data; and applied sessions followed data through fields such as agriculture and food security. Read together, they show discovery as a chain of decisions about identity, context and access, not merely a retrieval task.

That distinction matters for anyone revisiting the archive today. Interfaces have moved on, but the underlying questions endure: which identifier remains stable, which metadata schema carries enough meaning, and which service preserves the connection between a published claim and its evidence? The conference materials therefore remain useful as a framework for evaluating newer discovery platforms, even when particular products or projects are no longer active.

Discovery tools as a connected stack

Discovery works best when its layers perform distinct jobs. A broad catalogue can reveal an organisation’s holdings, while a repository manages objects and versions, and an API enables systems to exchange queries or records. The following functional comparison draws on the programme’s emphasis on searchable metadata, open interfaces and reusable research objects.

Discovery layer	Core purpose	Distinct value	Typical Australian setting
Institutional repository	Preserve and publish research outputs	Versioning, access controls and persistent identifiers	University theses, code and datasets
National catalogue	Search across institutions	Broader recall and shared metadata	Trove and sector-wide discovery
Research data commons	Register and describe datasets	Standard fields and visible access conditions	Research Data Australia and ARDC services
Disciplinary portal	Surface domain evidence	Specialist vocabularies and curated context	Agriculture, climate and health collections
API and workflow service	Connect catalogues to analysis	Automation and repeatable retrieval	Linking government and research records
Knowledge graph	Reveal relationships between entities	Linked people, places, organisms and outcomes	Tracing crops, rainfall and policy evidence

The layers are complementary rather than competitors. A researcher may begin with a national catalogue, follow a record into an institutional repository, inspect a dataset’s provenance, and call an API to assemble a study cohort. Search relevance can narrow the first step; durable identifiers and good metadata protect every later step. The strongest discovery tools make that route visible to users who are not information specialists.

Big data and deep learning add scale, but they do not remove these requirements. Training sets need clear provenance, versions and licensing; learned representations need enough documentation for scrutiny; and predictive results need a path back to the source material. The stack shown in the programme suggests that algorithmic sophistication should sit on transparent records rather than obscure them. It is also why federated systems matter: a central index can guide people to data held in many different places.

Metadata, citation and context

Metadata is the practical language of discovery. Descriptive fields tell someone what a dataset contains; structural fields explain how it is organised; administrative fields identify its creator, owner and licence; and provenance records document how it was produced or changed. A search interface can make rich metadata feel simple, yet the depth behind each result determines whether the data can be reused responsibly.

Citation performs a related task by giving credit and creating a stable trail. In digital agriculture, where observations may combine field sensors, satellite imagery, soil samples and farmer records, a citation needs to connect the published statement to the exact data and methods used. The conference’s attention to global food security made this concrete: evidence about yields, water use or land conditions has little value if its origin cannot be established.

For a practical companion, the conference archive provides archived data citation guidance. Its focus on digital agriculture illustrates why a conventional paper reference may not be enough. Teams can use persistent identifiers, dataset versions and explicit methods to create a citation trail that remains intelligible across repositories and research organisations.

Citation also helps with a common Australian research problem: knowing that information exists without knowing which institution holds the authoritative version. Cross-repository identifiers, clear version labels and standard contributor information reduce duplicate effort. They also support automated systems, which need a dependable way to distinguish an original dataset from a derived copy, teaching subset or later revision.

An Australian research lens

Australia already has a substantial discovery layer with which the 2017 themes can be compared. In Canberra, data.gov.au supports discovery across government datasets, while Libraries Australia and Trove expose records held by libraries, archives and special collections. Research Data Australia and the Australian Research Data Commons add research-specific registration and metadata services. Melbourne’s strong university and research network makes the commons model especially visible.

Australian fieldwork adds conditions that sharpen the need for context. A trial on the Darling Downs is not interchangeable with one in the Victorian dairy belt or the NSW wheat belt; rainfall timing, soil type, cultivar and drought history shape the meaning of an observation. Researchers often describe solving these gaps as “data plumbing”, while a “fair go” for regional institutions means access to standards and stewardship support as well as open licences.

The market is similarly mixed. CSIRO, universities, government agencies and agribusiness firms each maintain records, while commercial providers often hold the most detailed farm-level or sensor data. Some resources sit behind subscriptions, licences or negotiated agreements rather than public catalogues. Shared standards can lower that friction without pretending every organisation has the same budget or infrastructure.

That sharing must accommodate different forms of authority. Public datasets, commercially confidential records, personal information and Indigenous cultural materials cannot be treated as interchangeable assets. Metadata policies need to make those distinctions visible before a researcher downloads, combines or republishes material. This is where a common research language becomes more than technical interoperability: it requires clear rules about consent, ownership, attribution and responsible use.

A practical evaluation framework for Australian teams

A useful way to apply the archived lessons is to judge a service by the research decisions it supports. Australian teams can test discovery tools against a real question, such as locating reliable rainfall observations for a regional crop model or tracing policy datasets used in a health study. This keeps evaluation tied to work rather than feature counts.

A discovery workflow can follow four moves:

• Start with the decision, dataset type, geographic scale and acceptable evidence before choosing a service.
• Search across a repository, national catalogue, disciplinary portal and specialist registry, then compare their coverage.
• Inspect metadata for creator, collection date, version, licence, provenance, quality notes and persistent identifier.
• Record the final path from the discovery record to its data, documentation, code and citation before analysis begins.

Those moves can be assessed against four criteria:

• Findability: can relevant objects be located through names, concepts, identifiers and cited metadata?
• Interoperability: do schemas, vocabularies and APIs support exchange without repeated manual conversion?
• Reusability: are access conditions, provenance, methods and community expectations clear enough for responsible reuse?
• Governance: are ownership, privacy, Indigenous data sovereignty and long-term preservation actively managed?

Findability is only the opening move. Interoperability and reusability determine whether evidence survives the trip into a workflow, while governance decides who may use it and under what conditions. For Australian datasets, that last point includes Indigenous data governance and commercial confidentiality, not just open-access labels. Teams should record failed searches and inaccessible leads as well as successful datasets, because those gaps reveal where metadata and public access need work.

Why the 2017 lessons still apply

The archived program is more than a historical agenda. Its schedule, plenaries, presentation materials and practical information about check-in, transport and local recommendations preserve the conference as a working environment. Conference programmes also function as discovery records in their own right. The CMCEE 2018 programme offers a useful parallel for tracing how a specialist community documents methods and connections, while the IASSIST archive preserves the original sequence of sessions.

Many individual platforms from 2017 have changed, yet the demand for interoperable metadata has expanded with open data, machine learning pipelines and linked bibliographic records. The strongest idea to retain is that discovery should preserve context as it travels. A searchable description, a resolvable identifier, a clear licence and a documented transformation chain collectively make a dataset more trustworthy. Without those elements, scale simply makes confusion faster.

IASSIST 2017 also showed why a common research language requires participation from different domains. Digital agriculture, food security and statistical science may ask different immediate questions, but all depend on finding evidence, evaluating its fitness and explaining its origin. The conference’s value for an Australian audience lies in that durable pattern: connect catalogues, repositories, identifiers and people, then make every retrieval and transformation visible enough for the next researcher to follow.