Archiving Social Science Data: Lessons from IASSIST 2017 Conference

In May 2017, librarians, data curators, and researchers gathered in Lawrence, Kansas for the IASSIST annual conference, a meeting long regarded as a forum for practical conversations about preserving and sharing quantitative and qualitative material. The University of Kansas hosted the event under the banner "Data in the Middle: The Common Language of Research," and the programme covered everything from digital agriculture and deep learning to the quieter work of keeping social science datasets alive for the next generation of scholars.

For data archivists, the conference carries particular weight because social science material tends to be messier than laboratory output. Surveys, interview transcripts, census microdata, and historical records all carry contextual fragility: variables change meaning over time, codes drift, and the people who originally collected the data may be retired or deceased. A well-attended workshop on preserving survey instruments alongside their documentation can determine whether a dataset remains interpretable decades later, or whether it becomes an orphaned relic in a cold storage rack.

In Brisbane, Canberra, and Hobart, data managers at the Australian National University, the University of Melbourne, and the University of Queensland face similar questions to their American counterparts, with the added complexity of operating across multiple jurisdictions and funding bodies. The Australian Data Archive has long curated sociologists familiar with the National Centre for Longitudinal Data, while the Australian Bureau of Statistics manages one of the richest troves of population and economic data in the Asia-Pacific region, including surveys that reach communities across the bush. Cross-disciplinary borrowing between the two has shaped how Australian repositories think about sensitive material, particularly around Indigenous data governance.

The lessons from IASSIST 2017 sit squarely in this kind of cross-jurisdictional conversation. Some of the most useful sessions were not the headline plenaries but the side meetings, where curators swapped notes about migration tools, restricted-access agreements, and the politics of metadata. The archived programme remains available for those who could not attend. For a sense of the conference social events and the informal conversations that shaped many of the formal outcomes, the organisers kept a record of the receptions, dinners, and hallway gatherings in Lawrence.

The lifecycle question: from collection to reuse

A recurring theme at IASSIST 2017 was that archiving is not a final step but a continuum that begins the moment a research project is funded. Curators who treat the archive as a downstream warehouse tend to inherit documentation gaps, inconsistent variable labels, and consent forms that no longer satisfy contemporary ethical standards. The alternative, now common across Australian repositories, is to embed data management planning at the proposal stage so consent language, file formats, and storage expectations are negotiated before fieldwork begins.

This shift changes the role of the data steward. Rather than waiting passively for deposits, the steward becomes a partner in design, asking researchers what their files will look like in five years and what reuse they hope to enable. Sessions in Lawrence highlighted examples from the United States and Europe where early-stage engagement reduced post-hoc cleanup costs substantially, freeing time for the harder problems of linking datasets across decades.

The Australian Research Data Commons has pushed this thinking into national policy through the FAIR principles, encouraging researchers to make data Findable, Accessible, Interoperable, and Reusable from the outset. In Adelaide and Perth, university research offices now routinely require data management plans as a condition of grant approval, a practice that migrated from the National Health and Medical Research Council into the social sciences and humanities. A younger generation of PhD students now treats the data archive as a collaborator rather than a last resort.

Standards, metadata, and the politics of description

No conversation about archiving social science data can avoid DDI, the Data Documentation Initiative, and the 2017 programme gave the standard plenty of airtime. DDI provides a shared structure for describing surveys, codebooks, and variable lists, and its latest iterations support provenance tracking and controlled access workflows. Practitioners in the room pointed out that adoption is uneven: smaller institutions often lack the technical capacity to implement the full standard, and researchers sometimes resent the extra burden of structured metadata.

The Lawrence discussions explored how to lower that barrier without diluting the standard. Lightweight profiles, modular templates, and guided web forms were among the tools showcased. Australian practitioners have experimented with similar approaches through the Australian Data Archive's online deposit system, which walks researchers through consent, file format, and documentation requirements in plain English. The trade-off between rigour and usability remains live, and IASSIST working groups continue to circulate proposals for review.

Beyond DDI, sessions touched on Dublin Core for general discovery, PREMIS for preservation metadata, and the practical realities of crosswalking between schemas when a repository inherits a collection from a defunct project. Repositories that succeed in this area share a common habit: they document their crosswalks publicly so downstream users can understand exactly how a variable was transformed. The conference made space for the kind of detailed technical exchange that rarely finds a home in journal articles.

Restricted access, ethics, and Indigenous data sovereignty

Sensitive data presented some of the most thoughtful conversations in Lawrence. Social science archives increasingly hold material that cannot be openly shared: identifiable interview transcripts, location data for vulnerable populations, and health-adjacent surveys covered by privacy legislation. The conference explored how curators can make such material available to legitimate researchers while honouring consent agreements and legal constraints.

Australian contributors brought a distinctive lens, foregrounding Indigenous data sovereignty and the principles articulated by the Maiam nayri Wingara collective. These frameworks assert that data about Aboriginal and Torres Strait Islander peoples should be governed by those communities, with curation practices reflecting cultural protocols around storytelling. Several sessions examined how the Australian Institute of Aboriginal and Torres Strait Islander Studies has built access frameworks aligned with these principles, including protocols for material from remote communities in the outback, offering a model that has drawn attention from Canadian and New Zealand archivists facing parallel questions.

The discussions also addressed the tension between openness as a default and protection as an exception. While open data advocates argue that restricted access slows scientific progress, ethical researchers counter that careless release can cause real harm. The Lawrence conversations did not resolve this tension but offered practical templates for graduated release, secure computing facilities, and data use agreements that protect participants without shutting down legitimate research. Curators left with a clearer sense of the trade-offs and a stronger network of peers.

Tools, workflows, and the practical craft of preservation

Plenty of sessions focused on the day-to-day craft of running a repository, from file format migration to bit-level preservation. Archivists discussed the merits of different preservation formats, the risks of proprietary software lock-in, and the value of checksums, fixity audits, and redundant storage across geographically separated facilities. The Australian National Data Service hosted a session on distributed storage models that has since informed infrastructure decisions at several universities in Sydney and Melbourne.

Automation featured prominently, including presentations on scripted metadata extraction, automated format identification using tools like Siegfried and FIDO, and continuous integration pipelines to validate new deposits. The promise is that automation frees curators from repetitive work, letting them focus on contextual judgement that machines struggle to replicate. The reality, as several presenters admitted, is that scripts break, schemas drift, and the human in the loop remains essential. The IASSIST community has responded by sharing scripts openly and documenting them as part of the archival record.

Sustainability emerged as a quieter but persistent theme. Many of the tools that archives rely on, including Archivematica, Preservica, and various DDI utilities, depend on small development communities whose funding can vanish with a single grant cycle. The conference provided a venue for archivists to compare roadmaps and coordinate maintenance. For Australian institutions operating at smaller scale, this kind of coordination offers a path to capability that would be hard to achieve alone.

What Australian archives can take from Lawrence

Looking back from a few years on, the value of IASSIST 2017 lies less in any single breakthrough and more in the consolidation of a community vocabulary. Curators who attended carried home shared examples, common metrics, and a stronger sense of how their local challenges compared with colleagues overseas. The archived presentations remain available through the conference website, and the standards collaboration network referenced in several sessions offers a useful starting point for those exploring international alignment.

For Australian institutions, the practical lessons cluster around three priorities: investing in early-stage data management planning, building metadata capacity in line with DDI and FAIR principles, and engaging seriously with Indigenous data governance frameworks. None of these are new ideas, but the Lawrence programme offered concrete templates, working code, and contact lists that make implementation more tractable. A repository that emerged with a sharper sense of direction would count the week as time well spent.

Australian practitioners often weigh several approaches when planning a preservation workflow:

Approach	Strengths	Weaknesses	Best fit
Embedded data steward	Early intervention, fewer gaps downstream	Requires staffing investment	Large funded research programmes
Lightweight metadata profiles	Lower depositor burden, faster intake	May lose provenance depth	Small to medium collections
Full DDI implementation	Rich documentation, strong reuse	High overhead, steep learning curve	Census-scale or longitudinal studies
Restricted-access enclave	Strong ethical safeguards	Limits access, infrastructure cost	Sensitive or identifiable data
Distributed storage across campuses	Redundancy, local control	Coordination overhead	Multi-institution collaborations

Challenges raised across sessions

• Inconsistent documentation from legacy deposits
• Difficulty sustaining technical staff over the long term
• Tension between open access defaults and ethical safeguards
• Limited funding for migration of obsolete file formats
• Fragmented governance across state and federal jurisdictions

Practical steps that emerged as low-regret investments

• Adopt a documented metadata profile based on DDI
• Build relationships with Indigenous data governance bodies early
• Negotiate storage redundancy across at least two sites
• Document all crosswalks and transformations publicly
• Schedule annual fixity audits rather than ad hoc checks
• Train at least one early-career researcher as a future steward