The IASSIST 2017 Hackathon And Collaborative Data Solutions

The IASSIST 2017 conference, held at the University of Kansas in Lawrence from 23–26 May, explored “Data in the Middle: The Common Language of Research”. Its programme brought together librarians, data specialists, researchers, archivists and information professionals working across disciplines. Within that setting, the hackathon offered a practical way to turn shared concerns about research data into working ideas, prototypes and reusable methods.

A collaborative data hackathon is less about building a polished application in a few days than creating a productive environment for solving problems together. Participants bring different knowledge of metadata, statistics, software, research practice and data stewardship. The value lies in combining those perspectives, documenting the process and leaving with something that another team can understand, test or extend.

A Practical Response To Research Data Problems

The IASSIST 2017 hackathon reflected a period when research data was becoming central to scholarly communication. Large datasets, computational methods and digital collections were expanding quickly, yet many institutions still struggled with inconsistent formats, unclear documentation and disconnected systems. A researcher might have excellent data but lack the metadata required for discovery, while a repository might hold valuable files that were difficult to interpret or reuse.

A hackathon creates a short, focused space for addressing these problems. Rather than discussing data management only in abstract terms, participants can examine a real dataset, define a specific obstacle and attempt a solution. The work may involve cleaning records, mapping fields, improving a search interface, testing a visualisation or developing a workflow for data citation.

The “common language” idea is important here. Researchers often speak in terms of variables, samples and methods. Librarians may focus on description, preservation and access. Developers think about schemas, APIs and maintainable code. Collaborative events help these groups clarify what they mean and identify the connections between their priorities.

For Australian institutions, this model is highly relevant. A university research office in Melbourne may be supporting projects in health, climate science and the humanities at the same time. A state agency in Brisbane may need data products that work for researchers, policymakers and community organisations. Shared working sessions can bridge those professional boundaries faster than a series of separate meetings.

From Messy Records To Reusable Data

Collaborative data solutions usually begin with an unglamorous task: understanding what is actually in the data. Files may contain duplicate entries, inconsistent dates, missing values, undocumented codes or personal information requiring careful handling. Before anyone builds a dashboard or applies machine learning, the team needs to establish the dataset’s provenance, limitations and intended use.

A useful hackathon workflow therefore starts with data discovery. Participants identify the source, owner, format and level of access. They then inspect the structure, agree on terminology and record assumptions. This process can produce a data dictionary, a metadata template, a cleaning script or a set of validation rules. These outputs may appear modest, yet they often save future users substantial time.

The same principle applies to the conference themes of big data, deep learning, digital agriculture and global food security. A sophisticated model cannot compensate for weak documentation or biased inputs. In digital agriculture, for example, sensor readings may vary according to field location, equipment and weather conditions. In food security research, data from households, markets and remote sensing platforms may use different geographic and temporal scales.

A hackathon makes these technical and conceptual issues visible. Team members can compare interpretations, test a small sample and record decisions while the discussion is fresh. That transparency supports reproducibility. It also makes the final resource more useful to someone who was not in the room.

Roles That Make Collaboration Work

The strongest teams combine complementary capabilities rather than assembling people with identical technical skills. A subject expert can explain why a variable matters. A data librarian can improve discovery and description. A developer can automate repetitive tasks. A statistician can identify analytical risks, while a project lead keeps the group focused on a deliverable that can realistically be completed.

Clear roles do not have to create rigid hierarchies. In a short event, participants may move between planning, coding, testing and documentation. What matters is that essential responsibilities are visible. Someone needs to look after version control, someone needs to capture decisions, and someone needs to check whether the proposed solution meets the needs of its intended users.

Communication is equally important. Teams can lose hours when they begin with an attractive tool instead of a defined problem. A concise challenge statement helps: make a collection easier to search, harmonise two incompatible datasets, improve a data deposit form or create a reproducible analysis example. The group can then choose methods that serve that purpose.

This approach suits Australian research environments, where projects often involve several organisations and funding agreements. A CSIRO team, a university laboratory and a government department may each hold part of the evidence needed for a project. An early agreement about ownership, access, Indigenous data governance and permitted reuse can prevent later disputes. The Australian Research Data Commons and institutional repositories provide useful reference points, but local policies and community expectations still matter.

Lessons For Australian Data Practice

Australian data work has distinctive geographical and institutional conditions. Researchers may be dealing with large distances, regional campuses, remote communities and uneven connectivity. A solution designed in Canberra or Sydney may need to function in a small regional council, an agricultural research station or a community organisation with limited technical capacity.

This makes lightweight, well-documented tools especially valuable. A reproducible spreadsheet workflow, a clear metadata profile or a small open-source script can be more practical than a complex platform requiring specialist support. Teams should consider offline access, file sizes, low-bandwidth environments and the skills available to maintain the product after the event.

Language and trust also affect adoption. Australian teams may describe an afternoon session as an “arvo”, work through a problem over a cuppa and rely on a straightforward, low-fuss style of communication. Those informal habits can support collaboration, but they should be matched with formal records of decisions, responsibilities and consent. Friendly discussion is not a substitute for governance.

Indigenous data requires particular care. A collaborative project involving Aboriginal and Torres Strait Islander communities should consider Indigenous data sovereignty, cultural authority and the right of communities to determine how information is collected, described and shared. A technically impressive prototype may still be inappropriate if it ignores cultural protocols or treats community knowledge as unrestricted material.

The local data market also shapes what can be sustained. Australian universities and public agencies often operate through grants, procurement panels, tenders and short project cycles. A hackathon outcome must therefore be easy to explain to executives, funders and service managers. A working demonstration is helpful, but a concise maintenance plan, licence, documentation package and estimate of ongoing costs may determine whether the idea receives support.

Turning A Short Event Into Lasting Practice

A hackathon is most effective when its outputs are treated as the beginning of a workflow rather than the finished product. The team should identify what happened after the event: who will test the prototype, where the files will live, which licence applies and how users can report problems. Without that follow-up, useful work can remain on personal laptops or disappear when participants return to their normal duties.

Documentation should be treated as a deliverable from the first day. A repository README, issue tracker, data dictionary and short demonstration can explain the purpose of the work to future contributors. If sensitive information was used, teams should publish sample or synthetic data instead of exposing restricted records. Good documentation also makes it easier to evaluate whether the solution can be adapted to another discipline.

Different collaborative formats serve different needs. A traditional hackathon is well suited to rapid prototyping, while a data clinic provides deeper support for a smaller number of projects. A sprint may focus on one technical task, and a working group may be better for policy or governance questions. The right format depends on the maturity of the data, the time available and the authority participants have to make decisions.

Collaborative format	Best suited to	Typical output	Main risk
Hackathon	Rapid exploration of a defined data problem	Prototype, script, workflow or proof of concept	The result is abandoned after the event
Data clinic	Hands-on advice for several project teams	Improved dataset, metadata or analysis plan	Limited time for deeper development
Technical sprint	A focused engineering or standardisation task	Tested feature, schema or integration	Technical success may overlook user needs
Working group	Policy, governance and long-term coordination	Guidelines, priorities or shared framework	Discussion continues without a practical deliverable
Community workshop	Co-design with users and data custodians	Requirements, shared priorities and agreed principles	Expectations exceed available resources

Teams planning a similar event can use the following principles to keep collaboration practical:

Practical Principles For A Reusable Outcome

• Define one specific problem and one realistic deliverable before work begins.
• Include data custodians, domain experts, technical contributors and likely users.
• Agree on access rules, licences, privacy requirements and Indigenous data responsibilities early.
• Keep a decision log so that assumptions and unresolved issues remain visible.
• Use version control and simple documentation that a new contributor can follow.
• Finish with a handover plan covering ownership, testing, hosting and future maintenance.

The enduring value of the IASSIST 2017 hackathon lies in this combination of practical experimentation and shared understanding. It showed how people from different parts of the research data ecosystem can work on the same problem without needing identical professional backgrounds. For Australian organisations, that lesson remains useful: collaborative data work succeeds when technical solutions are grounded in context, governance, documentation and the people who will rely on the result.