UniAgent at INLG 2027

This rendition is a task proposal. Venue and dates are to be announced, and the schedule is tentative until the proposal is accepted.

Synopsis/Call

Shared tasks produce standardized evaluation resources and thereby drive progress across many computer science disciplines. Their epistemological foundation, however, lies not in the runs they collect, but in the notebook and overview papers through which participants and organizers communicate and interpret their results. The second UniAgent rendition turns that written dialogue itself into the environment an agent acts in: participants build agents that summarize the research artifacts of other shared tasks into the textual and tabular summaries typically found in overview papers.

We invite you to participate in Agent in the Loop, proposed for INLG 2027, featuring one task with two sub-tasks.

Register for participation Read the task definition

Shared Tasks

Given the research artifacts produced during participation in or organization of a shared task, create an agent that summarizes the scientific contribution those artifacts document. The task has two sub-tasks: textual cross-participant summaries, and structured comparison tables derived from them.

Participation

Participation is open to research groups and individuals from NLP, IR, AI, digital humanities, and adjacent communities, including scholars interested in the epistemology and sociology of science.

  1. Register on TIRA. Sign up on TIRA, then register your team to Agent in the Loop.
  2. Develop your agent against the SemEval/CLEF-derived training and evaluation data described under Data & Corpus.
  3. Submit your agent as code (working software) via TIRA, which runs it in a sandbox for reproducibility, then, after submissions close, on the private, blinded test set drawn from ongoing shared tasks.
  4. Results and findings are presented at INLG 2027. You describe your approach in a notebook paper.

Important Dates

All dates are tentative and will be finalized upon acceptance; some proposed dates are inconsistent in year across task phases and will be corrected in a later revision.

17 October 2026The task is announced and the dataset is made available.
11 January 2027The submission system opens for automatic evaluation on the test set.
17 March 2027Software submissions close; manual evaluation begins (up to four weeks).
INLG 2027Presentation of results (venue and exact dates to be announced).

Awards

Winning teams are recognized with certificates, a presentation slot at INLG 2027, and a mention in the overview paper. We foresee no monetary prizes at this stage, though awards may be extended if sponsorship becomes available.

Results

Results will be announced after the manual evaluation period; see Important Dates.

Organizing Committee

Contact

To participate, register on TIRA as described under Participation. You can reach the organizers at uniagent@lists.its.uni-kassel.de. Releases are announced on this page.