Agent in the Loop 2027

This rendition is a task proposal. Venue and dates are to be announced, and the schedule is tentative until the proposal is accepted.

Synopsis

Shared tasks produce standardized evaluation resources and thereby drive progress across many computer science disciplines. Their epistemological foundation, however, lies not in the runs they collect, but in the notebook and overview papers through which participants and organizers communicate and interpret their results. Agent in the Loop is an interdisciplinary shared task that investigates how agentic AI systems affect this written dialogue, and to what extent they can support it.

Using selected passages from overview papers as generation targets, participants build agents that unify the accounts given by shared task participants and synthesize the terminology used to structure the solution space. Development uses a corpus built from SemEval and CLEF shared tasks and their available research artifacts. To prevent training–test leakage, submitted agents run on TIRA for blinded experimentation on material from ongoing shared tasks whose notebooks and overviews have not yet been published. Generated texts are assessed by volunteering organizers of these host shared tasks, independent reviewers, and LLM judges from TREC AutoJudge 2026/27.

Register for participation

Important Dates

All dates are tentative and will be finalized upon acceptance; some proposed dates are inconsistent in year across task phases and will be corrected in a later revision.

17 October 2026The task is announced and the dataset is made available.
11 January 2027The submission system opens for automatic evaluation on the test set.
17 March 2027Software submissions close; manual evaluation begins (up to four weeks).
INLG 2027Presentation of results (venue and exact dates to be announced).

Motivation: Revisiting the Epistemology of Shared Tasks under Generative AI

Donna Harman (1992) characterized the first Text Retrieval Conference (TREC-1) as “a friendly evaluation forum,” whose importance “is not to show any system to be superior, but to allow comparison across a very wide variety of techniques.” Events like TREC-1 came to be known as ‘shared tasks’—a name that captures the collaborative ethos of science, and shared tasks have since become a pillar of empirical research in many computer science disciplines.

We conceive of a shared task as a moderated dialogue among scientists that forms a collective understanding of a task and how progress on it can be measured: organizers open the dialogue with a (1) task description; participants reply by documenting their independently developed ideas in (2) notebook papers; organizers, in turn, quantitatively analyze the runs and qualitatively synthesize the analysis results and participants’ contributions in an (3) overview paper; and, off-stage, attendees continue the dialogue at a workshop through (4) talks and (5) discussions. Much of a shared task’s power arises from this information pooling: by inviting independent solutions, it collects the “Wisdom of the Crowds,” requiring organizers to defend only the importance of the task, not to anticipate its solution space.

Generative AI challenges this social epistemology. If the written dialogue of a shared task were removed, only its engineered artifacts would remain, absent of interpretation and critical reflection—a regression to the “show and tell” culture that Karen Spärck Jones (1981) criticized and that originally inspired Harman to invent the shared task. At the same time, agentic AI raises new questions for collaborative knowledge production: How independent are two participants’ solutions if they used the same LLM? How do scientists validate, and to which extent rely on, LLM-generated input? We cannot address all these questions at once, so we start with the dialogic and social nature of knowledge production, using the written dialogue turns between participants and organizers as points of reference for understanding in what capacity generative AI can support, rather than replace, the shared task dialogue.

Task

In its most generic form, the task we study can be described as follows:

Given a collection of research artifacts produced during participation in or organization of a shared task, create an agent that summarizes the scientific contribution documented by those artifacts.*

* Artifacts include notebooks and runs, and, if available or donated by volunteers, code, datasets, models, revision histories, issue tracker logs, or chat logs with teammates or bots.

Agents must produce two types of summaries that are typically found in shared task overview papers:

  1. Textual cross-participant summaries that summarize all approaches submitted to the host shared task in a consistent language and terminology.
  2. Structured comparison tables that, based on the textual summaries created in Task 1, derive a structured comparison of all approaches along to-be-discovered key dimensions.

Approaches may range from zero-shot prompting to tool-using agentic setups that plan, run, and interpret task-specific analyses before generating text. Submitted agents are run on TIRA against held-out artifacts and using LLMs we provide, ensuring blinded experimentation and reproducible execution. The resulting summaries are evaluated against the summaries we curate from the corresponding overview papers, measuring how faithfully the salient scientific contributions are recovered.

  Task 1: Textual Cross-Participant Summary Task 2: Structured Comparison
Input Notebook paper, code, and logs of Team A and B Text summaries of Team A and Team B from Task 1
Output Team A uses an LLM with retrieval on the training data for in-context learning to improve classification. Team B uses a CNN trained on the official training set. Team Model Training Data Usage
A LLM In-context learning
B CNN Model training
Example for the two subtasks: (1) textual cross-participant summaries, and (2) a structured comparison derived from them.

Data & Corpus

Our datasets pair descriptions of approaches that solve a given shared task with the summaries organizers wrote of those approaches. We derive them from two types of artifacts released during shared tasks: (1) notebook papers, which describe each approach in detail—potentially using different, incompatible language for the same underlying concept—and, where available, code implementing the described idea; and (2) the organizers’ overview paper, which summarizes all notebook papers in a unified, standardized language, typically both as a textual description per approach and as a table comparing all approaches along a shared feature space. These two summary types are the basis for our two subtasks: for Task 1 we extract the textual per-approach summaries from the overview paper, and for Task 2 we extract the comparison table, where one exists.

Corpus construction

We build the corpus from the proceedings of SemEval and CLEF, which document each shared task together with the papers of its participants. We collect the ACL Anthology and CEUR-WS volumes of SemEval and CLEF, one volume per edition, treating their index pages as the authoritative table of contents. Overview papers are paired with notebook papers through titles and the section structure of a volume; for CLEF, where a lab section contains exactly one overview paper, its remaining papers are treated as notebook papers, while sections with several overview papers (an ambiguous case specific to CEUR-WS’s ordering) are set aside for manual review. Every PDF is retrieved and its text layer converted to Markdown with liteparse, which preserves heading structure; figures and tables are extracted alongside the text. For each notebook paper we additionally record links to participants’ code, and for each overview paper the number of teams and runs it reports, where stated.

Preventing test data leakage

To prevent leakage into commercial and fine-tuned models, evaluation uses data from ongoing shared tasks for which neither notebook papers nor overview papers have been published yet. For vanilla open-weights models, their release date determines admissibility of a shared task as a test case (i.e., it must have been organized after that date). We use TIRA to ensure blinded experimentation, so that neither human nor AI participants know which cases are used for evaluation: you submit your agent as working software, which we deploy offline on a virtual machine we provide, without direct access to the test data during the task.

Protocol

Following prior shared tasks organized by our group at INLG (the TL;DR challenge and the Abstractive Snippet Generation task), we split the shared task into three phases:

  1. Participants develop agents using the provided training and evaluation data.
  2. We collect submissions via TIRA, which runs the agents in a sandbox to improve reproducibility.
  3. After all submissions are collected, we run them on the private test set with TIRA.

This means participants submit their agent as working software that generates summaries; the software is deployed offline on a virtual machine we provide, and during the task the test data is never available directly, only through the submission system.

Evaluation

We combine fully automated evaluation with evaluation involving members of the organizing committees of the selected host shared tasks:

  • Task 1 (textual cross-participant summaries). Automated evaluation compares generated summaries against ground-truth summaries with text summarization measures (e.g., variants of BLEU, BERTScore) and LLM-as-a-Judge, including Nugget-based evaluations collected as part of TREC AutoJudge.
  • Task 2 (structured comparisons). We use precision, recall, and F1 against the ground-truth comparison tables.

For the manual evaluation, we ask organizers of the selected host shared tasks to provide feedback on how well the generated content captures their own, in the form of a comparative analysis against their own overview paper summaries. This expert comparison lets us identify cases where generated content surfaces novel and interesting aspects that enrich the understanding of the host task’s solution space.

Submission

Participants submit their agent as working software via TIRA, which runs it in a sandbox for reproducibility and, after submissions close, on the private, blinded test set drawn from ongoing shared tasks. The full participation process is described on the rendition overview.

Agent in the Loop builds on prior summarization shared tasks organized by our group at INLG, and connects to ongoing evaluation efforts for LLM-based judging:

  • The TL;DR Challenge (INLG 2019). Summarization for social media, on which our three-phase protocol (development, sandboxed submission, blinded test) is modeled.
  • Abstractive Snippet Generation (INLG 2020). Generating abstractive snippets for web pages, the second prior INLG shared task organized by our group.
  • TREC AutoJudge (2026/27). A cross-task benchmark for comparing LLM judges for citation-grounded RAG systems; we draw on its Nugget-based LLM-as-a-Judge methodology and judges for our automated evaluation.
  • TIRA & TIREx. The information retrieval experiment platform we use to host blinded, reproducible submissions as working software rather than static runs.

Task Committee