Agent in the Loop 2027
Synopsis
Shared tasks produce standardized evaluation resources and thereby drive progress across many computer science disciplines. Their epistemological foundation, however, lies not in the runs they collect, but in the notebook and overview papers through which participants and organizers communicate and interpret their results. Agent in the Loop is an interdisciplinary shared task that investigates how agentic AI systems affect this written dialogue, and to what extent they can support it.
Using selected passages from overview papers as generation targets, participants build agents that unify the accounts given by shared task participants and synthesize the terminology used to structure the solution space. Development uses a corpus built from SemEval and CLEF shared tasks and their available research artifacts. To prevent training–test leakage, submitted agents run on TIRA for blinded experimentation on material from ongoing shared tasks whose notebooks and overviews have not yet been published. Generated texts are assessed by volunteering organizers of these host shared tasks, independent reviewers, and LLM judges from TREC AutoJudge 2026/27.
- Sub-tasks: textual cross-participant summaries, structured comparisons of the approaches, and quantitative analyses of their evaluation results.
- Communication: [UniAgent forum on TIRA, organizers]
- Submission: [software via TIRA, notebook papers]
Important Dates
| 21 October 2026 | The task is announced and the dataset is made available. |
| 11 January 2027 | The submission system opens for automatic evaluation on the test set. |
| 17 March 2027 | Software submissions close; manual evaluation begins (up to four weeks). |
| INLG 2027 | Presentation of results (venue and exact dates to be announced). |
Motivation: The Epistemic Role of Shared Tasks under Generative AI
Donna Harman (1992) characterized the first Text Retrieval Conference (TREC-1) as “a friendly evaluation forum,” whose importance “is not to show any system to be superior, but to allow comparison across a very wide variety of techniques, much wider than only one research group would tackle.” Events like TREC-1 soon came to be known as “shared tasks” and have since become a pillar of empirical research in many computer science disciplines. Beyond their structural elements rooted in Cleverdon’s Cranfield experiments, shared tasks are a prime example of collaborative knowledge production, and thus an ideal object for examining how “right kinds of social organization and social norms enhance the prospects of acquiring truth” (O’Connor et al., 2024).
We conceive of a shared task as a structured dialogue among experts that forms a collective understanding of a task, how it can be operationalized, and how progress on it can be measured: organizers open the dialogue with a (1) task description; participants independently develop solutions, submit runs, and reply with (2) notebook papers; organizers, in turn, analyze the runs, appraise the contributions, and synthesize the task’s insights in an (3) overview paper; and, off-stage, attendees continue at a workshop through (4) talks and (5) discussions. By pooling independent solutions, a shared task collects the “wisdom of the crowds” (Surowiecki, 2004), with the organizers ensuring that every contribution is duly reflected.
Despite this, many of today’s shared tasks are reduced to their leaderboards (Ethayarajh and Jurafsky, 2020), although the comparison Harman envisioned requires understanding the systems, not just ranking them. This synthesis is documented in overview papers, which resemble scientific reviews of contributions made comparable a priori through a common task, data, and evaluation: they normalize terminology across differently described approaches, relate methodological choices to outcomes, and make visible patterns that emerge only across systems—including insights that leaderboards hide, such as a promising idea from a lower-ranked system.
Generative AI challenges this social epistemology. With agentic AI, a single researcher can explore a task’s solution space far more systematically than before, but if everybody relies on the same AI, will the pooling of independent thought become homogenized, raising questions of “epistemic reliance” (Wilholt, 2013; Greco, 2021)? And how do organizers make sense of, validate, and rely on LLM-generated contributions? Agent in the Loop makes knowledge synthesis an experiment in itself, contrasting for the first time human-authored overview papers with agent-generated syntheses, and reflecting on the questions this raises together with our participants and the organizers of other shared tasks.
Task
In its most generic form, the task we study can be described as follows:
Given a collection of research artifacts produced during participation in or organization of a shared task, create an agent that summarizes the scientific contribution documented by those artifacts.*
Agents must produce three types of summaries that are typically found in shared task overview papers:
- Cross-participant summaries that describe the individual approaches submitted to the host shared task and their results in a unified way.
- Structured comparisons of all approaches along salient dimensions.
- Quantitative analyses of the evaluation results of the submitted approaches, resulting in a critical appraisal.
An agent can be developed to address each subtask individually, but it can also use its own output of any subtask to solve another iteratively: all three subtasks share a unified input environment into which the output of each subtask is written (see the example below).
Approaches may range from zero-shot prompting to tool-using agentic setups that plan, run, and interpret task-specific analyses before generating text. Submitted agents are run on TIRA against held-out shared tasks and using LLMs we provide, ensuring blinded experimentation and reproducible execution. The three outputs are evaluated against the overview papers of designated test shared tasks, measuring and manually assessing how faithfully the salient scientific contributions are recovered.
| Subtask | 1: Cross-participant summary | 2: Structured comparison | 3: Quantitative analysis | ||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Input environment | The notebook papers, code, runs, and (optionally pre-computed) evaluation results of Teams A and B | ||||||||||||||||||||
| + outputs of Subtasks 2 and 3 | + outputs of Subtasks 1 and 3 | + outputs of Subtasks 1 and 2 | |||||||||||||||||||
| Output example | Team A uses an LLM with retrieval on the training data for in-context learning to improve classification. Team B uses a CNN trained on the official training set. |
|
Although Team A’s F1 score is slightly lower, their approach is superior due to its faster runtime. |
||||||||||||||||||
Data & Corpus
Our datasets pair descriptions of approaches that solve a given shared task with the summaries organizers wrote of those approaches. We derive them from two types of artifacts released during shared tasks: (1) notebook papers, which describe each approach in detail—potentially using different, incompatible language for the same underlying concept—and, where available, code implementing the described idea; and (2) the organizers’ overview paper, which summarizes all notebook papers in a unified, standardized language, typically as a textual description per approach, as a table comparing all approaches along a shared feature space, and as an appraisal of the approaches’ performance. These summary types are the basis for our three subtasks: for Task 1 we extract the textual per-approach summaries from the overview paper, for Task 2 the comparison table, where one exists, and for Task 3 the tabular and, if available, textual appraisal of the approaches’ performance.
Corpus construction
We build the corpus from the proceedings of regular shared task venues: SemEval, CLEF, TREC, NTCIR, FIRE, MediaEval, and the SISAP Indexing Challenge, which document each shared task together with the papers of its participants. For each edition, we collect the official proceedings—the ACL Anthology volumes of SemEval, the CEUR-WS working notes of CLEF, the NIST proceedings of TREC, the NII online proceedings of NTCIR, the FIRE archive and CEUR-WS working notes, the MediaEval history and CEUR-WS proceedings, and the SISAP challenge pages and conference proceedings—treating their index pages as the authoritative table of contents. DBLP serves only as a supplemental check that may confirm metadata; it never adds a paper or decides which task a paper belongs to. Overview papers are paired with notebook papers through titles and the section structure of a volume; for CLEF, where a lab section contains exactly one overview paper, its remaining papers are treated as notebook papers, while sections with several overview papers (an ambiguous case specific to CEUR-WS’s ordering) are set aside for manual review. Every PDF is retrieved and its text layer converted to Markdown with liteparse, which preserves heading structure, renders tables as Markdown, and extracts the images embedded in the PDF; captioned figures and tables, including vector graphics, are additionally extracted with PDFFigures2. For each notebook paper we additionally record links to participants’ code, and for each overview paper the number of teams and runs it reports, where stated; the participants’ runs themselves are not part of the corpus.
Preventing test data leakage
To prevent leakage into commercial and fine-tuned models, evaluation uses data from ongoing shared tasks for which neither notebook papers nor overview papers have been published yet. For vanilla open-weights models, their release date determines admissibility of a shared task as a test case (i.e., it must have been organized after that date). We use TIRA to ensure blinded experimentation, so that neither human nor AI participants know which cases are used for evaluation: you submit your agent as working software, which we deploy offline on a virtual machine we provide, without direct access to the test data during the task.
Protocol
Following prior shared tasks organized by our group at INLG (the TL;DR challenge and the Abstractive Snippet Generation task), we split the shared task into three phases:
- Participants develop agents using the provided training and evaluation data.
- We collect submissions via TIRA, which runs the agents in a sandbox to improve reproducibility.
- After all submissions are collected, we run them on the private test set with TIRA.
This means participants submit their agent as working software that generates summaries; the software is deployed offline on a virtual machine we provide, and during the task the test data is never available directly, only through the submission system.
Evaluation
For all shared tasks used as test cases, the human-authored passages extracted from their overview papers serve as reference rather than gold standard. Overview authors make choices about terminology, abstraction, salience, and dimensions of comparison, so alternative outputs may remain faithful and informative despite differing from the published reference. We therefore combine automatic, manual, and qualitative evaluation.
Automatic evaluation
- All three tasks. Generated outputs are compared with the corresponding reference passages extracted from the organizers’ overview papers using text summarization measures (e.g., variants of BLEU and BERTScore) and LLM-as-a-Judge approaches, including Nugget-based evaluations from TREC AutoJudge.
- Task 2 (structured comparisons). In addition, the generated comparisons are evaluated against the reference comparison tables using precision, recall, and F1.
Manual evaluation
Volunteering organizers of the selected host shared tasks assess the generated outputs against both the underlying participant contributions and their own reference summaries or tables. This allows us to identify outputs that are faithful but differ from the published reference, including cases where generated content exposes useful relationships or distinctions not made explicit in the organizers’ original synthesis. We further employ third-party assessors to enable a blinded comparison between generated and organizer-authored content.
Qualitative evaluation
Finally, we conduct a qualitative evaluation using entry and exit questionnaires for volunteering organizers and, if possible, expert interviews.
Submission
Participants submit their agent as working software via TIRA, which runs it in a sandbox for reproducibility and, after submissions close, on the private, blinded test set drawn from ongoing shared tasks. The full participation process is described on the rendition overview.
Related Work
Agent in the Loop builds on prior summarization shared tasks organized by our group at INLG, and connects to ongoing evaluation efforts for LLM-based judging:
- The TL;DR Challenge (INLG 2019). Summarization for social media, on which our three-phase protocol (development, sandboxed submission, blinded test) is modeled.
- Abstractive Snippet Generation (INLG 2020). Generating abstractive snippets for web pages, the second prior INLG shared task organized by our group.
- TREC AutoJudge (2026/27). A cross-task benchmark for comparing LLM judges for citation-grounded RAG systems; we draw on its Nugget-based LLM-as-a-Judge methodology and judges for our automated evaluation.
- TIRA & TIREx. The information retrieval experiment platform we use to host blinded, reproducible submissions as working software rather than static runs.
