Agentic AI in University Administration 2026
Synopsis
Build retrieval systems (Task 1) and agents (Task 2) to support selected administrative tasks at a university. We are collaborating with administrative staff at the University of Kassel to ensure that the systems work on real data for real information needs and administrative tasks. The two tasks are independent: A submission to Task 1 does not require a submission to Task 2 or vice versa.
Register for participation Join the UniAgent forum on TIRA
Important Dates
| Event | Date |
|---|---|
| Registration Opens; spot-check datasets available on TIRA | now |
| Registration Closes | |
| Submission of the Systems | |
| Notification of Results | 3 November 2026 |
| Notebook Paper Submission | TBD |
| Presentation at the CIKM 2026 AnalytiCup | 8 November 2026 |
All times are in Anywhere on Earth (AoE) timezone. The CIKM 2026 main conference runs from 9 to 11 November 2026.
Task
In its most generic form, the class of tasks we study can be described as follows:
Given a digital environment and a problem or task (the objective) that is typically tackled by a person acting in the environment, develop an intelligent agent that achieves the objective on the person's behalf by planning and executing the necessary steps in accordance with the environment's codes of conduct.
Topic areas
The first edition of UniAgent covers three administrative tasks at a university:
- Business travel: review of travel permit requests before the trip (e.g., hotel rate limits, private extensions), based on the submitted forms and supporting documents.
- Privately advanced costs: reimbursement of out-of-pocket expenses paid by university staff, decided on the receipts.
- Internal procurement: review of procurement requests before ordering (e.g., comparison offers, framework contracts).
Example topic
Tasks
- Task 1 (Retrieval): Given a query that is intended to help solving an administrative task at a university, retrieve relevant documents for this query from the public website of the University, from the private intranet of the university, and state laws and regulations. Output a standard TREC-style run file.
- Task 2 (Solving): Build a working agent that solves complete administrative university tasks in a batch. For each task in the batch, the agent has all forms and administrative documents of the user as input and should output the administrative solution to the task.
Logging of Events and Tool Calls
To support evaluations that go beyound effectiveness in the target task, we ask that the executed software (in both tasks) produces an event log of tool calls (can be skipped if no tools are used). We plan to make this event log (potentially anonymized) public after the shared task, so that the community can work on new evaluation measures that can also incorporate if the logs of an agent are helpful (for instance, to decide if a decision of an agent is right for the right reason). The event log must be in gzip-compressed JSON Lines format in a file run-trace.jsonl.log.gz. Each line is one event (e.g., a tool call). The log can be used for auditability, i.e., the log can, additionally to the raw facts of what tool was called, also incorporate which aspects that help to describe why an tool call yielded which decision. If the hosted model reports token-usage metadata, they could be included as well. The full description of the required event log is available in the event logging contract.
Agentic Development
We want to support the development of approaches via coding agents. (Our baselines are also developed via coding agents.) You can point your agent to the AGENTS.md file and to the baselines. Agents should try to implement the software so that it follows the event logging contract.
Data
Document collections
As the administrative tasks are collected at the University of Kassel, the document collection that an Agent has access to are the documents that administrative staff at the university of Kassel usually uses to solve their tasks. This collection of documents includes (1) the University of Kassel’s public website (pages in German and English), (2) the University of Kassel’s intranet (pages in German and English), and (3) legal documents such as laws and court rulings of the State of Hesse (pages in German). All documents are crawled via the tech stack of the OpenWebSearch.eu project. We parse all documents to their main content in plain text. During the execution, the parsed documents are mounted into the execution and a software is expected to build the retrieval pipeline for the documents on their own (we provide baselines for this using PyTerrier and Pyserini).
Topics
We compile a collection of topics across the three task areas above from administrative staff trained in the University of Kassel's “AI -FrAIdays” seminar series. The topics are fully anonymized. Each topic consists of a user query (with any attached documents). Additionally, we have the expected output and annotations for query type, complexity, and rule area that are part of the ground-truth for evaluation (i.e., systems can not use this).
The case documents are the main input for a case. They consist of forms, receipts, invoices, and correspondence, usually as text-based PDFs (so that the agent needs to parse them). Image-only documents such as scans and photos are excluded, so no OCR or vision component is needed.
Available Datasets for Task 1 (Retrieval)
Please see tira.io/datasets?query=uniagent respectively the github repository for an overview of all datasets. All internal retrieval datasets will be organized in the same way. We make three spot-check datasets available:- retrieval-de-spot-check (GitHub)
- retrieval-en-spot-check (GitHub)
- retrieval-hessian-law-de-spot-check (GitHub)
Available Datasets for Task 2 (Solving)
Please see tira.io/datasets?query=uniagent respectively the github repository for an overview of all datasets. All internal solving datasets will be organized in the same way. An agent has three potential cases to resolve, determined by their directory name:dienstreiseantrag-SOME-ID: directories starting withdienstreiseantragcontain a business trip case.beschaffungsantrag-SOME-ID: directories starting withbeschaffungsantragcontain a procurement case.auslagenerstattung-SOME-ID: directories starting withauslagenerstattungcontain an expense reimbursement case.
retrieval-corpora directory that contains a set of document corpora that can be indexed and used for retrieval tools.
We make three spot-check datasets available:
- business-travel-spot-check (GitHub)
- procurement-spot-check (GitHub)
- expense-reimbursement-spot-check (GitHub)
Submission
We do not publish the underlying case data, neither for development nor for evaluation, due to privacy reasons (only the spot-check datasets above are available).
Please submit your agent as code submission via TIRA (i.e., a docker image build from the code). In TIRA, we then run the submissions in a sandbox against the data. See TIRA’s participation guide for details.
Evaluation runs on the UniAgent Test Set. The test data stays inside TIRA; participants submit code and TIRA runs it against the test data. The test set is still being finalized, and we will post its size and topic distribution here once it is finalized.
Execution environment
Submissions run in a sandbox without outbound network access, so external LLM APIs cannot be called. We host a small set of open-weight models, among them gpt-oss-20b and Qwen2.5-7B-Instruct, and expose them through a REST API inside the environment. Your software needs to read environment variables to connect to the REST API of the LLM.
Each execution gets 4 CPU cores and 40 GB of RAM, plus access to the hosted LLM model via REST API. The time limit is 5 minutes per case and 4 hours for a complete execution (the test dataset contains ca. 50 topics). Your agent can use whatever tools or protocol it has embedded in its docker image.
Output Format
Input: a task description (a user request, optionally with attached documents). Output depends on the chosen sub-task and follows prescribed formats: for Retrieval, the retrieved documents in the TREC run format; for Solving, the synthesized solution together with action logs in a common protocol format, the retrieved documents, and reasoning tokens as plain text.
Baselines
We provide simple TIRA-compatible baselines for all tasks. For retrieval (Task 1), we provide a PyTerrier baseline with standard lexical ranking models such as BM25. For Task 2 (Solving), this includes a deterministic always-rejected baseline and a tool-augmented smolagents baseline, using simple policies as rules. For details, please check here: Github
Submission
Submissions are handled via TIRA. We use code-submissions in TIRA to collect the systems: TIRA builds a Docker image from your source code and uploads this image to the Docker registry of TIRA. Please have a look at the baselines that show how this process works. Each team can contribute up to five submissions for the final evaluation (you can submit more systems, but when you have more then five submissions we ask you to prioritize them before the official evaluation results are out).
The full participation process is described on the participation overview.
Evaluation
We combine manual and automatic evaluation.
- Task 1 (Retrieval): The retrieval runs are evaluated against classical qrel-style relevance judgments that allow to evaluate diverse IR measures. Participants can decide which measure they want to focus on.
- Task 2 (Solving): Solutions are scored by accuracy: does the agent reach the same decision as the university's administrative staff? Accuracy is the one dimension we can evaluate rigorously, because it is the one for which we have ground-truth data, and it is the only criterion in this first iteration of UniAgent.
Whether an agent is usable in practice, however, depends on far more than accuracy. In the longer term we want to encourage the community to develop evaluation measures that go beyond it, so we ask agents to output not only their decision but also logs of their tool use, reasoning, and intermediate steps. We would like to make these logs available for follow-up research, for instance to study whether an agent's logs help administrative staff judge whether a decision was reached for the right reasons. Any such release requires review and clearance by the university; we will announce details in due course.
Across both tasks we also intend to factor in efficiency. The aspects that matter most to us are the wall-clock time an agent takes to solve a task and the number of input and output tokens it consumes against the hosted LLM.
Related Work
This task builds on and complements prior shared tasks:
- TREC Enterprise tracks (2005 to 2008). We revisit the enterprise setting with a more holistic aim: rather than retrieving information for a query, we automate achieving the objective, of which retrieval is a sub-task.
- TREC Million LLM track (2025 to 2026). Retrieving expert LLMs for specific tasks complements ours, as an agent may consult such an expert while executing its plan.
- TREC RAG track (2024 to 2026). A RAG response is a synthesized solution integrating retrieval, but does not include tool calling or richer environment interactions.
- TREC AutoJudge track (2026) Similar LLM-as-a-judge behaviour, just generic, whereas we only focus on administrative tasks at the university.
