Agentic AI in University Administration 2026
Synopsis
Build retrieval systems (Task 1) and agents (Task 2) to support selected administrative tasks at a university. We are collaborating with administrative staff at the University of Kassel to ensure that the systems work on real data for real information needs and administrative tasks. The two tasks are independent: A submission to Task 1 does not require a submission to Task 2 or vice versa.
Register for participation Join the UniAgent forum on TIRA
Important Dates
| Event | Date |
|---|---|
| Registration Opens; demo dataset and public development data available on TIRA | now |
| Registration Closes | 25 September 2026 |
| Submission of the Systems | 15 October 2026 |
| Notification of Results | 3 November 2026 |
| Notebook Paper Submission | TBD |
| Presentation at the CIKM 2026 AnalytiCup | 8 November 2026 |
All times are in Anywhere on Earth (AoE) timezone. The CIKM 2026 main conference runs from 9 to 11 November 2026.
Task
In its most generic form, the class of tasks we study can be described as follows:
Given a digital environment and a problem or task (the objective) that is typically tackled by a person acting in the environment, develop an intelligent agent that achieves the objective on the person's behalf by planning and executing the necessary steps in accordance with the environment's codes of conduct.
Topic areas
The first edition of UniAgent covers three administrative tasks at a university:
- Business travel: travel permit requests, compliance checking (e.g., hotel rate limits, private extensions), and reimbursement processing. Topics are derived from labeled mail exchanges between administrative staff and university staff.
- Privately advanced costs: reimbursement of out-of-pocket expenses paid by university staff.
- Internal procurement: identification, selection, and ordering of goods through the university's procurement system.
Tasks
- Task 1 (Retrieval): Given a query that is intended to help solving an administrative task at a university, retrieve relevant documents for this query from the public website of the University, from the private intranet of the university, and state laws and regulations. Output a standard TREC-style run file.
- Task 2 (Solving): Build a working agent that solves complete administrative university tasks in a batch.
For each task in the batch, the agent has all forms and administrative documents of the user as input and should output the administrative solution to the task.
For each Solving topic, submissions must write a machine-readable event log as JSON Lines in a file named
run_trace.jsonl. Each line is one event. The log is used for auditability: it should show which inputs, retrieval results, model calls, and tool calls led to the submitted decision. It must not contain hidden chain-of-thought. If the hosted model reports token-usage metadata, including reasoning-token counts, these counts should be recorded as described below.Each event should follow this structure:
{ "trace_version": "1.0", "run_id": "team42-systemA-2026-10-15T12:34:56Z", "case_id": "dienstreiseantrag-03", "event_id": "evt-0007", "parent_event_id": "evt-0006", "timestamp": "2026-10-15T12:35:14.230Z", "event_type": "model_call", "actor": "agent", "name": "draft-decision", "input": { "prompt": "Check the travel request for completeness and rule compliance.", "documents": ["antrag.pdf", "bahn-ticket.pdf"] }, "output": { "response": "The request is incomplete because the return trip is missing...", "cited_sources": [ { "source_id": "antrag.pdf", "page": 1, "quote": "Ende Dienstgeschäft: 11.02.2026" } ] }, "usage": { "input_tokens": 1842, "output_tokens": 211, "reasoning_tokens": 96, "total_tokens": 2149 }, "status": "ok", "error": null }The field
event_typeshould be one ofinput,retrieval,model_call,tool_call,observation,decision, orerror. Fields that are not applicable may be set tonullor omitted. For deterministic review, all prompts, model responses, tool arguments, tool outputs, retrieved document identifiers, cited passages, errors, and token counters should be logged verbatim where possible.The final event of each trace should have
event_type: "decision"and contain the submitted answer, the decision label, and the evidence used for the decision:{ "trace_version": "1.0", "run_id": "team42-systemA-2026-10-15T12:34:56Z", "case_id": "dienstreiseantrag-03", "event_id": "evt-0012", "parent_event_id": "evt-0011", "timestamp": "2026-10-15T12:36:02.018Z", "event_type": "decision", "actor": "agent", "name": "final-answer", "output": { "decision": "deny", "answer": "The request must be returned for correction...", "evidence": [ { "source_id": "antrag.pdf", "page": 1, "relevance": "The end date of the business activity is before the travel date." }, { "source_id": "bahn-ticket.pdf", "page": 1, "relevance": "Only the outbound trip is documented." } ] }, "usage": null, "status": "ok", "error": null }
Example topic
Data & Tools
The environment comprises a document collection, a set of topics, and a hosted language model.
Document collection
As the administrative tasks are collected at the University of Kassel, the document collection that an Agent has access to are the documents that administrative staff at the university of Kassel usually uses to solve their tasks. This collection of documents includes (1) the University of Kassel’s public website (pages in German and English), (2) the University of Kassel’s intranet (pages in German and English), and (3) legal documents such as laws and court rulings of the State of Hesse (pages in German). All documents are crawled via the tech stack of the OpenWebSearch.eu project. We parse all documents to their main content in plain text. During the execution, the parsed documents are mounted into the execution and a software is expected to build the retrieval pipeline for the documents on their own (we provide baselines for this using PyTerrier and Pyserini).
Topics
We compile a collection of topics across the three task areas above, drawing on the hands-on expertise of administrative staff trained in the University of Kassel's “AI FrAIdays” seminar series. The topics are fully anonymized. Each topic consists of a user query (with any attached documents). Additionally, we have the expected output and annotations for query type, complexity, and rule area that are part of the ground-truth for evaluation (i.e., systems can not use this).
The case documents for a topic are part of the input: forms, receipts, invoices, and correspondence, usually as text-based PDFs (so that the agent needs to parse them). Please note that the document collection above is already parsed with plain text extracted as they come from a homogenous source of text, whereas the case documents are from heterogenous text sources, so it is required for the agents to process the case documents from the PDF. Image-only documents such as scans and photos are excluded, so no OCR or vision component is needed.
Datasets
For Task 1 (Retrieval), we provide public spot-check datasets for testing retrieval submissions.
Each dataset contains topics in queries.jsonl and relevance judgments in standard TREC qrels format (qrels.txt).
For Task 2 (Solving), we provide a small public
business-trip spot-check dataset. It contains input documents similar to the real cases underdecision-trail/, as well as judgments and justifications (ground-truth.jsonl).
Please submit your agent as code (a Docker image), and TIRA runs it against the data. See TIRA’s participation guide for details.
Execution environment
Submissions run in a sandbox without outbound network access, so external LLM APIs cannot be called. We host a small set of open-weight models, among them gpt-oss-20b, and expose them through a REST API inside the environment. Your software needs to read environment variables to connect to the REST API of the LLM.
Each execution gets 4 CPU cores and 40 GB of RAM, plus access to the hosted LLM model via REST API. The time limit is 5 minutes per case and 4 hours for a complete execution (the test dataset contains ca. 50 topics). Your agent can use whatever tools or protocol it has embedded in its docker image.
Output Format
Input: a task description (a user request, optionally with attached documents). Output depends on the chosen sub-task and follows prescribed formats: for Retrieval, the retrieved documents in the TREC run format; for Solving, the synthesized solution together with action logs in a common protocol format, the retrieved documents, and reasoning tokens as plain text. Validation tools are provided.
Baselines
We provide simple TIRA-compatible baselines for all tasks. For retrieval (Task 1), we provide a PyTerrier baseline with standard lexical ranking models such as BM25. For Task 2 (Solving), this includes a deterministic always-rejected baseline and a tool-augmented smolagents baseline, using simple policies as rules. For details, please check here: Github
Submission
Submissions are handled via TIRA. We use code-submissions in TIRA to collect the systems: TIRA builds a Docker image from your source code and uploads this image to the Docker registry of TIRA. Please have a look at the baselines that show how this process works. Each team can contribute up to five submissions for the final evaluation (you can submit more systems, but when you have more then five submissions we ask you to prioritize them before the official evaluation results are out).
The full participation process is described on the participation overview.
Evaluation
We combine manual and automatic evaluation.
- Task 1 (Retrieval): The retrieval runs are evaluated against classical qrel-style relevance judgments that allow to evaluate diverse IR measures. Participants can decide which measure they want to focus on.
- (2) Solving. The synthesized solution is scored via accuracy, i.e., would the agents make the same decisions as the administrative staff of the university. Accuracy is one measure that we can properly evaluate, because we have the ground-truth data for this. However, If an Agent can be used in practice can depend on many more dimensions than accuracy, and, in the long term (not in the first iteration), we also want to encourage the community to research on evaluations that go beyound accuracy. For this, we ask that agents do not output only their decision (which is the only thing we look at for the evaluation in the first iteration of UniAgent), but also output logs of their tool-use, reasoning, etc. We plan to make (anonymized) outputs of agents publically available after the shared task, so that the community can work on evaluation measures that go beyond accuracy, and, for example, take into consideration if the logs of an Agent are helpful for administrative staff, for instance, to decide if a decision was reached for the right reasons.
Across the two tasks we also intend to include efficiency into the evaluation. The main aspects of efficiency for us are the overall elapsed time to solve a task and the number of input and output tokens against the hosted LLM that an Agent used during its execution.
Related Work
This task builds on and complements prior shared tasks:
- TREC Enterprise tracks (2005 to 2008). We revisit the enterprise setting with a more holistic aim: rather than retrieving information for a query, we automate achieving the objective, of which retrieval is a sub-task.
- TREC Million LLM track (2025 to 2026). Retrieving expert LLMs for specific tasks complements ours, as an agent may consult such an expert while executing its plan.
- TREC RAG track (2024 to 2026). A RAG response is a synthesized solution integrating retrieval, but does not include tool calling or richer environment interactions.
- TREC AutoJudge track (2026) Similar LLM-as-a-judge behaviour, just generic, whereas we only focus on administrative tasks at the university.
