UniAgent at CIKM 2026
Synopsis/Call
Agentic AI systems combine large language models, information retrieval, planning, and tool use to act on behalf of users in complex digital environments. Yet their evaluation remains difficult for lack of reproducible, bounded laboratories. A university administration is a closed-world, socio-technical environment in which many tasks follow standardized, well-documented processes. Such an environment is therefore particularly suited to a Cranfield-style evaluation of AI agents. The controlled, reproducible evaluation of agentic AI in such enterprise-like settings is an underexplored yet highly relevant problem.
This shared task offers the first reusable laboratory for the controlled evaluation of agentic AI in a university environment, namely the University of Kassel. Participants build agents that solve standardized administrative tasks (e.g., travel permits and reimbursements) by retrieving the relevant regulations and planning the necessary steps, while complying with the institution's codes of conduct. The document collection is compiled together with the OpenWebSearch.eu project.
We invite you to participate in the first UniAgent shared task, part of the CIKM 2026 AnalytiCup in Rome, 8 to 11 November 2026, featuring two sub-tasks.
Register for participation Join the UniAgent forum on TIRAShared Tasks
- Agentic AI in University Administration — four administrative task areas and two sub-tasks (retrieval, solving)
Important Dates
| Event | Date |
|---|---|
| Registration Opens; demo dataset and public development data available on TIRA | now |
| Registration Closes | 25 September 2026 |
| Run Submission | 15 October 2026 |
| Notification of Results | 3 November 2026 |
| Notebook Paper Submission | TBD |
| Presentation at the CIKM 2026 AnalytiCup | 8 November 2026 |
All times are in Anywhere on Earth (AoE) timezone. The CIKM 2026 main conference runs from 9 to 11 November 2026.
Participation
Participation is open to research groups and individuals from IR, NLP, AI, and adjacent communities, as well as industry teams.
- Register on TIRA. Sign up on TIRA, then register your team to the UniAgent 2026 task. The form asks for a team name (up to 30 characters), your email, affiliation, country, and the other team members. You may enter either of the two sub-tasks and revise that choice later.
- Develop your agent against the public part of the data and the demo dataset (see Data & Tools). You build retrieval on the mounted crawl with your own pipeline, and you may ship your own tools inside your image.
- Submit your runs as code (a Docker image) via TIRA, in the prescribed formats together with the agent behavior log. You may submit and run as often as you like during development. For the official evaluation, each team submits up to five runs, which you select yourself before any results are known.
- Submit a client-side cache with every official run, recording the model and tool calls that produced it: prompts, responses, and retrieved sources, verbatim. This is more than the reasoning-token log. We rank runs by success, and we also check whether that success was reached for the right reasons, which needs the full trace. Without a cache, a run cannot be told apart from a zero-shot prediction. This follows the caching approach used by the TREC AutoJudge track.
- Results and findings are presented at the CIKM 2026 AnalytiCup session. You describe your approach in a notebook paper.
Paper Submission
Results and findings are presented at the CIKM 2026 AnalytiCup session. You describe your approach in a notebook paper.
- As the paper title, use the format
<Your Team Name> at UniAgent: <Brief Description of your Approach>. - As the paper subtitle, use
UniAgent at CIKM 2026. - Detailed submission instructions will be published closer to the submission deadline; see Important Dates for the current schedule.
Program
The shared task results and findings will be presented during the CIKM 2026 AnalytiCup session on 8 November 2026 in Rome. The detailed schedule is to be announced.
| 8 November 2026 | CIKM 2026 AnalytiCup session, detailed program to be announced |
| TBA | Task overview and results presentation |
| TBA | Participant system presentations |
Keynotes
To be announced.
Awards
Winning teams are recognized with certificates, a presentation slot at the CIKM 2026 AnalytiCup session, and a mention in the overview paper. We foresee no monetary prizes at this stage, though awards may be extended if sponsorship becomes available.
Results
Results will be announced after the evaluation period; see Important Dates.
Resources
- Document collection (OpenWebSearch.eu): to be announced
- Baselines and code repository: to be announced
- Reproducibility platform (TIRA): to be announced
Organizing Committee
Contact
To participate, register on TIRA as described under Participation. Questions and discussion take place in the UniAgent forum on TIRA. You can also reach the organizers at uniagent@lists.its.uni-kassel.de. Releases are announced on this page.
