LLM-Assisted Subjective Annotation Toolkit

Description

In complex multi-stakeholder systems, patients’ subjective experiences provide rich but unstructured data. To systematically incorporate these insights into iterative design, we propose a Subjective Annotation Toolkit, integrated directly into the Design Cycle of artifact creation and evaluation. The aim of this project is to develop and evaluate a Subjective Annotation Toolkit that systematically captures and structures patients’ experiential feedback for design improvement.

Tasks

• Implement a Pipeline for Patient-Centered Annotation on Artifacts
  • Patients interact with digital mockups/prototypes of dashboards, visualizations, or interfaces.
  • They annotate directly on the artifacts: mark confusing areas, suggest alternative visual encodings, highlight features they find useful or unnecessary, and express emotional responses (e.g., frustration, trust, satisfaction).
  • Annotations can include free-text or sketches, preserving natural patient expression.
• Enable LLM-Mediated Translation and Structuring
  • Translate subjective, colloquial, or metaphorical comments into structured thematic codes (e.g., “hard to find the lab results” → “navigation clarity issue”).
  • Map annotations to visualization-relevant dimensions, such as interpretability, trust, cognitive load, or engagement.
  • Summarize recurring themes across patients, generating multi-level outputs.
  • Experts and patients can review annotations and LLM-synthesized themes together in real-time collaborative sessions, enabling bidirectional knowledge transfer and immediate clarification of misinterpretations.
• Evaluation of the Approach
  • Conduct a small-scale study to assess the effectiveness, usability, and impact of the patient-centered annotation pipeline, including the quality of LLM-mediated interpretations and the value of insights for design improvements.
     

Requirements

• Good knowledge of machine learning/AI (especially NLP/LLMs).
• Familiarity with UX research and participatory design.
• Potentially, familiarity with qualitative research methods (coding, thematic analysis, diary studies).
   

References

https://arxiv.org/pdf/2507.15821

https://dl.acm.org/doi/full/10.1145/3613904.3641960

https://dl.acm.org/doi/abs/10.1145/3726302.3729916