Making co-design more responsible: a case study on developing an AI-based decision support system in dementia care (Preprint)

Abstract

Background: Emerging technologies like artificial intelligence (AI) require early-stage assessment of potential societal and ethical implications to increase their acceptability, desirability and sustainability. This paper explores and compares two of these assessment approaches: the responsible innovation framework originating from technology studies and the co-design approach coming from design studies. While the responsible innovation (RI) framework has been introduced to guide early-stage technology assessment through anticipation, inclusion, reflexivity and responsiveness, co-design is a commonly accepted approach in the development of technologies to support the care for frail older adults. However, there is limited understanding about how co-design contributes to anticipation of implications. Objective: This paper empirically explores how the co-design process of an AI-based decision support system (DSS) for dementia caregivers is complemented by explicit anticipation of implications. Methods: The case investigated is a international collaborative project that focused on the co-design, development, testing and commercialization of a DSS that is intended to provide actionable information to formal caregivers of people with dementia. In parallel to the co-design process, an RI exploration took place, which involved examining project members’ viewpoints on both positive and negative implications of using the DSS, along with strategies to address these implications. Results from the co-design process and RI exploration were analyzed and compared. In addition, retrospective interviews were held with project members to reflect on the co-design process and the RI exploration. Results: Our results indicate that, when involved in exploring requirements for the DSS, co-design participants naturally raised various implications and conditions for responsible design and deployment: protecting privacy, preventing cognitive overload, providing transparency, empowering caregivers to be in control, safeguarding accuracy and training users. Yet, when comparing the co-design results with insights from the RI exploration, we also found limitations to the co-design results, for instance regarding the specification, interrelatedness and context-dependency of implications and strategies to address implications. Conclusions: This case study shows that a co-design process that focuses on opportunities for innovation rather than balancing attention for both positive and negative implications, may result in knowledge gaps related to social and ethical implications and how these can be addressed. In the pursuit of responsible outcomes, co-design facilitators could broaden their scope and reconsider the specific implementation of the process-oriented RI principles of anticipation and inclusion.