Real-Time Visualizations of the Vocal Tract for Clinical Interventions (The VTV Project)
Research Team Lead
Melanie Baljko
Melanie Baljko
York University
Speech impairments often arise due to difficulties in controlling and coordinating the most important speech organ, the tongue. We plan to devise novel modes of visualization for use in a clinical setting for interventions focused on tongue movements. Our goal is to develop visualization techniques that provide salient feedback both to clients and to clinicians, for use in the client-clinician therapeutic dyad. This visualization-based approach is expected to be particularly beneficial for patients with multiple severe motor and sensory disabilities and who are less responsive to traditional therapy. A software-supported framework would provide an enhanced treatment tool and the opportunity for client engagement in therapeutically-beneficial exercises and be more time effective for the clinician.
Research Description:
The visualizations will be based on real-time data, streamed from sensors attached to the tongue and recording tongue movements. We plan to use the recently-developed WAVE system (NDI, Canada), which provides real-time tracking of the tongue and other speech articulators. To date, this system has been employed solely in speech-science laboratory settings and not in clinical applications. Our group has access to and expertise in the use of the device.

The visualization techniques will be audience-tailored; those intended for clients will emphasize aspects of the dynamic process that are relevant to embodied understanding. We will test hypotheses concerning the clinical value of various degrees of selective abstraction in these visualizations (as opposed to those that high degree of anatomical and process fidelity). Our group has already developed key modelling competencies that will be needed for the planned visualization techniques of the vocal tract (Yunusova, Baljko, et al) and has the expertise to incorporate these models using appropriate computer graphics and animation techniques (Faloutsos) and in an interactive system that makes use of appropriate UI design methodologies (Baljko). The client and clinician visualizations will each have their own functionalities (e.g., clients will likely control the speed of the animations and degree of selective abstraction, clinicians will need to annotate the data for clinically-relevant events). Selective abstraction will be viewed, in part, through the lens of aesthetic experience.

Baljko-1 We will experiment with the augmentation of the visualizations with tangible objects. Our group has expertise in the fabrication of anatomically-correct models of the vocal tract (LeBouthillier) and in the aesthetics of tangible, digital objects (Tenhaaf). The models are presently used in a surgical training domain, but we hypothesize their use in this clinical domain. Our group has an established collaboration concerning questions about embodied interactant experience (Tenhaaf, Baljko) and the development of tangible artifacts for use in clinical settings (LeBouthillier, Faloutsos). The artists among our collaborative team have an interest to also pursue these objects as art objects in themselves (Tenhaaf, LeBouthillier).

We will use MRI data to inform the fabrication of the tangible objects, a process with which our group has expertise (Faloutsos, LeBouthillier).

We will evaluate the visualizations using clinical evaluation criteria, in terms of the client's acquisition and maintenance of speech sounds that are especially dependent on tongue trajectory. Our group has expertise in clinical deployment (Yunusova) and in the evaluation of technologies for use by those with impairments.
Petros Faloutsos
Faculty Researcher, York University
Yana Yunusova
Faculty Researcher, University of Toronto
Francis LeBouthillier
Chair, Sculpture and Installation, OCADU
Nell Tenhaaf
Full Professor, Department of Visual Arts, York University
Alireza Moghaddam
Doctoral Candidate, Computer Science, York University
Brandon Haworth
Doctoral Candidate, Computer Science, York University
Elaine Kearney
Doctoral Candidate, Department of Speech Language Pathology, University of Toronto
Madhura Kulkarni
Lab Manager, Speech Production University of Toronto
Mehrnaz Zhian
M.Sc Student, Department of Electrical Engineering and Computer Science, York University
Rojin Majd Zarringhalam
M.Sc Student, Department of Electrical Engineering and Computer Science, York University
Vincci Tau
Research Assistant, Speech Production Lab, University of Toronto
Yue Zou
M.Sc Student, Department of Electrical Engineering and Computer Science, York University
UHN - Toronto Rehabilitation Institute