Development of the Digidiadem digital application for telemedical self-examination of speech and memory with self-assessment using artificial intelligence for quick detection of cognitive disorders – study protocol

Summary

Aim: With the growing number of older adults and the declining number of younger specialists, assessment of memory and language functions in-person will become increasingly challenging. Digital applications will therefore gain importance, yet such tools are currently lacking in the Czech Republic. The aim of this study is to present the methodology of a unique digital application designed to independently examine and automatically evaluate memory and language functions in a home environment without the presence of an administrator. Methods: The Digital Diagnostics of Dementia, with the acronym Digidiadem derived from the initial parts of the words, is based on tasks that can be administered in the form of a structured interaction with a computer. It includes sentence learning and recall, scene description and recall of information from the scene, naming and recall of 20 pictures, names of animal, and repetition of pseudowords. This will be followed by in-person assessment using standardized instruments: the Repeatable Battery for the Assessment of Neuropsychological Status (RBANS), the Mississippi Aphasia Screening Test (MAST), and the certified tests Amnesia Light and Brief Assessment (ALBA) and Picture Naming and Immediate Recall (PICNIR), along with mood and functional independence questionnaires. The study will include cognitively normal individuals without neuropsychiatric history and with normal RBANS scores, as well as patients with mild cognitive impairment and mild dementia. Digital and in-person test results will be correlated and subsequently compared using the area under the receiver operating characteristic curve (AUC). Results: Scores from the in-person battery will characterize cognitive performance across groups and at the same time verify the concurrent validity of the Digidiadem. The discriminative validity of the Digidiadem application will be evaluated using inferential statistics and AUC analysis. Conclusion: The project will result in a modern digital application, the Digidiadem, intended for self-assessment of memory and language functions across various settings. Automated self-evaluation using artificial intelligence methods will provide users with information about their cognitive functions and enable convenient detection of cognitive impairment.

Keywords:

dementia – Alzheimer‘s disease – memory – Speech – Telemedicine – Picture Naming and Immediate Recall (PICNIR) – cognitive test – Amnesia Light and Brief Assessment (ALBA) – mild cognitive impairment (MCI) – digital electronic testing – digital dementia diagnostics (Digidiadem)

This is an unauthorised machine translation into English made using the DeepL Translate Pro translator. The editors do not guarantee that the content of the article corresponds fully to the original language version.

Introduction

The growing elderly population in the Czech Republic is accompanied by an increase in the incidence of chronic diseases and cognitive disorders. Screening for these conditions can take place at several levels. General practitioners offer the widest availability, thanks to their large numbers [1–4]. Another option is to visit specialists with expertise in this area—neurologists, psychiatrists, and geriatricians [5]. An interesting option in the Czech Republic is to have one’s memory assessed at certified pharmacies throughout the country [6,7].

General practitioners and other specialists are recommended to use very short tests lasting up to five minutes, such as the Amnesia Light and Brief Assessment (ALBA), Picture Naming and Recall (PICNIR), and Mini-Cog, the administration of which by general practitioners is covered by health insurance companies [1,3,4,8,9]. The innovative and original Czech tests ALBA and PICNIR are very brief, sensitive to the detection of mild cognitive impairments, and certified by the Czech Ministry of Health. A more detailed assessment of cognitive functions is possible within 10–30 minutes using Czech versions of international tests [10–22].

The final level involves the option of assessing cognitive function in the home environment, either independently or with the support of a loved one. Given that physicians are also aging and their numbers are declining, digitization and the use of artificial intelligence (AI) offer a potential solution. AI-based digital applications can be used for the early detection of cognitive disorders as well as for prevention, directly in the patient’s home environment. Furthermore, telemedicine, supported by modern digital technologies, expands access to care even to remote areas with limited access to healthcare services. Another advantage is that if the tasks are adapted for computer-based administration, they do not need to be performed by a trained specialist.

This innovative approach to testing already exists in the Czech Republic. The unique and first Czech test, ALBAV, was developed for the Czech context as an electronic self-assessment of memory; it has been validated for the electronic detection of mild cognitive impairment [25,26]. The Czech digital application called ALBAV is available for computers on the website [27]. Another example of the use of electronic testing is the electronic data collection for the Czech comprehension test TEPO, which is designed to assess sentence comprehension in children aged 3–8 years [23]. Furthermore, we have verified that, with certain adaptation, the ALBA, PICNIR, and Addenbrooke’s Cognitive Test (ACE) can be administered remotely by an examiner using two monitors, yielding results similar to those obtained during in-person testing [24].

Another approach to telemedical memory assessment is digital self-testing with automatic result evaluation. Currently, AI methods can provide such analysis. A systematic review has shown that AI methods for speech and language processing yield promising results in predicting cognitive decline in Alzheimer’s disease. However, only a few such approaches have been successfully implemented in clinical practice so far. The main limitations of the field include insufficient standardization, limited comparability of results, and a partial disconnect from clinical applications. The use of AI in analyzing voice recordings from neuropsychological examinations makes it possible to develop a process leading to fully automated assessment. This can serve as an accessible and effective tool for both screening and predicting the progression of cognitive impairment [28,29]. Digital applications for cognitive screening are already being developed abroad, but no tool specifically adapted to the Czech language currently exists. The aim of this study was therefore to develop our own methodology and a digital application in which the computer assumes the role of the examiner. The computer conducts the interview, provides instructions, displays images, and records the examinee’s spoken responses. These are then transcribed into text and automatically analyzed using AI methods, enabling a form of telemedical testing similar to an in-person examination. Another objective of the study is to verify the psychometric properties of the Digidiadem digital application based on the proposed methodology and to prepare it for routine use in the Czech Republic.

 

Methodology

The application is called "Digital Dementia Diagnostics," abbreviated as "Digidiadem" from the first letters of the words. It is based on tasks that can be performed in the form of a conversation between the person being tested and a computer or tablet. The development of the application consists of several phases. First, it was necessary to create the Digidiadem web application, which consists of several speech and memory tests based on predetermined rules. In the next step, the psychometric properties of the Digidiadem application will need to be validated by testing as many patients with cognitive impairments and volunteers with normal cognitive functions as possible. At the same time, we will strive to achieve the best possible conversion of the subjects’ speech utterances into text. Finally, the text will be analyzed using AI methods. The study is designed as a prospective cross-sectional validation study of the Digidiadem digital diagnostic application.

The application is being developed at three sites. The Memory Disorders Clinic at the Department of Neurology of the Královské Vinohrady University Hospital (FNKV) and the Third Faculty of Medicine of Charles University (LF UK) in Prague is responsible for designing tasks for the Digidiadem application and for examining two groups of volunteers—healthy individuals and patients with mild cognitive deficits. The Faculty of Applied Sciences at the University of West Bohemia in Pilsen is in charge of the technical implementation of the application, the conversion of spoken language into text, and its analysis. The Institute of Physics of the Czech Academy of Sciences will use machine learning methods to classify individuals based on their speech.

In designing the application, we drew on our many years of experience developing the certified ALBA and PICNIR tests and supplemented them with semi-structured and free-speech tests. The result is a digital battery of speech and memory tests. The computer gradually provides audio and written instructions and displays various images on the monitor. The assessment consists of several subtests. First, vision and hearing are tested. This is followed by a single repetition and learning of a 10-word sentence: “A month ago, an older man met a woman he knew after 17 years.” The next task involves freely describing the picture for one minute and then recalling as many words and objects from the picture as possible. Next, 20 black-and-white pictures appear one after another, each to be named with a single word, and the task is to recall as many of the picture names as possible within half a minute. This is followed by a half-minute task involving the verbal naming of animals. The next task involves recalling an introductory sentence consisting of 10 words. The test concludes with the repetition of several nonsensical words, known as pseudowords.

 

Design and Rationale for Tasks in the Digidiadem Digital Application

When developing the app, we took several factors into account simultaneously:

1)
Inclusion of tasks requiring only spoken responses;

2)
selecting various types of speech—ranging from structured speech production through semi-structured to free, unstructured speech—to enable conversion to text and analysis by artificial intelligence;

3)
Inclusion of memory testing as the most important cognitive function;

4)
utilizing the principles and items from the original Czech and innovative cognitive tests ALBA and PICNIR, converting them into electronic format;

5)
compiling a set of several different tasks to assess multiple abilities and to maintain motivation to complete the test;

6)
maintaining a reasonable total battery duration of up to 15 minutes for several reasons (appropriate workload, not keeping the examinee waiting too long, and maintaining attention and motivation to complete the entire battery);

7)
creating a setup that allows for independent administration without the presence of an administrator, using a custom digital voice that provides the examinee with instructions on which task to perform;

8)
creating a dialog form between a computer and a person.

 

Based on these rules, we developed the Digidiadem application, which assesses several types of speech and memory over the course of approximately 11 minutes. Its content is schematically illustrated in Fig. 1. The assessment includes tasks based on structured speech (repeating and recalling sentences, naming and recalling pictures, repeating nonsense words—so-called pseudowords), semi-structured speech (verbal production of animal names, recalling items from a scene in a picture), and free, unstructured speech (describing a scene in a picture).

The structured speech assessment is based on two certified tests: ALBA and the “hedgehog” version of PICNIR [16,19]. Both tests have been adapted for a digital environment, while their basic principles have been preserved. At the beginning of the assessment, the participant repeats and learns a 10-word sentence, compared to six words in the ALBA test. The longer sentence increases the difficulty of the task, which the computer generates automatically without difficulty. While the distracting task involving gestures lasts approximately one minute in the ALBA test, in the Digidiadem application, the delayed recall of the sentence is placed near the end of the test with an extended interval of about 10 minutes. This increases the test’s sensitivity. In the meantime, several other tasks are performed, including the PICNIR test, which has been modified in several ways compared to the original version. Unlike the paper-and-pencil version, the 20 black-and-white pictures from the “hedgehog” version of the PICNIR are displayed on the screen in reverse order and one at a time. This allows the app to capture the verbal naming of the pictures and simultaneously distinguish the names of individual pictures. These pictures were selected so that they could be named in Czech using a single word [30,31]. This is followed by picture name recall. To facilitate verbal recall, the interval was shortened to half a minute compared to one minute in the original version, where the recalled picture names had to be written by hand. The results of the shorter verbal recall and the longer written recall should be comparable [24]. The ALBA test lacks a nonverbal gesture test. The structured speech assessment at the end of the application is supplemented by the repetition of nonsense words, which preliminary analyses have shown to be a useful indicator for distinguishing between individuals with intact and impaired cognitive functions [32].

The assessment includes a very brief but diagnostically informative verbal production task involving animals, which is an example of semi-structured speech. In this task, the subject is asked to name as many different animal names as possible as quickly as possible within half a minute, compared to the usual one minute during an in-person examination [33]. Shortening the time intervals for selected tests also reduces the overall duration of the application.

To assess free, unstructured speech production, it was necessary to create a special picture. The scene in the picture includes a storyline, living creatures, and inanimate objects that are universal, sex-neutral, familiar to both Czech and foreign conceptions of life, and timeless. The advantage of such free speech based on a pictorial stimulus is the ability to easily assess the nature of thinking, reasoning, and vocabulary. The test subject’s task is to describe the picture for one minute. This is followed by a picture recall task, during which the subject has half a minute to list as many events and elements from the picture as possible, such as living creatures, plants, and animals.

 

Development of the “Life on the Shore” Scene for Description by Test Subjects

The scene titled “Life on the Shore” is based on the actual setting of a flooded sandpit, which was first captured in a photograph. As the activity on the shore was observed repeatedly, additional elements were added. After several sketched versions, the final form depicted in Fig. 1 [32] was created.

The illustration depicts a family enjoying time by the water on a hot summer day. The scene features three environments—land, water, and air—in which various situations unfold. On the shore, the mother is lying on a blanket, sunbathing and reading a book. Next to her is a picnic basket with snacks. Two children, a boy and a girl, are tossing a ball back and forth. A toddler, who was originally playing with buckets and a shovel near the blanket, is quietly wading into the water while no one is watching. A man is swimming on the surface, and nearby is a duck with her ducklings. The water is teeming with life: a fish leaps out of the water in the middle, and at the water’s edge, a frog jumps from a rock into the reeds. There is also a fisherman in the reeds trying to catch fish, but his empty landing net suggests he has not yet had any success. A dog is running around the meadow, chasing a squirrel that is trying to escape into a tree. The dog is still on a leash, indicating that it has run away from someone. In the sky, you can see flying ducks, an airplane, and the sun partially hidden behind clouds. Several trees grow at the edge of the meadow, with a pigeon perched on one of them. Two symbols of potential danger have been successfully incorporated into the entire scene: the aforementioned child falling/climbing into the water and the squirrel dashing up a tree to escape the dog.

The way the picture is described is important. We observe whether people notice more details or describe the picture comprehensively, and whether they also observe the relationships between the characters and the causal connections. An example of such a connection is the fisherman who has a fish on the line, causing the rod to visibly bend. To illustrate this, Table 1 [32] presents descriptions of the image by a healthy person and a person with dementia.

 

The examination process using the Digidiadem application

The assessment takes the form of an interaction with a computer. Since the Digidiadem application is still in development, it will be necessary for an administrator to supervise the assessment and, if needed, assist with its operation in our study for the time being. However, the administrator’s role will be purely passive, limited to addressing any technical issues that may arise. The administrator will not interfere with the tasks themselves and will not respond to requests from participants for clarification of the instructions or similar questions. The administrator’s presence is planned only for the duration of the study to verify that the application is functioning correctly and to provide feedback for its further development. In the future, the goal is for participants to be able to complete the Digidiadem test on their own. The test subject listens to instructions from the speaker, which are simultaneously displayed on the monitor, and responds aloud into the microphone. For verification purposes, the session is recorded on an external voice recorder. The test does not require typing on a keyboard or complex computer operations—answers are verbal only. This type of telemedicine testing is convenient and very similar to an in-person examination. Tasks are time-limited or voice-activated, and the user switches between tasks with a mouse click.

 

Research Examination Procedure and Methods Used

The entire examination will include the collection of sociodemographic and medical history data (10 min), electronic testing using the Digidiadem application (10–15 min), and a neuropsychological battery we have developed to validate the application under development (40–60 min). We estimate that the assessment of healthy individuals will take approximately one hour, and that of patients approximately one and a half hours.

Several methods were selected for the validation of the application to ensure they align with the application’s tasks while providing a comprehensive and effective assessment of cognitive functions in the shortest possible time. The goal was to keep the examination duration around one hour so that a sufficient number of individuals could be assessed without causing undue burden, particularly for older patients.

Our test battery consists of the following methods: the Repeatable Battery for the Assessment of Neuropsychological Status (RBANS) [34–36], the Mississippi Aphasia Screening Test (MAST) [37–39], and a personal assessment using the certified Amnesia Light and Brief Assessment (ALBA) tests and the face-to-face version of the written Picture Naming and Attributes Test (PICNIR) [16,19,40]. We purchased the Czech version of the RBANS; the other methods are freely available at no cost. We will also include questionnaires to assess participants’ emotional state and functional status (Geriatric Depression Scale [GDS], Beck Anxiety Inventory [BAI], and Functional Activity Questionnaire –⁠ CZ [FAQcz]) [33]. Using these questionnaires, we assess mood disorders and anxiety, and evaluate participants’ self-sufficiency—both through self-assessment in healthy volunteers and by measuring the degree of impairment in patients with cognitive impairment. The examination procedure and order of methods are shown in Fig. 2.

 

Research Cohorts

The Digidiadem application will be validated primarily by comparing two groups of individuals—healthy individuals living in a home environment with no history of neuropsychiatric disorders and normal cognitive functions (hereinafter abbreviated as NOS) and patients with cognitive impairments. A more detailed breakdown of these groups is shown in Figure 3. The goal is to examine at least 200 individuals. Identifying, examining, and processing data for a single individual takes 4–8 hours, depending on the degree of cognitive impairment. Inclusion criteria for the NOS group include being over 45 years of age and having Czech as a native language. Individuals in the NOS group who retain their independence are classified, based on the results of cognitive tests, into NOS with normal cognitive functions (hereinafter referred to as NOS-NKF) and NOS with mild cognitive impairment (NOS-MKP, with at least one low score on the RBANS). Exclusion criteria for individuals in the NOS group include a history of brain injury, psychiatric history, or current psychiatric medication. To recruit participants, we use contact databases from previous studies and various campaigns and recruitment events to identify new candidates (e.g., employees of FNKV and the Third Faculty of Medicine, Charles University; the internet; articles or television appearances; social media; and acquaintances or relatives of patients). Their examinations take place in person at the Memory Disorders Outpatient Clinic at the Neurology Department of FNKV and the Third Faculty of Medicine, Charles University, in Prague. They receive financial compensation for their participation.

Other groups will consist of patients with mild cognitive impairment (MCI) and mild dementia, as defined by the international DSM-5 criteria, from the Memory Disorders Outpatient Clinic at the Neurology Department of the FNKV and the Third Faculty of Medicine, Charles University in Prague [41]. An important prerequisite is that cognitive deficits be only mild in severity, so that such patients can manage to complete the electronic assessment. This is also consistent with the app’s purpose of identifying patients in the early stages of cognitive impairment. Patients with more severe impairment would not be able to interact with the computer and the app. Patients have various diagnoses, primarily Alzheimer’s disease and frontotemporal dementia.

Transcriptions of the speech describing the image and the repetition of nonsense words will be manually transcribed into accurate text form. These texts will serve as a reference source for comparison with the automatic conversion of spoken speech to text. The manually transcribed records will be used to train and improve the speech-to-text converter. At the same time, this will make it possible to assess the accuracy of speech-to-text conversion. All these steps will lay the groundwork for remote speech self-assessment with automatic evaluation and classification of individuals into those with and without cognitive deficits.

 

Statistical Analysis and Evaluation of the Diagnostic Accuracy of the Digidiadem Application

The data analysis plan includes descriptive statistics for individual groups, correlation analysis to assess concurrent validity between the results of the digital application and neuropsychological and cognitive tests, and an evaluation of discriminant validity using the receiver operating characteristic (ROC) curve and the area under the curve (AUC). The effectiveness of the Digidiadem app will be evaluated based on its ability to discriminate between cognitively healthy individuals and patients with mild cognitive impairment or dementia, using AUC values and effect size as measured by Cohen’s d.

 

Discussion

The goal was to design a methodology and develop a digital application in which the computer takes on the role of administrator. In designing the Digidiadem digital application, we drew on our many years of experience developing the certified ALBA and PICNIR cognitive tests [10–19,42] and supplemented them with semi-structured and free-speech tests. The inclusion of a picture-description task to assess free speech is well-founded. Internationally, narrative pictures are part of several fee-based diagnostic batteries, such as the Boston Diagnostic Aphasia Examination with the “Cookie Theft” picture or the Western Aphasia Battery with the “Picnic scene” picture. According to international research, the picture description task is useful in the diagnosis of mild cognitive impairment as well as dementia. However, it also has applications in the differential diagnosis of various types of dementia [43]. Story pictures are also part of screening tests, such as the MAST [37–39]. Based on the results of international studies, describing a story picture can be considered a high-quality diagnostic tool that allows for the assessment of a wide range of cognitive deficit features in spontaneous speech. Examples include the assessment of word count, coherence, speech fluency, syntax, prosody, utterance length, articulation rate, use of figurative language, and more [44,45].

The automatic detection of diseases and disorders based on speech and memory functions is an intensively studied issue across various languages. Telemedical examinations offer significant benefits in the screening for neurodegenerative brain diseases. However, it is important to note that this type of examination does not replace a comprehensive neuropsychological evaluation by a specialist. The reason is that both the administration and evaluation of the results are performed by a computer, which interprets only the test scores achieved according to pre-set criteria and is therefore unable to place the result within a broader diagnostic context. A significant limitation is the absence of clinical observation of the examinee by an experienced clinician. Computer processing cannot account for a person’s behavior during testing, such as fatigue, test anxiety, a tendency toward frustration upon failure, and so on. These manifestations are important for differential diagnosis—for example, distinguishing a cognitive disorder from symptoms of depression or anxiety. Another pitfall may be differences in digital literacy among older adults. Self-administered testing in a home setting also presents a problem due to the lack of control over external variables. Unlike standardized testing in a clinic, results can be skewed by disruptive factors, such as noise or the presence of other people, or by technical parameters such as microphone quality and internet connection quality. Technical issues with the application may prevent the examination from being started or completed.

Despite these limitations, this telemedicine examination plays an irreplaceable role in initial screening. Its goal is not to establish a definitive diagnosis, but rather to identify at-risk individuals early on, who are then referred for a comprehensive medical and neuropsychological examination to complete the differential diagnostic process.

 

Conclusion

Our app innovatively combines clinical practice, machine learning, and speech processing methods to automatically detect the presence of cognitive impairments in users. We will validate the Digidiadem app by comparing results between patients at the FNKV and 3rd Faculty of Medicine, Charles University outpatient clinic for memory disorders and adult and elderly volunteers with normal cognitive function. Participants’ results obtained through the Digidiadem app will be compared with neuropsychological test scores to verify the concurrent validity of the app.

The goal is to create the Digidiadem digital app, which will enable self-assessment of memory and speech disorders with AI-supported automatic evaluation. The app may be useful not only for patients with cognitive difficulties but also for other groups, such as stroke survivors. If the application is successfully developed and validated, it could become part of routine medical practice. It could thus offer an effective screening tool for the growing population of older adults with cognitive difficulties at a time when there is a shortage of physicians and their availability continues to decline.

 

Ethical Considerations

This work was conducted in accordance with the 1975 Declaration of Helsinki and its revisions in 2004 and 2008. The project was approved by the Ethics Committee of the Královské Vinohrady University Hospital on November 1, 2023, under reference number EK-VP/51/0/2023 (Digidiadem project).

 

Grant Funding

This work was supported by a project of the Technology Agency of the Czech Republic under the SIGMA DC3 Program [TQ01000332 Telemedical Self-Assessment of Speech and Memory for the Rapid Detection of Cognitive Disorders Using Machine Learning], COOPERATIO Q38 of Charles University and RVO [Královské Vinohrady University Hospital, 00064173].

 

Conflict of Interest Statement

Aleš Bartoš developed the ALBA, PICNIR, and ALBAV tests and owns the stimuli used in the Digidiadem digital application. Marie Krejčová and Michaela Zapletalová have no conflict of interest. The application will be deployed in clinical practice in the future for a fee. This is a project funded by the TAČR grant agency, which requires the commercialization of the results.

 

Table 1. Examples of differing image descriptions by a healthy person and a patient with dementia.

Note the relational description provided by the healthy individual. The description contains elaborate sentences with numerous adjectives. In contrast, the description by the patient with dementia is much more sparse. It offers only a list of elements. Sentence agrammatisms, repetitions of statements, and anomic pauses indicating difficulty with naming are also evident.

 

Description of the picture by a healthy person

“Mom is lying on a lounge chair under a beach umbrella, reading a book. Dad is fishing. Someone is swimming in the water. There’s a frog, fish, and a duck with her ducklings. A picnic is set up on a blanket. A little girl and a boy are playing with a ball, throwing it back and forth. A dog is chasing a squirrel. An airplane is flying; the sun is shining. A flock of birds. The children have a red-and-white ball. One of the children falls into the water. There’s a little shovel there, so they were playing with the sand. Mom’s book is red. There are four trees and some reeds. Dad’s fishing rod is bending right now, so he’s probably caught a fish. The swimmer is doing the crawl. A frog is jumping from a rock into the water. There’s cattail growing there.”

Description of the picture by a patient with severe dementia

“Here, the kids are playing. Here, here’s a fisherman. Here… there’s… it’s going to be… like… Water here. Here, that one’s a swimmer. Here’s a fish. Here’s a frog. Here, here it’s going, here’s some kind of animal that’s walking, that’s kind of climbing a tree. And here’s a dog, well. Here, the boys are playing some kind of game. Here, here’s… The lady is reading a book, eating something right now. Well, here’s a shovel. Here’s the sun. Here’s some kind of bird.”