Robotic systems have shown their potential to support people in a variety of daily household tasks, such as cooking [1], [2], drinking [3], [4], cleaning [5], dressing [6], [7], etc. With the recent advances of Large Language Models in robotics [8], [9], the communication between humans and robots can be more natural using speech. For example, Kodur et al. [10] developed a framework that maps spoken commands to robotic actions for a cooking scenario. Therefore, robots can understand and act on spoken requests, which enables non-expert users to interact easily with robots.
Although several robotic tasks related to activities of daily living (i.e., activities focused on taking care of one's own body) have been explored in the field of assistive robots, the assistance of users with the preparation for instrumental activities of daily living (IADL) that support daily life within the community [11], such as getting ready to go to work, school, outdoor events, shopping for groceries, running errands, etc. has not been investigated [12]. Modern daily routines for getting ready for work, school, a doctor's appointment, or a social or recreational activity include packing necessary objects, which is a time-consuming process and prone to human error. An intelligent robotic assistant can mitigate these issues by learning the objects required for each IADL and streamlining the preparation process for each individual.
In this paper, we propose an intelligent robotic assistant that learns the necessary objects to prepare for a daily activity via spoken instructions and retrieves them when the user requests. The robot requires one-shot instruction on the objects needed for an IADL. The user informs the robot of USA mkyrarini@scu.edu their intent to prepare for a learned activity, and the robot retrieves the objects associated with this activity. A pilot study evaluated the proposed robotic system, and preliminary results are presented, showing its potential usefulness.
Intelligent mobile manipulator systems have recently been developed to learn user preferences for object manipulation tasks. Mobile manipulators are robots that consist of a mobile base with a robotic arm. Although no system is focused on assisting users in preparing for IADL outside the home, such as getting ready to go to work, school, or other locations, several systems have been developed for robot learning of multi-object tasks from spoken instructions.
TidyBot [5] is a personalized robotic system that learns how to clean up and tidy up rooms from a few-shot interactions with a user. The robotic system is capable of picking up objects and putting them away. The target location of the objects is learned based on a few interactions with the user. The proposed system uses textual input from the users to generate the necessary robotic actions for picking up objects from the floor and putting them away. The framework achieved an 85% success rate in real-world test scenarios during a user study with 40 participants.
Kodur et al. [10] developed a framework that uses speech commands as input and automatically generates robotic actions for object manipulation, which become part of a graph for a specific task. The graph represents the learned sequence of actions for a collaborative cooking task. The proposed framework uses speech-to-text by Google to convert speech commands into text. Subsequently, BERT [13] and GPT Neo [14] are used to recognize valid speech commands and generate robotic commands. A robot control module translates robotic commands into object manipulation. The framework was evaluated in a user study with 30 participants [15], [16].
A grocery reminder mobile manipulator is proposed by Ayub et al. [17]. The authors developed a Graphical User Interface (GUI) that enables a user to move the mobile manipulator in different poses so that the robot can learn the context locations. The learned knowledge is stored and is used to predict missing objects in the days ahead. When an object is missing, the GUI informs the user so that they can buy it. The proposed system was not evaluated but was planned as future work.
In this paper, we expand on the robotic framework proposed by Kodur et al. [10] by developing an interactive robot learning phase that enables the user to teach necessary objects for desired IADL. During the robot working phase, the user can describe the specific IADL, and the robot retrieves the associated objects.
The proposed intelligent robotic system for IADL expands the work developed by Kodur et al. in [10], [15], in which a single command type was supported, allowing users to request the retrieval of an object using speech commands. The robot working phase in Fig. 1 describes the system developed by Kodur et al. [10]. Building on this foundation, the proposed system has the following functionalities:
1) Single Item Retrieval: This command remains unchanged from the previous architecture, allowing users to request the retrieval of a single object. 2) Teach a New Task: This new command allows users to teach the robot a new task, which it can learn and execute when requested. 3) Execute a Learned Task: This command enables users to instruct the robot to perform a previously learned task. To achieve additional functionalities, a robot learning phase is developed based on a rule-based system, a type of rudimentary artificial intelligence that uses conditional statements "if-then," or rules, to make decisions based on an input [18]. When a user gives a command to teach the robot a new task, the robot will ask the user a series of questions to create and store the new task for future use. As a task, it is considered an IADL that requires the manipulation and delivery of several objects. For our proposed system, the rule-based system, shown in Fig. 1
• Rule Engine: This is the system's core where the rules are processed. The Rule Engine processes the user's speech commands and outputs the learned task into the robot's memory. The Rule Engine consists of Rule Base and Error Handling. • Rule Base: The rule base acts as the system's knowledge base to make decisions. It processes the user's commands and, based on defined rules, creates a new task, modifies an existing task, or retrieves an existing task. The Rule Base is designed to be easily adaptable, allowing new activities to be added or existing ones to be modified as the system evolves. This flexibility is crucial for accommodating new tasks and improving the system's capabilities. • Error handling: The error handling ensures that if any part of the process fails, the system can provide meaningful feedback to the user. For example, if the robot does not have access to a requested object needed for a task, the system will inform the user and ask the user to request a different object. • Robot Memory: This is where the system stores the processed data, including the user's input and the tasks created by the user. It is implemented using a CSV database, where learned tasks and their corresponding details are stored. This allows the robot to recall and execute previously taught tasks without requiring the user to repeat the teaching process. The Robot Memory provides the robot with information about the learned tasks. During the robot learning phase, when the user selects to teach a new task, the robot will ask the following three questions: (1) "What is the name of the task?" (2) "How many items are needed for the task?" and (3) "Which items are needed?". During the robot working phase, the user can command the robot to complete a task, such as retrieving an object or a list of objects. Alternatively, the user can switch to the robot learning phase to teach the robot a new task.
A pilot study was conducted to evaluate the proposed method. This section discusses the experimental setup, the pilot study, and the results. a) Experimental Setup: Fig. 2 shows the experimental setup for preparing for an IADL. The setup consists of two tables: one designated as the user area where the participant sits and the other where all the items needed for IADL are placed. A mobile base equipped with a 7-Degrees of Freedom robotic arm, two RGB-Depth cameras, and a container for transporting multiple objects is used for the study, as shown in Fig. 2. The items used for the pilot study are the following: wallet, car keys, calculator, cellphone, hand sanitizer, medicine, umbrella, and a laptop. QR codes are used for mobile base localization and object pose estimation. The participant communicates with the robotic system by using a microphone. To respect the participant's privacy, the user taps the microphone when they want to instruct the robot and taps it off when they finish their interaction.
b) Pilot Study: For the pilot study, 10 adult participants (4 female and 6 male) of average age 28.7 ± 15.4 years old were recruited with the approval of the Institutional Review Board (IRB) at Santa Clara University (IRB #23-02-1902). It is worth noting that the mobile manipulator moves at a slow speed to ensure human safety during the pilot study. After the participant read and signed the consent form, the participant was instructed to request one item from the robot. Subsequently, the participant taught the robot a task and then asked the robot to execute the learned task. The users used predefined speech commands to communicate with the system. A demo video for the single item retrieval can be found in www.youtube. com/watch?v=7ieUHwGYHr8, for teaching a task in www.youtube.com/watch?v=3ZwdYYUvZRw and for executing a learned task in www.youtube.com/watch? v=I1os_-2GVwU.
After the interaction, the participants completed the System Usability Scale (SUS) questionnaire [20], which is widely used to determine a system or product's perceived usability and user-friendliness. SUS consists of a 10-item questionnaire with 5-point Likert scale response options ranging from "Strongly agree" (for 5) to "Strongly disagree" (for 1). A SUS score above 68 is generally considered above average, while a SUS score below 68 is considered below average. Additionally, the participants were requested to fill out the Human-Robot Collaboration (HRC) Questionnaire, which was proposed by Kodur et al. [15] and is shown in Table I. It consists of 14 statements (Q1-Q14) with the 5point Likert scale, similar to the SUS. c) Preliminary Results & Discussion: All participants in the pilot study successfully completed all the required interactions with the robot on their first attempt. Therefore, the success rate of our system was 100 % for the ten trials. The average SUS score was 81.5 ± 15.3, demonstrating that the participants found the proposed system to be useful. Table I shows the average score and standard deviation for each of the statements in the HRC Questionnaire. For the statements Q2-Q8, Q10, and Q14, the results are above the score of 4, which shows the users felt safe and in control, and the system was easy to use. For Q1, the participants did not agree or disagree that they accomplished the given tasks rapidly. This may be due to the robot's slow speed. Since no identifiable data were collected during the study, participants were not concerned about their privacy (Q13). Additionally, they did
TABLE I THE 5-POINT LIKERT SCALE STATEMENTS FOR THE HUMAN-ROBOT COLLABORATION QUESTIONNAIRE [15] AND THE AVERAGE AND STANDARD DEVIATION RESULTS FROM THE PILOT STUDY.
Results (Average ± Standard Deviation) Perceived Usefulness Q1. I accomplished the given tasks rapidly.
3.3 ± 1.4 Q2. I accomplished the given tasks successfully.
4.5 ± 0.7 Perceived Safety and Trust Q3. The robot's actions were predictable.
4.4 ± 0.8 Q4. I felt safe using the robot.
4.9 ± 0.3 Q5. I trusted the robot's suggestions.
4.4 ± 0.7 Perceived Ease of Use Q6. I found the robot easy to use.
4.2 ± 0.8 Q7. The robot learned how to assist me.
4.1 ± 0.7 Q8. The robot met my expectations.
4.6 ± 0.5 Perceived Interaction Q9. I had to learn more about robots in order to be able to interact with the system. 1.3 ± 0.4 Q10. I felt my voice volume was normal.
4.7 ± 0.5 Q11. I had to speak slowly to interact with the robot.
1.5 ± 0.9 Ethical Considerations Q12. It is acceptable for the robot to have much information about the user.
3.8 ± 1.2 Q13. I am concerned about my privacy when using the robot.
1.6 ± 0.7 Q14. I should have full control of when and how the robot will assist me.
4.4 ± 0.9 not feel they had to speak slowly to interact with the robot (Q11) and did not have to learn more about robots to interact with the system (Q9). Our findings from the pilot study demonstrate that the system has the potential to be useful and easy to use without requiring user training. However, as this is a pilot study with a small number of participants, a study with more users is needed to validate our preliminary results.
This paper proposes an intelligent robotic system that assists people with preparation for IADL. A rule-based system was developed to enable a robot to learn tasks via speech commands and to execute the learned tasks. A pilot study was conducted with 10 participants and demonstrated that the system is easy to use and useful based on user feedback. The proposed system has the potential to support people with Alzheimer's disease by providing reminders for specific tasks [21], such as getting ready for IADL. In the future, we plan to enable a more proactive robotic behavior where the robot will access the user's weekly calendar (with the user's approval) and can bring the required items without the user's request. For example, if a person needs to go to the doctor at 9 am, the robot would have already retrieved the items. This functionality will be important for individuals who suffer from Alzheimer's disease, and we will conduct a user study to evaluate it.
Authorized licensed use limited to: Santa Clara University. Downloaded on July 14,2025 at 19:36:03 UTC from IEEE Xplore. Restrictions apply.