skip to main content


Title: Multimodal Information Integration for Indoor Navigation Using a Smartphone
We propose an accessible indoor navigation application. The solution integrates information of floor plans, Bluetooth beacons, Wi-Fi/cellular data connectivity, 2D/3D visual models, and user preferences. Hybrid models of interiors are created in a modeling stage with Wi-Fi/ cellular data connectivity, beacon signal strength, and a 3D spatial model. This data is collected, as the modeler walks through the building, and is mapped to the floor plan. Client-server architecture allows scaling to large areas by lazy-loading models according to beacon signals and/or adjacent region proximity. During the navigation stage, a user with the designed mobile app is localized within the floor plan, using visual, connectivity, and user preference data, along an optimal route to their destination. User interfaces for both modeling and navigation use visual, audio, and haptic feedback for targeted users. While the current pandemic event precludes our user study, we describe its design and preliminary results.  more » « less
Award ID(s):
1827505 1737533
NSF-PAR ID:
10185780
Author(s) / Creator(s):
; ; ; ; ; ;
Date Published:
Journal Name:
IRI2020 - The 21st IEEE International Conference on Information Reuse and Integration for Data Science
Page Range / eLocation ID:
59-66
Format(s):
Medium: X
Sponsoring Org:
National Science Foundation
More Like this
  1. null (Ed.)
    The iASSIST is an iPhone-based assistive sensor solution for independent and safe travel for people who are blind or visually impaired, or those who simply face challenges in navigating an unfamiliar indoor environment. The solution integrates information of Bluetooth beacons, data connectivity, visual models, and user preferences. Hybrid models of interiors are created in a modeling stage with these multimodal data, collected, and mapped to the floor plan as the modeler walks through the building. Client-server architecture allows scaling to large areas by lazy-loading models according to beacon signals and/or adjacent region proximity. During the navigation stage, a user with the navigation app is localized within the floor plan, using visual, connectivity, and user preference data, along an optimal route to their destination. User interfaces for both modeling and navigation use multimedia channels, including visual, audio, and haptic feedback for targeted users. The design of human subject test experiments is also described, in addition to some preliminary experimental results. 
    more » « less
  2. Indoor navigation systems are very useful in large complex indoor environments such as shopping malls. Current systems focus on improving indoor localization accuracy and must be combined with an accurate labeled floor plan to provide usable indoor navigation services. Such labeled floor plans are often unavailable or involve a prohibitive cost to manually obtain. In this paper, we present IndoorWaze, a novel crowdsourcing-based context-aware indoor navigation system that can automatically generate an accurate context-aware floor plan with labeled indoor POIs for the first time in literature. IndoorWaze combines the Wi-Fi fingerprints of indoor walkers with the Wi-Fi fingerprints and POI labels provided by POI employees to produce a high-fidelity labeled floor plan. As a lightweight crowdsourcing-based system, IndoorWaze involves very little effort from indoor walkers and POI employees. We prototype IndoorWaze on Android smartphones and evaluate it in a large shopping mall. Our results show that IndoorWaze can generate a high-fidelity labeled floor plan, in which all the stores are correctly labeled and arranged, all the pathways and crossings are correctly shown, and the median estimation error for the store dimension is below 12%. 
    more » « less
  3. Wi-Fi is one of the key wireless technologies for the Internet of things (IoT) owing to its ubiquity. Low-power operation of commercial Wi-Fi enabled IoT modules (typically powered by replaceable batteries) is critical in order to achieve a long battery life, while maintaining connectivity, and thereby reduce the cost and frequency of maintenance. In this work, we focus on commonly used sparse periodic uplink traffic scenario in IoT. Through extensive experiments with a state-of-the-art Wi-Fi enabled IoT module (Texas Instruments SimpleLink CC3235SF), we study the performance of the power save mechanism (PSM) in the IEEE 802.11 standard and show that the battery life of the module is limited, while running thin uplink traffic, to ~30% of its battery life on an idle connection, even when utilizing IEEE 802.11 PSM. Focusing on sparse uplink traffic, a prominent traffic scenario for IoT (e.g., periodic measurements, keep-alive mechanisms, etc.), we design a simulation framework for single-user sparse uplink traffic on ns-3, and develop a detailed and platform-agnostic accurate power consumption model within the framework and calibrate it to CC3235SF. Subsequently, we present five potential power optimization strategies (including standard IEEE 802.11 PSM) and analyze, with simulation results, the sensitivity of power consumption to specific network characteristics (e.g., round-trip time (RTT) and relative timing between TCP segment transmissions and beacon receptions) to present key insights. Finally, we propose a standard-compliant client-side cross-layer power saving optimization algorithm that can be implemented on client IoT modules. We show that the proposed optimization algorithm extends battery life by 24%, 26%, and 31% on average for sparse TCP uplink traffic with 5 TCP segments per second for networks with constant RTT values of 25 ms, 10 ms, and 5 ms, respectively. 
    more » « less
  4. Vision-based localization approaches now underpin newly emerging navigation pipelines for myriad use cases, from robotics to assistive technologies. Compared to sensor-based solutions, vision-based localization does not require pre-installed sensor infrastructure, which is costly, time-consuming, and/or often infeasible at scale. Herein, we propose a novel vision-based localization pipeline for a specific use case: navigation support for end users with blindness and low vision. Given a query image taken by an end user on a mobile application, the pipeline leverages a visual place recognition (VPR) algorithm to find similar images in a reference image database of the target space. The geolocations of these similar images are utilized in a downstream task that employs a weighted-average method to estimate the end user’s location. Another downstream task utilizes the perspective-n-point (PnP) algorithm to estimate the end user’s direction by exploiting the 2D–3D point correspondences between the query image and the 3D environment, as extracted from matched images in the database. Additionally, this system implements Dijkstra’s algorithm to calculate a shortest path based on a navigable map that includes the trip origin and destination. The topometric map used for localization and navigation is built using a customized graphical user interface that projects a 3D reconstructed sparse map, built from a sequence of images, to the corresponding a priori 2D floor plan. Sequential images used for map construction can be collected in a pre-mapping step or scavenged through public databases/citizen science. The end-to-end system can be installed on any internet-accessible device with a camera that hosts a custom mobile application. For evaluation purposes, mapping and localization were tested in a complex hospital environment. The evaluation results demonstrate that our system can achieve localization with an average error of less than 1 m without knowledge of the camera’s intrinsic parameters, such as focal length. 
    more » « less
  5. In an era of ubiquitous digital interfaces and systems, technology and design practitioners must address a range of ethical dilemmas surrounding the use of persuasive design techniques and how to balance shareholder and end-user needs [2], [5]. Similarly, the increasing user concerns about unethical products and services [1] is paralleling a rise in regulatory interests in enforcing ethical design and engineering practices among technology practitioners, surfacing a need for further support. Although various scholars have developed frameworks and methods to support practitioners in navigating these challenging contexts [3], [4], often, there is a lack of resonance between these generic methods and the situated ethical complexities facing the practitioner in their everyday work. In this project, we designed and implemented a three-hour cocreation workshop with designers, engineers, and technologists to support them to develop bespoke ethics-focused action plans that are resonant with the ethical challenges they face in their everyday practice. In developing the co-creation session, we sought to answer the following questions to empower practitioners: • How can we support practitioners in developing action plans to address ethical dilemmas in their everyday work? and • How can we empower designers to design more responsibly? Building on these questions as a guide, we employed Miro, a digital whiteboard platform, to develop the co-creation experience. The final c o-creation e xperience w as d esigned w ith the visual metaphor of a “house” with four floors and multiple rooms that allowed participants to complete different tasks per room, all aimed towards the overall goal of developing participants' own personalized action plan in an interactive and collaborative way. We invited participants to share their stories and ethical dilemmas to support their creation and iteration of a personal action plan that they could later use in their everyday work context. Across the six co-creation sessions we conducted, participants (n=26) gained a better understanding of the drivers for ethical action in the context of their everyday work and developed an action plan through the co-creation workshop that enabled them to constructively engage with ethical challenges in their professional context. At the end of the session, participants were provided the action plans they created to allow them to use it in their practice. Furthermore, the co-design workshops were designed such that practitioners could take them away (the house and session guide) and run them independently at their organization or another context to support their objectives. We describe the building and the activities conducted in each floor below and will provide a pictorial representation of the house with the different floors, rooms, and activities on the poster presentation. a) First floor-Welcome, Introduction, Reflection: The first floor of the virtual house was designed to allow participants to introduce themselves and to reflect on and discuss the ethical concerns they wished to resolve during the session. b) Second floor-Shopping for ethics-focused methods: The second floor of the virtual house was designed as a “shopping” space where participants selected from range of ethicsfocused building blocks that they wish to potentially adapt or incorporate into their own action plan. They were also allowed to introduce their own methods or tools. c) Third floor-DIY Workspace: The third floor was designed as a DIY workspace to allow the participants to work in small groups to develop their own bespoke action plan based on building blocks they have gathered from their shopping trip and by using any other components they wish. The goal here was to support participants in developing methods and action plans that were resonant with their situated ethical complexities. d) Fourth floor-Gallery Space: The fourth floor was designed as a gallery to allow participants to share and discuss their action plans with other participants and to identify how their action plans could impact their future practice or educational experiences. Participants were also provided an opportunity at this stage to reflect on their experience participating in the session and provide feedback on opportunities for future improvement. 
    more » « less