Today’s 3D printers make it fairly easy to conjure, say, a chess set into existence. But these printers are largely fixed in place. So if someone wants to add 3D-printed elements to a room — a footrest beneath a desk, for instance — the project gets more difficult. A space must be measured. The objects must then get scaled, printed elsewhere and fixed in the right spot. Handheld 3D printers exist, but they lack accuracy and come with a learning curve.

91̽�� researchers created , a mobile 3D printer that can automatically measure a room and print objects onto its floor. The team’s graphic interface lets users design objects for a space that the robot has mapped out. The prototype, which the team built on a modified consumer vacuum robot, can add accessibility features, home customizations or artistic flourishes to a space.

The team Tuesday, Oct. 15, at the ACM Symposium on User Interface Software and Technology in Pittsburgh.

“Digital fabrication, like 3D printing, is pretty mature at this point,” said , a doctoral student in the Paul G. Allen School of Computer Science & Engineering. “Now we’re asking: How can we push it further and further into the world, and lower the barriers for people to use it? How can we change the built environment and tailor spaces for peoples’ specific needs — for accessibility, for taste?”

The prototype system can add accessibility features, such as tactile markers for blind and low-vision people. These might provide information, such as text telling conference attendees where to go, or warn of dangers such as staircases. Or it can create a ramp to cover an uneven flooring transition. MobiPrint also allows users to create custom objects, such as small art pieces up to three inches tall.

Before printing an object, MobiPrint autonomously roams an indoor space and uses LiDAR to map it. The team’s design tool then converts this map into an interactive canvas. The user then can select a model from the MobiPrint library — a cat food bowl, for instance — or upload a design. Next, the user picks a location on the map to print the object, working with the design interface to scale and position the job. Finally, the robot moves to the location and prints the object directly onto the floor.

For printing, the current design uses called PLA. The researchers are working to have MobiPrint remove objects it’s printed and potentially recycle the plastic. They’re also interested in exploring the possibilities of robots that print on other surfaces (such as tabletops or walls), in other environments (such as outdoors), and with other materials (such as concrete).

“I think about kids out biking or my friends and family members who are in wheelchairs getting to the end of a sidewalk without a curb,” said , a professor in the Allen School. “It would be so great if in the future we could just send Daniel’s robot down the street and have it build a ramp, even if it was working just for a short period of time. That just shows you how reconfigurable environments can be.”

, an assistant professor at Purdue University, who was a doctoral student in the Allen School while doing this research, is a co-author on this paper. This research was funded by the National Science Foundation.

For more information, contact Zamora at danielcz@cs.uw.edu and Froehlich at jonf@cs.uw.edu.

Big Tech’s race into augmented reality (AR) grows more competitive by the day. This month, Meta of its headset, the Quest 3. Early next year, Apple its first headset, the Vision Pro. The announcements for each platform emphasize and that merge the virtual and physical worlds: a digital board game imposed on a coffee table, a movie screen projected above airplane seats.

Some researchers, though, are more curious about other uses for AR. The 91̽��’s is applying these budding technologies to assist people with disabilities. This month, researchers from the lab will introduce multiple projects that deploy AR — through headsets and phone apps — to make the world more accessible.

Researchers from the lab will RASSAR, an app that can scan homes to highlight accessibility and safety issues, on Oct. 23 at the in New York.

Shortly after, on Oct. 30, other teams in the lab will present early research at the conference in San Francisco. One app and the other aims to for low-vision users.

91̽��News spoke with the three studies’ lead authors, and , both 91̽��doctoral students in the Paul G. Allen School of Computer Science & Engineering, about their work and the future of AR for accessibility.

What is AR and how is it typically used right now?

Jae Lee: I think one commonly accepted answer is that you use a wearable headset or a phone to superimpose virtual objects in a physical environment. A lot of people probably know AR from “Pokémon Go,” where you’re superimposing these Pokémon into the physical world. Now Apple and Meta are introducing “mixed reality” or passthrough AR, which further blends the physical and virtual worlds through cameras.

Xia Su: Something I have also been observing lately is people are trying to expand the definition beyond goggles and phone screens. There could be AR audio, which is manipulating your hearing, or devices trying to manipulate your smell or touch.

A lot of people associate AR with virtual reality, and it gets wrapped up in discussion of the metaverse and gaming. How is it being applied for accessibility?

JL: AR as a concept has been around for several decades. But in ’s lab, we’re combining AR with accessibility research. A headset or a phone can be capable of knowing how many people are in front of us, for example. For people who are blind or low vision, that information could be critical to how they perceive the world.

XS: There are really two different routes for AR accessibility research. The more prevalent one is trying to make AR devices more accessible to people. The other, less common approach is asking: How can we use AR or VR as tools to improve the accessibility of the real world? That’s what we’re focused on.

JL: As AR glasses become less bulky and cheaper, and as AI and computer vision advance, this research will become increasingly important. But widespread AR, even for accessibility, brings up a lot of questions. How do you deal with bystander privacy? We, as a society, understand that vision technology can be beneficial to blind and low-vision people. But we also might not want to include facial recognition technology in apps for privacy reasons, even if that helps someone recognize their friends.

Let’s talk about the papers you have coming out. First, can you explain your ?

XS: It’s an app that people can use to scan their indoor spaces and help them detect possible accessibility safety issues in homes. It’s possible because some iPhones now have (light detection and ranging) scanners that tell the depth of a space, so we can reconstruct the space in 3D. We combined this with models to highlight ways to improve safety and accessibility. To use it, someone — perhaps a parent who’s childproofing a home, or a caregiver — scans a room with their smartphone and RASSAR spots accessibility problems. For example, if a desk is too high, a red button will pop up on the desk. If the user clicks the button, there will be more information about why that desk’s height is an accessibility issue and possible fixes.

JL: Ten years ago, you would have needed to go through 60 pages of PDFs to fully check a house for accessibility. We boiled that information down into an app.

And this is something that anyone will be able to download to their phones and use?

XS: That’s the eventual goal. We already have a demo. This version relies on lidar, which is only on certain iPhone models right now. But if you have such a device, it’s very straightforward.

JL: This is an example of these advancements in hardware and software that let us create apps quickly. Apple announced , which creates a 3D floor plan of a room, when they added the lidar sensor. We’re using that in RASSAR to understand the general layout. Being able to build on that lets us come up with a prototype very quickly.

So RASSAR is nearly deployable now. The other areas of research you’re presenting are earlier in their development. Can you tell me about ?

JL:  It’s an app that uses an AR headset to enable people to speak more naturally with voice assistants like Siri or Alexa. There are all these pronouns we use when we speak that are difficult for computers to understand without visual context. I can ask “Where’d you buy it from?” But what is “it”? A voice assistant has no idea what I’m talking about. With GazePointAR, the goggles are looking at the environment around the user and the app is tracking the user’s gaze and hand movements. The model then tries to make sense of all these inputs — the word, the hand movements, the user’s gaze. Then, using a , GPT, it attempts to answer the question.

How does it sense what the motions are?

JL: We’re using a headset called HoloLens 2 developed by Microsoft. It has a gaze tracker that’s watching your eyes and trying to guess what you’re looking at. It has hand tracking capability as well. In a paper that we submitted building on this, we noticed that we have a lot of problems with this. For example, people don’t just use one pronoun at a time — we use multiple. We’ll say, “What’s more expensive, this or this?” To answer that, we need information over time. But, again, you can run into privacy issues if you want to track someone’s gaze or someone’s visual field of view over time: What information are you storing and where is it being stored? As technology improves, we certainly need to watch out for these privacy concerns, especially in computer vision.

This is difficult even for humans, right? I can ask, “Can you explain that?” while pointing at several equations on a whiteboard and you won’t know which I’m referring to. What applications do you see for this?

JL: Being able to use natural language would be major. But if you expand this to accessibility, there’s the potential for a blind or low-vision person to use this to describe what’s around them. The question “Is anything dangerous in front of me?” is also ambiguous for a voice assistant. But with GazePointAR, ideally, the system could say, “There are possibly dangerous objects, such as knives and scissors.” Or low-vision people might make out a shape, point at it, then ask the system what “it” is more specifically.

And finally you’re working on a system called . What is it and what prompted this research?

JL: This is going even more into the future than GazePointAR. ARTennis is a prototype that uses an AR headset to make tennis balls more salient for low vision players. The ball in play is marked by a red dot and has a crosshair of green arrows around it. Professor Jon Froehlich has a family member that wants to play sports with his children but doesn’t have the residual vision necessary to do so. We thought if it works for tennis, it’s going to work for a lot of other sports, since tennis has a small ball that shrinks as it gets further away. If we can track a tennis ball in real time, we can do the same with a bigger, slower basketball.

One of the co-authors on the paper is low vision himself, and he plays a lot of squash, and he wanted to try this application and give us feedback. We did a lot of brainstorming sessions with him, and he tested the system. The red dot and green crosshairs is the design that he came up with, to improve the sense of depth perception.

What’s keeping this from being something people can use right away?

JL: Well, like GazePointAR, it’s relying on a HoloLens 2 headset that’s $3,500. So that’s a different accessibility issue. It’s also running at roughly 25 frames per second and for humans to perceive in real time it needs to be about 30 frames per second. Sometimes we can’t capture the speed of the tennis ball. We’re going to expand the paper and include basketball to see if there are different designs people prefer for different sports. The technology will certainly get faster. So our question is: What will the best design be for the people using it?

For more information, contact Lee at jaewook4@cs.washington.edu, Su at xiasu@cs.washington.edu and Jon Froehlich at jonf@cs.washington.edu.

Smartwatches offer people a private method for getting notifications about their surroundings — such as a phone call, health alerts or an upcoming package delivery.

Now 91̽�� researchers have developed , a smartwatch app for deaf, and hard-of-hearing people who want to be aware of nearby sounds. When the smartwatch picks up a sound the user is interested in — examples include a siren, a microwave beeping or a bird chirping — SoundWatch will identify it and send the user a friendly buzz along with information about the sound.

The team Oct. 28 at the ACM conference on computing and accessibility.

“This technology provides people with a way to experience sounds that require an action — such as getting food from the microwave when it beeps. But these devices can also enhance people’s experiences and help them feel more connected to the world,” said lead author , a 91̽��doctoral student in the Paul G. Allen School of Computer Science & Engineering. “I use the watch prototype to notice birds chirping and waterfall sounds when I am hiking. It makes me feel present in nature. My hope is that other d/Deaf and hard-of-hearing people who are interested in sounds will also find SoundWatch helpful.”

The team started this project by designing a system for d/Deaf and hard-of-hearing people who wanted to be able to know what was going on around their homes.

“I used to sleep through the fire alarm,” said Jain, who was born hard of hearing.

The first system, called , uses Microsoft Surface tablets scattered throughout the home which act like a network of interconnected displays. Each display provides a basic floor plan of the house and alerts a user to a sound and its source. The displays also show the sound’s waveforms, to help users identify the sound, and store a history of all the sounds a user might have missed when they were not home.

The researchers tested HomeSound in the Seattle-area homes of six d/Deaf or hard-of-hearing participants for three weeks. Participants were instructed to go about their lives as normal and complete weekly surveys.

Based on feedback, a second prototype used machine learning to classify sounds in real time. The researchers created a dataset of over 31 hours of 19 common home-related sounds — such as a dog bark or a cat meow, a baby crying and a door knock.

“People mentioned being able to train their pets when they noticed dog barking sounds from another room or realizing they didn’t have to wait by the door when they were expecting someone to come over,” Jain said. “HomeSound enabled all these new types of interactions people could have in their homes. But many people wanted information throughout the day, when they were out in their cars or going for walks.”

The researchers then pivoted to a smartwatch system, which allows users to get sound alerts wherever they are, even in places they might not have their phones, such as at the gym.

Because smartwatches have limited storage and processing abilities, the team needed a system that didn’t eat the watch’s battery and was also fast and accurate. First the researchers compared a compressed version of the HomeSound classifier against three other available sound classifiers. The HomeSound variant was the most accurate, but also the slowest.

To speed up the system, the team has the watch send the sound to a device with more processing power — the user’s phone — for classification. Having a phone classify sounds and send the results back to the watch not only saves time but also maintains the user’s privacy because sounds are only transferred between the user’s own devices.

The researchers tested the SoundWatch app in March 2020 — before Washington’s stay-at-home order — with eight d/Deaf and hard-of-hearing participants in the Seattle area. Users tested the app at three different locations on or around the 91̽��campus: in a grad student office, in a building lounge and at a bus stop.

People found the app was useful for letting them know if there was something that they should pay attention to. For example: that they had left the faucet running or that a car was honking. On the other hand, it sometimes misclassified sounds (labeling a car driving by as running water) or was slow to notify users (one user was surprised by a person entering the room way before the watch sent a notification about a door opening).

The team is also developing , which uses augmented reality to provide real-time captions and other sound information through HoloLens glasses.

“We want to harness the emergence of state-of-the-art machine learning technology to make systems that enhance the lives of people in a variety of communities,” said senior author , an associate professor in the Allen School.

Another current focus is developing a method to pick out specific sounds from background noise, and identifying the direction a sound, like a siren, is coming from.

The SoundWatch app is available for free as an . The researchers are eager to hear feedback so that they can make the app more useful.

“Disability is highly personal, and we want these devices to allow people to have deeper experiences,” Jain said. “We’re now looking into ways for people to personalize these systems for their own specific needs. We want people to be notified about the sounds they care about — a spouse’s voice versus general speech, the back door opening versus the front door opening, and more.”

Additional researchers on the HomeSound and SoundWatch projects are: , an associate professor in the 91̽��human centered design and engineering department; and , doctoral students in the Allen School; , a high school senior at Bishop Blanchet High School; , who worked on this project as an undergraduate design major at the UW; , a doctoral student in the 91̽��human centered design and engineering department; and , 91̽��undergraduate students studying computer science and engineering; and , a freelance user experience designer. This research was funded by the National Science Foundation and a Google Faculty Research Award.

For more information, contact Jain at djain@cs.washington.edu and Froehlich at jonf@cs.washington.edu.

Grant number: IIS-1763199

But unless you’re one of those people, you might not know how hard it is to get around your city.

Now people can help map out accessibility here in Seattle. 91̽�� researchers have led the development of , an online crowdsourcing game that lets anyone with an internet connection use Google Street View to virtually explore neighborhoods and label curb ramps, missing or rough sidewalks, obstacles and more. Project Sidewalk first launched in , and it’s now available in — near Portland — and Seattle.  The team will present its results from the Washington, D.C., deployment May 7 at the 2019 in Glasgow, Scotland.

“A lot of people think this is something where you walk around your neighborhood and take pictures of accessibility problems with your smartphone,” said corresponding author , an assistant professor in the Paul G. Allen School of Computer Science & Engineering. “But Project Sidewalk is not like that at all. There’s no assumption that you have any physical experience with what you’re reporting on. That is the key difference. Anyone can do it from anywhere, as long as they have a web browser.”

To get started on Project Sidewalk, the team interviewed people with mobility impairments to learn about how accessibility — or a lack of it — affects their lives. From there the researchers came up with a method to use crowdsourcing to collect street-level data about accessibility in cities.

Project Sidewalk relies on volunteers to log accessibility issues across a city. So the team used a video game model to make it more fun. Players go on missions where they audit 500 to 1,000 feet of a city at a time.

“Your first mission is a guided mission,” Froehlich said. “We have to teach you how you walk around and how to label things. But then we also need to help you understand what accessibility means: What is a curb ramp? What does it mean to have a missing curb ramp?”

Then players are sent out on solo missions — they’re either dropped into the city in an area where there aren’t already a lot of labels, or they can choose to go to a specific part of the city. For their first few missions, players receive helpful tips about the interface and shortcuts to make their labeling faster. Project Sidewalk also displays a progress bar that shows players how far they’ve gone on a mission.

“We’ve found that people love seeing that progress bar,” said first author , a doctoral student in the Allen School. “They say it makes it more fun and feel more like a game.”

After the team launched the Washington, D.C., version of Project Sidewalk in August 2016, 797 players added 205,385 labels to the city’s streets over the 18-month deployment. Players placed labels accurately about 72% of the time and were most likely to find and label curb ramps.

“We’re still working on analyzing the data,” Saha said. “But when we look at all the labels on the map, we can immediately start to see which portions of the city might be having issues.”

Players also made a variety of common errors, such as labeling a driveway as a curb ramp or labeling surface problems on the street when the sidewalks were fine. These errors prompted the team to develop a new verification “minigame” in the Seattle and Newberg versions of Project Sidewalk, in which players verify 10 labels that someone else placed.

“I want it to be like Super Mario Brothers 2, which has these fast little minigames that pop up between levels,” Froehlich said. “It gives people time to breathe and do something different. It’s something you could do on the bus.”

Because the data from Project Sidewalk is available to anyone, the researchers envision that it could serve multiple purposes, from helping government officials decide which areas to investigate first to enhancing the independence of people with mobility impairments.

Data from Seattle’s Project Sidewalk could inform other accessibility projects in the area, Froehlich said. For example, , which provides directions for pedestrians and wheelchair users looking to avoid hills, construction sites and other accessibility barriers, could use data from Project Sidewalk to be able to create better directions.

Eventually the team would like to have computers use machine learning to help people add labels on Project Sidewalk and make accessibility audits go faster. The researchers hope to use the Project Sidewalk data to train an algorithm that would teach computers how to do their own audits.

“My ambitious vision is for anyone in the world to click on their city and have our system provide a visualization and an accessibility assessment,” Froehlich said. “It shouldn’t matter if you live in Paris, France, Beijing, China or Cairo, Egypt. If Google Street View has driven there, you should be able to get a map visualizing the city’s sidewalk accessibility.”

This research received a among submissions to the CHI conference. Other co-authors are and at the UW; , and at the University of Maryland; at Intelsat, who completed this research while at the University of Maryland; Ryan Holland at UCLA, who completed this research while at Montgomery Blair High School; at the University of Michigan, who completed this research while at the University of Maryland; and at Singapore Management University, who completed this research while at the University of Maryland. This research was funded by the National Science Foundation, a Singapore Ministry of Education AcRF Tier 1 Grant and a Sloan Research Fellowship.

###

For more information, contact Froehlich at jonf@cs.uw.edu.

Grant number: IIS-1302338