Project Sidekick

Sidekick is a project developed by NASA and Microsoft, started in December 2015 on the International Space Station, which provides virtual assistance to astronauts using Microsoft HoloLens augmented reality headsets.^[1]

Sidekick has two modes of operation. Remote Expert Mode uses the functionality of the Holographic Skype application—voice and video chat, real-time virtual annotation—to allow a ground operator and space crew member to collaborate directly over what the astronaut sees, with the ground operator able to see the crew member's view in 3D, provide interactive guidance, and draw annotations into the crew member's environment. In Procedure Mode, animated virtual illustrations display on top of objects as a crew member interacts with them. This mode can be used for guidance and instructional purposes in standalone scenarios.^[2] Applications include inventory management, in which the system recognizes an inventory item, and can display a path to where the item should be stored.^[3] Previously, crew members would rely primarily on printed instructions and voice-based communication when performing complex tasks. The capabilities provided by Sidekick have been promoted as potential enabling features allowing for reduced crew training requirements and increased efficiency in deep space missions in which communication delays can complicate difficult operations.^[2]

After having performed simulated reduced-gravity testing in its Weightless Wonder C9 aircraft,^[2] NASA attempted to launch a pair of HoloLens units aboard the SpaceX CRS-7 launch to the International Space Station on June 28, 2015,^[4] but the Falcon 9 rocket carrying the payload exploded at 2 minutes and 19 seconds into the flight.^[5] Sidekick was tested at the Aquarius laboratory from late July to early August 2015 as part of the two-week long NASA Extreme Environment Missions Operations 20 expedition, demonstrating its operation in tasks such as equipment checks and setup.^[3][2] HoloLens' hardware was successfully delivered to the ISS in the Cygnus CRS OA-4 commercial resupply mission on December 9, 2015.^[1]

Despite Sidekick's clear advantages, there are potential drawbacks described in the B. Nuernberger research. Some concerns are centered around technical issues, like latency, the device's field of view, and the robustness of the HoloLens in the challenging environment of the ISS. The project also relied on the continuous evolution and development of the HoloLens, which, like any piece of technology, has its limitations and occasional bugs.^[6]

Another concern was about astronaut autonomy. There's a fine line between assisting astronauts and over-reliance on ground support, which could potentially undermine the development of problem-solving skills needed for long-duration missions, like those planned for Mars where communication delays with Earth are significant.^[6]

• Augmenting Astronaut Performance in Space by JPL Ops Lab