Project Sidekick

Last updated November 24, 2023

Sidekick is a project developed by NASA and Microsoft, started in December 2015 on the International Space Station, which provides virtual assistance to astronauts using Microsoft HoloLens augmented reality glasses.^[1]

Functionality

Sidekick has two modes of operation. Remote Expert Mode uses the functionality of the Holographic Skype application—voice and video chat, real-time virtual annotation—to allow a ground operator and space crew member to collaborate directly over what the astronaut sees, with the ground operator able to see the crew member's view in 3D, provide interactive guidance, and draw annotations into the crew member's environment. In Procedure Mode, animated virtual illustrations display on top of objects as a crew member interacts with them. This mode can be used for guidance and instructional purposes in standalone scenarios.^[2] Applications include inventory management, in which the system recognizes an inventory item, and can display a path to where the item should be stored.^[3] Previously, crew members would rely primarily on printed instructions and voice-based communication when performing complex tasks. The capabilities provided by Sidekick have been promoted as potential enabling features allowing for reduced crew training requirements and increased efficiency in deep space missions in which communication delays can complicate difficult operations.^[2]

History

After having performed simulated reduced-gravity testing in its Weightless Wonder C9 aircraft,^[2] NASA attempted to launch a pair of HoloLens units aboard the SpaceX CRS-7 launch to the International Space Station on June 28, 2015,^[4] but the Falcon 9 rocket carrying the payload exploded at 2 minutes and 19 seconds into the flight.^[5] Sidekick was tested at the Aquarius laboratory from late July to early August 2015 as part of the two-week long NASA Extreme Environment Missions Operations 20 expedition, demonstrating its operation in tasks such as equipment checks and setup.^[3]^[2] HoloLens' hardware was successfully delivered to the ISS in the Cygnus CRS OA-4 commercial resupply mission on December 9, 2015.^[1]

Critics

Despite Sidekick's clear advantages, there are potential drawbacks described in the B. Nuernberger research. Some concerns are centered around technical issues, like latency, the device's field of view, and the robustness of the HoloLens in the challenging environment of the ISS. The project also relied on the continuous evolution and development of the HoloLens, which, like any piece of technology, has its limitations and occasional bugs.^[6]

Another concern was about astronaut autonomy. There's a fine line between assisting astronauts and over-reliance on ground support, which could potentially undermine the development of problem-solving skills needed for long-duration missions, like those planned for Mars where communication delays with Earth are significant.^[6]

Related Research Articles

The International Space Station (ISS) is the largest modular space station in low Earth orbit. The project involves five space agencies: the United States' NASA, Russia's Roscosmos, Japan's JAXA, Europe's ESA, and Canada's CSA. The ownership and use of the space station is established by intergovernmental treaties and agreements. The station serves as a microgravity and space environment research laboratory in which scientific research is conducted in astrobiology, astronomy, meteorology, physics, and other fields. The ISS is suited for testing the spacecraft systems and equipment required for possible future long-duration missions to the Moon and Mars.

Mixed reality (MR) is a term used to describe the merging of a real-world environment and a computer-generated one. Physical and virtual objects may co-exist in mixed reality environments and interact in real time.

Flight controllers are personnel who aid space flight by working in such Mission Control Centers as NASA's Mission Control Center or ESA's European Space Operations Centre. Flight controllers work at computer consoles and use telemetry to monitor various technical aspects of a space mission in real-time. Each controller is an expert in a specific area and constantly communicates with additional experts in the "back room". The flight director, who leads the flight controllers, monitors the activities of a team of flight controllers, and has overall responsibility for success and safety.

Akihiko Hoshide is a Japanese engineer, JAXA astronaut, and former commander of the International Space Station. On August 30, 2012, Hoshide became the third Japanese astronaut to walk in space.

The electrical system of the International Space Station is a critical resource for the International Space Station (ISS) because it allows the crew to live comfortably, to safely operate the station, and to perform scientific experiments. The ISS electrical system uses solar cells to directly convert sunlight to electricity. Large numbers of cells are assembled in arrays to produce high power levels. This method of harnessing solar power is called photovoltaics.

Kjell Norwood Lindgren is an American astronaut who was selected in June 2009 as a member of the NASA Astronaut Group 20. He launched to the International Space Station (ISS) as part of Expedition 44/45 on July 22, 2015.

Astronaut training describes the complex process of preparing astronauts in regions around the world for their space missions before, during and after the flight, which includes medical tests, physical training, extra-vehicular activity (EVA) training, procedure training, rehabilitation process, as well as training on experiments they will accomplish during their stay in space.

Andrew Richard "Drew" Morgan is a NASA astronaut from the class of 2013.

SpaceX CRS-7, also known as SpX-7, was a private American Commercial Resupply Service mission to the International Space Station, contracted to NASA, which launched and failed on June 28, 2015. It disintegrated 139 seconds into the flight after launch from Cape Canaveral, just before the first stage was to separate from the second stage. It was the ninth flight for SpaceX's uncrewed Dragon cargo spacecraft and the seventh SpaceX operational mission contracted to NASA under a Commercial Resupply Services contract. The vehicle launched on a Falcon 9 v1.1 launch vehicle. It was the nineteenth overall flight for the Falcon 9 and the fourteenth flight for the substantially upgraded Falcon 9 v1.1.

OA-5, previously known as Orbital-5, was the seventh planned flight of the Orbital Sciences' uncrewed resupply spacecraft Cygnus and its sixth flight to the International Space Station under the Commercial Resupply Services contract with NASA. The mission launched on 17 October 2016 at 23:45:36 UTC. Orbital Sciences and NASA jointly developed a new space transportation system to provide commercial cargo resupply services to the International Space Station (ISS). Under the Commercial Orbital Transportation System (COTS) program, Orbital designed and built Antares, a medium-class launch vehicle; Cygnus, an advanced maneuvering spacecraft; and a Pressurized Cargo Module which is provided by Orbital's industrial partner Thales Alenia Space.

Dragon 2 is a class of partially reusable spacecraft developed and manufactured by American aerospace manufacturer SpaceX, primarily for flights to the International Space Station (ISS). SpaceX also launches private missions, such as Inspiration4 and Axiom Mission 1. There are two variants of the Dragon spacecraft: Crew Dragon, a spacecraft capable of ferrying four crewmembers, and Cargo Dragon, a replacement for the original Dragon 1 used to carry freight to and from space. The spacecraft consists of a reusable space capsule and an expendable trunk module. The spacecraft launches atop a Falcon 9 Block 5 rocket and the capsule returns to Earth through splashdown.

Microsoft HoloLens is an augmented reality (AR)/mixed reality (MR) headset developed and manufactured by Microsoft. HoloLens runs the Windows Mixed Reality platform under the Windows 10 operating system. Some of the positional tracking technology used in HoloLens can trace its lineage to the Microsoft Kinect, an accessory for Microsoft's Xbox 360 and Xbox One game consoles that was introduced in 2010.

Dragon, also known as Dragon 1 or Cargo Dragon, was a class of fourteen partially reusable cargo spacecraft developed by SpaceX, an American private space transportation company. The spacecraft flew 23 missions between 2010 and 2020. Dragon was launched into orbit by the company's Falcon 9 launch vehicle to resupply the International Space Station (ISS).

The Roll Out Solar Array (ROSA) and its larger version ISS Roll Out Solar Array (iROSA) are lightweight, flexible power sources for spacecraft designed and developed by Redwire.

NG-12, previously known as OA-12, was the thirteenth flight of the Northrop Grumman robotic resupply spacecraft Cygnus and its twelfth Commercial Resupply Services flight to the International Space Station (ISS) for NASA. The mission launched on 2 November 2019 at 13:59:47 UTC). This was the first launch of Cygnus under the Commercial Resupply Services 2 (CRS-2) contract.

NG-14, previously known as OA-14, was the fifteenth flight of the Northrop Grumman robotic resupply spacecraft Cygnus and its fourteenth flight to the International Space Station under the Commercial Resupply Services (CRS-1) contract with NASA. The mission was launched on 3 October 2020, at 01:16:14 UTC.

SpaceX CRS-22, also known as SpX-22, was a Commercial Resupply Services (CRS) mission to the International Space Station (ISS) that launched at 17:29:15 UTC on 3 June 2021. The mission is contracted by NASA and is flown by SpaceX using a Cargo Dragon 2. This is the second flight for SpaceX under NASA's CRS Phase 2 contract awarded in January 2016.

NG-15, previously known as OA-15, was the fifteenth launch of the Northrop Grumman robotic resupply spacecraft Cygnus and its fourteenth flight to the International Space Station (ISS) under the Commercial Resupply Services (CRS) contract with NASA. The mission launched on 20 February 2021 at 17:36:50 UTC. This is the fourth launch of Cygnus under the CRS-2 contract.

SpaceX CRS-23, also known as SpX-23, was a Commercial Resupply Service mission to the International Space Station, successfully launched on 29 August 2021 and docking the following day. The mission was contracted by NASA and was flown by SpaceX using the Cargo Dragon C208. This was the third flight for SpaceX under NASA's CRS Phase 2 contract awarded in January 2016. It was the second mission for this reusable capsule.

SpaceX CRS-25, also known as SpX-25, was a Commercial Resupply Service mission (CRS) to the International Space Station (ISS) that was launched on 15 July 2022. The mission was contracted by NASA and was flown by SpaceX using their reusable spacecraft, the Cargo Dragon. The vehicle delivered supplies to the crew aboard the ISS along with multiple pieces of equipment that will be used to conduct multiple research investigations aboard the ISS.

References

1 2 Franzen, Carl (9 December 2015). "Here's How Astronauts Will Use Microsoft's HoloLens Aboard The Space Station". Popular Science. Retrieved 14 December 2015.
1 2 3 4 Ramsey, Sarah, ed. (25 June 2015). "NASA, Microsoft Collaborate to Bring Science Fiction to Science Fact". Jet Propulsion Laboratory . Retrieved 28 June 2015.
1 2 Metz, Rachel (8 September 2015). "Why NASA Wants Microsoft's HoloLens in Space". MIT Technology Review . Retrieved 12 September 2015.
↑ Buhr, Sarah (28 June 2015). "SpaceX's Failed CRS-7 Mission Had Something To Do With An "Overpressure Event"". TechCrunch. Retrieved 17 February 2016.
↑ Chang, Kenneth (June 28, 2015). "SpaceX Rocket Explodes After Launching". New York Times. Retrieved 28 June 2015.
1 2 Nuernberger, Benjamin; Tapella, Robert; Berndt, Samuel-Hunter; Kim, So Young; Samochina, Sasha (2020-01-01). "Under Water to Outer Space: Augmented Reality for Astronauts and Beyond". IEEE Computer Graphics and Applications. 40 (1): 82–89. doi:10.1109/MCG.2019.2957631. ISSN 0272-1716.

External links

Augmenting Astronaut Performance in Space by JPL Ops Lab

This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.

[Franzen1214-1] 1 2 Franzen, Carl (9 December 2015). "Here's How Astronauts Will Use Microsoft's HoloLens Aboard The Space Station". Popular Science. Retrieved 14 December 2015.

[Sidekick_PR-2] 1 2 3 4 Ramsey, Sarah, ed. (25 June 2015). "NASA, Microsoft Collaborate to Bring Science Fiction to Science Fact". Jet Propulsion Laboratory . Retrieved 28 June 2015.

[Metz0912-3] 1 2 Metz, Rachel (8 September 2015). "Why NASA Wants Microsoft's HoloLens in Space". MIT Technology Review . Retrieved 12 September 2015.

[4] Buhr, Sarah (28 June 2015). "SpaceX's Failed CRS-7 Mission Had Something To Do With An "Overpressure Event"". TechCrunch. Retrieved 17 February 2016.

[5] Chang, Kenneth (June 28, 2015). "SpaceX Rocket Explodes After Launching". New York Times. Retrieved 28 June 2015.

[:0-6] 1 2 Nuernberger, Benjamin; Tapella, Robert; Berndt, Samuel-Hunter; Kim, So Young; Samochina, Sasha (2020-01-01). "Under Water to Outer Space: Augmented Reality for Astronauts and Beyond". IEEE Computer Graphics and Applications. 40 (1): 82–89. doi:10.1109/MCG.2019.2957631. ISSN 0272-1716.

[1]

[2]

[3]

[4]

[5]

[6]