Tesla Autopilot, an advanced driver-assistance system for Tesla vehicles, uses a suite of sensors and an onboard computer. It has undergone several hardware changes and versions since 2014, most notably moving to an all-camera-based system by 2023, in contrast with ADAS from other companies, which include radar and sometimes lidar sensors.
Initially, the ADAS used a combination of cameras capturing the visual spectrum, forward-facing radar, ultrasonic proximity sensors, and a Mobileye EyeQ3 computer as Hardware 1, fitted to Model S vehicles starting in October 2014. After Mobileye ended its partnership with Tesla in 2016, Tesla began shipping cars equipped with an Nvidia Drive PX 2 computer and an increased number of cameras as Hardware 2. In 2019, Tesla shifted to a computer using a custom "FSD Chip" designed by Tesla, branded as Hardware 3. Starting in 2021, Tesla stopped installing the radar sensor in new vehicles, and the ADAS was updated to drop radar support. In 2022, Tesla announced it also would drop support for the ultrasonic sensors, moving the ADAS to an all-visual system. The most recent sensor and computer implementation is Hardware 4, which began shipping in January 2023.
Vehicles manufactured after late September 2014 are equipped with a single camera mounted at the top of the windshield, forward looking radar [1] [2] in the lower grille, and 12 ultrasonic acoustic location sensors in the front and rear bumpers that provide a 360-degree view around the car. [3] The computer is the Mobileye EyeQ3; [4] as implemented, this chip is built on a 40 nm process with a TDP of 2.5 W and a clock speed of 500 MHz. [5] : 5 This equipment allows suitably equipped Tesla Model S and Model X vehicles to detect lane markings, obstacles, and other vehicles, enabling advanced driver-assistance functions branded Autosteer (automatic lane-keeping), Auto lane change, Autopark (parallel parking robot), and Side-collision warning. [3] The EyeQ3 used a neural network approach, relying primarily on inputs from the camera to recognize and label objects to determine which potential areas in the camera's field of view are unoccupied and possible for the vehicle to travel through. [5] : 4
Auto lane change can be initiated by the driver turning on the lane changing signal when safe (due to the ultrasonic sensors' 16-foot limited range capability), and then the system completes the lane change. [6] In 2016 the system did not detect pedestrians or cyclists, [7] and while Autopilot detects motorcycles, [8] there have been two instances of HW1 cars rear-ending motorcycles. [9]
Tesla released a new version of Autopilot in September 2016 that changed the object detection algorithm to more fully use the radar sensor; previously, primary obstacle detection responsibilities fell to the cameras and the radar was used in a secondary role to confirm their presence, but was not given the authority to initiate emergency braking alone. After the update, the radar data was given an equal role in object detection and made capable of identifying "dense obstacles," including "other vehicles, moose, or even alien spaceships," according to Musk. He added that Tesla "believes it would have" prevented the fatal May 2016 underride crash in Williston, Florida, in which Autopilot failed to detect a white trailer against the sky. [10]
Mobileye ended its partnership with Tesla in 2016, stating that Tesla was "going to hurt the interests of [Mobileye] and hurt the interests of an entire industry, if a company of our reputation will continue to be associated with this type of pushing the envelope in terms of safety". [11] Tesla responded that Mobileye backed away after learning Tesla was developing its own vision-based sensor system. [12] Speculation immediately following the announcement included partnering with Nvidia and potentially designing its own ADAS computer. [13] After Hardware 2 was released, an upgrade from Hardware 1 to Hardware 2 was not offered as it would have required substantial work and cost. [14]
HW2, included in vehicles manufactured after October 2016, includes an Nvidia Drive PX 2 [15] GPU for CUDA based GPGPU computation. [16] [17] Tesla claimed that the hardware was capable of processing 200 frames per second. [18] Elon Musk called HW2 "basically a supercomputer in a car", referring to its capacity of up to 12 trillion operations per second. [19] The Autopilot computer hardware, housed just above the glovebox, is replaceable to allow for future upgrades. [20] [21] Tesla claimed the HW2 suite of sensors and computation provided the necessary equipment to allow FSD at SAE Level 5. [22]
The hardware includes eight cameras covering an aggregate view of 360° around the car and 12 ultrasonic sensors, in addition to forward-facing radar with enhanced processing capabilities. [22] The radar is able to observe beneath and ahead of the vehicle in front of the Tesla; [23] the radar can see vehicles through heavy rain, fog or dust. [24] The eight cameras are mounted in various locations around the vehicle: three forward-facing, next to the central rearview mirror mounted on the windshield; two front/side cameras, one each mounted in the left and right B-pillars; two rear/side cameras, mounted in the left and right front fender turn-signal repeaters; and one rear camera, above the license plate. [19]
When "Enhanced Autopilot" was enabled in February 2017 by the v8.0 (17.5.36) software update, testing showed the system was limited to using one of the eight onboard cameras—the main forward-facing camera. [25] The v8.1 software update released a month later enabled a second camera, the narrow-angle forward-facing camera. [26] With all eight cameras enabled, data extracted from Autopilot in debugging mode showed the cameras provide a black-and-white feed to the computer, possibly to improve image processing speed. [27]
The Tesla Model 3, introduced in 2017, and related Model Y, introduced in 2019, are equipped with an additional driver-facing in-cabin camera. This was disabled at launch and was intended to monitor the cabin remotely while the owner was operating the vehicle as an autonomous robotaxi, [28] but was activated in May 2021 to monitor driver attentiveness while using Autopilot in vehicles without radar sensors. [29]
In August 2017, Tesla announced that HW2.5 included a secondary processor node to provide more computing power and additional wiring redundancy to slightly improve reliability; it also enabled dashcam and "sentry mode" capabilities. [30] During this time, the supplier for the system's radar components was changed from Bosch to Continental, [31] using the ARS4-B unit. [32]
Tesla will upgrade HW2.5 vehicles to HW3 at no extra cost if the owner purchases Full Self-Driving.
According to Tesla's director of Artificial Intelligence (AI) Andrej Karpathy, Tesla had, as of Q3 2018, trained large neural networks but they could not be deployed to Tesla vehicles built up to that time due to insufficient computational resources. HW3 was designed to run these neural networks. [33] Overall, Tesla claims HW3 has 2.5× improved performance over HW2.5, with 1.25× higher power and 0.2× lower cost. [34]
HW3 is based on a custom Tesla-designed system on a chip called "FSD Chip", [35] fabricated using a 14 nm process by Samsung. [36] Jim Keller and Pete Bannon, among other architects, have led the project since February 2016. [37] FSD Chip features twelve ARM Cortex-A72 CPUs operating at 2.6 GHz, two systolic arrays (not unlike the approach of TPU) [38] operating at 2 GHz and a Mali GPU operating at 1 GHz. [34] Tesla claimed that FSD Chip processes images at 2,300 frames per second (fps), which is a 21× improvement over the 110 fps image processing capability of HW2.5. [37] [39] The firm described FSD Chip as a "neural network accelerator" custom-designed for Tesla AI processing. [18] Each of the two systolic arrays on a single FSD Chip are capable of 36 trillion operations per second, [35] and there are two FSD Chips for redundancy. [40]
HW3 cars are equipped with eight cameras, in the same locations and covering the same directions as the HW2 and HW2.5 cameras. Each of the eight cameras supplied with HW3 use the same AR0136A image sensor supplied by Onsemi, which has a maximum resolution of 1280×960 (1.2-megapixel) and a 3.75 μm pixel size. [41] Initial versions of HW3 also included a Continental ARS4-B radar module. [41] The HW3 system board is the same physical size as the HW2.5 board, but carries more components. [41] This makes it possible to upgrade from HW2 / HW2.5 to HW3, which was not possible during the prior generational change from HW1 to HW2/2.5.
The company claimed that HW3 was necessary for FSD, but not for "enhanced Autopilot" functions. [42] The first availability of HW3 was April 2019. [43] Customers with HW2 or HW2.5 who purchased the FSD package were promised an upgrade to HW3 without cost; [44] however, when FSD Beta was released in 2021, Tesla owners who had subscribed to FSD and owned a vehicle with HW2 or HW2.5 were required to pay US$1500 to upgrade to HW3. Owners who had purchased FSD with a one-time fee as part of their initial vehicle acquisition were eligible for a free upgrade to HW3. [45] [46] Within days, Tesla reduced the cost of the upgrade to US$1000. [47] In 2022, a Tesla owner in the State of Washington won a default judgment against Tesla which ordered the company to upgrade his vehicle to HW3 without cost; in the ruling, the judge cited Tesla's promises that all cars sold since 2016 included the hardware for FSD. [48]
In late May 2021, Elon Musk posted to Twitter that "Pure Vision Autopilot" was starting to be implemented. [49] The system, which Tesla brands "Tesla Vision", eliminates the forward-facing radar starting in May 2021 from the Autopilot hardware package on Model 3 and Model Y vehicles built for the North American market. [50] The Washington Post reported in March 2023 the immediate result was "an uptick in crashes, near misses and other embarrassing mistakes by Tesla vehicles suddenly deprived of a critical sensor." [51] For vehicles without the forward radar, temporary limitations were applied to certain features such as Autosteer, and other features (Smart Summon and Emergency Lane Departure Avoidance) were disabled, but Tesla promised to restore the features "in the weeks ahead ... via a series of over-the-air software updates".
In response, Consumer Reports delisted the Model 3 from its Top Picks in 2021, and the Insurance Institute for Highway Safety (IIHS) announced plans to delist the Model 3 as a Top Safety Pick+, [52] [53] but after further testing, both organizations restored those designations. [54] NHTSA rescinded the agency's check marks for forward collision warning, automatic emergency braking, lane departure warning, and dynamic brake support, applicable to Model 3 and Model Y vehicles built on or after April 27, 2021, [55] but as of October 2022 [update] , those check marks have been restored. [56] [57]
In December 2021, the New York Times reported that Musk had made a unilateral decision to pursue the camera-only approach and had "repeatedly told members of the Autopilot team that humans could drive with only two eyes and that this meant cars should be able to drive with cameras alone." Several autonomous vehicle experts have denounced the analogy. [58] According to former employees, Tesla engineers attempted to convince Musk that removing the radar could lead to crashes if the cameras became obscured, but Musk "was unconvinced and overruled" their objections. [51] Brad Templeton noted that lidar "will never fail to see a train or truck, even if it doesn't know what it is. It knows there is an object in front and the vehicle can stop without knowing more than that." [51] In contrast, Tesla Vision relies on the "Autopilot labeling team", [59] hundreds of Tesla employees that view short video clips recorded by the cameras to label visible signs and objects, which trains the machine vision interpreter. [51] [60]
Tesla announced in October 2022 they would remove ultrasonic sensors by 2023; vehicles without the ultrasonic sensors would be shipped without Autopark, Park Assist, Summon, and Smart Summon features initially. [61] Vehicles shipped in 2023 featured Park Assist, however it relied on the cameras to determine the distance from objects. In late 2023, Tesla rolled out High Fidelity Park Assist for AMD-equipped vehicles. Throughout 2024, additional features were rolled out, including Summon (renamed to Dumb Summon) and the revamped Autopark and Smart Summon (named Actual Smart Summon). [62] [63]
Samsung manufactures the processor for Hardware 4 (HW4) in Hwasung, South Korea, on a 7 nm process. The custom System on a chip (SoC) is called "FSD Computer 2". [64] According to a teardown of a production HW4 unit in August 2023, the board has 16 GB of RAM and 256 GB of storage, which are two and four times the RAM and storage in HW3, respectively. [65] Musk stated that HW4 computational capabilities are three to eight times more powerful than HW3. [66]
In June 2022, Tesla filed an application with the US Federal Communications Commission (FCC) to use a new radar module in future vehicles. [67] Based on filings with the Chinese government and a prototype Tesla Model 3 spotted in December 2022 while undergoing testing in Santa Cruz, it is believed that HW4 could include radar, high-resolution cameras, and fans and/or heaters fitted to the cameras to prevent fogging. [68] While automotive radar modules used in other manufacturer's cars have up to 200 channels (giving them capabilities similar to lidar), [32] an updated filing with the FCC in 2023 indicates the new radar module that might be included with HW4 is not likely to be high-resolution. [69]
Tesla started shipping cars with HW4 in January 2023, starting with the refreshed Model S and Model Y; [66] however, FSD was not available initially. [70] It took six months before HW4-based cars ran camera-based software. [66] [71] Despite the increased image sensor resolution with HW4-equipped cars, HW4 runs the FSD software by emulating HW3, including downsizing the camera images, as Tesla has postponed training based on the new HW4 cameras. [72] [66] Musk stated that the HW4-specific training of FSD will start after the new data center is finished at the Texas Gigafactory. [66]
Musk announced Hardware 5 (HW5), being named AI5, during the Tesla annual meeting on June 13, 2024. [66] [73] Musk stated it is scheduled for release in January 2026, and will be ten times more powerful than HW4. [66] Musk also stated that it will use up to 800 watts when processing complex environments (versus 300 watts for HW3 and HW4). [66]
Hardware | Year | Function | Description [74] | Requirements | ||||
---|---|---|---|---|---|---|---|---|
HW1 | HW2 | HW3 | AP | EA | FSD | |||
2014 | Over-the-air updates | Autopilot updates received as part of recurring Tesla software updates. | ||||||
2014 | Safety Features | If Autopilot detects a potential front or side collision with another vehicle, bicycle or pedestrian within a distance of 525 feet (160 m), it sounds a warning. [75] Autopilot also has automatic emergency braking that detects objects that may hit the car and applies the brakes, and the car may also automatically swerve out of the way to prevent a collision. | ||||||
2014 | Visualization | System generates a visualization of what it sees around it, including lane lines and vehicles in front, behind and on either side of it (in other lanes). It also displays lane markings and speed limits (via its cameras and what it knows from maps). On HW3, it displays stop signs and traffic signals. It distinguishes pedestrians, bicyclists/motorcyclists, small cars, SUVs, pickup trucks, buses, and large semi-trucks. | ||||||
2014 | Speed Assist | Front-facing cameras detect speed limit signs and compare them against map data to display those limits on the dashboard center display even if no signs are detected. [75] | ||||||
2018 | Obstacle Aware Acceleration | Reduces acceleration when an obstacle is detected in the path of travel while driving at low speeds. | ||||||
2019 | Blind spot monitoring | Sounds warning chime when an obstacle is detected while changing lanes. | ||||||
2014 | Traffic Aware Cruise Control [80] | Also known as adaptive cruise control, the ability to maintain a safe distance from the vehicle in front of it by accelerating and braking as that vehicle speeds up and slows down. It also slows on tight curves, on interstate ramps, and when another car enters or exits the road in front of the car. It can be enabled at any speed between 0 mph and 90 mph. By default, it sets the limit at the current speed limit and adjusts its target speed according to changes in speed limits. If road conditions warrant, Autosteer and cruise control disengage and an audio and visual signal indicate that the driver must assume full control. | ||||||
2014 | Autosteer | Steers the car to remain in whatever lane it is in (lane keeping). It is able to safely change lanes when the driver taps the turn signal stalk. [81] On divided highways, HW2 and HW3 cars limit use of the feature to 90 mph (145 km/h), and on non-divided highways, the limit is five miles over the speed limit or 45 mph (72 km/h) if no speed limit is detected. [82] If the driver ignores three audio warnings about controlling the steering wheel within an hour, Autopilot disables until a new journey is begun. [83] | ||||||
2019 | Emergency Lane Departure Avoidance | Steers to prevent departing from a lane that may have a chance of collision. | ||||||
2019 | Lane Departure Avoidance | Steers to maintain lane centering. | ||||||
2014 | Lane Departure Warning | Warns the driver when the vehicle begins to move out of lane centering. | ||||||
2016 | Navigate on Autopilot | A set of features consisting of automatically 'guiding through highway interchanges and exits' including 'lane changes on certain roads'. [90] As of 2019, it navigates freeway interchanges fully from onramp to offramp, including automatic lane changes. [89] [91] [92] | ||||||
2014 | Automatic Lane Change | Lane changing without driver initiation. [93] [6] | ||||||
2015 | Autopark | Parks the car in perpendicular or parallel parking spaces on either side of the road, with the car backing into spaces, without driver supervision. [97] [98] In 2024, Tesla enabled vision-based Autopark for vehicles running Hardware 3 and higher. [63] | ||||||
2014 | Summon | Moves car forward or backward in tight spaces using the key fob or the Tesla app, without the driver in the car. [99] [100] | ||||||
2019 | Smart Summon | Enables remote car retrieval in parking lots using the key fob or Tesla phone app from up to 150 ft (46 m) away. Smart Summon uses the ultrasonic sensors for navigation and requires users to maintain line of sight with the vehicle. [102] [103] [104] | ||||||
2019 | Traffic Signs Aware | Traffic light, stop sign and yield sign recognition. | ||||||
2020 | Traffic Light and Stop Sign Control [80] [106] | When using Traffic-Aware Cruise Control or Autosteer, this feature will stop for stop signs and red traffic lights, [107] and will proceed through green lights. [108] Even when Autopilot is not engaged, the system can chime when the traffic light turns green. [109] | ||||||
2023 | Autosteer on city streets | Enables in-city navigation under FSD. [110] | ||||||
2024 | Actual Smart Summon | Enables remote vehicle retrieval in parking lots using the Tesla phone app from up to 280 ft (85 m) away. Actual Smart Summon uses the vehicle's onboard cameras for navigation and requires users to maintain line of sight with the vehicle. [111] [112] |
Hardware name | Hardware 1 | Hardware 2.0 [a] | Hardware 2.5 [b] | Hardware 3 [c] | Hardware 4 | ||
---|---|---|---|---|---|---|---|
Initial availability date | 2014 | October 2016 | August 2017 | April 2019 | May 2021 | October 2022 | January 2023 [66] |
Computers | |||||||
Platform | MobilEye EyeQ3 [116] | Nvidia DRIVE PX 2 AI computing platform [117] | Nvidia DRIVE PX 2 with secondary node enabled [114] | Two identical Tesla-designed "FSD 1 Chip" (12 core) processors | Two identical Tesla-designed "FSD 2 Chip" (20 core) processors [118] | ||
Sensors | |||||||
Forward radar | 160 m (525 ft) [74] | 170 m (558 ft) [74] | None [119] [d] | Model S & X: 300 m (984 ft) [123] Model 3 & Y: None | |||
Ultrasonic | 12 surrounding with 5 m (16 ft) range | 12 surrounding with 8 m (26 ft) range | None [124] | ||||
Front / Side camera | N/A | RCCC [74] | RCCB [74] | ||||
Forward cameras | 1 monochrome with unknown range | 3:
| All:
| ||||
Forward looking side cameras | N/A |
| 5 megapixel | ||||
Rearward looking side cameras | N/A |
| 5 megapixel |
A self-driving car, also known as a autonomous car (AC), driverless car, robotaxi, robotic car or robo-car, is a car that is capable of operating with reduced or no human input. Self-driving cars are responsible for all driving activities, such as perceiving the environment, monitoring important systems, and controlling the vehicle, which includes navigating from origin to destination.
Tesla, Inc. is an American multinational automotive and clean energy company. Headquartered in Austin, Texas, it designs, manufactures and sells battery electric vehicles (BEVs), stationary battery energy storage devices from home to grid-scale, solar panels and solar shingles, and related products and services.
The Tesla Model X is a battery electric mid-size luxury crossover SUV built by Tesla, Inc. since 2015. Developed from the full-sized sedan platform of the Tesla Model S, the vehicle notably uses falcon wing doors for passenger access.
Mobileye Global Inc. is an Israeli autonomous driving company. It is developing self-driving technologies and advanced driver-assistance systems (ADAS) including cameras, computer chips, and software. Mobileye was acquired by Intel in 2017 and went public again in 2022.
The Tesla Model 3 is a battery electric powered mid-size sedan with a fastback body style built by Tesla, Inc., introduced in 2017. The vehicle is marketed as being more affordable to more people than previous models made by Tesla. The Model 3 was the world's top-selling plug-in electric car for three years, from 2018 to 2020, before the Tesla Model Y, a crossover SUV based on the Model 3 chassis, took the top spot. In June 2021, the Model 3 became the first electric car to pass global sales of 1 million.
The Tesla Powerwall is a rechargeable lithium-ion battery stationary home energy storage product manufactured by Tesla Energy. The Powerwall stores electricity for solar self-consumption, time of use load shifting, and backup power.
The Tesla Model Y is a battery electric mid-size crossover SUV produced by Tesla, Inc. since 2020. The vehicle was presented in March 2019 as the company's fifth production model since its inception after the Roadster, Model S, Model X and Model 3.
Nvidia Drive is a computer platform by Nvidia, aimed at providing autonomous car and driver assistance functionality powered by deep learning. The platform was introduced at the Consumer Electronics Show (CES) in Las Vegas in January 2015. An enhanced version, the Drive PX 2 was introduced at CES a year later, in January 2016.
Tesla Autopilot is an advanced driver-assistance system (ADAS) developed by Tesla that amounts to partial vehicle automation. Tesla provides "Base Autopilot" on all vehicles, which includes lane centering and traffic-aware cruise control. Owners may purchase or subscribe to Full Self-Driving (FSD) which adds semi-autonomous navigation that responds to traffic lights and stop signs, lane change assistance, self-parking, and the ability to summon the car from a garage or parking spot.
The Tesla Semi is a battery electric semi-trailer truck built by Tesla, Inc. since 2022. The truck is powered by three motors, and according to Tesla has approximately three times the power of a typical diesel semi truck, a range of 500 miles (800 km), and can operate at an energy use of less than two kilowatt-hours per mile (1.2 kW⋅h/km).
The Tesla Roadster is an upcoming battery electric four-seater sports car to be built by Tesla, Inc. The company said it will be capable of accelerating from 0 to 60 mph in 1.9 seconds, which would be quicker than any street legal production car to date at its announcement in November 2017. The Roadster is the successor to Tesla's first production car, the 2008 Roadster.
In road-transport terminology, lane centering, also known as lane centering assist, lane assist, auto steer or autosteer, is an advanced driver-assistance system that keeps a road vehicle centered in the lane, relieving the driver of the task of steering. Lane centering is similar to lane departure warning and lane keeping assist, but rather than warn the driver, or bouncing the car away from the lane edge, it keeps the car centered in the lane. Together with adaptive cruise control (ACC), this feature may allow unassisted driving for some length of time. It is also part of automated lane keeping systems.
DeepScale, Inc. was an American technology company headquartered in Mountain View, California, that developed perceptual system technologies for automated vehicles. On October 1, 2019, the company was acquired by Tesla, Inc.
openpilot is an open-source, semi-automated driving software by comma.ai, Inc. When paired with comma hardware, it replaces advanced driver-assistance systems in various cars, improving over the original system. As of 2023, openpilot supports 250+ car models and has 6000+ users, accumulating over 90 million miles (140,000,000 km).
Tesla, Inc. has been criticized for its cars, workplace culture, business practices, and occupational safety. Many of the criticisms are also directed toward Elon Musk, the company's CEO and Product Architect. Critics have also accused Tesla of deceptive marketing, unfulfilled promises, and fraud. The company is currently facing criminal and civil investigations into its self-driving claims. Critics have highlighted Tesla's downplaying of issues, and Tesla's alleged retaliation against several whistleblowers.
Tesla Dojo is a supercomputer designed and built by Tesla for computer vision video processing and recognition. It is used for training Tesla's machine learning models to improve its Full Self-Driving (FSD) advanced driver-assistance system. According to Tesla, it went into production in July 2023.
require "stripping down the entire car and replacing 300+ parts"
Musk did say that the new vehicles will eventually be able to upgrade the new onboard Autopilot computer since the access has been made relatively easy
Dashcam is only available for Model S and Model X cars manufactured after August 2017...
If you purchased Full Self-Driving capability and have Autopilot computer 2.0 or 2.5, you are eligible for a complimentary upgrade to FSD computer. Certain early production vehicles may also receive complimentary camera replacements. A complimentary hardware upgrade is not available for Full Self-Driving capability subscriptions; however, you may be eligible to upgrade at your cost.
Our goal with the introduction of this new hardware and software is not to enable driverless cars, which are still years away from becoming a reality. Our system is called Autopilot because it's similar to systems that pilots use to increase comfort and safety when conditions are clear. Tesla's Autopilot is a way to relieve drivers of the most boring and potentially dangerous aspects of road travel – but the driver is still responsible for, and ultimately in control of, the car.