Tesla’s head of AI has released new footage of the automaker’s auto labeling tool for its self-driving effort.

It’s expected to be an important accelerator in improving Tesla’s Full Self-Driving Beta.

Labeling data for self-driving

Tesla is often said to have a massive lead in self-driving data thanks to having equipped all its cars with sensors early on and collecting real-world data from a fleet that now includes over a million vehicles.

The automaker is able to use the extensive data set to improve its neural nets powering its suite of Autopilot features, and it ultimately believes it will lead to full self-driving capability.

However, that data is a lot more valuable when it is “labeled” – meaning that the information in the images collected by the fleet is being tagged with information, such as vehicles, lanes, street signs, etc.

If the images are properly labeled – for example, if you can consistently recognize a speed sign and label it as such – you can feed a bunch of different images of different speed signs to a computer vision neural net in order to be able to recognize them.

Labeling has been a focus of Tesla’s Autopilot team.

Andrej Karpathy, Tesla’s head of AI and computer vision, revealed last year that Tesla only has “a few dozen” engineers working on neural networks, but they have a “huge” team working on labeling.

Tesla is trying to automate a lot of the labeling in order to be able to use a lot of the data that is being collected by the fleet.

Last year, Tesla CEO Elon Musk said that drivers are effectively labeling just by driving through intersections:

Essentially, the driver when driving and taking action is effectively labeling — labeling reality — as they drive and [make] them better and better. I think this is an advantage that no one else has, and we’re quite literally orders of magnitude more than everyone else combined.

But Tesla also has employees manually labeling data to be fed to its neural nets.

The automaker has reportedly hired thousands of labelers, many working out of Gigafactory New York.

Auto-Labeling

Even with thousands of employees manually labeling videos, Tesla is still leaving a lot of good data on the table.

The automaker has now over a million vehicles on the road collecting video footage that can be used to improve its neural nets.

The holy grail of labeling is developing an auto-labeling system that can automatically and accurately label large quantities of footage.

Tesla said that it’s working on such a tool, especially to work with its Dojo supercomputer.

It looks like the company is making progress.

In a new series of tweets, Karpathy released images from Tesla’s new auto labeling tool:

Karpathy wrote about the new footage:

Some panoptic segmentation eye candy from a new project we are bringing up. These are too raw to run in the car, but feed into auto labelers. Collaboration of data labeling a large (100K+), clean, diverse, multicam+video dataset and engineers who train the models.

The multicam + video data, temporal continuity of a slowly moving viewpoint, close collaboration with data sourcing and labeling, and the infinity-sized dataset of unlabeled clips dramatically expands creative modeling opportunities on the neural net side.

Karpathy, who leads tesla’s computer vision team, said that it’s still early in the deployment of this technology and he appears to be sharing the footage in an effort to recruit more people for his team.

Subscribe to Electrek on YouTube for exclusive videos and subscribe to the podcast.