Ex-Meta Researchers Set to Launch Visual AI in Manufacturing Environments

Artificial intelligence is transforming our environment, yet its effects have primarily been confined to the digital realm thus far. However, a growing number of startups are working to integrate AI into the physical realm.

Perceptron, a startup co-founded by two ex-scientists from Meta, is a prime example of this endeavor. Launched in November 2024, the company specializes in creating sophisticated vision models aimed at improving machine interactions within their physical settings.

This week, the startup introduced its latest model, Isaac 0.5, which its creators claim allows machines to “perceive, reason, and act” in industrial environments. In particular, the software aids vision-guided robots in maneuvering through intricate settings such as warehouses and factory floors. Furthermore, it supports companies in deriving visual insights from the footage captured by these robots.

Isaac 0.5 is being launched as an open-weight model, granting anyone access to its parameters and training data.

The startup, which recently raised $21 million in a funding round led by Bessemer Venture Partners, was co-founded by Armen Aghajanyan and Akshat Shrivastava, both of whom have experience in Meta’s Fundamental AI Research (FAIR) division. They aspire for their software to define the future of automated industrial deployment.

“The current landscape of physical AI presents a false dichotomy: generalist foundation models demanding extensive dedicated cloud GPUs for each application, or specialized models that address either perception or control, but not both,” the company explains.

Aghajanyan and Shrivastava assert that their solution distinguishes itself from existing models as it is versatile, designed for general usage rather than confined to a singular, repetitive task. They emphasize that the model can adapt to diverse environments or circumstances.

In a recent discussion, Shrivastava encouraged a deeper consideration of the steps involved in a basic task such as sorting boxes: “Picture a robot tasked with organizing packages right now. What would its necessary actions be?”

This seemingly simple task entails multiple steps. The robot must first read the package label, execute spatial analysis to identify the locations of the boxes, and choose which box to lift. If it needs to lift multiple boxes, it has to strategize which ones to select and in what order.

Perceptron’s software is designed to assist robots in navigating each step of this process. While the industry already has software that can help machines with most of these tasks, only a few programs possess the adaptability needed for fluctuating conditions.

Where does the information for this algorithmic advancement stem from?

Models like Isaac 0.5 develop operational capabilities by analyzing vast volumes of video training data. Perceptron claims that its latest model was trained on over a million hours of general video to teach its algorithm how to recognize specific environments, visuals, and situations. The company also heavily utilized what is known as ego video—footage captured, typically from a GoPro or wearable camera, from the perspective of a person engaged in a physical task—as well as UMI video, which is similarly used to train AI systems on movement by recording repetitive human actions.

Although Perceptron is keeping its training data sources confidential, Shrivastava indicated that the company has “internally developed petabyte-scale datasets that cover various modalities, whether it’s images, text, video, and so forth, including robotic trajectories.”

The opportunities for software that enhances robot efficiency in warehouses are significant, and Perceptron believes it is uniquely positioned to lead this automation wave. The startup is keen to market its software to diverse vendors, potentially incorporating its intelligence framework across various sectors.

These sectors encompass manufacturing, logistics and warehousing, security, mobility, as well as media and entertainment.

“Nothing like this truly exists in the current market,” Aghajanyan reflected. “We’re immensely excited about it.”