Can Brainwaves Influence the Evolution of Physical AI?

Visualize a Jenga tower nestled within a warehouse in San Leandro, California, representing the cutting edge of physical AI advancements.

This site acts as the headquarters for Encord, a company focused on crafting data tools for training AI models. Andrew Ceja, dubbed a pilot—Encord’s designation for robotic trainers—carefully removes blocks from the delicate tower while donning a headset that captures his visual experiences. While this method is traditional, it employs extra sensors to monitor his brain activity as he carefully disassembles the tower.

Encord differentiates itself from emerging rivals by asserting that the critical challenge for humanoid and warehouse robots lies not in the design of their models, but in the lack of real-world physical training data. Rather than just delivering existing datasets to robotics firms, Encord prioritizes the creation of essential data that is currently missing.

The brainwave headset used by Ceja is produced by Zander Labs, a German neuroscience startup. They believe that by studying brain activity—capturing cognitive factors like error detection, intention, and surprise—they can generate richer datasets crucial for model training. Encord and Zander are collaborating on a project to create an initial dataset annotated with brainwave data, which will be tested against client robotic models to evaluate its effect on performance prior to full-scale implementation.

Lucas Gehrke, a neuroscientist from Zander leading this initiative, is hopeful that insights gained from recorded brain activity during various tasks can offer invaluable guidance to model developers on when to deploy their most advanced models.

Vineeth Velmurugan, Encord’s head of robotic learning, views this effort as a groundbreaking move to resolve the data bottleneck in robotics. After his time at OpenAI’s robotics lab and Berkshire Grey, a firm specializing in warehouse automation, Velmurugan joined Encord to establish its internal data generation team.

Initially, Encord was founded to support organizations engaged in machine vision through data annotation and model evaluation. However, they quickly recognized a pressing need for their own training data as clients sought all-encompassing solutions for robotic manipulation tasks. “The data simply does not exist,” Velmurugan underscores.

The idea that generative AI might achieve its chatbot success in robotics faces a similar barrier. Large Language Models (LLMs) were trained on a vast range of Internet texts. However, finding equivalent foundational materials for training neural networks focused on physical manipulation is challenging: companies developing self-driving cars typically accumulate their data, but scaling this process can be taxing. While video data could serve as a training source, it often lacks the accuracy of real-world datasets. Velmurugan suggests that transformative outcomes would require a dataset approximately five times larger than the entire YouTube database, stressing that data generation is primarily a business issue, not merely a research concern.

Meeting Your Egocentric Data Needs

At present, organizations creating robotic intelligences mainly depend on two primary data sources: “Egocentric” video footage recorded by individuals with cameras, often combined with various perspectives and metrics, as well as data obtained from remote-controlled robots. Encord utilizes both methods, collecting egocentric data from factories around the globe while leveraging its San Leandro facility to explore novel approaches like brainwave data and dataset generation for skill acquisition.

During a recent visit from TechCrunch, pilots were using leader-follower setups—paired robotic arms, with one operated by a human and the other mimicking its moves—to gather data for tasks like pouring coffee into mugs (a challenging endeavor) and stacking poker chips. “Every humanoid robotics company has expressed interest in these skills,” Velmurugan noted.

Storage areas brimmed with boxes filled with artificial flowers in vases, books, plastic vegetables, kitty litter trays, scoops, and bundles of wires—the essential tools for training robots to handle household tasks.

At one station, another pilot, Sofia Infante, skillfully directs robotic arms to connect and disconnect ethernet cables from a server’s rear—an operation that data center managers would eagerly automate if robots could manage cable handling with the necessary precision. After attempting to navigate the setup myself, I quickly recognized why this task remains challenging: robotic claws are significantly less flexible than human fingers and lack the range of motion we often take for granted.

Encord is also investigating a new data modality using sensors affixed to the forearm to detect electrical signals in muscles. Human hand movements are often poorly recorded on video, but Velmurugan aims to build a 3D model of hand positions at any instant using arm sensors, leading to a more complete dataset for model training.

Encord’s datasets are rigorously annotated with detailed descriptions of actions depicted in each video—like “right hand tightens bolt”—to assist LLM-based models in comprehending activities. Velmurugan estimates that this level of annotation detail is valued at 100 times that of “low-quality ego data” for specific task training, while costing only 20 times more to produce, making it seem advantageous initially.

However, “20 times more” still signifies considerable expenses, highlighting the challenge: extracting text from the Internet—a strategy employed by LLM developers who gather data from platforms like Stack Overflow—incurs minimal costs for large labs. In contrast, generating physical training data is not economically sustainable, limiting the ability of physical AI frameworks to closely mimic LLMs. This type of data must be created rather than merely collected, altering the economic landscape of model development.

Velmurugan contends that advancement is occurring—thanks to Encord’s insights into various industry initiatives, he observes that both startups and established labs are adopting strategies that enhance physical AI models. This unique perspective, connecting numerous robotics firms, strengthens Encord’s value proposition by proactively identifying emerging data methodologies ahead of individual clients.

This will keep Encord’s team of roughly a dozen pilots fully engaged. Both Infante and Ceja are part of a growing workforce laying the groundwork for neural networks; they transitioned from Scale, another AI data annotation firm, to Encord.

Ceja previously worked in waste management, where his passion for technology led him to manage a robotic waste sorting system. Now, while interacting with the Jenga tower, he expresses enthusiasm for tackling robot training challenges, stating, “It’s something new every day!”