Data Fuels Robotics Research

AI chatbots have exploded into the mainstream, but the technology is just now starting to trickle down into other areas, such as robotics. One of the major hurdles to adopting AI and other digital technologies in these fields is data. LLMs have vast amounts of data available via the internet, but robotics data takes more time. “You can't get enough data from just doing physical robot tests,” says Gary Fedder, the Howard M. Wilkoff Professor of Electrical and Computer Engineering at Carnegie Mellon University. “You'd be here forever generating data.”

Fedder is currently involved in efforts to gather the data needed to advance robotics, as he is not only a professor at CMU, but also the faculty director of the Manufacturing Futures Institute (MFI). Part of Carnegie Mellon University’s College of Engineering, the MFI brings together different areas of study to collaborate on projects covering subjects such as robotics, additive manufacturing, digital twins and more. “We’re here to catalyze advanced manufacturing research within the university for the benefit of the country and the benefit of the region,” Fedder explains. “We've been very broad-based to date, looking at advanced manufacturing from the bottom up to see what's going on, what are people doing, and we want to be able to support those efforts by seeding research.” The institute performs this research, translates the results to industry, and builds the necessary relations to make everything happen.

Much of MFI’s work has focused on digitalization in manufacturing, which, according to Fedder, is experiencing renewed interested thanks to maturation of AI. “You're seeing it in your everyday life,” he notes, “and the same is true of how that can impact manufacturing and a lot of that ties to the need for data.”

Armed and Ready

MFI is a collaborator on a project called the AI Data Foundry, along with Catalyst Connection, an economic development agency. The project is led by the ARM (Advanced Robotics for Manufacturing) Institute. ARM is a Manufacturing USA manufacturing innovation institute, which are public-private partnerships that bring together government, industry and academia to collaborate and share resources to advance the manufacturing industry in America. ARM focuses specifically on strengthening U.S. manufacturing through robotics and artificial intelligence. ARM, Catalyst Connection and MFI are separate entities but share resources, including space at Mill 19, a redeveloped steel mill just outside downtown Pittsburgh.

The concept of the AI Data Foundry started in early 2024 and is intended to be “a national center that collects and aggregates data for robots working in manufacturing settings in order to accelerate and develop AI and machine- learning algorithms that are relevant to various manufacturing tasks,” Fedder explains. This data will come from robots at ARM’s facility, as well as ARM’s consortium members, and other credible sources to make robots more effective at performing common manufacturing tasks such as machine tending, welding, grinding, polishing, painting and more.

This data will provide the foundation for algorithms that will enable robots to work with challenging workpieces (such as flexible parts) or complete complex tasks (plugging cables into sockets or screwing nuts and bolts together). These tasks are more challenging to automate because they aren’t perfectly repetitive and therefore require so much data to program that gathering enough real-world data could be prohibitive. According to Fedder, synthetic, or computer-generated data could bridge this gap. Researchers can use synthetic data for tests and validate the results with real-world data.

The end goal is to produce pre-trained models which can be quickly and easily customized for specific end uses. For example, researchers could create a pre-trained model for plugging in a USB cable, which a manufacturer could take and adapt to its robot, environment and task. “By having these pre-trained models, it's a lot faster to provide that solution to the manufacturer,” Fedder says.

The AI Data Foundry’s work will initially be available to the Institute’s consortium members and collaborators. While manufacturers are typically reticent to share data about their work, Fedder explains that robotics data often doesn’t reveal proprietary information about a part. “For a lot of the manufacturers, robotics is a means to an end,” he notes. “So, it may surprise you that there are a lot of companies that are willing to share data in that sector.” Many are also committed to advancing AI. “They realize, too, that in order to reap the benefits of AI, one needs to cooperate and share data and aggregate data,” he adds. 

Twinning

Robotic data is also key for developing digital twins. “A digital twin, strictly speaking, is going to be a living, predictive model of a physical twin,” Fedder says. “The key part of this is it's bi-directional, meaning you're collecting data from the physical twin and updating your model.” This concept is familiar to many roboticists, who have been using static models in their work for some time, but a “living” model makes their work more accurate and precise.

Digital twins aren’t a new concept, but only recently has “the technology has caught up with the vision,” Fedder says. “The technology is the convergence of our ability to more easily network, collect data and compute. Moore's Law has improved over the years, to the point now where no one's blinking at giant amounts of data like they used to.”

A digital twin of a robot gives its computer (which functions as its “brain”) a better understanding of the robot, so it can better predict the outcomes of its actions. “For everything the robot needs to manipulate, it needs to understand the consequence,” explains Changliu Liu, an assistant professor at CMU’s Robotics Institute and director of its Intelligent Control Lab. “What if I do this, what if I do that, how that will impact the future evolution of the system?”

Robots cannot make these predictions without data. “They need to have a world model, maybe it’s data driven, maybe some human has predefined it for them, to understand what will happen,” Liu continues. This is particularly challenging in applications that combine manual and automated operations. Liu describes one of her research projects, which involves robotic finishing of manual welds. “It's hard to get a digital twin,” she explains. “We have the CAD model, but since the weld is done manually, there's huge variation.” Local sensors help build a model of the workpiece while a laser in the end effector provides real-time measurements.

Digital twins bridge the gap between the physical and digital worlds. “If you have digital twins, you have that anchor into the physical world, but it's an anchor that computers understand,” Fedder notes. This anchor is essential because stakes are high in the physical world. “A robot could actually hurt someone, or be damaged, or screw up your manufacturing line and create a bunch of scrap or other issues,” Fedder notes. “So, this idea of hallucinating, if you want to call it that, or just not being perfect, can't really be tolerated as much in the physical world.” But with enough data, digital twins can help provide safeguards. “If there were enough digital twins out there, we could start realizing this future of having intelligence that actually does manipulate the physical world in ways that are safe,” he adds.

Manufacturers can extend the concept of the digital twin to maximize its benefits. “If you can also then have a digital twin of the environment around the robot, it becomes even more powerful,” Fedder adds. Digital twins can even aid in robotic maintenance. “While the robot is in motion, the gears may be in friction, and then the motors or the mechanical part may wear out, and all of this needs to be tracked,” Liu explains. A digital twin can also help the robot understand its physical limits at a given time.

Academia and Industry

Academia, and Carnegie Mellon University specifically, is well-positioned to advance the digitization of robotics and manufacturing. For one, the school has a strong AI program (U.S. News and World Report ranks it the best in the nation).

Academia generally also has advantages when it comes to research. Industry needs to consider its bottom line, which makes long-term research challenging to undertake. This constraint also gives academics more flexibility in what they study. Academia is where “the crazy ideas become not crazy ideas,” Fedder says.

According to Fedder, CMU provides a remarkably collaborative environment for cross-disciplinary research. “People are not pigeonholed, so you get a lot of creativity that attracts people that love that creative environment,” he explains. “It’s not something you can manufacture. Over time, it happens.” The school’s history, cutting-edge research and collaborative environment have attracted a “clustering of like-minded geniuses who feed off each other,” he adds. This environment is particularly beneficial for multi-disciplinary fields such as robotics.