LOS ANGELES — The greater the distance, the more significant the delay between a human operator’s instructions and a robot carrying them out. One solution is to give robots greater autonomy. SE4 Inc. today introduced a robot operating system that combines virtual reality, computer vision, and artificial intelligence for accelerated remote robotic control. The company said that its Semantic Control product “provides adaptive operational capacity for dynamically changing environments at any distance.”

Tokyo-based SE4 is developing software that uses AI to understand an operator’s actions in virtual reality (VR), enabling a robot to conduct a complete process and mitigating latency. The company is starting with construction applications, with the goal of bringing such capabilities to space colonization.

“One thing that motivated us to make SE4 is the concept of robots as explorers of the future,” said Lochlainn Wilson, CEO of SE4. “NASA and SpaceX have concerns about latency and think you need astronauts in orbit to control robots, say, on the Moon or Mars. They look at latency as a force of nature or a fact of life, because of the speed of light.”

“We’ve created a framework for creating physical understanding of the world around the machines,” he told The Robot Report. “With semantic-style understanding, a robot in the environment can use its own sensors and interpret human instructions through VR.”

SE4 is part of the NVIDIA Inception accelerator, which supports AI startups with expertise and discounted hardware. The startup received its initial investment from Mistletoe Inc. after its founding in 2018.

“We’re just emerging from stealth and are still in the early seed stage,” Wilson said. “We are looking for computer vision employees and ‘adventure capitalists’ who are looking for a mission with a return on investment that’s more than five years out.”

SE4 Semantic Control lightens operator load

SE4 said Semantic Control eliminates the need for complex programming of robot behaviors, as well as the need to dedicate skilled staffers to such programming. Instead, operators use VR in a simulated 3D environment, where they can annotate objects and perform actions on them.

“Human supervision is still really important,” said Pavel Savkin, chief technology officer at SE4. “AI is usually good at accuracy for low-level tasks, like moving a block to a place. Humans are still better than AI in terms of comprehending or explaining high-level concepts, such as deciding which block to move in a changing space. Semantic Control is how we ask robots to do something.”

“The majority of robot teaching programs use 2D methods — such as a touchscreen pendant — to interact with 3D environments,” he explained. “We control objects in 3D space using computer vision and VR, which is more efficient and eliminates the learning curve. It also increases safety because tasks are sanity-checked first by the simulator, and it limits the potential for catastrophic failure. Instructions have to make physical sense before they can be executed.”

AI interprets the actions, which are organized and sent to the robot as a series of tasks, like a to-do list, said SE4. The robot itself can determine the best way to accomplish each task and in what order. It can use local information and adapt to changes in its workspace. SE4 is using collaborative robots, or cobots, for prototyping its software.

“The user interface is a step up from reaching into a 3D space with a VR controller,” said Wilson. “The semantic representation understands the relationships among objects, such as a where to put an item on a plane or desk. We can connect building blocks in any configuration, as well as purely virtual parts in a spacing template. When we attach blocks, the robot knows the relevant axes.”

“What I’ve been doing at SE4 is making all of these functions available in a more simplified way,” said Nathan Quinn, research engineer at SE4. “We can just reach out and interact with objects in VR. You don’t need to write lines of code to do something — we’ve translated concrete motion to abstractions behind the scenes.”

“Orchestrated autonomy allows one person to control many robots or many people to control one robot,” Savkin said. “With Semantic Control, we’re making an ecosystem where robots can coordinate together. The human says what to do, and the robot decides how.”

Creating order from chaos

SE4’s operating system is targeted at both new and existing robots equipped with the necessary sensor technology. It is designed to make robotics more useful in remote locations with high latency or in settings that are inhospitable to humans.

“Most robots work in highly organized environments,” said Wilson. “We’re working on a robot operating system [not to be confused with the open-source Robot Operating System or ROS] that starts with a disorganized starting condition and enables a robot to assemble a structure and connect everything inside with minimal human input.”

“We are talking with large companies in the construction industry,” said Savkin. “Our technology could be applicable to other industries, such as small-batch manufacturing, where you need to reorganize a production line, but a general-purpose robot won’t beat a specialist human or a specialist robot.”

“In Japan, the average age of construction workers is 55,” Wilson said. “It’s one of the least automated industries today.”

SE4 is initially targeting repetitive tasks in excavation, where autonomous diggers can be told via a virtual environment which sector to excavate, to what depth, and where to put the excavated earth. Once the robot has received its instructions, the operator can walk away, while it could dig and adapt to environmental changes. If the digger were to come across a boulder, for example, it could contact the operator for additional guidance.

“Another lovely thing about VR is that you can change scale,” said Quinn. “You can be a giant moving shipping containers or 5cm tall soldering a circuit board. With teleoperation, it doesn’t matter what it is — you could control robots, drones, or vehicles.”

“Since 90% of human labor is moving something from Point 1 to Point 2, you could save time for construction,” he added. “It’s Minecraft meets reality.”

From the Earth to Mars

The company’s founders expect their approach to be useful in space exploration. By uploading a directive to a robot and providing it with operational flexibility, there is no need for a stream of constant instructions and the resulting delays, which in outer space can take 12.5 minutes one way from Mars to Earth, making direct and real-time control impossible.

“Imagine a robotic construction project on Mars right now — it is like King George telling the colonists in the New World how to build a single structure with a one-month sea delay each way. That’s the status of where we are now,” Savkin said. “[With SE4], a general directive can go out to a single robot or to many devices for simultaneous control, using command queueing and automated delegation.”

“There is a limit to the number of instructions we can send in a sequence,” said Wilson. “I see latency as an expanding bubble of light seconds and light minutes. The ‘latency horizon,’ beyond which control becomes inefficient, is now only a half-second for tele-operation — for example, pouring a glass of water. For assembling a satellite dish on Mars, if one step fails, then the subsequent steps fail 40 minutes later. We’re pushing it out. With SE4, a robot could take multiple steps.”

“Operator training is specific to the application,” Savkin noted. “We would expect a child to be able to use our system and direct robots to make a moon base [or something] underwater, in a mine, or a nuclear power plant.”

“With our technology, you can copy and paste buildings once you have the initial knowledge and raw materials,” said Wilson. “You could copy the template for a home and make minor modifications.”

SE4 will demonstrate its robot operating system at SIGGRAPH’s Real-Time Live! in Los Angeles today.

“We want to see what’s out there and the reactions to our technology,” Wilson said. “We’re looking for partners to scale up the technology and use it in real-world situations.”