Inside Trey: The future of forklifts, augmented by Nvidia

Our goal has always been to expand the human capacity to move goods by automating the most complex human-operated workflows.

To achieve this, we need to always be on the lookout for ways to improve our technology and make the most of our innovations.

Recently, we identified a set of software solutions from Nvidia that help optimize the vision, semantic, and localization technology we developed with Nvidia hardware for our autonomous forklift Trey, specialized in loading and unloading trailers at the loading dock.

But before we go into detail about how Trey works, let’s take a quick look at the benefits of autonomous forklifts for loading bay operations.

Autonomous forklifts and workplace safety

How does having an autonomous forklift benefit your operation? In addition to streamlining loading and unloading operations, autonomous forklifts reduce costly workplace accidents at the loading bay.

According to stats from the Occupational Safety and Health Administration (OSHA), forklift accidents cause between 35,000 to 62,000 injuries each year.

In terms of severity, workplace accidents caused by traditional forklifts can range from “non-serious” (61,800) and those that result in serious injuries (34,900).

But manned forklift accidents also resulted in 128 fatal workplace accidents between 2022-2023.

Most commonly, forklift accidents are caused by rollovers, pedestrian impacts, and a lack of proper training.

And while the tragic and avoidable loss of human life is impossible to put a value on – the cost of workplace accidents can be calculated.

Cost of forklift accidents

In addition to the immeasurable value of human lives, forklift accidents in the workplace cost businesses money.

According to OSHA figures, the average cost of a worker compensation claim is $41,003. Add to this the average OSHA safety violation cost of $13,494.

Forklift accidents also cause equipment damage requiring repairs which can reach thousands of dollars. But it’s not just repairs – you also lose productivity while your forklift is out of commission.

How an autonomous forklift improves safety and productivity

Autonomous forklifts like Trey use a suite of technological solutions for full 360°environmental and situational awareness.

This awareness decreases the chances of costly workplace accidents occurring that could result in injuries and loss of productivity.

A tech stack that elevates safety and helps get the job done faster

 Trey’s autonomy stack consists of several crucial elements that combine to make autonomous mobile robots (AMR) a reality.

They key stack elements are:

• Vision module: How Trey “sees” its surroundings;
• Localization: 3D imaging of Trey’s surroundings in relation to Trey, and;
• Semantics: Object detection within a 3D field of view that helps Trey navigate the area.

These, and other Gideon’s custom-built and fine tuned pieces of technology are the foundations that make it possible for Trey to see its surrounding environment, understand it, and then localize in it – and ultimately, handle the work all on its own.

How Nvidia helps boost our autonomy stack

At Gideon we’ve identified Nvidia as a key provider of technology and solutions that help boost Trey’s ability to operate as an AMR.

In addition to running our robot and server-based deep learning models with Nvidia’s powerful graphics processing units (GPUs), we also employ Jetson Xavier NX as a part of the vision module.

The vision module is responsible for the image processing and stereometric depth estimation that makes it possible for Trey to navigate spaces using visual data.

We used Nvidia’s CUDA, an API used to program applications running on Nvidia GPUs, to run image processing and stereo depth estimation for the vision module.

Further, Nvidia’s TensorRT is a framework that optimizes deep learning models which are used for semantic segmentation, object detection, and pose estimation, as well as feature extraction and matching for 3D SLAM.

Recently, this was built into the Nvidia Isaac Perceptor.

What is Nvidia Isaac Perceptor?

Nvidia Isaac Perceptor is a set of perception-related software components for AMRs. This includes:

• AI based stereo depth perception
• 3D reconstruction in real time
• Accelerated stereo visual odometry

This development will allow Nvidia to offer deployment-ready components for AMRs in the future.

Essentially, this means that a company that plans on building a robot will not have to develop all the components, like we had to. Instead, they can use these software components free of charge when they run their systems using Nvidia hardware.

“At Gideon we developed our own version of the perception components offered by Nvidia. We expect that these versions will be more robust for our applications, especially after in-depth testing in production environments.

 The stereo depth perception developed by Nvidia is interesting, as it is based on a deep learning model that was trained on a high volume of synthetic and genuine camera images, resulting in detailed and smoother depth maps.” – Tomislav Haus, PhD – VP Software at Gideon

Additionally, the 3D reconstruction from Nvidia relies on GPU – as opposed to the CPU in our implementation. This optimizes resources for faster 3D data.

For the full integration of Nvidia components in our stack, our plan is to conduct a thorough evaluation and testing throughout the summer of 2024, focusing on stereo depth perception for depth prevision and depth map density – and potentially real-time 3D reconstruction.

In time, we trust that the Nvidia layer will accelerate Trey’s 3D vision capabilities to enhance its use in real-world applications.

But how much acceleration are we talking about? And what is the benefit of an accelerated and more detailed 3D map?

How much time does the Trey autonomous forklift save?

Compared to traditional, manned forklifts, autonomous forklifts can accelerate loading and unloading operations by as much as 80%.

This boost in speed will solve one of the bigger challenges in the supply chain that is still recovering from the impact of pandemic lockdowns.

More benefits of autonomous forklifts

While cutting back on time spent loading and unloading, autonomous forklifts also help streamline operations by cutting back – on injuries.

Thanks to safety sensors and leading machine learning autonomous driving capabilities, forklifts like Trey can navigate spaces more safely, unlimited by some of the constraints human operators face. These include blind spots, fatigue, and inadequate training.

Human error as the leading cause of workplace forklift injuries is all but eliminated by introducing AMR forklifts.

Autonomous forklifts also improve your carbon footprint.

Traditional forklifts are powered by fossil fuels – mainly gasoline, diesel, propane or natural gas. This can result in CO2 emissions of more than 5,000 grams per hour. This means that over the course of a couple of shifts, a forklift operator can generate their weight in CO2 – as well as other harmful fossil fuel emissions.

AMR forklifts are a more sustainable alternative to traditional forklifts because they run on battery power, lowering the negative environmental impact of loading and unloading operations.

Trey, the future of autonomous forklifts

Trey is Gideon’s autonomous mobile robot forklift developed to help streamline dock loading and unloading operations in supply chains with workplace safety and efficiency top of mind.

At Gideon, we recently began using Nvidia Isaac Perceptor software to optimize the performance of Nvidia hardware running Trey’s critical components – i.e. the vision module, 3D imaging and semantic processing.

Based on a deep learning model trained on a large number of synthetic and real-camera images, Nvidia’s AI-based stereo depth perception looks very promising. As a result, we expect it could produce denser and smoother depth maps compared to our current solution.

Nvidia’s real-time 3D reconstruction is also interesting because it is designed to run on GPU instead of CPU, as we originally envisioned. This makes it fast, while optimizing valuable CPU resources for enhanced operation of AMRs.

More efficient processing will lead to faster, real-time decision making, resulting in a safer and more productive AMR – which has always been our goal to produce.