Journey to Autonomy: The Power of Xwing Air Data for Deep Learning

In this latest article in our Journey to Autonomy series, we dive into deep learning and look at the advantage that real-world air cargo operations provide to the development of our autonomous flight technology.

We recently posted about our upcoming Supplemental Type Certificate (STC) that will enable us to install sensor pods on our commercial cargo aircraft. This will be a key enabler in collecting real-world data from our operations and improving our autonomous technology stack.

In our previous blog post The Potential of Vision-Based Systems, we explained why computer vision can unlock the full potential of autonomous flight. We presented some example applications and examined one of them in detail: vision-based landing. We also mentioned how deep learning has been a key enabler for that task thanks to unparalleled performance. In this article, we dive deeper into the benefits and requirements of deep learning and explore why the data collected by our cargo operations at Xwing Air facilitates our journey to autonomous flight.

The components of deep learning are not new, but a succession of key research findings has revolutionized the field since 2012 and unlocked a plethora of new applications. Thanks to those breakthroughs, it is not uncommon for deep learning algorithms to now outperform human experts in specialized tasks: medical diagnoses, satellite imagery analysis, plant disease classification, board games, video game strategies, and many others. One of the most mediatic has been the successful use of deep learning for autonomous driving. Self-driving cars operate in diverse environments from intricate city streets to countryside highways and often have to deal with the unexpected, known as edge cases.

Airports are a much more regulated and constrained environment, making it a potentially easier application, with fewer unforeseen behaviors. Here at Xwing, we expect deep learning to provide a substantial advantage in our journey to autonomy, dramatically improving the safety of autonomous flight. We carefully select which components can benefit from redundancy and deep learning approaches. In the end, deep learning components represent only a few subsystems out of the more comprehensive and integrated flight stack.

As opposed to machine learning, deep learning learns data features and representations on its own

Classical computer vision algorithms use rules and features which are designed and handcrafted by subject matter experts. On the other hand, deep learning algorithms go one step further and learn those features, building their own intermediate representation of the data. Taking the example of vision-based landing, a traditional machine learning approach might gather airport experts and pilots to design helpful features in detecting a runway: the color of the runway strip, runway markings, runway lighting system, grass surrounding the runway, etc. This manual process is complex, error-prone, and might need to be repeated several times before finding a set of features with adequate detection performance. Even top experts could fall short in understanding contextual cues or faint signals hidden deep in the data, which could help an algorithm detect a runway. Conversely, in that same example of runway detection, a deep learning model would automatically learn a hierarchy of features, with multiple layers of increasing complexity. For example, the first layers of the model might focus on high-level features of images such as edges. Subsequent layers then learn to combine those low-level features to form corners, shapes, or textures. Finally, the last few “deep” layers may extract the high-level semantic meaning of the image (e.g. person, car, runway).

Deep learning learning applications require huge amounts of data to be robust

Deep learning models learn this hierarchy of features by being shown examples of inputs and their corresponding outputs, this is called the training phase. Continuing with our example of vision-based landing, the inputs are images from onboard cameras, while the outputs are the location of the runway in those images. The expected outputs are often called ground truth labels or annotations. They are usually manually annotated beforehand and help supervise the model toward correct predictions. The image below shows such an input/output pair, with the ground truth label for the runway overlaid in blue. During the training phase, the model uses the ground truth to calculate its current performance and to adjust its parameters for the next iteration. Each layer of a deep learning model can have several millions of parameters. Training consequently requires large amounts of data to tune all these knobs and reach a robust model performance. Having a large amount of data is also required to truly cover the operational space of the larger system, and make it safe. For example, it is pretty common for modern deep-learning datasets to contain hundreds of thousands of images or more.

To see deep learning in action for vision-based landing, check out our post on autoland.

An example training image with the main runway label in blue.

While the model requires a lot of data to perform correctly, the data also has to be diverse. The training process conditions the model performance to extrapolate reasonably well to cases never seen before, but which remain similar to the training data. However, we cannot expect the model to generalize to entirely different cases. For example, if the training data only contains images of paved runways, the model will most likely fail to detect grass landing strips as it never learned to extract their specific features in training. Having diversity in the data is useful for several reasons. It first makes the model robust to work in different operating conditions. It also prevents the model from learning hidden or biased representations of the data. For example, if all your training data is at the same airport, on the same runway, your deep learning model might learn that the runway is always to the right of the control tower. This model would obviously fail to generalize to airports with a different layout, or without a control tower. Having diverse training data prevents the model from using such latent representation shortcuts.

Example of runway diversity in cloud coverage and lighting conditions.

For our runway detection example, the variables below might help capture most of the data distribution and its diversity:

• Weather and lighting conditions (clear, cloudy, foggy, rainy)

• Seasonality (spring, summer, fall, winter)

• Time of day (day or night, twilight)

• Sun azimuth

• Approach angles and deviations from the glideslope

• Airport (size, location)

• Towered vs. non-towered airport

• Runway type (paved, grass, dirt)

• Runway markings (present, washed off, absent)

• Runway lighting system type

The Xwing Air Advantage

In order to train the deep learning models mentioned above, we now understand that we need data, a lot of it. This is where our cargo operation Xwing Air comes into the picture.

Xwing Air conducts piloted commercial cargo flights, shuttling thousands of packages across select regional U.S. routes, daily. As part of UPS’ regional air cargo network, we fly 35 Caravans about 400 flights a week to 60 locations across the U.S. to gather hands-on operations experience. Those routes are flown in 14 different states by our in-house expert pilots who provide critical feedback on our use case. We also operate aircraft repair stations in Addison, Texas, and Lansing, Michigan. Digging into the numbers, our aircraft have flown more than 13,000 missions and counting, which represent 1.5 million miles. The data and the lessons learned from those flights help us test and refine our technology’s capabilities on a continual basis.

Flight data collected from thousands of commercial cargo flights per month, provide our systems with the ability to learn, adapt, and improve continuously based on real-world scenarios.

In order to further leverage our cargo operations, we have designed a data collection pod that will be seamlessly integrated into our existing Xwing Air fleet. This sensor pod is equipped with the different sensors (cameras, lidars, radars) used by our autonomy stack. The pod does not run Superpilot and is not linked to the controls of the aircraft. Its sole purpose is to be turned on or off to separately record data during flights, without interfering with the regular aircraft operations and pilot’s tasks.

Example Cessna 208B equipped with data collection pods under the wings

Looking at other examples in the autonomous vehicles industry, self-driving cars have been using similar data collection approaches. Waymo’s fleet has accumulated more than 20 million miles on real roads since 2009 and has even made part of that data available for research. For its part, Tesla deploys Autopilot and data collection software directly onto customer vehicles. Those vehicles gather 100,000 miles worth of data every day. This data can be used to refine the deep learning models based on specific situations their drivers experience. In the same fashion, running data collection as part of our cargo operations helps us in different ways. It first provides invaluable diversity to our training data, capturing both planned and unforeseen scenarios. This improves our coverage of the data distribution, especially the tail or edge cases which can be hard to plan for, but arise as part of daily operations when you run a cargo airline. Second, it provides us with training data that is identical to our operational use case: we can train our models on the exact same type of data they are expected to be deployed on.

Training on such diverse data will enable us to democratize Superpilot’s autoland to small runways, as opposed to working only on large and well-equipped airports. This data is usable not only for autoland but also for other perception algorithms of our autonomy stack. Beyond training deep learning models, the data can be used to develop more traditional algorithms or validate requirements. In addition to setting up data collection with our regular cargo operations, we can also perform targeted data collection with our experimental aircraft. This type of data collection is more flexible and can be used to prototype algorithms that fall outside of our regular operations. In the journey to autonomy, it is not rare to see the companies with the best data get ahead. In that regard, Xwing Air is providing us with a clear data advantage to develop the best technology to connect communities with safe autonomous flight.

.