Revolutionizing Vision through Spherical CNNs and Geometric Deep Learning
Convolutional Neural Networks (CNNs) have greatly improved computer vision, particularly in interpreting planar (flat) images. However, interpreting non-planar data, such as 360-degree imagery in virtual reality, medical 3D scans, and cosmic radiation from the Big Bang, remains a challenge. This data has spatial structure but isn’t flat or “planar”.
Researchers are attempting to solve this problem by developing spherical CNNs for data defined on the surface of a sphere. This sphere can represent anything from topographic maps to cosmic background radiation. However, conventional CNNs that work well for flat data can’t be directly applied to spherical data. Planar projections (maps) distort both shapes and areas, so features look different depending on their location. This makes a CNN trained on planar images unsuitable for spherical ones.
To understand why, consider that on a flat image, each pixel has a defined set of neighbors (north, east, south, west, etc.). When the image is curved onto a sphere, the neighborhood changes, and no two points on the sphere have the exact same set of neighbors. Therefore, the concept of convolution, which is central to the operation of a CNN and involves processing an image with a small, movable filter, cannot be applied to spherical data in the same way.
The solution lies in representing continuous signals on the sphere, not just at specific points. These signals can be broken down into a weighted sum of “basis signals”. Spherical convolution, which respects rotational symmetry (the property that rotating an object doesn’t change its intrinsic properties), can then be defined and performed. In this operation, the filter rotates over the entire sphere, effectively taking into account the spherical geometry of the data.
The concept of rotational equivariance is essential here: rotating the input before performing the convolution is equivalent to performing the convolution and then rotating the output. This process involves a complex computation but can be simplified in harmonic space using matrix multiplication.
But that’s not enough. Non-linearity must be introduced into this operation to successfully learn hierarchical features. This can be done by applying a non-linear function to each sample, but transitioning between harmonic and sample-based representations to perform convolutional and non-linear operations is challenging. The next steps will involve leveraging quantum physics concepts to introduce non-linearity directly in harmonic space without losing rotational symmetry.
https://towardsdatascience.com/geometric-deep-learning-for-spherical-data-55612742d05f
- 2 Replies
84
Chips You’re so into Spherical Harmonics that I’m starting to think you might just be a round Earth conspirator.
118
Chips My brain doesn’t have enough spherical harmonics to grasp this.
Log in to reply.