Capsule Deep Dive – Spatial Tracking

Monitoring sea ice, wildfires, troop movements or spying: the many uses of satellite video tracking.

Most action thrillers have at least one scene where the geek on the team hacks into some secretive society hell-bound on destroying the earth, downloads a satellite video, and – with a few key strokes – cleans it up to reveal where the evil mastermind fled to and is hiding.

As the viewer you wonder “can they actually do that?” Not the hacking, we know that’s possible, it’s “cleaning up the video with a few key strokes” that seems unlikely. For one, the objects you are looking for on a video are generally very small – only a few pixels – which makes it difficult to make out any discriminative features and secondly, the object of interest might not be in plain view but at least partially obscured. 

Most state-of-the-art algorithms look at single frames rather than considering the additional information provided in subsequent frames due to the motion of the object. Siamese tracker approaches, however, address this problem by extracting an appearance template from the current frame, which is used to localize the target in the next frame.

Tracking moving objects through multiple frames from satellites

One such Siamese tracker approach can be found in the Code Ocean Compute Capsule library.  Compute Capsules are computational units that contain everything needed (code, environment, data and results) to run a model. In their Capsule Shao and colleagues provide a runnable model for a “lightweight parallel network with a high spatial resolution to locate the small objects in satellite videos.” This network models the video sequence in time to detect the moving targets in pixels and track them. 

Other than foiling the evil plans of secretive societies these networks can be used for any number of applications. The authors mention object detection in remote, difficult to access areas where cameras are hard to install and maintain, e.g. the Arctic for sea ice monitoring.

This technology could also be used to detect wildfires and how they are moving to give firefighters the most current information on where to deploy their resources. They also mention tracking fishing activities globally to stop overfishing and using it to detect human trafficking across borders. The Russian invasion of Ukraine is an example of how satellite video could be used for military purposes, e.g. for better monitoring enemy troop movements or troop build-up. 

Static satellite images can reveal a lot of information, but videos add additional elements such as improved object recognition and classification based on data from multiple angles, they can mitigate the effects of patchy clouds partially obscuring the object and identify direction of movement.

Of course, satellites equipped with such advanced artificial intelligence algorithms can also be used for more nefarious purposes, such as spying.

Your turn: Play with the code and see if you can get it to track your pet in your personal videos 

The capsule at a glance

Title: HRSiam: High-Resolution Siamese Network, Towards Space-borne Satellite Video Tracking

Link: https://codeocean.com/capsule/4794578/tree/v1

Authors: Jia Shao

Publication: J. Shao, B. Du, C. Wu, M. Gong and T. Liu, “HRSiam: High-Resolution Siamese Network, Towards Space-Borne Satellite Video Tracking,” in IEEE Transactions on Image Processing, vol. 30, pp. 3056-3068, 2021, doi: 10.1109/TIP.2020.3045634.