Long-term Behaviour Recognition in Videos with Actor-focused Region Attention

Long-Term activities involve humans performing complex, minutes-long actions. Differently than in traditional action recognition, complex activities are normally composed of a set of sub-actions, that can appear in different order, duration, and quantity. These aspects introduce a large intra-class variability, that can be hard to model. Our approach aims to adaptively capture and learn the importance of spatial and temporal video regions for minutes-long activity classification. Inspired by previous work on Region Attention, our architecture embeds the spatio-temporal features from multiple v... Mehr ...

Verfasser: Ballan, Luca
Strafforello, O.
Schutte, Klamer
Farinella, Giovanni Maria
Radeva, Petia
Braz, Jose
Bouatouch, Kadi
Dokumenttyp: conferencePaper
Erscheinungsdatum: 2021
Schlagwörter: Netherlands / Knowmad Institut
Sprache: unknown
Permalink: https://search.fid-benelux.de/Record/base-29597947
Datenquelle: BASE; Originalkatalog
Powered By: BASE
Link(s) : https://www.openaccessrepository.it/record/131971

Long-Term activities involve humans performing complex, minutes-long actions. Differently than in traditional action recognition, complex activities are normally composed of a set of sub-actions, that can appear in different order, duration, and quantity. These aspects introduce a large intra-class variability, that can be hard to model. Our approach aims to adaptively capture and learn the importance of spatial and temporal video regions for minutes-long activity classification. Inspired by previous work on Region Attention, our architecture embeds the spatio-temporal features from multiple video regions into a compact fixed-length representation. These features are extracted with a 3D convolutional backbone specially fine-tuned. Additionally, driven by the prior assumption that the most discriminative locations in the videos are centered around the human that is carrying out the activity, we introduce an Actor Focus mechanism to enhance the feature extraction both in training and inference phase. Our experiments show that the Multi-Regional fine-tuned 3D-CNN, topped with Actor Focus and Region Attention, largely improves the performance of baseline 3D architectures, achieving state-of-the-art results on Breakfast, a well known long-term activity recognition benchmark.