People create large amounts of digital video data recently. Such data comes from many sources e.g., surveillance videos, personal videos, commercial videos, and etc. Many of videos are human-centered. Automatic analysis of videos, e.g., for indexing and searching, is thus an interesting and critical problem. It is also very challenging due to its unconstrained nature and sheer scale. Human action recognition is one of the tasks, in this genre, which has gained substantial attention in recent years [6,36,39,44]. Most of such works have addressed third-person videos while there are some works on egocentric videos as well [11,40,54].
Drones are becoming more popular and readily available for purchase in the consumer market. Similar to the existing human-borne camera videos, it is desirable to automatically analyze drone-captured videos. However, drone-captured videos present distinct challenges due to continuous and typical motions, perspectives, and distortions. Thus they are very different from human-borne camera videos (Fig. 1a).
In this paper, we focus on an unsupervised video domain adaptation setting. We aim to leverage the existing largescale annotated datasets of third-person videos 1 , to help perform action recognition on challenging drone-captured videos. Since acquiring and annotating videos in any new domain is an expensive and time-consuming task, under such domain adaptation settings, we aim to minimize the annotation efforts.
The large domain differences between the source domain of third-person videos and the target domain of dronecaptured videos (Fig. 1a), motivate us to also investigate the case of semi-supervised domain adaptation [15]. In the semi-supervised domain adaptation setting, we assume that a limited amount of annotated target data is available during training in contrast to the unsupervised domain adaptation setting.
In addition to the case where both source and target have the same label sets, we also address the challenging setting where the label sets are different (Fig. 1b). To reduce the domain gap between source and target data, we employ a domain classifier and adversarial loss in the both problem settings, i.e., same and different label sets. We use standard cross-entropy loss in the same label set setting, while we use an embedding-based framework in the different label sets case. The input in the latter case is agnostic of the specific class annotations of the training examples. We care only about dis-/similarities between examples, i.e., if they belong to different/same classes irrespective of the particular classes. By employing an embedding-based method, our classifier can generalize to new categories in the target domain.
We also propose to do both full video-based as well as instance-based adaptation. The full video-based method has the merit that exploits correlated context while the instancebased approach is motivated by the argument that focusing on the actor itself is more critical for better performance.
To evaluate the presented methods, we also propose a novel dataset of human actions captured by drones: NEC-DRONE. The NEC-DRONE dataset consists of 5250 videos. We evaluate the proposed method on this challenging dataset and show that we can successfully perform domain adaptation from mostly third-person videos to dronecaptured videos. We further evaluate the proposed method on a publicly available Charades-Ego dataset [37]. We show qualitative results on the NEC-DRONE dataset to better understand the behaviors of the methods.
To summarize, we make the following three contributions of this work.
• We introduce a new problem of unsupervised and semi-supervised domain adaptation for action recognition from drones with two settings, i.e., same and different source and target label sets.
Drone-based video datasets. A few drone-based video datasets have been proposed [3,26,30,56]. However, there is only one dataset for drone-based human action recognition that we are aware of -the OKUTAMA-ACTION dataset [3]. The OKUTAMA-ACTION dataset is an outdoor dataset, and it is 43 minutes total while ours (NEC-DRONE) is 256 minutes. The number of actors is 9 vs. 19 actors (ours), and actions are 12 vs. 16 actions (ours). To the best of our knowledge, the proposed dataset is the largest dronecaptured dataset for human action recognition.
Action recognition. After the success of deep networks in the image domain, many works have addressed action recognition in videos [2,4,6,9,12,14,18,19,20,36,38,39,43,44,49,50,51,52]. This is in contrast to the earlier handcrafted features [47,48].
Most of these methods use third-person videos to train their models. In this work, we show that such third-person models do not accurately transfer to novel domains. We propose methods to make models to generalize better using domain adaptation, utilizing a large amount of annotated third-person data.
Cross-view modeling. Understanding object, scene, and action across different views has drawn attention in computer vision. There have been works on aerial and ground view matching [22,29], albeit the tasks are not human action recognition. For human actions, recent approaches use multi-stream networks to model first and third person videos jointly [1,10,35]. However, most of them require a dataset of paired videos across views.
We also want to learn view-invariant representations. However, collecting paired videos across different views such as a drone view, a third-person view, and a first-person view is expensive. Thus, we aim to leverage the existing labeled third-person videos while using only unlabeled target videos (from drones), for learning representations. Domain adaptation. Many works have addressed the problem of domain adaptation for the case of image classification [13,15,24,25,31,34,45,46,55] and object detection [8,28,53]. However, not much work has been done on domain adaptation for video-related tasks. A few approaches deal with an image to video domain adaptation [24,42]. Our work is different as we are interested in a video to video domain adaptation with the target videos being captured by drones.
There are a few works on video domain adaptation [17,7]. Similar to them, we also use the basic adversarial learning framework. However, we are also dealing with more challenging problem setting where we have different source and target label sets. We are also different in that we propose to use instance-based domain adaptation as our NEC-DRONE dataset has more significant domain gap.
Open set domain adaptation. Open set domain adaptation is the setting where both source and target datasets have 'unknown' classes, and unseen class examples are all classified together into one 'unknown' category [27,5,32]. However, we are interested in classifying the unknown examples in different novel classes in the target domain (e.g., 'exchanging backpack').
Our aim is to do domain adaptation from a source domain where we have class annotated training data (x s , y s ) ∈ X s × Y s , where Y s is the source label set, and unannotated data or a very limited amount of annotated data from the target domain (x t , y t ) ∈ X t × Y t with Y t being the target label set. We address two cases of domain adaptation: (i) when the source and target label sets are the same i.e., Y s = Y t , and (ii) when they are different i.e., Y s = Y t . We partition the target annotated data into three parts, the usual train and test sets and a third support set (X t N , Y t N ). We use the support set only in the case of unsupervised domain adaptation with different source and target label sets, to do k-NN classification in the target domain. We report the target performances on the target test set, which we again stress, has no overlap with the support set.
Our overall architecture (Figure 2) leverages the advances made in both video representations as well as domain adaptation. The system takes a video with T frames, denoted as V = {v 1 , v 2 . . . v T } where v i ∈ R h×w×c are the height h, width w, and c channel frames, as an input and splits it into small, potentially overlapping,
where L is the clip length. Then we pass the clips through a state-of-the-art video CNN, denoted as ψ(•) to obtain feature representations, ψ(x) of the clips. We pass the clip features to a softmax with classification loss or an embedding-based metric learning loss, as well as to a discriminator network with domain adversarial loss. We describe the different cases in the following.
The first case is when the K classes are the same in the source and target domains i.e., Y s = Y t (Figure 2a). Even in this case, the domain differences are substantial due to the various challenges such as variations in appearance, perspective, motion, etc. In this case, the system learns representations with a combination of cross-entropy loss for classification in the source domain along with the domain adversarial loss, i.e., binary cross-entropy loss, between examples of source and target domains. Formally, denoting the classifier by f C (•) with parameters θ c , and the discriminator by f D (•) with parameters θ d , we define the losses as,
The optimization problem is then given by,
where, θ f are the feature extractor parameters of ψ, and λ is a hyper-parameter for the trade-off between the crossentropy and the domain adversarial losses. We mark optimal parameters θ with a symbol * in a superscript.
The optimization learns a classifier by minimizing the classification loss, a discriminator by minimizing the adversarial loss and a feature extractor by minimizing the classification loss and maximizing adversarial loss, to learn domain invariant and discriminative representations. We use the gradient reversal layer [13] for adversarial training.
Semi-supervised adaptation. We also evaluate semisupervised domain adaptation, where, in addition to the unlabeled target examples, some annotated target examples are available for training as well. We use the target annotated examples with cross-entropy loss, and the target unannotated examples with domain adversarial loss only.
In the second case, the domain differences are due to the difference in labels sets i.e., Y s = Y t (Figure 2b) as well as the variations such as appearance, perspective, motion, etc. The source and target label sets could be different with some or potentially no overlap. In this case, we propose to learn embeddings of the videos which are agnostic of the specific classes but are aware of being similar (when examples come from the same class) or dissimilar (when they come from different classes). To do this we use a standard metric learning loss, i.e., the triplet loss [33], which takes a triplet of examples (x a , x p , x n ) with x a being the anchor and x p , x n being the positive (same class as anchor) and negative (different class than the anchor) examples respectively. In the embedding space, the triplet loss forces the smaller distance between the anchor and the positive example by a margin of δ , than the distance between the anchor and the negative example. Formally the loss and optimization problem are given as,
In a minibatch, we sample examples from both the source as well as the target domain. All samples contribute to minimizing the adversarial loss, while the samples from the source domain construct the triplet examples and contribute to minimizing the triplet loss.
Once we have trained the network, the system classifies query examples, by first, obtaining the embeddings from a forward pass of the base CNN, and then performing k-NNbased classification in the embedding space, using the target support set (X t N , Y t N ). Semi-supervised adaptation. We also evaluate semisupervised setting in a different source, and target label sets setting, similar to the same label sets setting. The two differences are, (i) we use the cross-entropy loss for target classes as well, and (ii) we do not use the support set, as now the system can directly do target class classification.
Since we are interested in human actions, the discriminative visual regions in the frames are expected to be around humans. We could expect that focusing on the humans in frames would give better performance by eliminating noise from the background. On the other hand, the background might contain correlated elements which could potentially contribute to better recognition. Since both the human fore-ground as well as the background have potential merits, we propose to do both 'video-based' and 'instance-based' adaptation. In the video-based adaptation we give the full clip as the input to the system, while for the instance-based case, we first perform human detection using a state-of-theart pre-trained human detector [16] and then feed only the human spatio-temporal tube (i.e., a clip made by cropping out human from every frame) as an input the the system.
We independently train video-based and instance-based domain adaptation models. During testing, we perform late-fusion of the two predictions from video-based and instance-based models. We empirically show that both have advantages, especially when some amount of target annotated data is available (semi-supervised setting), and their combination consistently improves over either of them alone.
We propose a new dataset, NEC-DRONE, of videos taken from drones for the task of domain adaptation from third-person videos to drone videos. Figure 3 shows some The actions are both single as well as two-person actions. The partial motivation of defining the actions was to keep surveillance scenarios in mind, e.g., two people getting together and exchanging a backpack could be an interesting event to tag, or retrieve. There are 10 single person actions, i.e., walk, run, jump, pick up a backpack and go, leave a backpack and go, sit on a chair, talk on a mobile phone, drink water from a bottle, throw something, pick up a small object, and 6 two-person actions, i.e., shake hands, push a person, hug, exchange a backpack, walk toward each other and stay, stand together leave.
We recorded each action instance by two drones simultaneously flown in an unconstrained manner by relatively new pilots. The videos in the dataset are from varied perspectives with the drones flying at varying distances and heights from the actors. The drones used were 'DJI Phantom 4.0 pro v2' and the videos were recorded at 30 fps at a resolution of 1920 × 1080 pixels. We manually annotated the actions of all videos.
Finally we have a total of 5250 videos with a total of more than 460k frames. We split the videos into 1188 train, 437 val, and 454 test sets with labels, and 3171 videos without labels. We make sure that the actors in the train, val and test sets are disjoint. We evaluate the performance on the dataset as the mean class accuracy.
The proposed dataset is challenging for the following main reasons. First, view point is different from typical action datasets, and it changes heavily over time due to the flying drone. Second, due to the continuous and often erratic drone motion, the videos have jitters and motion blur. Third, often the person(s) of interest are not centered and are relatively small. To the best of our knowledge, the NEC-DRONE is the largest drone dataset for human action recognition. We plan to release it publicly upon acceptance of the paper.
In Table 1, we show a significantly larger domain gap between Kinetics and the NEC-DRONE dataset, compared to the domain gap between Kinetics and UCF-101. We perform a nearest neighbor classification on the UCF-101 [41] and NEC-DRONE datasets. Note that we do not learn any parameters. We use ℓ 2 normalized mixed5c activations of the I3D network [6] pre-trained on Kinetics as our feature.
Video-based nearest neighbor classifier can achieve 72.3% accuracy on UCF-101 dataset (split 1). However, the video-based nearest neighbor can achieve only 8.2% accuracy on the NEC-DRONE dataset. Using instance-based nearest neighbor, we can achieve 10.8% accuracy. This significant difference is due to the large domain gap between the NEC-DRONE dataset and typical third-person video datasets. Furthermore, the existing third-person video datasets such as UCF-101 and Kinetics have correlated backgrounds for different actions. However, the NEC-DRONE dataset has a similar background for all the actions. Thus the dataset is very challenging, as without capturing the human motion, it is difficult to recognize the different human actions in the NEC-DRONE dataset.
Abbreviations. We use the following abbreviations. DA: domain adaptation, UDA: unsupervised domain adaptation, SSDA: semi-supervised domain adaptation, src.: source, tgt.: target, vid.: video-based, inst.: instance-based, sup.: supervised finetuning.
Same label set for source and target. We use the Kinetics [21] dataset as the source dataset and the NEC-DRONE dataset as the target dataset. Since the two datasets do not share the same classes, we subsample the two datasets to obtain similar classes. We choose 13 classes from Kinetics [21] dataset and 7 classes from the NEC-DRONE dataset which have similar actions to construct the source and target datasets. See supplementary for the details.
Different label sets for source and target. When we work with different label sets settings, we use the UCF-101 [41] as our source dataset. UCF-101 dataset is mainly a third-person dataset and contains 13, 320 videos from 101 action classes. The domain gap between the UCF-101 and NEC-DRONE datasets is significant (as we also show quantitatively below) and the label sets of the UCF-101 dataset and NEC-DRONE datasets are entirely disjoint. Hence this makes a challenging and practical domain adaptation setting from third-person videos to drone-captured videos.
For both settings, we use labels for m target examples per class, in addition to the unannotated target and annotated source examples, for semi-supervised adaptation.
Implementation details. We use state-of-the-art I3D network [6] as our base network for feature extraction with L = 16 frame clip inputs for drone experiments and L = 32 frame clip inputs for Charades-Ego experiments. We attach the domain discriminator to mixed5c layer of the I3D network. We use a 4 layer MLP for domain classifier where the hidden fully connected layers have 4096 units each.
When aligning features at the instance-based, we extract the human tubes by running per frame detectors and making tracks based on the overlaps of the detections in the successive frames. We use a Mask R-CNN [16] pre-trained on MS-COCO dataset [23] for person detection.
We set λ = 1.0 for the gradient reversal layer [13], δ = 0.5 for the margin parameter of the triplet loss and the embeddings. We use a batch size of 10 and sample minibatches as follows. In the case of triplet loss only, 7 out of 10 examples are from anchor class, and the rest 3 are from different classes. In the case of triplet loss with unsupervised domain adaptation, 5 (3 same class, 2 different classes) out of 10 examples are source examples and rest 5 are target examples. We use SGD optimizer with the momentum of 0.9. For the source pre-training and semisupervised finetuning, we use an initial learning rate of 10 -4 , and for the unsupervised domain adaptation training, we use an initial learning rate of 10 -6 . We reduce the learning rate by 1/10 after 5 epochs.
Same source and target label sets. We first perform an ablation study of video-based DA and instance-based DA and their combination with different number m of target annotated examples per class used during training. We also include the results without any domain adaptation. Since it is an ablation study, we perform experiments on the val set. The column where m = 0 is the unsupervised domain adaptation setting while the columns where m > 0 are semisupervised domain adaptation settings.
The results show the contribution of different adaptation components. The video-based adaptation achieves 13.6% in the unsupervised case, the instance-based achieves 16.5%, while the combination of the two gives 15.1%. The performances rise rapidly as even a small number of annotated examples from the target domain are provided during training. With only m = 3 examples the performance of the combined method increases to 32.0% which further increases to 41.8%, 54.9%, 58.3% on m = 5, 10, 20. The m = 20 performance of 58.3% is still far from the full target supervised performance of 76.7%; in the latter case, the average number of examples per class is 80. We also note that the combination of the video-based adaptation with the instancebased adaptation is always greater than either of them indicating complementary information in the two methods.
Table 3, third column, gives the final test performances of the same label set setting for the different methods on the NEC-DRONE dataset for m = 5. We see that the video-based adaptation improves the source only classifier from 13.6% to 27.2%, while the instance-based adaptation achieves 29.4%. The combination of both gets the best per- formance of 32.0%. This is still quite far from the target fully supervised value of 69.3% indicating that still, a large domain gap exists even after semi-supervised adaptation.
Different source and target label sets. Table 4 shows The trends among the methods are similar to the previous case of the same source and target label sets. The source only classifier performs very poorly at 8.2%, cf. 6.25% for a random chance for this 16 class case. The contrast is much higher in this case compared to the previous as (i) it is a harder setting where completely new classes are predicted, and (ii) in general embedding-based methods perform lower than cross entropy-based 1-of-C class classifiers. Compared to the source only classifier, the video-based method improves performance by 29.2% relatively, while the instancebased method improves by 74.3% relatively. The combination of the two further improves 76.8% relatively.
Table 5 gives the semi-supervised domain adaptation results for the setting when the source and target label sets 2. With a small number of annotated examples used during training, we can improve the performance compared to the unsupervised domain adaptation (14.5% for m = 0 vs. 39.7% for m = 20).
We compare the performance with other methods on a publicly available Charades-Ego dataset [37] in Table 6. Please note that Charades-Ego is a paired dataset. Therefore every first person and third person video is paired with its counterpart. Our method does not require paired dataset, thus more general than Actor and Observer [35]. Also note that the reported performance in [35] is invalid because the authors evaluated on the wrong splitfoot_1 . Thus, we run the authors' code and report mAP on the valid test set, which is 20.0%. We obtain 17.9% mAP with our video-based unsupervised domain adaptation. Using annotated target data improves performance. With only 10% of the labeled tar- get data, our semi-supervised domain adaptation achieves 20.4% mAP, outperforming the Actor and Observer. Both Actor and Observer and our semi-supervised domain adaptation method use some of supervision: Actor and Observer uses paired videos for supervision, while our method uses x% of the target label for supervision. Our semi-supervised domain adaptation method is more general than requiring paired third-person and first-person videos. Therefore, our method is more suitable for action recognition from novel domains where getting paired video dataset is difficult e.g., drone-captured videos.
We qualitatively demonstrate the contribution of domain adaptation in Figure 4, showing (i) misclassified examples when not using domain adaptation but are correctly classified with domain adaptation (top row), (ii) examples where video only adaptation method fails but combined instance and video-based adaptation method succeeds. We can observe the large domain gaps in terms of the perspective and the area occupied by the actor in the example frames. While the typical third-person videos have a direct perspective and almost centered actor as the major content of the video, the videos in the proposed dataset have challenging perspectives and can also be taken from far. The proposed method addresses these domain gaps and improves performance.
With our embedding-based method, we also obtain a common space where we can compare videos from the source and target domains. To demonstrate it, we show the cross-domain action category level retrieval results in Figure 5. Given a query from 'marching' action class of the Kinetics (first column of the top-left block), we show the top nearest neighbors from the NEC-DRONE dataset. In each block, the first and second row show the retrieval results without and with domain adaptation respectively. We can observe that the domain adpated model can successfully retrieve 'walking' class videos from the NEC-DRONE dataset despite a huge domain gap. Without domain adaptation, the retrievals contain more irrelevant videos from other classes.
We addressed the task of human action recognition from drones in the setting where we do not have any labeled examples of drone dataset, or we have only a few labeled examples. We further explored a more challenging setting where the source and the target label sets are different. To deal with this challenging setting, we proposed to use metric learning loss and unsupervised domain adaptation along with instance-level action recognition.
Since a challenging large dataset of drone videos for human action recognition did not exist, we collected 5250 high-resolution videos from two drones with 16 predefined single person and two-person actions. We empirically showed that a large domain gap exists between third-person video datasets and the NEC-DRONE dataset. We will release the dataset upon acceptance to the community.
Our work is among the first to show encouraging domain adaptation results on challenging video domains. However, we also show that we are still far from the fully supervised classifier performances in the target domain of drone videos, and hence, there is much room for improvement.
We refer to existing action recognition datasets such as Kinetics and UCF-101 as third-person datasets while noting that they may contain some other perspective videos, e.g., first person, as well.
https://github.com/gsig/actor-observer/issues/7