Social sensing has emerged as a powerful sensing paradigm for collecting observations of the physical world through social media [1,2]. Examples of social sensing applications include city-wide traffic surveillance using Twitter feeds [3], urban anomaly detection using Foursquare check-ins [4], and community disease outbreak monitoring using Facebook posts [5]. Unlike other infrastructure-based sensing paradigms (e.g., CCTV cameras, remote sensing, wireless sensor networks), social sensing provides a pervasive and scalable solution for obtaining real-time damage information during disaster events [6]. In this paper, we focus on an emerging application in social sensing: streaming disaster damage assessment (streaming DDA) [7]. The goal of streaming DDA applications is to automatically assess the damage severity of affected areas in a disaster event on the fly by leveraging the streaming imagery data posted on social media.
The outputs of streaming DDA applications can be shared with emergency response agencies (e.g., Federal Emergency Management Agency (FEMA), fire departments) for timely rescue and recovery operations.
Recent advancements in artificial intelligence (AI) have helped in improving the performance of DDA applications [8,7,9,10]. In particular, compared with the traditional DDA solutions that largely rely on intensive manual labeling efforts from disaster specialists [11], the AI-driven DDA solutions significantly reduce the labeling costs while providing a reasonable assessment accuracy [12].
However, current AI-driven DDA solutions often require inputs from experts who are specialists in both AI models and DDA applications to design an appropriate neural network architecture for a particular DDA application. This manual neural network architecture design process is known to be both timeconsuming and suboptimal [13]. Figure 1 shows an example where the optimal neural network architecture in a streaming DDA application changes over time. In particular, we observe that the optimal neural network architectures for disaster-related images collected in consecutive timesteps in the same disaster event are different. In such scenarios, it is difficult for AI experts to predict and design an individual optimal neural network architecture for each newly arrived image on the fly. Motivated by the above observations, we study a dynamic optimal neural architecture searching (NAS) problem in streaming DDA applications where the goal is to dynamically determine the optimal neural network architecture that accurately estimates the damage severity for each newly arrived image without the inputs from AI experts.
In this study, we develop a crowd-driven dynamic neural architecture search (CD-NAS) system to address the above problem by exploring the collective intelligence of both AI and humans. The objective of our CD-NAS design is to leverage human intelligence from crowdsourcing systems to guide the discovery of the optimal neural network architecture for every image in a streaming DDA application. In particular, we observe that human perception is often more reliable and consistent than AI algorithms in terms of identifying the severity of disaster damage from the image (e.g., we can clearly determine the damage severity of images reported in Figure 1). Such human intelligence could possibly help us dynamically identify the optimal neural network architecture in Mechanical Turk) [14]. We refer to the human intelligence collected from the crowdsourcing platform as crowd intelligence in the remainder of this paper.
Two important technical challenges exist in designing such a crowd-driven NAS system, which are elaborated below.
Dynamic optimal neural architecture searching. The first challenge lies in the dynamic identification of the optimal instance of neural network architecture for each image in the streaming DDA application without knowing its ground truth label a priori. In particular, current NAS solutions in AI are mainly designed to identify a single best-performing neural network architecture for a given set of training data [15,16] and leverage the identified neural network architecture to estimate the damage severity for all testing data. However, such a one-size-fits-all neural network architecture could inaccurately estimate the damage severity for a non-negligible portion of images because the optimal neural network architecture often changes over time (as shown in Figure 1).
Recent advancements in dynamic neural networks could potentially be applied to address this issue [17,18]. However, a major limitation of these solutions is that they still require a large amount of high-quality training labels from the studied disaster event to periodically retrain their models to capture the dynamics of streaming data. However, such a high-quality training dataset is often not available for an unfolding disaster in streaming DDA because of the "cold start" problem [19] and the lack of real-time annotations due to cost and resource constraints [20]. Additionally, recent efforts in online deep learning have been made to dynamically update the learned network instances [21,22].
However, these solutions primarily focus on periodically optimizing the performance of a particular neural network architecture pre-defined by the AI experts on relatively simple tasks (e.g., tweet text classification and handwriting number identification). As a result, their performance may be suboptimal because of the potential bias and constraints of the manual network design process given the excessive damage characteristics and fine-grained details of disaster-related social media images [23]. Therefore, the dynamic identification of the optimal neural network architecture for each incoming imagery data in streaming DDA applications remains a nontrivial question.
Imperfect crowd intelligence-driven NAS. The second challenge lies in leveraging the imperfect crowd intelligence from potentially unreliable crowd workers to facilitate the identification of the optimal neural network architecture in streaming DDA applications. Unlike AI experts who are capable of designing effective neural network architectures, crowd workers are often limited to simplified annotation tasks (e.g., labeling damage severity levels for assigned images). More importantly, unlike the damage severity annotated by disaster specialists, the labels from crowd workers are often imperfect (biased, noisy, and even conflicting responses from different crowd workers) [24]. Additionally, the noise embedded in crowd intelligence can be amplified during the neural network architecture search process, leading to the selection of the poorly performed neural network architecture [15,25]. Therefore, the key question in our design is how to effectively transfer potentially imperfect crowd knowledge (e.g., noisy crowd labels) into an accurate neural network architecture selection for streaming imagery data.
To address the above challenge, we developed CD-NAS, a crowd-driven dynamic neural architecture searching approach that carefully explores crowd intelligence to solve the optimal neural architecture search problem and optimize the performance of streaming DDA applications. To address the first challenge, we develop a streaming neural network architecture search framework that recursively updates the optimal neural network architecture for each incoming image through a novel recursive maximum likelihood estimation model. To address the second challenge, we designed a novel crowd-AI fusion model that translates imperfect crowd intelligence to effective neural network architecture selection through a robust crowd-AI collaborative network searching process.
To the best of our knowledge, CD-NAS is the first dynamic crowd-driven NAS approach for solving the streaming DDA problem. We evaluated CD-NAS using a real-world streaming DDA application from a recent disaster event, Typhoon Hagupit. The evaluation results show that our CD-NAS consistently outperforms both state-of-the-art AI and NAS baselines by achieving the highest disaster damage assessment accuracy while maintaining the lowest computational cost under various evaluation scenarios.
A preliminary version of this study was published in [26]. The journal paper is a significant extension of previous work in the following aspects. First, we identify two new intrinsic challenges (i.e., dynamic optimal neural architecture searching and imperfect crowd intelligence-driven NAS ) to solve the dynamic optimal NAS problem and explicitly discuss how our scheme addresses these two challenges (Section 1 and Section 4). Second, we extend the dynamic optimal architecture searching (DOAS) module in CD-NAS by developing a dynamic neural network architecture searching scheme that adaptively updates the estimation of the optimal neural network for each image through a recursive estimation framework (Section 4). Third, we extend the evaluation in the conference paper by explicitly studying the performance of all compared schemes with a diversified set of crowdsourcing settings (i.e., different numbers of crowd workers and crowd query frequencies). The new results demonstrate the effectiveness of our scheme in explicitly leveraging crowd intelligence to guide the discovery of the optimal neural network architecture under different streaming DDA application scenarios (Section 5). Fourth, we add a new study to evaluate the computational cost of all compared schemes (i.e., the average computation time required to estimate the damage severity of an image). This is motivated by the fact that the computational cost is critical in streaming DDA applications, especially in the context of massive social media data inputs. The new results demonstrate that our CD-NAS scheme takes orders of magnitude less time to accomplish the streaming DDA task compared with other baselines (Section 5).
Fifth, we compare CD-NAS with two additional deep learning and NAS baselines (i.e., DenseNet and MnasNet) and demonstrate the performance gains achieved by CD-NAS compared with all baselines (Section 5). Sixth, we add a new robustness study to evaluate the robustness of the CD-NAS by varying one key parameter in our design: the size of the sliding window for streaming DDA applications (Section 5). Finally, we extend the related work by adding discussions on recent progress in social sensing and NAS Both of these topics are closely related to the theme of this study (Section 2).
Motivated by the recent advances in Internet and communication techniques (e.g., 4/5G, Internet of Everything (IoE)), as well as the proliferation of online social media (e.g., Twitter and Instagram), social sensing has emerged as a new sensing paradigm to obtain timely observations of the physical world from "human sensors" [27]. Examples of social sensing applications include monitoring real-time traffic conditions in a metro area using mobile crowdsensing to enhance traffic safety [3], obtaining situational awareness in the aftermath of a disaster using online social media for rapid disaster response [28], and detection of infectious disease outbreaks in big cities using location-based crowd track-ing services to improve public health [5]. Several key challenges exist in the current social sensing applications. Examples include real-time guarantee, data reliability, incentive design, privacy protection, and noise reduction [29,30,31].
However, the crowd-driven dynamic optimal NAS problem in streaming DDA applications remains an unsolved challenge in social sensing. In this paper, we address this problem by developing a novel crowd-AI collaborative NAS framework to accurately assess the damage severity of affected areas on the fly using streaming imagery data posted on social media.
Recent advances in AI and deep learning have been proved remarkably helpful in improving the performance of DDA applications [8,7,9,10,32,33]. For example, Li et al. developed a deep domain adaptation approach to estimate the damage severity of affected areas using online social media data via adversarial transfer learning [8]. Nguyen et al. proposed a deep convolutional network framework for disaster damage assessment of unfolding disaster events for timely disaster response [7]. Kumar et al. proposed a deep image classification framework to identify disaster-affected cultural heritage sites from social media imagery data via an end-to-end deep image processing system design [9].
Mouzannar et al. developed a deep neural network approach that utilizes both text and image data from social media posts for damage identification via multimodal convolutional neural networks [10]. However, current AI-driven DDA solutions often require extensive inputs from AI experts to design an effective neural network architecture for DDA tasks. Such a manual design process is known to be both error-prone and time-consuming in the presence of massive social data inputs in streaming DDA applications [13]. Efforts on dynamic neural networks in DDA are also relevant to our work [17,18]. However, two limitations prevent them from being applied to address our problem: i) those methods often require periodical model retraining that often cannot catch up with the large dynamics in our streaming DDA application settings [34]; ii) the performance of these models often drops significantly when they are retrained using the imperfect crowd labels [35]. In contrast, our CD-NAS framework effectively identifies the optimal neural network architecture for each image without the inputs from AI experts and in the absence of ground-truth labels of newly arrived images.
Our work is also related to the growing trend of utilizing pervasive and scalable human intelligence from crowdsourcing systems to solve complex realworld problems [36,37,38,39,40]. For example, Harris et al. leveraged mobile crowdsourcing to detect the defected and deteriorated urban infrastructure for smart city management [37]. Dos Reis et al. utilized citizen scientists to segment cancer cells from breast tumors in biomedical research [38]. Wang et al.
used road traffic information reported by common citizens to monitor real-time traffic congestion in intelligent transportation [40]. However, two fundamental limitations exist in current solutions that fully rely on human intelligence from crowdsourcing systems. First, these approaches may be too labor-intensive and costly compared to our CD-NAS which only requires crowd labels from a small subset of studied images to guide the discovery of the optimal neural network architecture for desirable DDA performance [12]. Second, unlike the professional annotations from disaster specialists, labels from crowd workers can be biased, noisy, and even conflicting because of the lack of sufficient expertise on disaster assessment and response [24]. As a result, the current crowdsourcing solutions could suffer from a non-trivial DDA performance drop by using only the imperfect responses from crowd workers. In contrast, our CD-NAS jointly integrates the inputs from crowd workers and AI models into a novel crowd-AI collaborative model that effectively fuses intelligence from both the crowd and AI to address the imperfect crowd response challenge and identify the optimal neural network architecture in DDA applications.
Our work also resembles the NAS technique that is used to automate the neural network design process in many AI-driven real-world applications [15,16,25,41,42]. For example, Zoph et al. developed a scheduled drop path mechanism to enable an effective neural network architecture search for semantic image segmentation [15]. Liu et al. proposed a differentiable architecture representation mechanism to effectively refine the neural network architecture during the NAS process in natural language modeling [16]. Tan et al. designed a lightweight NAS approach to incorporate model inference latency into the factorized hierarchical searching process for image object detection via multiobjective reinforcement learning [25]. Mo et al. proposed a recursive NAS approach to concurrently search for the optimal network architecture on layer and network block levels to improve the NAS performance in keyword spotting on smart devices [41]. To the best of our knowledge, CD-NAS is the first NAS solution that effectively transfers imperfect crowd intelligence to dynamic optimal neural network architecture selections in streaming DDA applications.
In this section, we formally define our crowd-driven dynamic NAS problem in streaming DDA applications. We first define a few key terms that will be used in the problem formulation.
Definition 1. Disaster-related social media images (X): We define X to represent the disaster-related images posted by common citizens on social media (e.g., Twitter) during a disaster event (as shown in Figure 2), where each posted image captures a specific scene of the studied disaster event.
Definition 2. Social media image stream (S): We define S = {X 1 , X 2 , ..., X T } as the set of streaming social media images collected during a disaster event, where X t represents the disaster-related social media image collected from the t th timestep and T is the total number of timesteps in the studied streaming DDA application (e.g., see Figure 1). Definition 4. Categories of damage severity level (K): Following a similar procedure in [7], the damage severity level in an image can be classified into one of the K pre-defined categories: L t ∈ {1, 2, ..., K}. For example, we can consider three categories of damage severity levels (i.e., K =3) that include severe damage, mild damage, no/minor damage as shown in Figure 2. in Figure 1). In this study, we leverage the neural network architecture design space (i.e., different configurations of adopting ImageNet-pre-trained convolutional layers for image classification tasks [43]), which is commonly adopted in the current AI-driven DDA solutions [7,11]. Definition 6. Damage severity estimation from AI ( L N ): We define L N as the damage severity level estimated by different neural network architectures in N . In particular, L Ne t represents the damage severity level estimated by the neural network architecture N e for the reported image X t . Definition 7. Dynamic optimal network architecture (N * ): We define N * as the set of optimal neural network architectures identified by our CD-NAS framework from N for different images in S. In particular, N t * represents the optimal neural network architecture that produces the most accurate damage severity estimation L N t * t for the image X t collected at the t th timestep (e.g., N t * is set to be architecture 1 at timestep 1 and architecture 2 at timestep 2 in Figure 1).
The goal of our crowd-driven dynamic NAS problem is to leverage human intelligence from the crowdsourcing systems to improve the performance of streaming DDA applications. In particular, our goal was to dynamically select the optimal neural network architecture for each image. We formally define our problem as follows:
This problem is challenging because of the difficulty of transferring the imperfect crowd intelligence to dynamically identify the optimal neural network architecture for streaming social media image data in the absence of ground-truth labels.
In this paper, we develop a CD-NAS system to address this problem, which is elaborated in the next section.
In this section, we present the CD-NAS framework to address the dynamic optimal neural architecture search problem in streaming DDA applications. We first present an overview of CD-NAS and then discuss its core modules in detail.
Finally, we summarize the CD-NAS framework using pseudocodes.
An overview of the CD-NAS is shown in Figure 3. In particular, it consists of two modules: 1) crowd-driven network architecture selection (CNAS)
and dynamic optimized architecture searching (DOAS). First, the CNAS module develops a novel crowd-AI integration model to effectively leverage imperfect crowd knowledge to facilitate the discovery of an optimal neural network architecture. Second, the DOAS module designs a dynamic neural network architecture searching scheme that adaptively updates the estimation of the optimal neural network for each image through a recursive estimation framework. In this subsection, we develop a principled crowd-AI integration model to explicitly leverage imperfect crowd intelligence to facilitate the discovery of the optimal neural network architecture in streaming DDA applications. In particular, we first define a key concept that is used in our CNAS module: Definition 8. AI-crowd fusion window (AF W ): The AF W is defined as a sliding window for the streaming DDA applications that includes the most recent I images from the social media stream S. In particular, we define AF W = {X 1 , X 2 , ..., X I }, where X i represents the i th image in the sliding window and I is the size of the AF W . We note that I is an application-specific parameter and will study its effect in Section 5.
Similar to the online video applications (e.g., YouTube) that often use a local data buffer to ensure a smooth streaming video service, the AF W here is designed to buffer a set of images in streaming DDA applications for the dynamic neural network architecture search. In particular, we add newly arrived images to the AF W until it is full. Then, we apply the first-in-first-out (FIFO) strategy to replace the oldest image in AF W with the newly arrived image. The optimal neural network architecture for each image in the AF W was identified when the image was evicted from the AF W . Such a design is performed to ensure that our CD-NAS can recursively improve the estimation of the optimal neural network for each image in the AF W .
In our CD-NAS system, we explicitly leverage human intelligence from a crowdsourcing system to guide the discovery of the optimal neural network architecture for each image in AF W . Hence, we further define a few concepts related to crowd intelligence as follows:
Definition 9. Crowd query (Q): We define a crowd query as a crowdsourcing task in which our system sends a subset of images in the AF W for the crowd workers to label their damage severity levels. The returned crowd labels are used to search the optimal neural network architecture for each image, which is discussed later in this section when we formally introduce our AI-crowd collaboration model design.
where Given the above definitions, the goal of our CNAS module is to select the neural network architecture in M with the highest assessment reliability as the optimal neural network architecture in our crowd-driven NAS problem. To that end, we further define P + u,k and P - u,k as the unknown probability that the member M u estimates the damage severity level of an image to be the k th level and the value other than the k th level given the ground-truth damage severity level of the image is the k th level, respectively. We formally define P + u,k and P - u,k as follows:
where L Mu t represents the estimated damage severity level by a member M u in M on an image X t in AF W . L i is the ground-truth damage severity level for X i . Given the above definition, P + u,k and P - u,k are related to the assessment reliability δ Mu using Bayesian theorem as follows:
where G Mu,k and G Mu,k represent the probability that a member M u estimates the k th damage severity level and values other than the k th level, respectively.
d k represents the prior probability that a randomly selected image belongs to the k th damage severity level. We note that we can learn the assessment reliability score δ Mu if we can obtain the values for the other parameters in the above equation. To that end, we formulate a crowd-AI maximum likelihood estimation (MLE) problem to estimate the unknown assessment reliability score δ Mu for each member M u in the AI-crowd collaboration committee and unknown damage severity level L as follows:
where L Mu indicates the damage severity estimated by a neural network architecture L Ne or labeled by a crowd worker L C b in M .
Given the crowd-AI MLE problem above, we further define the likelihood function L(θ; ω, Z) of our MLE problem as follows:
The above likelihood function represents the likelihood of the observed data ω (i.e., damage severity levels of images in the current AF W estimated by different neural network architectures and crowd workers) and the values of hidden variables Z (i.e., the actual damage severity level of an image) given the estimated parameter θ. Detailed explanations of the parameters in the L(θ; ω, Z) are summarized in Table 1.
In the previous subsection, we presented our crowd-AI MLE formulation to learn the assessment reliability for each neural network architecture in our AIcrowd collaboration committee. The next question involves adaptively solving the formulated crowd-AI MLE problem to learn the assessment reliability on the fly so that we can dynamically identify the optimal neural network architecture for each image. To that end, we propose a recursive expectation maximization (EM) solution to solve the crowd-AI MLE problem. In estimation theory [44],
the estimation parameter of an MLE problem can be recursively updated in indicator variable that is set to be 1 when a member Mu estimates the damage severity of a given image X i to be the k th level and is set to be 0 otherwise.
indicator variable that is set to be 1 when a member Mu estimates the damage severity of a given image X i to be the value other than k th level and is set to be 0 otherwise.
where θ t and θ t+1 indicate the estimation parameters θ at two consecutive timestep t and t + 1, respectively. X t+1 indicates the nearly arrived image at timestep t + 1. The estimation parameter θ t+1 is used to calculate the updated assessment reliability for each neural network architecture in the AI-crowd collaboration committee using Equation ( 4). I c (θ t ) -1 indicates the inverse of the Fisher information of the estimation parameter θ t at timestep t. Φ(X t+1 , θ t )
represents the score vector of the observed data (input image X t+1 ) at timestep t + 1 given the estimation parameter θ t from the last timestep t. The key idea of the above streaming formulation is to provide a dynamic solution to recursively update the estimation parameter θ on the fly.
To obtain the Fisher information I c (θ t ) and score vector Φ(X t+1 , θ t ), we first derive the log function of L(θ; ω, Z) by assuming that the correctness of the hidden variable (Z t,k ) can be correctly estimated when the number of members in the AI-crowd collaboration committee is sufficient. In particular, we can derive the log-likelihood function logL(θ; ω, Z) as:
Given the log-likelihood function logL(θ; ω, Z), we can derive the inverse of the Fisher information I c (θ t ) -1 for our problem as follows:
In addition, we can also derive the score vector Φ(M t+1 , θ t ) from logL(θ; ω, Z)
as follows:
Finally, we can plug in I c (θ t ) -1 and Φ(M t+1 , θ t ) into Equation ( 7) to obtain the recursive formula to update the estimation parameters θ (i.e., P + u,k and P - u,k ) as follows: ). In addition, we observe that Z t+1 i,k is unknown and can be estimated by its approximation Ẑt+1 i,k as follows:
where W t+1 n,k can be computed as follows:
In summary, the above recursive approach provides a dynamic solution for learning the estimation parameter θ of the crowd-AI MLE problem on the fly at each timestep using the estimation from the previous timestep and the images from the current image sliding window. Finally, we can derive the assessment reliability δ Mn for each member M u in M dynamically by plugging the updated θ t+1 to Equation (4) at each timestep. After obtaining the assessment reliability score for each neural network architecture, we select the neural network architecture with the highest assessment reliability score as the optimal neural network architecture N * for the image that is about to be evicted from the AF W as follows:
where X i t+1 represents the image that is about to be evicted from the AF W at timestep t + 1. δ Mu t+1 represents the updated assessment reliability score at timestep t + 1. In addition, the estimated damage severity L N i * from the optimal neural network architecture N i * was taken as the final output of our CD-NAS framework for image X i t+1 .
Finally, we summarize the CD-NAS framework in Algorithm 1. The inputs to the CD-NAS are the set of streaming social media images X t . The outputs are the dynamically identified optimal neural network architecture N t * and the estimated damage severity level L N t * generated by N t * for each X t .
In this section, we evaluate the performance of the CD-NAS framework using real-world streaming DDA applications from a real world disaster event. The results show that CD-NAS consistently outperforms the state-of-the-art AI and NAS baselines in terms of both damage assessment accuracy and computational cost under various application scenarios.
Algorithm 1 CD-NAS Framework Summary
Disaster Damage Assessment Dataset: In our evaluation, we used a real-world dataset on disaster damage assessment collected by [7] foot_0 . In particular, the dataset consists of social media images collected over the course of Typhoon Hagupit in Philippines (2014). The collected social media images have diversified damage characteristics (e.g., flooding damage, buildings and infrastructure damage, and vehicle damage) as shown in Figure 1. In the dataset, the ground-truth damage severity level of each social media image was manually classified by domain experts into three categories (i.e., severe damage, mild damage, and no/minor damage). In particular, the distributions of different damage severity levels in our dataset were as follows: severe damage: 11.2%; mild damage: 42.2%; and no damage: 46.6%. We keep the ratio of training to testing data as 3:1, the same as in [7]. The training dataset was used to train all the compared AI models for disaster damage assessment.
Amazon Mechanical Turk Platform: To obtain the crowd intelligence, we utilize Amazon Mechanical Turk (AMT)foot_1 , one of the largest crowdsourcing platforms that provides a large number of 24/7 freelance crowd workers to complete assigned tasks with reasonable incentives. In each crowdsourcing task, we ask the crowd workers to label the damage severity level of the image in the query. To ensure the crowd label quality, we select the crowd workers who have an overall task approval rate greater than 95% and have completed at least 1000 approved tasks to participate in our crowdsourcing tasks. We paid $0.20 for each worker per image in our experiment. In our evaluation, we study a diversified set of crowd query settings to create a challenging evaluation scenario for our CD-NAS framework. In particular, we vary the number of participating crowd workers who respond to each queried image (Definition 10) from 3 to 5 and vary the crowd query frequency β (Definition 11) from 1/5 to 1/3.
We compared CD-NAS with a set of representative deep neural network (DNN) and neural architecture searching (NAS) baselines in streaming DDA applications.
• DNN Baselines:
1. InceptationNet [45]: a popular deep learning model that accelerates the learning process of the DDA task through a convolution factorization mechanism.
2. DenseNet [46]: a widely used deep neural network approach that establishes dense connections among different network layers to boost the DDA accuracy.
3. VGG [11]: A representative deep convolutional network framework that utilizes recursive deep convolutional operations to ensure the sufficient network depth for a desirable DDA performance.
• NAS Baselines:
1. NashNetLarge/Mobile [15]: A state-of-the-art NAS approach that effectively refines the neural network architecture by introducing a scheduled drop path mechanism. In addition to the standard version of NashNet (NashNetLarge), we also consider the mobile version of NashNet (NashNetMobile) which achieves a better trade-off between the NAS performance and computational efficiency in streaming DDA applications.
2. Darts [16]: a representative NAS framework that introduces a differentiable architecture representation to ensure an effective NAS process.
3. MansNet [25]: A lightweight NAS approach to incorporate model inference latency into the factorized hierarchical architecture searching process via multi-objective reinforcement learning.
To ensure a fair comparison, the inputs to all compared schemes were set to be the same, which included: 1) the input social media images, 2) the ground truth labels of images in the training dataset, and 3) the labeled images from crowd workers. In particular, we retrained all compared baselines using the labels returned by the crowd query to ensure a fair comparison. In addition, we also consider the random baseline, which estimates the damage severity for each image by randomly selecting a damage severity level from the possible categories. In our system, we implemented our CD-NAS model using Tensorflow 2.0foot_2 , and trained our model using the NVIDIA Quadro RTX 6000 GPU. In our experiment, all hyperparameters were optimized using the Adam optimizer [47].
In particular, we set the learning rate to be 10 -6 . We also set the batch size to be 20, and the model was trained over 300 epochs.
To evaluate the performance of all compared schemes, we adopted three metrics that are widely used to evaluate the performance of multi-class image classification tasks in image processing: 1) F1-score, 2) Cohen's kappa Score (K-Score) [48], and Matthews correlation coefficient (MCC) [49]. We use K-Score and MCC in our evaluation because we have an imbalanced dataset, and these two metrics have been proven to be reliable for imbalanced data [50]. Higher F1-score, K-Score, and MCC indicate better performance.
In the first set of experiments, we studied the performance of all the compared schemes with different crowdsourcing settings. First, we vary the crowd query frequency β (Definition 11) from 1/5 to 1/3 for all compared schemes (e.g., we periodically send every one out of three images in the data stream when β is 1/3 in crowd query) while fixing the number of participating crowd workers B (Definition 10) to be 3. Second, we change the number of participating crowd workers B in the crowd query from three to five while fixing the crowd query frequency to be 1/3. We set the size of the AI-crowd fusion window AF W (Definition 8) to be 40. The evaluation results are presented in Tables 2 and3. We observed that our CD-NAS consistently outperformed all the compared baselines in all experimental settings. For example, the performance gain of CD-NAS compared to the best-performing baseline (i.e., DenseNet) when the crowd query frequency β = 1/3 and B = 3 on F1-Score, K-Score, and M CC are 5.76%, 7.48%, and 6.00%, respectively. The performance gains our scheme mainly come from the fact that it adaptively transfers the imperfect crowd intelligence to the optimal neural network selection for each image through the dynamic crowd-AI MLE design. In addition, we further evaluated the performance of our CD-NAS on additional settings of the two experimental variables (i.e., crowd query frequency β and the crowd worker numbers B). We also compared the performance of the CD-NAS with the best-performing baselines from the different categories (i.e., DenseNet for DNN baselines in Table 2 and Table 3, NasNetMobile for NAS baselines in Table 2 and NasNetLarge for NAS baselines in Table 3). The results are shown in Figure 4 and Figure 5, respectively. We observed that CD-NAS consistently outperformed the bestperforming baselines on different evaluation metrics for all evaluation settings.
Such evaluation results demonstrate the effectiveness of our scheme in leveraging the imperfect crowd knowledge to dynamically identify the optimal neural network architecture for each newly arrived image to provide accurate DDA results across different experimental variable settings.
In the second set of experiments, we compared the computational cost of all the compared schemes (except the trivial random baseline) in the studied streaming DDA application. We define the computational cost as the average computational time required to estimate the damage severity of an image.
To ensure a fair comparison, we evaluated all schemes using the same NVIDIA Quadro RTX 6000 GPU. The evaluation results are presented in Tables 4 and5, respectively. We observe that our CD-NAS scheme takes orders of magnitude less time to accomplish the DDA task compared to other baselines under different evaluation settings. This is because the compared baselines require additional computational time to retrain their models to capture the dynamics of the streaming data by leveraging the labels from crowd workers. In contrast, our CD-NAS designs a recursive expectation maximization solution that estimates the assessment reliability score of each neural network architecture on the fly without requiring any additional network retraining. In addition, we evaluated the computational cost of our CD-NAS for additional crowdsourcing settings.
Similar to the performance comparison in Section 5.3.1, we compare the performance of the CD-NAS with the best-performing baselines from each category in Tables 2 and3. The results are shown in Figure 6 and Figure 7. We observe that our CD-NAS achieves a clear performance gain compared to the best-performing baselines in all different settings, which further demonstrates the effectiveness of the dynamic neural network architecture searching scheme in maintaining the best DDA performance while maintaining the lowest computational time cost. provide a window for users of our CD-NAS scheme to select the AFW size to achieve a desirable DDA performance. In addition, we also note that the CD-NAS buffers few images in AFW its is too small, which often leads to suboptimal classification results. On the other hand, CD-NAS can buffer too many images in AFW when its size is too large, which often leads to a significantly reduced computation time. The actual selection of the AFW size will largely depend on the tradeoff between the classification accuracy and response time of the CD-NAS scheme that the users would like achieve in a particular DDA application. for the other scenarios are similar. Please note that we show the performance of CD-NAS from the 20 th timestep because our CD-NAS needs to explore the imagery data at the first few timesteps to overcome the cold start problem of the recursive EM algorithm. We observe that our CD-NAS can quickly boost the assessment performance and remain stable afterward, suggesting its effectiveness in recursively learning the optimal neural network architecture in the studied application.
We presented a CD-NAS framework to address a crowd-driven dynamic NAS problem and improve the QoS of streaming DDA applications. Our solution is accuracy and computational cost. We believe that CD-NAS will provide useful insights to explore the collective power of AI and crowd intelligence in a rich set of AI-driven streaming applications (e.g., disaster response, truth discovery, intelligent transportation).
https://crisisnlp.qcri.org/
https://www.mturk.com/
https://www.tensorflow.org/