An official website of the United States government
The term crowdsourcing – a compound contraction of crowd and outsourcing – is a new paradigm for utilizing the power of crowds of people to facilitate large-scale tasks that are costly or time consuming with traditional methods. This paradigm offers mobile application companies the possibility to outsource their testing activities to crowdsourced testers (crowdtesters) who have various testing facilities and environments, as well as different levels of skills and expertise. With this so-called Crowdsourced Mobile Application Testing (CMAT), some of the well-recognized issues in testing mobile applications, such as multitude of mobile devices, fragmentation of device models, variety of OS versions, and omnifariousness of testing scenarios, could be mitigated. However, how effective is CMAT in practice? What are the challenges and issues presented by the process of applying CMAT? How can these issues and challenges be overcome and CMAT be improved? Although CMAT has attracted attention from both academia and industry, these questions have not been addressed or researched in depth based on a large-scale and real-life industrial study. Since June 2015, we have worked with Mooctest, Inc., a CMAT intermediary, on testing five real-life Android applications using their CMAT platform – Kikbug. Throughout the process, we have collected 1013 bug reports from 258 crowdtesters and found 247 bugs in total. This paper will present our industrial study thoroughly and give an insightful analysis to investigate the successes and challenges of applying CMAT.
more »
« less
Crowdsourced Testing Services for Mobile Apps
Recent publications have pointed out a number of challenges and challenges for when engineers validate mobile apps using a conventional way inside a testing laboratory. Top two issues include: a) higher test costs due to the diversity of mobile devices and platforms, b) difficulty in conducting large-sale user-oriented performance and usability testing. A new testing approach, known as crowdsourced testing, provides a promising way to address these challenges and issues. This paper provides a comprehensive tutorial on crowdsourced test services, and informative concepts, insights, and detailed discussion about common questions raised by engineers and managers. It presents a clear comparative view between mobile crowdsourced testing with traditional lab-based mobile testing. In addition, it also summarizes and compares different major players, their commercial products, and solutions in mobile crowdsourced test services. Furthermore, it examines the major issues, challenges, and needs.
more »
« less
- Award ID(s):
- 1637371
- PAR ID:
- 10092502
- Date Published:
- Journal Name:
- 2017 IEEE Symposium on Service-Oriented System Engineering (SOSE)
- Page Range / eLocation ID:
- 75 to 80
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Telecommunication industries and spectrum regulation authorities are increasingly interested in unlocking the 12 GHz band for two-way 5G terrestrial services. The 12 GHz band has a much larger bandwidth than the current sub-6 GHz band and better propagation characteristics than the millimeter wave (mmWave) band. Thus, the 12 GHz band offers great potential for improving the coverage and capacity of terrestrial 5G networks. However, interference issues between incumbent receivers and 5G radio links present a major challenge in the 12 GHz band. If one could exploit the dynamic contexts inherent to the 12 GHz band, one could reform spectrum sharing policy to create spectrum access opportunities for 5G mobile services. This article makes three contributions. First, it presents the characteristics and challenges of the 12 GHz band. Second, we explain the characteristics and requirements for spectrum sharing at a variety of levels to resolve those issues. Lastly, we present several research opportunities to enable harmonious coexistence of incumbent licensees and 5G networks within the 12 GHz band.more » « less
-
The performance of Internet services—be it file download completion times, video quality, or lag-free video conferencing—is heavily influenced by network parameters. These include the bottleneck bandwidth, network delays, and how fairly the bottleneck link is shared with other services. However, current techniques to evaluate service performance in emulated and simulated networks suffer from three major issues: (a) testing predominantly in settings representing the "edge" of the Internet, and not the core; (b) focus on evaluating Congestion Control Algorithms (CCAs), neglecting the impact of application-level controls like Adaptive-Bitrate (ABR) algorithms on network performance; (c) testing in settings that do not necessarily reflect the network conditions experienced by services with expansive CDNs. The goal of this thesis is to improve the state of the art in emulated testing for a more up-to-date evaluation of Internet service performance. To highlight the need to perform Internet evaluations in settings representing congestion at the core of the Internet, we test CCAs with core Internet speeds and flow counts. We find that this dramatically alters fairness outcomes, and challenges long-standing assumptions about CCA behavior that were built on measurements performed at in settings representing the edge of the Internet, emphasizing the need to run Internet evaluations in more diverse settings. We then challenge the implicit assumption that CCA evaluations alone are suf- ficient to predict the network behavior of services that use them. We perform this analysis through the lens of fairness, and build Prudentia, an Internet fairness watch- dog, that measures how fairly two Internet services can share a bottleneck link. In addition to discovering extreme unfairness on the Internet today, we gain key insights into improving current testing methodology – (a) The most and least fair services both use variants of the same CCA, highlighting the need to test services in addition to CCAs; (b) network settings can drastically affect even service-level fairness outcomes, necessitating their careful selection. Lastly, we infer the network conditions experienced by users of Netflix, a global video streaming provider, and contrast them with those used in typical Internet evaluations. We find that Netflix users experience shorter RTTs, greater maximum observed queuing delay, and greater ACK aggregation, all parameters that play an important role in determining CCA behavior. This highlights the need for more service operators to run similar analyses and share their respective perspectives of prevalent network conditions, so that the networking community can include these settings in the design and evaluation of Internet services.more » « less
-
Cellular networks that offer ubiquitous connectivity have been the major medium for delivering emergency services. In the U.S., mobile users can dial an emergency call with 911 for emergency uses in cellular networks, and the call can be forwarded to public safety answer points (PSAPs), which deal with emergency service requests. According to regulatory authority requirements for the cellular emergency services, anonymous user equipment (UE), which does not have a SIM (Subscriber Identity Module) card or a valid mobile subscription, is allowed to access them. Such support of emergency services for anonymous UEs requires different operations from conventional cellular services, and can therefore increase the attack surface of the cellular infrastructure. In this work, we are thus motivated to study the insecurity of the cellular emergency services and then discover four security vulnerabilities from them. Threateningly, they can be exploited to launch not only free data service attacks against cellular carriers, but also data DoS/overcharge and denial of cellular emergency service (DoCES) attacks against mobile users. All vulnerabilities and attacks have been validated experimentally as practical security issues in the networks of three major U.S. carriers. We finally propose and prototype standard-compliant remedies to mitigate the vulnerabilities.more » « less
-
In recent years, mobile apps have become the infrastructure of many popular Internet services. It is now fairly common that a mobile app serves a large number of users across the globe. Different from web- based services whose important program logic is mostly placed on remote servers, many mobile apps require complicated client-side code to perform tasks that are critical to the businesses. The code of mobile apps can be easily accessed by any party after the software is installed on a rooted or jailbroken device. By examining the code, skilled reverse engineers can learn various knowledge about the design and implementation of an app. Real-world cases have shown that the disclosed critical information allows malicious parties to abuse or exploit the app-provided services for unrightful profits, leading to significant financial losses for app vendors. One of the most viable mitigations against malicious reverse engineering is to obfuscate the software before release. Despite that security by obscurity is typically considered to be an unsound protection methodology, software obfuscation can indeed increase the cost of reverse engineering, thus delivering practical merits for protecting mobile apps. In this paper, we share our experience of applying obfuscation to multiple commercial iOS apps, each of which has millions of users. We discuss the necessity of adopting obfuscation for protecting modern mobile business, the challenges of software obfuscation on the iOS platform, and our efforts in overcoming these obstacles. Our report can benefit many stakeholders in the iOS ecosystem, including developers, security service providers, and Apple as the administrator of the ecosystem.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Conference Paper:
- https://doi.org/10.1109/SOSE.2017.28
