An official website of the United States government
Recent publications have pointed out a number of challenges and challenges for when engineers validate mobile apps using a conventional way inside a testing laboratory. Top two issues include: a) higher test costs due to the diversity of mobile devices and platforms, b) difficulty in conducting large-sale user-oriented performance and usability testing. A new testing approach, known as crowdsourced testing, provides a promising way to address these challenges and issues. This paper provides a comprehensive tutorial on crowdsourced test services, and informative concepts, insights, and detailed discussion about common questions raised by engineers and managers. It presents a clear comparative view between mobile crowdsourced testing with traditional lab-based mobile testing. In addition, it also summarizes and compares different major players, their commercial products, and solutions in mobile crowdsourced test services. Furthermore, it examines the major issues, challenges, and needs.
more »
« less
Successes, challenges, and rethinking – an industrial investigation on crowdsourced mobile application testing
The term crowdsourcing – a compound contraction of crowd and outsourcing – is a new paradigm for utilizing the power of crowds of people to facilitate large-scale tasks that are costly or time consuming with traditional methods. This paradigm offers mobile application companies the possibility to outsource their testing activities to crowdsourced testers (crowdtesters) who have various testing facilities and environments, as well as different levels of skills and expertise. With this so-called Crowdsourced Mobile Application Testing (CMAT), some of the well-recognized issues in testing mobile applications, such as multitude of mobile devices, fragmentation of device models, variety of OS versions, and omnifariousness of testing scenarios, could be mitigated. However, how effective is CMAT in practice? What are the challenges and issues presented by the process of applying CMAT? How can these issues and challenges be overcome and CMAT be improved? Although CMAT has attracted attention from both academia and industry, these questions have not been addressed or researched in depth based on a large-scale and real-life industrial study. Since June 2015, we have worked with Mooctest, Inc., a CMAT intermediary, on testing five real-life Android applications using their CMAT platform – Kikbug. Throughout the process, we have collected 1013 bug reports from 258 crowdtesters and found 247 bugs in total. This paper will present our industrial study thoroughly and give an insightful analysis to investigate the successes and challenges of applying CMAT.
more »
« less
- Award ID(s):
- 1822137
- PAR ID:
- 10097670
- Date Published:
- Journal Name:
- Empirical software engineering
- Volume:
- 24
- Issue:
- 2
- ISSN:
- 1382-3256
- Page Range / eLocation ID:
- 537–561
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Mobile robot navigation is a critical aspect of robotics, with applications spanning from service robots to industrial automation. However, navigating in complex and dynamic environments poses many challenges, such as avoiding obstacles, making decisions in real-time, and adapting to new situations. Reinforcement Learning (RL) has emerged as a promising approach to enable robots to learn navigation policies from their interactions with the environment. However, application of RL methods to real-world tasks such as mobile robot navigation, and evaluating their performance under various training–testing settings has not been sufficiently researched. In this paper, we have designed an evaluation framework that investigates the RL algorithm’s generalization capability in regard to unseen scenarios in terms of learning convergence and success rates by transferring learned policies in simulation to physical environments. To achieve this, we designed a simulated environment in Gazebo for training the robot over a high number of episodes. The training environment closely mimics the typical indoor scenarios that a mobile robot can encounter, replicating real-world challenges. For evaluation, we designed physical environments with and without unforeseen indoor scenarios. This evaluation framework outputs statistical metrics, which we then use to conduct an extensive study on a deep RL method, namely the proximal policy optimization (PPO). The results provide valuable insights into the strengths and limitations of the method for mobile robot navigation. Our experiments demonstrate that the trained model from simulations can be deployed to the previously unseen physical world with a success rate of over 88%. The insights gained from our study can assist practitioners and researchers in selecting suitable RL approaches and training–testing settings for their specific robotic navigation tasks.more » « less
-
Abstract One of the major challenges in 3D printing is its lack of scalability both in size and speed, which directly impacts its economic feasibility for large-scale industrial applications. Cooperative 3D printing (C3DP) is an emerging paradigm that aims to address these issues by employing multiple mobile printers that work in parallel. However, a crucial step in enabling C3DP is the development of a collision-free communication framework between the printers during the manufacturing process. Many C3DP systems found in the literature develop solutions for collision-free printing that are specific to the setup being used, thus not allowing the solution to be transferred to other similar systems. In this paper, we formulate a general framework that generates four distinct collision-free communication strategies to enable arm-arm coordination for C3DP using robotic manipulators. We considered collisions both between the arms with themselves and between the arms and the part being printed. The strategies are general in that they are agnostic to the number of printers, their kinematics, and their spatial configurations in the manufacturing environment. We conducted a study of the four strategies using a two-printer scenario and then physically validated them with four test cases of varying geometries. The results show that the strategies successfully produce printed parts while being collision-free. The makespan reduction using our strategies when compared to a single printer varied from 20% to 42% depending on the strategy used. Finally, we discuss the limitations of the framework, as well as future research directions.more » « less
-
Many web applications use databases for persistent data storage, and using Object Relational Mapping (ORM) frameworks is a common way to develop such database-backed web applications. Unfortunately, developing efficient ORM applications is challenging, as the ORM framework hides the underlying database query generation and execution. This problem is becoming more severe as these applications need to process an increasingly large amount of persistent data. Recent research has targeted specific aspects of performance problems in ORM applications. However, there has not been any systematic study to identify common performance anti-patterns in real-world such applications, how they affect resulting application performance, and remedies for them. In this paper, we try to answer these questions through a comprehensive study of 12 representative real-world ORM applications. We generalize 9 ORM performance anti-patterns from more than 200 performance issues that we obtain by studying their bug-tracking systems and profiling their latest versions. To prove our point, we manually fix 64 performance issues in their latest versions and obtain a median speedup of 2× (and up to 39× max) with fewer than 5 lines of code change in most cases. Many of the issues we found have been confirmed by developers, and we have implemented ways to identify other code fragments with similar issues as well.more » « less
-
Actor concurrency is becoming increasingly important in the real world and mission-critical software. This requires these applications to be free from actor bugs, that occur in the real world, and have tests that are effective in finding these bugs. Mutation testing is a well-established technique that transforms an application to induce its likely bugs and evaluate the effectiveness of its tests in finding these bugs. Mutation testing is available for a broad spectrum of applications and their bugs, ranging from web to mobile to machine learning, and is used at scale in companies like Google and Facebook. However, there still is no mutation testing for actor concurrency that uses real-world actor bugs. In this paper, we propose 𝜇Akka, a framework for mutation testing of Akka actor concurrency using real actor bugs. Akka is a popular industrial-strength implementation of actor concurrency. To design, implement, and evaluate 𝜇Akka, we take the following major steps: (1) manually analyze a recent set of 186 real Akka bugs from Stack Overflow and GitHub to understand their causes; (2) design a set of 32 mutation operators, with 138 source code changes in Akka API, to emulate these causes and induce their bugs; (3) implement these operators in an Eclipse plugin for Java Akka; (4) use the plugin to generate 11.7k mutants of 10 real GitHub applications, with 446.4k lines of code and 7.9k tests; (5) run these tests on these mutants to measure the quality of mutants and effectiveness of tests; (6) use PIT to generate 26.2k mutants to compare 𝜇Akka and PIT mutant quality and test effectiveness. PIT is a popular mutation testing tool with traditional operators; (7) manually analyze the bug coverage and overlap of 𝜇Akka, PIT, and actor operators in a previous work; and (8) discuss a few implications of our findings. Among others, we find that 𝜇Akka mutants are higher quality, cover more bugs, and tests are less effective in detecting them.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/https://doi.org/10.1007/s10664-018-9618-5
