An official website of the United States government
Abstract The importance and complexity of spatial join operation resulted in the availability of many join algorithms, some of which are tailored for big-data platforms like Hadoop and Spark. The choice among them is not trivial and depends on different factors. This paper proposes the first machine-learning-based framework for spatial join query optimization which can accommodate both the characteristics of spatial datasets and the complexity of the different algorithms. The main challenge is how to develop portable cost models that once trained can be applied to any pair of input datasets, because they are able to extract the important input characteristics, such as data distribution and spatial partitioning, the logic of spatial join algorithms, and the relationship between the two input datasets. The proposed system defines a set of features that can be computed efficiently for the data to catch the intricate aspects of spatial join. Then, it uses these features to train five machine learning models that are used to identify the best spatial join algorithm. The first two are regression models that estimate two important measures of the spatial join performance and they act as the cost model. The third model chooses the best partitioning strategy to use with spatial join. The fourth and fifth models further tune two important parameters, number of partitions and plane-sweep direction, to get the best performance. Experiments on large-scale synthetic and real data show the efficiency of the proposed models over baseline methods.
more »
« less
Estimating Economic Characteristics with Phone Data
Historically, economists have relied heavily on survey-based data collection to measure social and economic well-being. Here, we investigate the extent to which the "digital footprints" of an individual can be used to infer his or her socioeconomic characteristics. Using two different datasets from Afghanistan and Rwanda, we show that phone data can be used to estimate the wealth of individuals in two very different economic environments. However, we find that such models are relatively brittle, and that a model trained in one country cannot be used to estimate characteristics in another. These results suggest several promising applications and directions for future work.
more »
« less
- Award ID(s):
- 1637360
- PAR ID:
- 10062319
- Date Published:
- Journal Name:
- AEA Papers and Proceedings
- Volume:
- 108
- ISSN:
- 2574-0768
- Page Range / eLocation ID:
- 72 to 76
- Format(s):
- Medium: X
- Sponsoring Org:
- National Science Foundation
More Like this
-
-
Policies designed to extend the lifetime of products—by encouraging reuse rather than disposal—are proliferating. Research suggests that reuse can ease pressure on natural resources and improve economic efficiency, all while preventing waste. In Maine, there are clear signs of a tradition of reuse that might be used to advance these goals. But beyond discrete observations, proverbs, and anecdotal stories, little data have been collected upon which to estimate the potential of Maine’s reuse economy. This paper draws upon findings generated during the first year of a five-year interdisciplinary, mixed-methods research project designed to explore the environmental, social, and economic dimensions of reuse in Maine. Our preliminary findings suggest that Maine does, indeed, have a vibrant but underestimated reuse economy. Less expected are findings that suggest reuse has promise to enhance economic resilience and contribute to culturally appropriate economic development.more » « less
-
Nonpoint source (NPS) pollution is a severe problem in the U.S. and worldwide. Best management practices (BMPs) have been widely used to control stormwater and reduce NPS pollution. Previous research has shown that socio-economic factors affect households’ adoption of BMPs, but few studies have quantitatively analyzed the spatio-temporal dynamics of household BMP adoption under different socio-economic conditions. In this paper, diverse regression approaches (linear, LASSO, support vector, random forest) were used on the ten-year data of household BMP adoption in socio-economically diverse areas of Washington, D.C., to model BMP adoption behaviors. The model with the best performance (random forest regression, R2 = 0.67, PBIAS = 7.2) was used to simulate spatio-temporal patterns of household BMP adoption in two nearby watersheds (Watts Branch watershed between Washington, D.C., and Maryland; Watershed 263 in Baltimore), each of which are characterized by different socio-economic (population density, median household income, renter rate, average area per household, etc.) and physical attributes (total area, percentage of canopy in residential area, average distance to nearest BMPs, etc.). The BMP adoption rate was considerably higher at the Watts Branch watershed (14 BMPs per 1000 housing units) than at Watershed 263 (4 BMPs per 1000 housing units) due to distinct differences in the watershed characteristics (lower renter rate and poverty rate; higher median household income, education level, and canopy rate in residential areas). This research shows that adoption behavior tends to cluster in urban areas across socio-economic boundaries and that targeted, community-specific social interventions are needed to reach the NPS control goal.more » « less
-
Abstract Policymakers everywhere are working to determine the set of restrictions that will effectively contain the spread of COVID-19 without excessively stifling economic activity. We show that publicly available data on human mobility—collected by Google, Facebook, and other providers—can be used to evaluate the effectiveness of non-pharmaceutical interventions (NPIs) and forecast the spread of COVID-19. This approach uses simple and transparent statistical models to estimate the effect of NPIs on mobility, and basic machine learning methods to generate 10-day forecasts of COVID-19 cases. An advantage of the approach is that it involves minimal assumptions about disease dynamics, and requires only publicly-available data. We evaluate this approach using local and regional data from China, France, Italy, South Korea, and the United States, as well as national data from 80 countries around the world. We find that NPIs are associated with significant reductions in human mobility, and that changes in mobility can be used to forecast COVID-19 infections.more » « less
-
Abstract Drug overdose deaths continue to increase in the United States for all major drug categories. Over the past two decades the total number of overdose fatalities has increased more than fivefold; since 2013 the surge in overdose rates is primarily driven by fentanyl and methamphetamines. Different drug categories and factors such as age, gender, and ethnicity are associated with different overdose mortality characteristics that may also change in time. For example, the average age at death from a drug overdose has decreased from 1940 to 1990 while the overall mortality rate has steadily increased. To provide insight into the population-level dynamics of drug overdose mortality, we develop an age-structured model for drug addiction. Using an augmented ensemble Kalman filter (EnKF), we show through a simple example how our model can be combined with synthetic observation data to estimate mortality rate and an age-distribution parameter. Finally, we use an EnKF to combine our model with observation data on overdose fatalities in the United States from 1999 to 2020 to forecast the evolution of overdose trends and estimate model parameters.more » « less
- Free Publicly Accessible Full Text
-
Accepted Manuscript1.0
- Journal Article:
- https://doi.org/10.1257/pandp.20181033
