Galaxy–Galaxy lensing in HSC: Validation tests and the impact of heterogeneous spectroscopic training sets

INTRODUCTION

Between the surface of last scattering (z ∼ 1100) and the present day (z = 0), the paths of all observed photons have been gravitationally influenced by the intervening "cosmic web" of matter. This gravitational lensing, and particularly weak lensing, is sensitive to the growth of structure and expansion history of the Universe and serves as a key probe of cosmology (e.g., see review by Mandelbaum 2018). In addition, weak lensing serves as an effective complementary NSF Graduate Research Fellow † E-mail: jspeagle@cfa.harvard.edu technique to other cosmological probes (e.g., the Cosmic Microwave Background or Type Ia supernovae) by helping to break degeneracies between cosmological parameters and providing constraints on the growth of large scale structure (e.g., DES Collaboration et al. 2017;Hildebrandt et al. 2018;Hikage et al. 2018, for some recent cosmic shear results).

The determination of accurate photometric redshifts (photo-z's) is a key challenge for deep lensing surveys. While shallow surveys (i < 24) can obtain spectroscopic follow-up for representative samples, deeper surveys face tougher challenges. In this paper, we focus on the challenges of deriving photo-z's for the first year source catalog of the HSC 1 survey which reaches an i-band depth of ∼ 26 AB magnitudes. As a precursor to LSST 2 , the HSC-SSP survey is a crucial testing ground for photo-z methods that will be applied for future precision cosmology analyses.

At the depths probed by HSC, there is a lack of adequate representative spectroscopic redshifts (spec-z's) available for training, validating, and testing photo-z methods (Masters et al. 2015;Tanaka et al. 2018). As a result, the HSC photo-z team instead supplements spec-z's taken from a variety of public surveys with grism/prism-based redshifts (g/prism-z's) along with photo-z's derived from deep, manyband photometry when training various photo-z algorithms and validating their performance. These unavoidable choices lead to a heterogeneous training set spanning a wide range of possibly redshift "quality".

Although mixing spec-z's and high-quality alternatives will likely occur in future surveys, the impact of using such a heterogeneous mixture on weak lensing has not yet been extensively explored. While the lack of high-quality spec-z's in regions of color and magnitude space makes it difficult to validate photo-z performance in those regions independently of the assumptions used to generate them, supplementing spec-z's in these regions with other methods that rely more heavily on these assumptions (see, e.g., Bezanson et al. 2016) will not alleviate this problem. This means that performance in these regions remains a "known unknown" that is difficult to directly validate. This problem is particularly acute for future cosmology surveys hoping to derive unbiased photoz's at the sub-percent level to the majority of their faint photometric samples.

Currently, there are several attempts in the literature to try to resolve this issue. These take two broad approaches. The first is an attempt to efficiently collect spec-z's to "fill in" regions of color space that currently do not have available data. The largest systematic approach is the C3R2 survey (Masters et al. 2017), which has so far collected ∼ 1300 highquality spectra in under-populated regions of color-space. The second is to assume that we can use cross-correlations of ensembles of galaxies that span the relevant redshift range, regardless of their color and/or magnitude, to characterize photo-z accuracy for a population of galaxies. This has proven to be promising but is not without challenges (Ménard et al. 2013;Newman et al. 2015;Hoyle & Rau 2018). Importantly, both of these methods assume that we can use ensembles of galaxies in specific regions of color and/or magnitude space to calibrate photo-z biases and uncertainties.

In this paper, we investigate how the use of heterogeneous training samples affects photo-z performance and galaxy-galaxy (gg) lensing analyses (e.g., Kwan et al. 2017;Leauthaud et al. 2017;Prat et al. 2018) using HSC-SSP data. In §2, we describe the photometry, shear, and redshift data used in this paper. In §3, we investigate the representativeness of current spec-z samples and examine the dependence on color and magnitude. We find strong evidence for evolution in the redshift distribution of galaxies at fixed color as a 1 The Hyper Suprime-Cam (HSC) Subaru Strategic Program (SSP) Survey (Aihara et al. 2018). See hsc.mtk.nao.ac.jp/ssp. 2 The Large Synoptic Survey Telescope (Ivezic et al. 2008). See lsst.org. function of magnitude. This leads us to develop a new framework, described in §4, for computing photo-z's from heterogeneous data that incorporates magnitude dependence and allows us to track how specific training objects contribute to individual photo-z predictions. We investigate the accuracy of photo-z probability density functions (PDFs) computed using this method in §5.

After discussing our photo-z tests, in §6 we outline the framework used for our lensing analysis. In §7, we then test whether our gg lensing measurements are robust to a variety of estimators, quality cuts, and spectroscopic incompleteness. We conclude in §8.

We assume a flat ΛCDM cosmology whenever appropriate with ΩΛ = 0.7, Ωm = 0.3, and h = 0.7. All magnitudes in the paper are AB magnitudes (Oke & Gunn 1983). For our lensing calculations (see §7), we assume physical coordinates to compute ∆Σ in 10 logarithmically spaced bins from 0.05 Mpc to 15 Mpc.

The weak lensing source sample

Our sample of galaxy sources is selected using the weak lensing cuts outlined in Mandelbaum et al. (2018). In brief, these are a series of quality cuts to ensure that composite model (i.e. CModel) photometry, point spread functions (PSFs), and measured object shapes are reliable. Observations are restricted to i < 24.5 mag to avoid using data with possibly unreliable shape measurements and to ensure "reasonable" spec-z coverage (although see §3). A "full-depth, full-color" (FDFC) cut was also imposed to eliminate sources that were not observed in all bands to full depth. Objects near bright stars were removed using the updated Arcturus bright star masks described in Coupon et al. (2018), as opposed to the original Sirius masks used in Mandelbaum et al. (2018), as those preserve more galaxies around bright stars. See Mandelbaum et al. (2018) for additional details regarding the construction and validation of the HSC-SSP S16A weak lensing shear catalog.

In addition to these weak lensing cuts, the photo-z's used in this work were only computed for objects with PSF-matched 1.1 aperture photometry available in all five bands. This effectively imposes an additional de facto cut on the seeing in all five bands. A variety of internal tests have found that this does not introduce a meaningful bias on weak lensing analyses (More et al., in prep.).

Redshift Training Data

The HSC-SSP Wide survey footprint is designed to maximize the overlap with other photometric and spectroscopic surveys while keeping the survey geometry simple. This allows HSC-SSP to exploit a large number of public specz's when constructing a redshift training set. In addition, (Ultra)Deep data taken in heavily observed fields such as COSMOS (Scoville et al. 2007) further allows HSC-SSP to include large numbers of fainter objects observed at higher signal-to-noise (S/N). These observations allow for more detailed modeling of the general population observed in the Wide survey, and are especially helpful for fainter sources.

A detailed description of the training sample can be found in Tanaka et al. (2018). We briefly summarize it here.

The training set contains spectroscopic, grism, and prism redshifts from a variety of overlapping public surveys including:

• zCOSMOS DR3 (Lilly et al. 2009),

• UDSz (Bradshaw et al. 2013;McLure et al. 2013),

• 3D-HST (Skelton et al. 2014;Momcheva et al. 2016),

• FMOS-COSMOS (Silverman et al. 2015),

• VVDS (Le Fèvre et al. 2013),

• VIPERS PDR1 (Garilli et al. 2014),

• SDSS DR12 (Alam et al. 2015),

• GAMA DR2 (Liske et al. 2015),

• WiggleZ DR1 (Parkinson et al. 2012), • DEEP2 DR4 (Newman et al. 2013), and • PRIMUS DR1 (Coil et al. 2011;Cool et al. 2013).

As each survey has its own flagging scheme to indicate redshift confidence, the different schemes were homogenized and used to select "secure" redshifts using the criteria outlined in Tanaka et al. (2018). In addition to these public surveys, a collection of private COSMOS spec-z's were also included exclusively for photo-z training (Mara Salvato & Peter Capak, private communication).

In addition to these spec-z's, grism-z's, and prismz's, the training set was supplemented with a set of highquality, many-band photo-z's taken from 3D-HST and COS-MOS2015 (Laigle et al. 2016) in order to maintain sufficient magnitude and color coverage down to i ∼ 24.5 (see §3). Without these photo-z's, the magnitude and color coverage of the training set fails to adequately span the relevant parameter space of the HSC-SSP data used in this analysis. The above heterogeneity in the spec-z's, grism-z's, prism-z's, and many-band photo-z's is one of the motivating reasons for the analysis presented in this work.

Objects were iteratively matched to this catalog within 1 at (1) UltraDeep, (2) Deep, and (3) Wide HSC-SSP depths in order to take advantage of higher-S/N data when available while avoiding possible duplicates. Our training data ultimately consists of ∼ 170k, 37k, and 170k high-quality spec-z, g/prism-z, and many-band photo-z's, respectively.

As described in Tanaka et al. (2018), to perform accurate cross-validation at HSC-SSP Wide depths, each object was assigned an "emulated" Wide-depth flux error based on the error properties of similar objects observed in the HSC-SSP Wide survey. Objects were also assigned an associated color-magnitude weight using a nearest-neighbor approach based on a representative subset of the weak lensing source sample to account for domain mismatches following the methodology described in §4.4. The original and reweighted redshift distributions of the HSC-SSP S16A training sample are shown in Figure 1.

HOW REPRESENTATIVE ARE EXISTING SPECTROSCOPIC REDSHIFT SAMPLES?

As discussed in §2.3, spectroscopic "completeness" within our training set varies strongly across magnitude and color. In other words, in a given color-magnitude "bin" the fraction of objects that come from more reliable sources such as spec-z's versus more unreliable sources such as many-band photo-z's can change rapidly. This behavior is concerning for several reasons. First, spec-z's generally have much smaller (often negligible) errors in redshift measurements compared to photo-z's, so our underlying knowledge of the redshift distribution at fixed magnitude and/or color degrades as the number and/or fraction of spec-z's decreases. Second, it is a well known issue that photo-z PDFs can be mis-calibrated (see, e.g., Tanaka et al. 2018). Third, there may be systematic biases of the redshift distribution of spec-z's relative to photo-z's in a given color-magnitude bin arising from selection effects that arise during the process of data collection. These involve choices that often generate the mismatch in the first place, from prioritizing spectroscopic targets (which often impose magnitude and color biases) to how non-detections are treated (which correlates strongly with redshift).

In particular, many studies assume that these pathological behaviors can be "calibrated out" by matching objects explicitly in terms of color (not magnitude) to obtain a representative spec-z sample (see, e.g., Lima et al. 2008). Surveys such as the Complete Calibration of the Color-Redshift Relation (C3R2) Survey (Masters et al. 2017) have expanded upon this strategy, explicitly sorting possible targets into bins in color space and then pursuing them assuming that spec-z's obtained at fixed color are representative of the entire photometric population in that given color bin.

While this strategy is efficient, it assumes that the intrinsic redshift distribution P (z|c) at fixed color c is representative over all relevant magnitudes m. This is a strong assumption given that the population of galaxies evolves as a function of redshift and that we expect brighter objects of fixed color to be (on average) at lower redshifts, all else being fixed. We begin by investigating to what degree P (mspec|c) differs from P (m|c) and whether or not these differences are of importance to current lensing surveys.

More formally, this assumption implies that P (z|c) = P (z|c, m)P (m|c)dm (1)

where K(z|zi,spec) is a kernel density estimate (i.e. smoothing scale) for each spec-z where zi,spec is a spectroscopic redshift drawn from a particular spec-z distribution P (mspec|c) = P (m|c) based on how the data were collected. Note that, in general, (2) is only guaranteed to be approximately valid if all the spec-z's comprise a representative sample from the underlying magnitude distribution at fixed color P (m|c).

To investigate these potential issues in our training sample, we will use manifold learning to sort our training galaxies into regions of color space to investigate possible trends in P (z|c, m). In §3.1, we describe the particular algorithm and procedure used to construct the manifold. In §3.2, we examine several examples of P (z|c, m) and find that there can be significant evolution as a function of magnitude in particular regions of color. Our results imply that current spec-z follow-up programs should be cognizant of these effects in order to avoid biasing photo-z predictions at fixed color. Since the redshift success rate for a given spectrograph may depend on the redshift, at a given magnitude and color bin even surveys that are relatively homogeneously selected may be subject to these subtle biases.

Manifold Learning and Self Organized Maps

For this study, we use a Self-Organizing Map (SOM; Kohonen 1982;Kohonen 2001) to both cluster our data in color space and learn a lower-dimensional 2-D projection that can be used for visualization purposes. We summarize the main As with Figure 1, we see that large portions of color/magnitude space do not have sufficient coverage within our dataset, necessitating the use of many-band photo-z's from surveys such as COSMOS to "bridge the gap" when computing photo-z's for HSC-SSP.

features of our specific implementation below, and direct the reader to Carrasco Kind & Brunner (2014a), Masters et al. (2015Masters et al. ( , 2017)), and Speagle & Eisenstein (2017) for more details concerning their applications to photo-z's.

The SOM is an unsupervised machine learning algorithm that projects high-dimensional data onto a lowerdimensional space using competitive training of a (large) set of "nodes" in a way that attempts to preserve general topological features and correlations present in the higherdimensional data. It consists of a fixed number of nodes N nodes = D i=1 N i nodes , where the product over i is taken over all dimensions D of the SOM, arranged on an arbitrary D-dimensional grid with N i nodes nodes in each dimension. Each node in the grid is assigned a position x on the SOM and is represented by a particular model F(x), where F is the set of observed features. In this paper, these are the set of grizy photometric flux densities comprising a particular galaxy Spectral Energy Distribution (SED) in the HSC filters.

Once the dimensions/shape of the SOM are chosen, training then proceeds as follows:

(i) Initialize the node models (most often randomly) and set the current iteration t = 0.

(ii) Draw (with replacement) a random object F and its associated errors σ from the input dataset.

(iii) Compute

across all nodes in the SOM over the available bands indexed by b, where the scale factor

renormalizes the model SED so that we are fitting in terms of flux density ratios (i.e. colors) rather than flux densities (i.e. magnitudes) directly. and plotted based on their relative likelihood of being associated with the node. We see a clear trend towards higher redshift as a function of magnitude. While some of the evolution at fainter magnitudes is due to intra-bin scatter from photometric uncertainty, this effect is minimal at brighter magnitudes where the trend is clearest.

(iv) Select the best-matching node

based on the minimum χ 2 (x) value. (v) Update the node models F(x) → F (x) based on a learning rate A(t) and neighborhood function H(x, x best |t) such that

(vi) If t Niter, increment t and repeat starting from (ii).

After training, objects are typically "mapped" onto the SOM by repeating steps (iii) and (iv) for every object in the input dataset, assigning each object to its best-matching node. We use a modified version of this approach where each object Fj is instead assigned to a set of nodes along with its corresponding weight wj(x) ∝ e -χ 2 (x)/2 for all nodes with wj(x) > fmin max(wj(x)). We take fmin = 10 -3 , which approximately corresponds to thresholding galaxies that are ∼ 2.5σ away from the best-fit. This "probabilistic mapping" allows us to better capture the uncertainty in an individual object's position on the SOM based on its photometric errors, resulting in smoother maps that are less sensitive to sampling noise and photometric errors relative to, e.g., Masters et al. (2017). A detailed discussion of these differences is beyond the scope of this paper and will be explored in future work. We choose our SOM to be 2-D with a 50 × 50 grid of nodes, and train it on the weak lensing source catalog photometry following the steps above for Niter = 10 5 iterations, which we find is enough to ensure the median χ 2 (x best ) value for objects across our map is approximately the number of colors (four) available. We choose our learning rate to be the weighted harmonic mean

for A0 = 0.5 and A1 = 0.1 and the neighborhood function to be a Gaussian kernel

with a standard deviation that goes as the weighted harmonic mean

with σ0 = 35 and σ1 = 1. Our final SOM is shown in Figure 2.

Redshift Evolution at Fixed Color

Using our SOM, we can now investigate the questions outlined in the beginning of this section. In particular, we want to examine whether the intrinsic redshift distribution at fixed color P (z|c) is insensitive to magnitude within our training data.

Although not common across our SOM, we do find some regions where there is evolution in P (z|c, m) at brighter magnitudes within spec-z-dominated samples. One such example is shown in Figure 3. For contrast, a more "typical" node shown for contrast in Figure 4. This confirms our basic intuition, formalized in Bayesian photo-z approaches such as BPZ (Benítez 2000), that the complicated evolution of galaxy SEDs and number densities as a function of time can lead to P (z|c, m) evolution as a function of magnitude if the underlying SED cannot be uniquely constrained. While this is likely possible in future multiwavelength datasets with full optical to near-infrared coverage (see, e.g., Hemmati et al. 2018), this likely remains a problem for current/planned weak lensing-oriented surveys such as HSC-SSP, DES, KiDS, and LSST.

Note that we do expect that noisy observations will naturally lead to a broadening of the redshift distribution at fainter magnitudes due to intra-bin scatter (i.e. an object's PDF gets "smeared" across multiple nodes on the SOM), and possibly to one whose mean distribution evolves strongly with magnitudes, mimicking a shift in the intrinsic P (z|c, m) distribution as a function of m.foot_1 This effect, however, should not impact the redshift distribution at brighter magnitudes (where measurement errors are small), which is where most of our spec-z's lie and where the trend seen in Figure 3 is the most apparent.

To quantify the extent to which possible redshift evolution can impact our redshift results, we focus on evolution in spec-z observations at i-band magnitudes brighter than m = 22.5 to mitigate redshift errors based on photometric measurement errors and incorrect redshift solutions. We compute the median redshift in bins of 0.5 mag, and fit linear trends for all SOM nodes where we could compute medians for 3 bins using 10 spec-z observations in each bin. We find that of the 709 nodes which fit this criteria (≈ 30% of the SOM), around 40% (287) display significant redshift evolution with dz/dm > 0.05. This trend is robust to different choices in dz/dm threshold and the required number of bins used in the fit, and is substantially higher than the few percent expected due to random variation. While this test is limited in scope, it highlights that the trend shown in Figure 3 is not an isolated case and needs to be taken seriously.

These results indicate that it can be dangerous to use re-weighted spec-z samples based on only a few broadband colors and expect to get the correct P (z|c) distribution, a danger which is indeed recognized by other weak lensing analyses (e.g., Troxel et al. 2017;Köhlinger et al. 2017;Hikage et al. 2018). This implies that spec-z samples may need to be representative in magnitude as well as color.foot_2

PHOTOMETRIC REDSHIFT FRAMEWORK

Based on the results in §3, we aim to develop a framework that allows us to explicitly incorporate magnitudedependence into our photo-z predictions to probe P (z|c, m) and alleviate possible mismatches at fixed color between spec-z's and many-band photo-z's present within our training set. At fainter magnitudes, however, almost all objects that make up our estimates for P (z|c, m) come from manyband photo-z's. As a result, we also want to track how individual objects in the training set propagate forward to our eventual photo-z predictions to investigate how much our many-band photo-z's are contributing to redshift predictions in different regions of color-magnitude space.

We adopt a Bayesian-oriented nearest-neighbors (NN)based approach that attempts to properly account for measurement errors within both training and testing sets when making photo-z predictions based explicitly on observed flux densities (magnitudes). In §4.1, we discuss the Bayesian underpinning of our approach. We describe our likelihood in §4.2 and our NN-based approximations to the likelihood/posterior in §4.3. We discuss our priors in §4.4.

All photometric redshifts (and SOMs) in this study were computed using an early development version (v0.1.5) of the Python photo-z package frankenzfoot_3 (Speagle et al. in prep.).

Photometric Likelihood

Assuming the errors on the measured fluxes Fg and Fh are independent and Normal (i.e. Gaussian) and ignoring the impact of selection effects (see, e.g., Leistedt et al. 2016), the log-likelihood for P (g|h) from our set of B bands indexed by b can be naively written as

+ B ln 2π This represents the standard χ 2 statistic often used in template-fitting methods (see §3.1) but with error contributions from both the target (g) and training (h) objects and including the relevant normalization term.

Unlike most template-fitting codes and contrary to the approach taken in §3, we deliberately chose not to include a free scaling parameter s to try and account for normalization offsets between g and h (i.e. fitting in magnitudes instead of colors). There are two reasons for this. The first is that the conditional prior P (s|h) we would want to impose over s when computing P (g|h) = P (g|h, s)P (s|h)ds is unclear. For instance, a fixed prior such as the uniform P (s|h) = P (s) = 1 prior used by most template-fitting codes is equivalent to assuming a fixed (and unphysical) luminosity function, which can create biases in inference. Trying to specify a color-dependent luminosity function P (s|h) directly, however, is extremely challenging because the integral over s often cannot be evaluated analytically. Fitting flux densities F directly avoids these complications.

More importantly for our purposes, however, is that our results from §3 show that there can be strong evolution in the underlying redshift distribution P (z|c, m) at fixed color c as a function of magnitude m. To mitigate this effect without attempting to deal with complicated priors, for simplicity we opt to keep the likelihood "close to the data" and fit directly in F.

Nearest-Neighbors Approximation

To avoid running over all M objects in the training set, we use a modified nearest-neighbors approach to preferentially select objects that have similar flux densities with respect to their errors. As the relative errors σ2 g + σ2 h of any two training/target objects g and h will differ, the relevant distance metric will be different for every pairwise training-target object combination. As nearest neighbor searches are typically done with respect to a fixed distance metric (often the Euclidean distance), this pairwise distance dependence poses a problem.

We deal with this by using Monte Carlo methods to search for neighbors across multiple realizations of the observed flux densities. We first generate a Monte Carlo realization Fg and Fh of the photometry for all objects in our target set and training set, respectively. We then determine the k nearest neighbors based on the Euclidean squared distance between our set of Monte Carlo-ed flux densities to a given observed galaxy g using a k-d tree (Bentley 1975), defining a set of k indices h(g). After repeating this process K times, we define an object's set of "photometric neighbors" as the union of the k nearest neighbors from each of the K Monte Carlo realizations:

Using our K Monte Carlo k nearest neighbors (KM-CkNN) approximation, we only compute photometric likelihoods to a small fraction of the data preferentially selected to contain the highest likelihoods. This procedure is most robust when the set of neighbors is roughly complete out to 3-5 standard deviations (relative to all possible pairwise galaxy combinations). An object can have at most k × K possible neighbors, with the exact number a strong function of the signal-to-noise of the target object g and the density of training objects h in its local region of color-magnitude space.

The sparse (kK N h ) nature of this KMCkNN approximation enables us to keep track of the individual loglikelihoods ln P (g|h) and their corresponding indices H(g) across all target objects. Because these have been computed exclusively using observables, we can subsequently use them to construct any associated ancillary quantities "after the fact". The most relevant example is the photo-z PDFs following equation ( 10), but this may include a whole range of other useful quantities such as those detailed in §4.5. A schematic diagram of our KMCkNN approach is shown in Figure 5.

One significant drawback of using a nearest-neighbors approach is that it is difficult to accommodate missing data. However, since the weak lensing source catalog used in this work is only defined in regions with full depth and full color coverage (Mandelbaum et al. 2018), this restriction does not impact our results.

Photometric Priors

We incorporate the KMCkNN approximation from §4.3 into our photometric prior P (h) by defining a new "sparse" prior Pg(h) = P (h) for h ∈ H(g) 0 otherwise (13)

Our photo-z PDFs can then be written as

Typically, the prior P (h) is defined to adjust for "domain mismatch" between the training and target datasets. Since spec-z training sets are significantly biased in both color and magnitude relative to most target photometric galaxy populations, this "reweighting" via P (h) traditionally substantially improves photo-z accuracy relative to cases where P (h) is assumed to be uniform (Lima et al. 2008).

We compute a photometric "prior" using the magnitudebased, KMCkNN approach described in §4.2 and §4.3 by computing the approximate Bayesian evidence under a uniform prior

where our roles for g and h have switched: we now treat our set of training galaxies indexed by h as "target" objects and a subsample of N ref reference objects indexed by g as "training" objects. We found the impact of the prior on our redshift predictions in internal testing was mostly unchanged for values of N ref Ntrain, and so opt to use N ref = 5 × 10 5 here.

To put this procedure another way, we determine P (h) by stacking the likelihoods of neighbors in the target data around individual training objects. This procedure, while not entirely proper (we are using a subset of galaxies in our sample to determine the prior), is sufficient for our purposes and represents the first step to a proper hierarchical model (Leistedt et al. 2016).

We find our prior-weighted training data are able to reproduce the B-dimensional distribution of our target data quite well as long as K and k are sufficiently large. See Tanaka et al. (2018) for additional details.

Goodness-of-Fit

By computing posterior quantities to every pair of trainingtarget objects, we can exploit goodness-of-fit tests used in a broad set of Bayesian model fitting applications. We will examine the two most basic indicators here: the maximum a posteriori (MAP) result Pmax(g) ≡ max{. . . , P (h|g), . . . } and the evidence P (g) ≡ h∈h P (g|h)P (h).

The MAP quantifies how good our best-fit result is, enabling us to determine if a given set of observables is represented in our training data. This is extremely useful when trying to remove objects with unreliable predictions that lie outside the parameter space spanned by our training data.

The evidence quantifies how well an object is represented across the entirety of our training data. This is useful when trying to identify objects which do not have meaningful "coverage" since objects that are only similar to a handful of training examples might have unreliable predictions.

In general, we find that the MAP and the evidence are highly correlated among our data: objects that are well-fit by at least one training example are very likely to be well-fit by others, and vice versa. Based on internal testing, we find that instituting a cut explicitly on the best-fit result based on the fitted χ 2 values removes the majority of poorly-fit objects from our sample. Our final cut is based on the 95th quantile P (χ 2 5 X) = 0.95, where χ 2 5 is the χ 2 distribution with 5 degrees of freedom, which is conceptually roughly equivalent to 2-sigma clipping.

Information Content

As mentioned in §4.3, our sparse KMCkNN approximation allows us to keep track of individual log-likelihoods computed between sets of training-target object pairs. It is then straightforward to transform these results into photo-z PDFs via equation ( 14).

More generally, however, keeping track of the relevant individual posterior predictions Pg(h|g) = P (g|h) Pg(h)

h P (g|h) Pg(h) allows us to compute almost any posterior-dependent result. This flexibility enables us to investigate auxiliary properties of interest.

In this work, we explore the impact of photo-z systematics on weak lensing using our heterogeneous HSC-SSP training data. In particular, we are worried about the impact many-band photo-z's might have on our results. In order to do this, we introduce two quantities to keep track of where the information content in a given photo-z prediction P (z|g) actually originates:

• Fphot: the fraction of neighbors in the training set with many-band photo-z.

• Pphot: the posterior-weighted fraction of neighbors in the training set with many-band photo-z.

Fphot and Pphot will help us to determine what kind of redshift (e.g. photo-z, spec-z, ..) any given object has been trained on. This will be an important ingredient for our lensing tests in Section 7.

Tuning: Feature Selection and (Hyper-)parameter Choices

The HSC-SSP catalog contains a variety of features that can be used for photo-z predictions, including a variety of photometry measurements and size information. In addition, the KMCkNN framework described in §4.3 involves several hyper-parameters that can impact performance. We conduct a variety of internal cross-validation and hold-out tests following §5.2 to determine the subset of features and hyper-parameters that give the best performance at a reasonable computational cost. Our results are summarized below:

• Our chosen flux density measurements were PSFmatched 1.1 aperture photometry among objects with successful forced photometry in all five bands. Adding additional features such as size or using different combinations of other photometry products (e.g., cmodel) gave comparable or worse results.

• We introduce a photometric smoothing kernel σ g,b = f b Fg,b for each object g in each band b to account for systematic uncertainties in measured photometric errors and to serve as a smoothing scale when computing likelihoods. We find that f b = 0.02 gives good photo-z PDFs in aggregate and constitutes an effective zero-point calibration uncertainty of ≈ 0.02 mag (although see Tanaka et al. 2018).

• To ensure optimal runtime, we want to make K and k only as large as necessary to obtain good magnitude/colorspace coverage for each object. The lower limit on K is set by the number of Monte Carlo realizations needed to roughly marginalize over the measurement uncertainties when searching for neighbors, while the lower limit on k is to ensure a reasonably large collection of neighbors. Based on internal testing, we find K = 25 Monte Carlo realizations and k = 10 neighbors selected at each iteration works as a reasonable compromise. The worst case performance (N nghbr ∼ 10) generally only occurs for bright and rare objects, which usually also have poor likelihoods. The typical number of unique neighbors is N nghbr ∼ 100 -200.

• We take our redshift kernels to be Normal distributions N z µ = ẑh , σ 2 = (∆z) 2 + σ2 h,z centered on the measured redshift ẑh with a variance set by a combination of an intrinsic width ∆z = 0.01, similar to the redshift spacing used when storing most photo-z PDFs, and the associated redshift measurement error σh,z . This allows us to propagate uncertainties from the many-band photo-z's to our final predictions. Following Tanaka et al. (2018), our redshift PDFs P (z|g) are evaluated over a redshift grid ranging from 0 z 6 with ∆z = 0.01 spacing. All redshift-based quantities described later in the text are derived from these discretized PDFs.

Calibration: Characterizing Behavior with Cross-Validation

As discussed in §2.3, to account for inhomogeneity and domain mismatch in the training set all objects are assigned an HSC-SSP Wide-depth emulated error following the procedure described in Tanaka et al. (2018) and an associated color-magnitude weight following §4.4. Unless stated otherwise, we utilize both quantities when computing any of the performance estimates reported here. We first randomly divided our training data into validation/hold-out testing sets comprised of (1 -f )/f fractions of the data for some hold-out fraction f . We then use two strategies to select our hyper-parameters and evaluate our performance within the validation set: k = 5-fold crossvalidation and internal leave-one-out tests. For k = 5-fold cross-validation, we randomly divided our validation set into k = 5 subsets. We then train on k -1 of these subsets to compute photo-z predictions to the remaining subset, cycling through each of the subsets until we had obtained predictions to the entire validation sample. For leave-one-out tests, we instead train on the entire validation set. However, when computing predictions to each object, we "mask out" its possible contribution within the selected group of neighboring objects used to compute the photo-z prediction. Both of these procedures, along with the final hold-out test set, attempt to mitigate over-fitting and ensure realistic performance estimates.

We find that the results from §5.1 are mostly insensitive to the chosen hold-out fraction f when f 0.5 (i.e. when our validation set consists of more than ∼ 150k objects). In addition, we also find that the performance on the hold-out test set is essentially identical to performance estimates within the validation set using both strategies when f 0.8, confirming that our approaches avoid over-fitting and that the information content appears to roughly saturate as our validation set exceeds ∼ 250k objects. Based on these results, we find that it is reasonable to treat our features and hyper-parameters from §5.1 as essentially fixed. Our reported performance is then estimated by applying the more conservative k = 5-fold cross-validation tests across the entire training sample (i.e. without the (1 -f )/f validation/testing split).

The 2-D stacked photo-z PDFs versus the input true redshifts (smoothed by their intrinsic uncertainties) along with the associated dispersion in ∆z/(1 + z) as a function of magnitude are shown in Figure 6. We see that our performance over the weak lensing sample is relatively robust, with an overall ∆z/(1 + z) ≈ -0.3% bias and with 68% of the PDFs contained within ∆z/(1 + z) = [-7.8%, 6.9%].

In addition to tests on the overall accuracy of our predictions, we also test the reliability of our individual PDFs. We opt to use the empirical cumulative distribution function (eCDF), which is constructed by evaluating the true redshift of each cross-validation object i ∈ i at the value of the predicted photo-z CDF ûi ≡

z i 0 P (z|i)dz (16) and computing Û (x) = i I(ûi x)

where I(•) is the indicator function which returns 1 if the condition is true and 0 if it is false. In the case where our PDFs are properly calibrated and have the expected coverage provided by any associated confidence interval, each CDF draw ûi ∼ Unif(0, 1) will be uniformly distributed from 0 to 1 and Û (x) will approximately define a straight line from 0 to 1. We show the eCDF results for our photo-z PDFs in Figure 7. These confirm that our PDFs are relatively robust and internally well-calibrated.

Application: Estimating Spectroscopic Incompleteness

We now turn our attention to the motivating issue behind the development of the photo-z framework outlined in §4 by investigating the distribution of Fphot and Pphot (see §4.5.2) within our HSC-SSP photo-z's.

We show the distribution of Fphot and Pphot as a function of magnitude in Figure 8. The results are as expected: many-band photo-z's in our training sample make up an increasing large fraction of neighbors and contribute an increasing amount to the photo-z PDFs at fainter magnitudes. We will use Fphot and Pphot to investigate the robustness of weak lensing measurements as a function of spectroscopic incompleteness (see §7). As expected, the photo-z's of objects at i > 23 are trained almost entirely on the many-band photo-z's in our training sample, while those at brighter magnitudes i < 23 tend to be trained on spec-z's and g/prism-z's. This transition happens more smoothly in Fphot than Pphot because the exponential nature of the likelihood tends to strongly favor a few photometric neighbors over others (see §4.2). This behavior is most apparent at brighter magnitudes, where even though ∼ 15% of neighbors come from many-band photo-z's, they tend to contribute very little to the overall photo-z prediction.

Computing the Galaxy-Galaxy Lensing Signal

Our computation of the lensing observable, ∆Σ, follows the methodology of Singh et al. (2017). We use the code dsigmafoot_4 , which was specifically written for computing gg lensing signals for HSC-SSP.

We compute ∆Σ as a function of physical radius R as

where ∆ΣL is the stacked signal around lens galaxies, ∆ΣR is the stacked profile around a much larger number of random positions, and f bias is a correction factor (see §6.2). The ∆Σ profile for both lenses and randoms are computed as follows:

where R Ls indicates a sum over all lens-source pairs with separation R. When computing ∆ΣR(R), we replace R Ls with R Rs since we instead sum over all random-source pairs. In Equation 19, γt is the tangential shear of a source galaxy:

where e1, e2, and φ are the two shear components and the angle from the direction of right ascension to the lenssource direction in sky coordinates measured by the HSC-SSP pipeline (Mandelbaum 2018;Bosch et al. 2018). Σcrit is the critical surface mass density:

which is computed using the angular diameter distance between the source and observer DA(zs), lens and observer DA(z l ), and source and lens DA(zL, zs). Each source galaxy is weighted by:

where σrms is the intrinsic shape dispersion per component and σe,Ls is the per-component shape measurement error (see Mandelbaum et al. 2018). R(R) is the shear responsivity factorfoot_5 that describes the response of galaxy ellipticity to a small amount of shear:

We compute and apply R independently for each radial bin.

The [1+K(R)] term is a correction for the multiplicative shear bias m:

where ms is a per source value that is calibrated using simulations. Please see Mandelbaum et al. (2018) for details about the calibration of HSC-SSP weak lensing catalog.

In this work, we use 10 5 random points to compute ∆ΣR sampled following the HSC-SSP S16A survey geometry. Random points are assigned redshifts following the redshift distribution of lenses. Although Singh et al. (2017) use the boost factor (Σ R Ls wLs/Σ R Rs wRs) to correct for dilution effects (see also Mandelbaum et al. 2005), we do not apply any boost factor corrections here. Instead, in Section 7.4 we test whether or not the signal varies as we impose more stringent lens-source separation cuts.

We assume physical coordinates and compute ∆Σ in 10 logarithmically spaced bins from 0.05 Mpc to 15 Mpc. Errors on all ∆Σ-related quantities are computed via bootstrap resampling. The dsigma code divides lenses and randoms to roughly equal-area regions. Here we use 40 regions with typical sizes of ∼ 2.5 deg and compute errors with NBs = 5000 bootstraps.

Corrections for Photometric Redshift Bias and Dilution Factors

Our procedure for estimating the bias on ∆Σ arising from photo-z's partially follows that of Mandelbaum et al. (2008), Nakajima et al. (2012), andLeauthaud et al. (2017). We summarize our approach here.

To correct for biases in ∆Σ arising from photo-z errors, a common procedure is to use set of galaxies with spectroscopic redshifts that have been re-weighted with appropriate color-magnitude weights (see §4.4) to match the source distribution. However, as shown in Figures 2 and 8, the currently available spec-z's in our training set are so underrepresented in some regions of color-magnitude space that it is impossible to properly re-weight them (spec-z's) to match the source sample.

For this reason, instead of a spectroscopic redshift catalog, we build a calibration catalog based on the set of manyband photo-z's in the COSMOS field matched with observations taken at HSC-SSP Wide depths with weak lensing cuts applied (see §2.2). Our assumption here is that the COS-MOS many-band photo-z's have narrow enough PDFs that they can be used to compute biases on ∆Σ for HSC. We refer to this catalog hereafter as the COSMOS calibration sample.

We compute the bias on ∆Σ as follows. Let ∆ΣP (Σcrit,P) represent the value of ∆Σ measured with photoz's and ∆ΣT (Σcrit,T) represent the true value of ∆Σ. We define f bias ≡ ∆ΣT/∆ΣP and estimate it via:

Ls wLs (Σcrit,T,Ls/Σcrit,P,Ls) Ls wLs

where the sum is performed over source galaxies drawn from the COSMOS calibration sample. Unlike other versions of this equation (e.g. Equation A3 in Leauthaud et al. 2017) there is no re-weighting factor to account for color mismatches between the source sample and the calibration sample because our COSMOS calibration sample is already representative.

For a given lens sample, we estimate f bias using Monte Carlo methods by randomly drawing sources from our COS-MOS calibration catalog and lens redshifts from the lens sample. We correct all ∆Σ values reported hereafter using f bias . This accounts for the dilution effect by sources that scatter above zL but which are actually located at redshifts below zL).

More explicitly, there are three issues: the impact of photo-z scatter and bias for sources that are above the lens redshift, dilution due to sources that are below the lens redshift but get scattered above it due to photo-z error, and dilution due to physically-associated sources. Our approach corrects for the first two of these, but not the third.

For our signals, typical values for f bias are around a few percent (∼ 2 -5%).

Comparing Lensing Signals

One of the primary concerns in this work is the robustness of the gg lensing signal with respect to possible photo-z biases. Since an absolute calibration does not exist, we instead aim to demonstrate the robustness of the signal to various cuts and choices for lens-source separation. We quantify this by considering the ratio of ∆Σ for two different computations i and j: fi,j ≡ ∆Σi/∆Σj.

(26)

This ratio test assumes that when we change how we calculate the gg lensing signals (by, e.g., tweaking the source sample selection or the redshift estimator), the true ∆Σ(R) should be the same (i.e. ∆Σi(R) = ∆Σj(R) for all R). This relies on the assumption that ∆Σ(R) does not vary much across the sample within the lens redshift bins. In other words, we assume that changing the source sample in a way that emphasizes different redshifts within the lens sample does not meaningfully change ∆Σ(R) given the same photoz quality across both source samples.

While it is straightforward to take the ratio between two lensing signals with different source cuts, ∆Σi and ∆Σj will be highly correlated. To deal with this effect, we derive the covariance matrix for f via bootstrap resampling using the same bootstrap regions as described previously.

We assume that fi,j is a constant (we only consider amplitude changes) and solve for the maximum-likelihood result (MLE). We fit for amplitude shifts over our full radial range (denoted f all ) and also over the radial range R = [0.1, 1] Mpc (denoted finner) and R = [1, 10] Mpc (denoted fouter).

RESULTS: HOW ROBUST IS THE GALAXY-GALAXY LENSING SIGNAL?

We now investigate how robust gg lensing signals are to various photo-z estimators and quality cuts. After exploring changes in the gg lensing sensitivity to a variety of choices ( §7.2- §7.6), we subsequently use the results of those choices to define a "fiducial" sample (see §7.7). Our results are presented in terms of the stability of the gg lensing signal based on other possible choices with respect to our fiducial sample.

The various cuts that we test comprise 15 unique lenssource samples. These are described in detail below and summarized in Table 1.

Since we perform a large number of tests (3 × 3 × 14 = 126) in the subsequent sections, many often correlated, we might expect by pure statistical chance that some of the results reported might have deviated from the expected null result by a large amount. We attempt to control against Table 1. Redshift estimates and selection criteria used to construct robustness tests for galaxy-galaxy lensing. V indicates that the quantity is varied for a particular test, while F indicates that it is kept fixed. These comprise a total of 15 unique lens-source samples.

Photometric Redshift Point Estimates

In lensing analyses, ∆Σ is often computed with respect to fixed point estimates derived from the photo-z PDFs to avoid having to integrate over all photo-z PDFs P (z|g)'s.

We study whether or not the particular choice of a point source estimate impacts the gg lensing signal. We compare five point estimates in this paper:

• zmean, the first moment (mean) of the photo-z PDF,

• z med , the 50th percentile (median) of the photo-z PDF,

• z mode , the redshift corresponding to the maximum value of the photo-z PDF,

• z best , the redshift estimator that minimizes the loss assuming a Lorentzian kernel in ∆z/(1 + z) with a width of σ = 0.15 (see Tanaka et al. 2018 for additional details), and

• zmc, a Monte Carlo draw from the photo-z PDF.

A comparison with integrating over the PDF is beyond the scope of this work and is discussed further in More et al. (in prep.). Most of these point estimates have been used to varying degrees in weak lensing analyses in the literature, each with various benefits and drawbacks. Here we will briefly outline the arguments for each estimator (see also Tanaka et al. 2018).

While beyond the scope of this paper, it is well known that the mean estimate zmean is the optimal point estimate for a PDF assuming "squared error" (L2) loss. In other words, if we introduce a penalty proportional to (zest-ztrue) 2 and assume ztrue follows our PDF, then zest = zmean is the estimator that is "best" given the PDF. This particular result is optimal for Gaussian distributions.

In general, however, most photo-z PDFs are not Gaussian, but instead can have asymmetric tails and/or extended shapes. The mean zmean is particularly sensitive to these tails, and so estimates that are more "robust" are sometimes preferred. As with the mean, it can likewise be shown that the median z med is the optimal point estimate under "absolute" (L1) loss where the penalty is proportional to |zest -ztrue|. This reduced penalty makes z med less sensitive to the tails. The mode z mode can likewise be shown to be the optimal point estimate under "unforgiving" loss (L0) where the penalty is maximized and constant for all zest = ztrue. This penalty makes z mode only sensitive to the peak of the PDF where the probability is maximized.

While these various estimators are optimal under different assumptions for how much we want to penalize "incorrect" guesses, none of them are specifically tuned for photo-z estimation. In particular, most PDFs and photo-z applications tend to have a dependence on |zest -ztrue|/(1 + ztrue) rather than just |zest -ztrue|, and also care about being accurate relative to a given "tolerance" σ. z best is the point estimate that minimizes the loss relative to these conditions.

Finally, we may want a point estimate that "explores" the entire PDF, rather than attempting to "summarize" it. Assuming "uniform" loss (i.e. a flat penalty everywhere), any Monte Carlo sample zmc from the PDF serves as a reasonable point estimate. These may better capture the behavior of PDFs by allowing us to probe, e.g., the tails of the distribution but lead to some (additional) amount of random noise being introduced.

The ∆Σ ratio estimates computed based on each of these various redshift point estimates with respect to our fiducial sample in each redshift bin are shown in the top two rows of Figures 10,11,and 12. We find that, with the exception of zmc, all of these choices result in negligible ( 1%), albeit sometimes statistically significant (at 3σ), differences in the computed ∆Σ. This is likely due to the general quality of our PDFs, which are reasonably well-constrained and unimodal for the majority of objects (see Figure 6) and also well-calibrated against the expected underlying redshift distribution (Figure 7), leading to very similar point estimates.

In general, using Monte Carlo redshifts zmc tends to lead to an underestimate of the ∆Σ signal by an increasing amount as a function of the lens redshift. This is due to the (exponentially) increasing sensitivity of the ∆Σ signal at close lens-source separations as well as increasing photo-z uncertainties at higher redshifts. Since Monte Carlo redshifts scatter sources around based on their PDFs, these tend to dilute the computed signals relative to the more "stable" point estimates above.

Since there is no (relevant) statistical difference between the photo-z point estimates excluding zmc, we decide to use the z best estimate due to its superior performance relative to the other estimators when predicting redshifts for individual objects within the full HSC-SSP S16A Wide sample. For additional comparisons between the per-object accuracy of these photo-z point estimates, see Tanaka et al. (2018).

Photometric Redshift Uncertainties

Although our photo-z PDFs are well-calibrated with respect to the underlying redshift distribution ( §5.2), in general using point estimates to summarize broad PDFs can lead to distortions in the underlying redshift population (Carrasco Kind & Brunner 2014b). In addition, broad photo-z PDFs where the probability density spans a large redshift range are generally seen as more unreliable, with more potential for miscalibrations that can lead to under/overestimated uncertainties on the prediction compared to narrower PDFs. As a result, it is common in many gg lensing analyses to remove "unreliable" photo-z's based on the width of their PDFs.

We define two main sources of uncertainty that con-tribute to unreliable PDFs. The first is systemic uncertainty: having a poor understanding of the object in question and therefore an unreliable redshift prediction. This can occur if the object is not well-represented within the training set, which leads to them having large χ 2 values when comparing to their closest color-magnitude neighbors. We can exploit this fact to flag and remove these sources explicitly.

The second source of uncertainty is statistical uncertainty: utilizing a point estimate that does not accurately represent the PDF. This can occur if the redshift PDF is overly broad or multi-modal with several possible redshift solutions. We quantify this source of uncertainty by defining the "risk" (Tanaka et al. 2018) that the point estimate is incorrect is the integral over the PDF with respect to the associated loss

where the particular loss function

is taken to be a Lorentzian kernel with a width of γ = 0.15. The best redshift estimate and the associated risk z risk are then defined jointly as

Sources with higher z risk generally have broader PDFs with multiple peaks. See Tanaka et al. (2018) for additional discussion.

We divide our sample into a number of sub-samples based on a range of photo-z quality cuts. These are:

• basic: χ 2 5 6. This is the χ 2 5 value corresponding to the 95% cut discussed in §4.5 since P (χ 2 5 6) ≈ 0.95 for a chi-square distributed random variable with 5 degrees of freedom. As expected, this removes ∼ 5% of sources. The majority of these sources are at brighter magnitudes and lower redshifts and do not not contribute significantly to the gg lensing signal.

• medium: In addition to basic, this selection also imposes a cut on the "risk" of a particular photo-z point estimate z risk < 0.25. This generally removes overly broad PDFs and leaves ∼ 75% of the sample.

• strict: In addition to basic, this selection imposes a stricter cut of z risk < 0.15, restricting our estimates to even fouter is displayed in the bottom-right corner along with a Gaussian distribution for reference. Most of the computed ratios are consistent with the expected null result at 3σ; those that disagree are highlighted in bold. Although these disagreements are statistically significant, some (e.g., with respect to z med ) are negligible in practice since their impact is 1%. In general, fouter is more unbiased for all samples than f and f inner . See §7 for additional discussion and details. narrower PDFs than medium. This leaves ∼ 60% of the sample.

The ∆Σ ratio estimates computed for each of these global photo-z quality cuts with respect to our fiducial sample are shown in the second row of Figures 10, 11, and 12. We find that the computed ∆Σ signals appear insensitive to the global photo-z quality cut chosen, and are statistically consistent with the null result (at 3σ). As with §7.2, this is likely due to the fact that our PDFs are both relatively well-constrained and well-calibrated for the majority of our sample, especially since any outlying PDFs are removed by our initial basic quality cuts.

Since our performance is similar across different global photo-z quality cuts, we opt to use our medium cut for our fiducial sample to compromise between sample size and PDF quality.

Lens-Source Separation

In addition to possible biases based on how the photo-z point estimates trace the underlying source population, it is also imperative to ensure that any possible differences we observe are not dominated by dilution effects or contamination from correlated objects (see §6.2). This is often done by imposing cuts that aim to ensure that the bulk of any source galaxy PDF lies behind the lens population (e.g., Medezinski et al. 2018; see also §7.2).

We parameterize this cut using two parameters. The first is a summary statistic detailing the redshift below which X% (z lowX ) of the source galaxy PDF lies, which we use to establish how confidently we can place a source galaxy behind a given lens. In other words, X% of the photo-z PDF is above a redshift of z lowX , which should be greater than the redshift of the lens z lens . The second is a "buffer" ∆z lens to establish a minimum separation threshold between the source z lowX and the lens z lens . This term is used to avoid being extremely sensitive to photo-z biases and possible miscalibrations in the corresponding PDFs since the dependence of Σcrit is highly non-linear when zsource and z lens are very close together.

We test four lens-source separation cuts:

for X = 68% and 95% (roughly 1 and 2-sigma) and ∆z lens = 0.1 and 0.2, listed roughly in order from most aggressive to most conservative.

Our results are shown in the third and fourth rows of Figures 10, 11, and 12. We find that all cases are consistent (at 3σ) with the null result with the exception of the z low95 + 0.2 > z lens cut for the zbin24 lens sample, which is smaller by ≈ 1.5%. As the most conservative cut in the lowest redshift bin (which should be least-sensitive to photo-z issues), it is somewhat surprising that we see a noticeable suppression. While it is possible that this is just statistical noise, it might also be the case that the photo-z's in the training set have systematic discrepancies at intermediate redshifts that are accentuated when only the low-z sources are removed.

In general, however, these results support the gg lensing signal being mostly insensitive to these specific combination of lens-source separation cuts for our HSC-SSP S16A data. This implies that the majority of our sources are correctly selected to be behind the bulk of the lens sample, providing additional (indirect) support that our photo-z PDFs are well-calibrated. These results also suggest that our gg lensing signals do not require any boost factor corrections.

Given that these lens-source separation cuts perform comparably, we again opt to use a compromise for our fiducial sample by choosing X = 95% and ∆z lens = 0.1.

High and Low Redshift Sources

One additional concern is our gg lensing analysis may be sensitive to degrading photo-z quality as a function of redshift. This is in general due to a combination of spectroscopic incompleteness at higher redshifts and fainter magnitudes as well as broader PDFs arising from noisier photometry (see §3). This is a particularly acute concern for this work due to our reliance on many-band photo-z's at the magnitudes and redshifts probed by a significant majority of our weak lensing source galaxies.

To investigate this effect, we divide our source sample into high-redshift and low-redshift samples to investigate this effect defined by:

• zlow: z best 1, which leaves ∼ 50% of the sample.

• zhigh: z best > 1, which leaves ∼ 50% of the sample.

The results for our high and low-redshift samples are shown in the fourth row of Figures 10, 11, and 12. Although we find the ∆Σ signals from zlow to be systematically higher than those from zhigh, the effect is not statistically significant and both agree with the null result (at 3σ). This provides us with confidence that we can utilize photo-z's for source galaxies at all redshifts when constructing our fiducial sample.

Origin of Training Redshifts

One benefit of the KMCkNN framework outlined in §4 is that we actually have a direct proxy of spectroscopic incompleteness through metrics such as Fphot and Pphot, in addition to indirect proxies such as the high/low redshift split used in §7.5. This allows us to examine how robust our gg lensing signals are depending on the information content used to estimate the photo-z's of individual source galaxies.

One complication of using Pphot to select source galaxies directly is that it tends to be strongly correlated with magnitude and redshift, with sources with lower redshifts and brighter magnitudes tending to also have lower Pphot. We attempt to alleviate this issue by limiting our analysis to the zhigh subset of galaxies (z best > 1). While this substantially reduces the sample (by 50%), it mitigates some of the extreme differences that can arise due to these effects.

As a compromise between preserving number density and maximizing differences between sources that are manyband photo-z-dominated versus those that are not, we ultimately split our source galaxies into three subsamples based on spec-z and g/prism-z information content:

• "Great": Pphot < 0.5 (i.e. > 50% of information comes from spec-z's and g/prism-z's), which leaves ∼ 10% of the sample.

• "Moderate": 0.5 Pphot < 0.85 (moderately photo-zdominated), which leaves ∼ 15% of the sample.

• "Poorest": Pphot 0.85 (completely photo-zdominated), which leaves ∼ 25% of the sample.

The results for these subsamples are shown in the bottom two rows of Figures 10,11,and 12. We find these signals are entirely consistent with the null result (at 3σ). This demonstrates that our gg lensing signals are stable to the origin of the training redshifts (e.g. spec-z, photo-z, etc.) used to compute redshifts for source galaxies. We note, however, that the spec-z samples used to train our photo-z's tend to have targeted very specific populations of galaxies even at higher redshift compared with the broader photometric sample (see §3). This can lead to low Pphot serving as a proxy for selecting galaxy samples in particular regions of colormagnitude space (and thus having different intrinsic properties). These changes in the underlying galaxy population could mask some of the expected impacts from many-band photo-z's alone and make it difficult to extrapolate conclusions beyond this current work.

In general, as we find that the computed ∆Σ signals are consistent with null results across all lens redshift and Pphot subsamples, we opt to include all sources when constructing our fiducial catalog.

Fiducial Lensing Cuts

Based on the results above, we now define the fiducial sample that all other samples are compared to in Figures 10, 11, and 12. Our reasoning is as follows:

• All point estimates (excluding zmc) investigated in this work give gg lensing signals with similar amplitudes. We thus opt to use the z best point estimates ( §7.2) given their improved performance across the broader photometric sample as outlined in Tanaka et al. (2018).

• Since the three basic quality cuts give similar ∆Σ estimates, we select the medium photo-z quality cuts ( §7.3) as a fiducial choice. This represents a compromise between retaining a larger sample size and removing overly broad photo-z PDFs.

• All four combinations of lens-source separation cuts give gg lensing signals that are consistent with each other. As with the global photo-z cuts, we decide to then compromise by selecting z low95 + 0.1 > z lens ( §7.4), which guarantees the vast majority of the PDF is located behind the lens while being slightly less conservative about the enforced ∆z lens separation.

• Our tests over high (z best 1) and low (z best < 1) subsamples of source galaxies do not show any sign of distortion by photo-z biases arising from changing populations of objects in our training data ( §7.5). To maximize sample size, we thus opt to use galaxies at all available redshifts.

• Finally, our tests using subsamples binned by Pphot at z best > 1 also do not find evidence for differences in ∆Σ among the varying subsamples ( §7.6). As a result, we opt to include all photo-z's regardless of their spectroscopic information content.

These cuts are implemented as defaults in dsigma.

CONCLUSION

Determining accurate photometric redshifts (photo-z's) remains a key challenge for deep lensing surveys such as HSC-SSP and LSST. At the depths probed by HSC-SSP, there remains a dearth of spectroscopic redshifts available for training, validating, and testing photo-z methods across the colors and magnitudes covered by weak lensing photometric samples. To reach the required coverage to compute photoz's to these objects, the HSC-SSP photo-z team constructed a heterogeneous training set derived from an amalgamation of public spec-z and g/prism-z surveys along with photo-z's derived from deep, many band COSMOS data.

Since mixing spec-z's and high-quality alternatives (g/prism-z's, photo-z's) will likely occur in future surveys, in this paper we sought to thoroughly investigate their impact on gg lensing analyses through a variety of methods. Our conclusions are as follows:

(i) Using Self-Organizing Maps (SOMs), we examine the color/magnitude-space coverage of our HSC-SSP training data relative to the HSC-SSP S16A weak lensing photometric sample ( §3). We find that, as expected, our spec-z coverage is highly non-representative relative to the overall sample, with the majority of our redshift information for "typical" galaxies in the weak lensing photometric sample coming from manyband photo-z's.

(ii) We then investigated whether current spectroscopic survey strategies, which seek to systematically fill in underpopulated regions of color space, can resolve this problem ( §3.2). We find that the assumption that the intrinsic redshift distribution at fixed color is constant as a function of magnitude does not always hold. This mismatch implies that certain regions of color space will likely require spec-z's that also probe the magnitude distribution of future weak lensing samples, complicating current efforts. This effect is in addi-tion to redshift-dependent success rates at fixed color and magnitude.

(iii) Based on these results, in §4 we develop a hybrid machine learning/Bayesian framework for tracking how subsets of galaxies in our training set contribute to individual photoz predictions explicitly as a function of magnitude. We show that in §5 our approach gives reasonable photo-z predictions and well-calibrated PDFs.

(iv) Using our fits, we are able to define metrics to reject objects that are poorly represented by the training data and further identify how "reliable" results are based on how significantly many-band photo-z's contribute to the derived PDFs ( §5.2). The results imply that photo-z's computed for objects with i 23 tend to be spec-z-dominated, while those at i 23 tend to be photo-z-dominated.

(v) Finally, using the full sample of LOWZ and CMASS BOSS galaxies, we investigate the impact various photo-z estimators, quality cuts, lens-source separation constraints, redshift subsamples, and spectroscopic information content can have on gg lensing signals. We find that most cases give results that are consistent with a fiducial baseline sample, indicating that biases in the gg lensing signal due to photo-z bias and scatter are sub-dominant to statistical uncertainties in the HSC-SSP S16A weak lensing data.

Although we do not find cause for concern in the analysis presented here, we hope that these methods can be used in future work to investigate similar issues when dealing with larger, deeper, and more complex samples leading up to future precision cosmology-oriented surveys such as LSST.