INTRODUCTION

Where should we send the police? Who should we give housing to? How should we educate our children? Who should we give unemployment benefits to? Which families should we investigate for child abuse? AI-based predictive algorithms are being used or are being considered for use across all of these everyday public sector decisions [19,28,56,93,126]. Many of these technologies have faced public scrutiny and opposition. For example, in St. Paul, Minnesota, an algorithm intended to assess which children were at risk of getting involved in the juvenile justice system was blocked by a group of impacted parents and teachers who organized to oppose it [96]. While some government agencies have established track records of community engagement around the deployment of new technologies, the perspectives of stakeholders who will be most impacted by algorithms are not always adequately considered [20,57,109,138].

In this paper, we aim to address the following research question: What do impacted stakeholders think about data-driven technologies in the child welfare system? To do so, we held seven workshops with 35 expert stakeholders who are personally impacted by child protective services (CPS) and/or work in CPS. We first explained to our participants how current data-driven predictive risk models (henceforth PRMs) are designed and used. We then talked with participants about their perspectives on these technologies. We also encouraged participants to weigh in on whether current PRMs address the main problems they see in CPS, and to imagine other possibilities for data and data-driven tools beyond current PRMs. Prior work with impacted stakeholders has explored the design and use of PRMs [20]. Our study is the first in academic ML and HCI to ask stakeholders whether these technologies should be used at all and to imagine new futures beyond them. Yet, these conversations have been ongoing outside these academic disciplines [3,129]. 1Our participants brought up several important themes: In Section 5.1, we note that most participants opposed current PRMs because they saw them as exacerbating existing problems in CPS. These findings are consistent with, yet more specific and more critical than, prior work [20]. We present these first as a primer to more novel, constructive suggestions in Sections 5.2, 5.3, and 5.4. In Section 5.2, we present participants' suggestions for new data-driven tools beyond PRMs which better support impacted communities, e.g. to evaluate the child welfare system and the people who work in it, to recommend mandated reporters when not to make a report, and to allocate resources to families to prevent child maltreatment. In Section 5.3, participants recommended guidelines to mitigate possible harms of PRMs if they must be used in the future. In Section 5.4, participants suggested low-tech and no-tech alternatives better address the problems that motivate the use of PRMs. Overall, our work advances ongoing discussions around data-driven tools in CPS. We argue against current PRMs, and give new avenues to work in solidarity with impacted communities, beyond just designing algorithms for CPS agencies.

RELATED WORK 2.1 Algorithms in child welfare

CPS agencies have been using checklist-style actuarial risk assessments (henceforth diagnostic checklists), such as Structured decisionmaking (SDM) [90], for decades to assess how likely they think a family is to harm their children. Many agencies also use practice models such as Signs of Safety (SofS) and Safety Organized Practice (SOP) [127] as decision-making guides, often in conjunction with diagnostic checklists [79]. For a case study of diagnostic checklists, see [114]. Saxena et al. [113] note that predictive risk models (PRMs) which apply machine learning to administrative data have grown in popularity since around 2015. Some PRMs have been developed by private companies [31,60,124]. However, due to high error rates and proprietary opacity, many have been dropped [61,81,82]. Other PRMs have been developed through public-academic partnerships [28,101,130,131]. PRMs are currently being used or deployed in (at least) Pennsylvania, New York, Florida, Washington, Oregon, Colorado, and California [110]. For an extensive list of algorithms used in the U.S. child welfare system, see [110] or [113]. PRMs have been deployed in response to racial biases and disparities [38,70], inaccurate and inconsistent decisions, child fatalities [73], etc. Proponents of PRMs argue they make more accurate decisions than both workers and diagnostic checklists; and that they make more consistent, objective, and equitable decisions [28,34,58,88,121]. Some critics disagree with these points, arguing that PRMs are still discriminatory and still too inaccurate [29,45,83]. Others argue that PRMs risk "coding over the cracks" without addressing the foundational flaws in child welfare, and that communities should instead organize around systemic improvements to address these flaws [47]. Others still argue that CPS is not a flawed system but a carceral one that plays a dual, paradoxical role [32,94,103,105] to police families while supporting them -and that the supportive, "welfare" side is an over-stated veneer to cover up the real carceral side [108]. These critics argue that PRMs introduce new ways for CPS to police Black, Indigenous, and poor families [2,107,108].

Participatory algorithm design

Influenced by action research and the work of Paulo Freire [46], participatory design developed around the 1970s by Scandinavian researchers working to gain workers more power over the design of technologies they use on the job [16,50,75,111]. Participatory methods have since become a mainstay in HCI and CSCW [69,80], but have been broadened beyond their Marxist roots [17,120]. More recently, many have called for increased participation to ensure that diverse stakeholders' perspectives, needs, and values are reflected in the design of AI systems [74,78,91,132,137,138]. Yet, without clear political motivations beyond "democratization" of AI governance, participatory work in ML differs widely based on "which stakeholders are involved" and "what is on the table" [36,118,136]. Some propose consulting "the public" or broadly-defined "stakeholders" on their preferences around specific, technical design decisions [12,14,51,59,63,65,66,77,85,109]. Others intentionally work with specific groups who are most impacted by these technologies, yet still do not empower impacted stakeholders to engage in broader design decisions [20,23,27,52,57,115,119,119]. While more common across HCI and CSCW, less work in participatory ML empowers stakeholders to decide on the "scope and purpose for AI, including whether it should be built or not" [36]. Specifically around the design of algorithms in child welfare, 2 prior participatory work has either collaborated with government agencies or solely engaged with government workers in their studies [20,26,67,68,115]. 3 Most similar to our work, Brown et al. [20] partnered with a CPS agency to aid the development of a PRM by conducting participatory design workshops where they asked workers and community stakeholders about scenarios related to specific design choices. Our work differs from Brown et al. [20] in that we: 1) worked independently of a CPS agency, 2) asked whether PRMs should be used in the first place, and 3) asked open-ended questions about other technologies or non-technical changes beyond just designing algorithms for CPS agencies. Our approach can be seen as human-centered [23][24][25]: where the humans that we center are impacted communities, not government agencies. Drawing from standpoint theory [30,53] and the Marxist roots of participatory design [50], 4 we engaged with parents and workers who were most impacted by, but most disempowered around, decisions on data and technologies in CPS to better understand a "view of technology from below" [1]. 5 These methodological differences may have led to novel suggestions in Sections 5.2, 5.4, and 5.3, which go beyond those uncovered in prior community-engaged research.

BACKGROUND

Figure 1 demonstrates how data-driven predictive risk models (PRMs) work and how they are used currently in child welfare. Many U.S. child welfare agencies currently use PRMs, mostly to assist workers in "front end" decisions, such as which families to investigate or how to investigate them [31,110,131]. A few agencies are starting to use PRMs to allocate services to families before they are reported or to prevent foster care placement [2,87,134]. No agencies currently use PRMs in decisions after investigation, e.g. in court; however, there are currently no regulations around how PRMs can or cannot be used. Figure 1b demonstrates how a typical PRM is developed and used in CPS. Different agencies or PRMs can use different kinds of data. However, most algorithms use family demographics (excluding race) and past CPS data, e.g. about prior reports on the family [28,48,113]; some use other governmental data, e.g. criminal, public health, or public benefits data [131]. Many PRMs are designed to predict the likelihood of some observable proxy for abuse or neglect, which are often vague and rarely observable [113]. A machine learning (ML) algorithm then uses this data to train a model (the PRM). Finally, this PRM is applied to new case data and the PRM's assessment -interpreted as the likelihood of some proxy for abuse or neglect-is shown to CPS workers, who use it when making decisions [67,113]. Although no PRM is currently used to fully automate decisions, some suggest this is possible [26,35,45,84]. Others note that automation is a spectrum: CPS agencies can pressure workers to conform to PRMs' recommendations in some cases more than others [26,67].

METHODS

Our work takes a human-centered, participatory approach to the design and use of predictive risk models (PRMs) and data-driven technologies in child welfare. We conducted 7 workshops with 35 participants total. Workshops were conducted over Zoom, each with 4 to 7 participants who were impacted by or worked in CPS.

Recruitment & Demographics. Our participants were mostly impacted parents and caseworkers, plus a few private service providers, psychologists, attorneys, students, one former foster youth, and one adoptee. Table 1 describes participants' personal and job experiences in CPS. See Table 3 in Appendix A.1 for participants' demographics. The majority of participants were Black and/or Latina women in New York or California, however there was a mix of racial/ethnic backgrounds, genders, and locations represented. 14 participants said they were impacted parents. 20 said they worked for a CPS agency or had an education in child social work -of these, at least 8 worked in public agencies. Only 2 participants had significant technical knowledge about PRMs. Children under 18 were excluded. We recruited 23 participants through an online recruitment form distributed via email using a snowball sampling approach. We reached out to multiple existing contacts who work in CPS or teach in schools of social work in the U.S to distribute our recruitment form. We also recruited 13 participants through an existing contact in an organization for impacted parents in the northeastern region of the U.S. This agency also trained parent advocates, which is likely why many parents in our study also said they worked in CPS. Many participants had a mix of CPS experiences, e.g. workers who had been investigated. Thus, each participant reflects deep knowledge of multiple aspects of CPS and impacted communities.

Protocol. See Figure 2 for an illustration of our study protocol. Participants were given an almost identical short survey before and after the workshop to gauge their opinions on CPS and PRMs. See Appendix B for a full list of survey questions and a description of responses. 6 Workshops were semistructured, starting with 10 minutes of background on PRMs (similar to Section 3) including showing Figures 1a and1b, followed by a 60-minute conversation led by three questions about CPS and PRMs (see below), ending with a 20-minute design activity to elicit ideas about how to use and design PRMs, and how to improve child welfare beyond PRMs. Throughout each of these study activities, we tried to present information about PRMs and avoid value judgments of participants' responses to not sway participants.

In the Questions phase (Activity 2) of the workshop, we asked participants three questions to center conversations:

(1) What do you think are the goals or outcomes of an ideal system for protecting children? (2) What are some pros and cons of PRMs?

(3) How should workers and interventions look in an ideal system for protecting children?

Although the workshops were centered around PRMs, the first and third questions did not specifically mention PRMs in order to give space for participants to bring up comments or concerns about CPS in general. For each of these questions, we shared a document with participants to add their comments to. Our team of facilitators also took notes in real-time. We did not erase the documents between workshops, so that participants in later workshops could comment on past participants' thoughts.

In the Co-design activity (Activity 3), we asked participants to write down at least 4 ideas to change PRMs or CPS. Then, we asked each participant to share 2 of their ideas and write those on a shared document. Finally, we asked participants whether they agreed or disagreed with other participants' ideas, and asked the group to make one collective list of ideas (without mandating consensus). Our design activity was based on Crazy Eights [71,72]. Though there may be drawbacks to these kinds of open-ended design activities [55], we draw inspiration from abolitionists in "imagining a safer world" for Black and other minoritized people [108].

Ethics & Institutional Review. To minimize the risk of unintended harms to participants, we consulted domain experts and impacted parents when designing our study [55,95]. Two workshops included only impacted parents to reduce the risk of conflicts or power imbalances with other kinds of stakeholders. We also worked with leaders of the parent organization who helped with recruiting assist in facilitating these two workshops. We did not ask participants to disclose personal experiences with CPS (besides whether they had been investigated), due to potential harms of such disclosures [55]. However, as a result, participants may have had additional relevant personal experiences that they did not disclose to us. This study, including all questions, study materials, and recruitment methods, was approved by the Institutional Review Board of Carnegie Mellon University.

Qualitative Analysis. We transcribed all 10.5 hours of online workshop recordings into text, then used thematic analysis [18] to analyze our data. We conducted open coding on the data, generating over 1000 codes. We performed an affinity mapping process, comparing and clustering alike codes, then identified themes that emerged from this affinity mapping. Examples of themes include: problems with diagnostic checklists, labels and stigmatization, and decisions not to use PRMs for. In Section 5, we present a subset of these themes which are most relevant to FAccT readers, leaving out some themes specific to child welfare which did not pertain to PRMs nor future design work.

Positionality. Most of the authors of this paper are academic ML and HCI researchers, white or Asian, and have little personal CPS experience (although one author is also Black and Latina and works in CPS). Our participants are mostly frontline caseworkers or Black and Latina mothers who have been in the system. The lead author, who ran all workshops, is a white man, which may have influenced participants' responses [89]. We anonymize participants' responses so that they could speak freely (especially workers who may be retaliated against). At the same time, we acknowledge that this may mean we quote and get academic credit for the ideas of participants with different lived experiences than most of us. Yet, we also consider "the researcher as an active participant throughout the research process" [32]. We see this work as depicting a conversation between us "socially-minded" technological researchers and our participants, who are impacted by the technologies that our field has (or we have) created.

Limitations. One limitation of our work is that we recruited few foster youth and adoptees, who may have differing views from the mostly parents and workers we spoke with. Another is that our study was not geographically restricted. Because CPS differs by location, our participants' responses do not necessarily reflect a specific community (nor do we claim them to). Future work may include qualitative studies focused on former foster youth and adoptees, or focused on a specific locale (e.g. one county or city).

RESULTS

In this section, we present prominent themes that emerged from the workshops, which we believe to be most interesting to FAccT readers. See Table 2 for a summary of suggestions. Section 5.1 outlines participants' concerns with PRMs, which many viewed as exacerbating existing problems in CPS. In Section 5.2, participants offer suggestions for new work that researchers can do for impacted communities, beyond creating PRMs for CPS agencies. Section 5.3 includes suggestions on how to mitigate potential harms caused by PRMs if they continue to be used. Section 5.4 offers no-tech or low-tech alternatives which may better address many of the problems that have motivated the use of PRMs.

Concerns that PRMs reinforce systemic problems in CPS

19 of 32 participants who responded to our survey disagreed that current PRMs would lead to better outcomes in CPS; only 5 agreed (8 were neutral). Personal experiences led many participants to hold negative views of CPS, e.g. P1 who explained her views simply by: "31 years working in the system." Like Brown et al. [20], our participants disliked PRMs due to "system-level concerns;" yet, our participants gave more pointed criticisms.

Participants disliked PRMs for further entrenching CPS in what they saw as punishment, undersupport, and disempowerment. Participants (both parents and workers) said that CPS often punishes families instead of supporting them. P24, a parent, said, "so many people have been treated badly... when they were on a good foot, but because they don't have enough support, certain things got out of hand, and they wasn't given the opportunity to pick up the pieces... They just automatically get scolded and child removed. " P9, a caseworker, echoed this, using the disparate treatment of foster families versus original families as an example: "we punish the [original] parents for not doing all the right things" while "our foster family agencies have a plethora of resources and support and funding to ensure that that child's needs are met. " 7Participants said current PRMs would not help support families. P12 said that CPS' "goal is really to support families, and I just don't think this tool plays any role in actually supporting families." Rather, participants said PRMs widen surveillance by encouraging CPS to process more cases and intervene more (P1,P13,P20,P30), getting more families involved in CPS (P7,P11,P12,P13), and getting more families stuck in the system for too long (P7,P14,P27,P29,P33). P12 said that she worried PRMs would cause "more monitoring, more surveillance, more intervention in Black and Brown and poor communities." P27 said, "I don't trust the algorithm, because it's... been set up to just surveil Brown and Blacks." McMillan explains: "It's surveillance:... Coming into someone's home, checking their drawers, cabinets, and strip searching their children, how is that support?" [3]. A number of our participants said this exact scenario happened to them or happens regularly to people they work with, e.g. P26 said CPS came to "strip my kid butt naked and go through my cabinets and uproar and turn my whole house upside down." P12 described calling a domestic violence hotline for help, and instead getting investigated by CPS and having her child removed. PRMs claim to improve efficiency: Participants said this could be helpful if it got families out of the system quicker, but harmful if it got more families investigated (P7,P11,P12,P13). Participants saw similarities between PRMs in CPS and the criminal system (e.g. [6,76,122,123]) and the use of criminal data in PRMs in CPS (e.g. [131]) as further solidifying CPS as a carceral institution (P33,P35,P36). P33 worried PRMs would embolden CPS workers and police, allowing them to act like "attack dogs" on families with high risk scores. P12 said PRMs would add an "extra layer" for parents to fight through: "not only are you fighting the... agency, now you're gonna have to fight this computer system. " Workers also worried about an extra layer of blame if they disagreed with a PRM, reinforcing a "Cover Your Ass" mentality (P3,P7,P9,P19). 89 Participants said PRMs reinforce caseworkers' power over families and their role as gatekeepers. P29 said, "the power holder is the caseworker that's inputting the information and so it's already starting from a standpoint of they're the end-all be-all. " Finally, participants said PRMs allow designers and CPS leadership to control on-theground decisions and justify harms. P12 said, "Computers don't make decisions; people make decisions and program the computers to carry out those decisions. So we're not going to turn around and say, 'Wow. Oh, it's the computer that's creating this decision and this is why 80% of the children who go into foster care are from the Black communities. "' Participants said PRMs perpetuate existing biases and racism in CPS. 26 participants said they did not trust CPS to make unbiased decisions; only 1 participant said they did. P12, whose child was placed in foster care, said, "I've been through the system, I know how harmful it is and how racist it is and how destructive it is to Black and Brown and marginalized communities and even poor people." Participants thought PRMs would not address the most prominent causes of biases based on race or class, such as laws and policies which justify differential treatment of poor and Black families within CPS, or biased reporting outside CPS. Participants also thought that PRMs would not eliminate workers' biases because they still allowed for worker discretion (P1,P5,P6,P7,P9,P15,P16), 10and would even exacerbate racial biases because of biased or "dirty" data. Brown et al. [20] this (P1,P2,P5,P33). Overall, most participants suggested that PRMs were at best ineffectual, and at worst counterproductive, at mitigating existing discrimination and disparities based on race and class (see [37]). Some participants thought PRMs would perpetuate or exacerbate other existing biases, e.g. against former foster youth (P1,P5,P36) or people with mental illnesses (P36).

There were some exceptions to these overall sentiments. Though, even those who liked PRMs said they might reduce individual workers' biases and improve decision-making, but they would not address systemic issues. P4 said larger reforms were needed to address systemic discrimination, but that these changes would not happen overnight. "In the meantime, " P4 thought PRMs could help day-to-day decisions now, especially if they used the "right data": "If you put the correct data points in... maybe we can take some of that subjective bias out of it." A few other participants echoed similar sentiments about incremental benefits of PRMs coupled with systemic changes (P13,P14,P31). This sentiment of PRMs helping "in the meantime" has been echoed by proponents of PRMs, including CPS agencies defending their use [7]. Beyond these exceptions, most participants saw current PRMs in CPS as exacerbating what they saw as CPS' tendency to punish instead of support families, particularly poor, Black, Brown, and Indigenous ones.

Beyond PRMs: New directions to work in solidarity with impacted communities

Although most participants opposed current PRMs, many gave constructive suggestions on how researchers and designers can use data and technologies to support impacted communities, beyond just designing PRMs for CPS agencies.

Participants suggested that researchers and designers should work in solidarity with impacted families and communities to use data to oppose CPS (P19,P24,P25,P36). For a number of participants, the desire for researchers to work with communities manifested through suspicion that us authors were working with CPS agencies or did not have communities' interests at heart. 11 P11 speculated that our study was being conducted by the "inventors of [PRMs]" in order to "anticipate... the objections... of potentially skeptical people [so that] the sponsors will be [better equipped]... to resist the objections" in order "to further develop their tools and sell them, and thus become prominent in their academic fields, or make money, or both. " In another workshop, P24 asked, "What is the point of all this data-driven focus mess?" then asked the lead author to consider whether they were doing this work to publish a paper and further their academic career or whether it was work which could actually benefit families harmed by CPS. 12 Given that most prior work on CPS in ML and HCI has been conducted to help develop algorithms to assess families or in partnership with CPS agencies [20,26,67,114,115], these suspicions seem justified. Instead, participants suggested specific ways researchers could better work in solidarity with communities. For example, participants suggested using data about families who have successfully fought CPS to produce strategies and suggestions for other impacted families to do the same (P13,P20,P24). Others suggested using data to help parent advocates verify or disprove negative and/or erroneous claims that CPS agencies make about parents (P25,P36).

Participants suggested using data to evaluate the child welfare system and the people who work in it, including reporters of alleged abuse, foster parents and homes, CPS workers, agencies, interventions, services, etc (P1,P2,P4,P6,P9,P12,P25,P28,P29,P30,P31,P33). Participants said administrative data collected on families reflect more on CPS and other governmental systems than they do on individual parents (P1,P3,P4,P12). P1 said, "if you've had 6 open cases, that means [CPS has] had 6 times where we weren't helpful to a family. It's measuring the system... It doesn't tell us anything about the people." Participants thought that data and data-driven tools (such as PRMs) should be used to assess harms caused by CPS and help communities push for change (P1,P4,P10,P23,P28). P10 said, "it doesn't make sense at all to me, why high or low risk is even what anyone thinks is being predicted... [PRMs] could just as easily be measuring the extent of racism, the extent of surveillance. " This hearkens back to Roberts' [103] call to "measure the extent of community damage caused by the child welfare system." For example, data-driven tools could be used to evaluate CPS workers, like they have been used on other street-level bureaucrats [22].

Participants suggested designing an algorithm for mandated reporters to recommend whether and where to make a report (P14,P19). P19 said such an algorithm should address the following questions: "Is this something I should make a call on? Is this something I should reach out to a prevention agency or agency that could possibly service the family prior to just calling it into [CPS]?" P14 and P19 said the goal here is to reduce the number of families in the system, either by recommending not to report or rerouting calls somewhere else.

Participants suggested using PRMs to allocate resources, but some worried this would expand surveillance and stigmatization (P1,P2,P4,P14,P31). Although many participants said PRMs should not be used for coercive interventions, e.g. investigations or home removals, some suggested using PRMs to connect families to resources and services. P2 said, "what I would want to see in the future is using these tools to decide on resource allocation, like who should have priority for access to services; instead of starting an investigation, more framing it from a more positive and supportive side." Specifically, participants wanted more direct assistance to help with childcare or alleviate poverty, which many viewed as a common root cause of neglect and abuse (which is backed by prior work [39]). P7 said, "the goal would be to... have finances available to support families in need as a preventative measure, or housing, or employment, or... medical services" or even something like "Supernanny [to] go into homes and be there to help the family." Beyond individual assistance, some participants suggested community-or neighborhood-based approaches [64,104]. P1 suggested to "use data to find the top 3 zip codes where child protection is involved and get some of our local Fortune 500 companies to create living wage jobs in those zip codes. "

However, participants also worried that expanding services provided by CPS or connected to CPS through mandated reporters would expand surveillance and place a stigma on families. 13 P15, a caseworker, said, "those who... have more contact with systems... are the ones who get reported on constantly. " P1, a private CPS worker, 13 Some participants advocated for getting rid of anonymous (or all) mandated reporting to decrease the chances that assistance would lead to CPS intervention. said, "people can't ask for help without a report. " P33, a parent, said, "[PRMs] put a stigma on people themselves... You know, it's not like anybody's saying, 'Well, I want my significant other to run out and leave me with the child by myself and I struggle, so I had to get on welfare. ' ... Basically to survive, I get a stigma. "foot_15 P20, a parent, said this leads "communities [to] hide in their struggles [rather] than say they need support or reach out for needed resources. " Prior work describes this tension where families want more supportive resources but fear more CPS intervention [21,106,108]. Recent work shares our participants' fear that PRMs used to allocate services will "[sweep] into the carceral net low-risk individuals who previously would not have been on the government's punitive radar at all" [2,107]. Empirical work suggests that PRMs which use data on public services may lead to over-surveillance of Black families [26]. Some participants (P14, P19) worried about using PRMs for "preventive services," which are services CPS agencies offer to prevent child maltreatment or future CPS involvement [8,100,125,133]. Recent work [2] suggests that PRMs will increasingly be used for preventive services, due to funding from the newly-enacted Family First Prevention Services Act (FFPSA), early examples in New York and Pittsburgh to look to [2,87,134], and to avoid criticism like that of PRMs used for screening or investigations [45]. P14, an administrator, confirmed their agency is doing exactly this: "The [FFPSA] is... requiring a lot more evidence-based preventive services... One of the things that [our agency is] looking at is 'What about primary or secondary prevention?' In Allegheny County, they have another... preventive risk modeling tool called Hello Baby" [87].

Guidelines for mitigating harms of PRMs

As stated in Section 5.1, most participants opposed current PRMs. Yet, many said that if these tools were to continue being used, they would like more guidelines around their use and design to reduce harms.

Participants wanted stricter rules on how data and PRMs can and cannot be used, so that data collected, or tools designed, for one purpose do not end up being used for another purpose (P2,P13,P33). P2 said they "would like to see some sort of policies to be put in place that would prevent tools like this being misused in the future... [and] really strict guidelines about how we can use these tools. " For example, local governments could implement legislation like Community Control Over Police Surveillance (CCOPS), which requires elected representatives to approve any government data or surveillance technologies (including PRMs) [4]. Some participants said PRMs should not be used for placement decisions (P10,P12,P26,P33) nor day-to-day decisions in general (P12,P17).

Participants said PRMs should be evaluated before and regularly after deployment (P4,P14,P30,P35). One big reason agencies have said they use PRMs is to mitigate workers' biases and address racial disparities in the system. Our participants suggested evaluating PRMs on whether they actually do this. Recent work suggests this, as well [40,49]. See, for example, prior work auditing PRMs [26,48]. However, P6 thought that evaluating whether algorithms help or harm may be difficult, especially if overall group effects such as racial disparities are improved, but individual families are harmed more. Future work on auditing algorithms should clarify how best to measure group and individual impacts.

Participants wanted more impacted families involved in CPS policy and technology decisions. Participants recommended that involving communities to make decisions and set policies can help mitigate biases at the unit-or agency-level (P5,P10,P14,P17,P24,-P26,P29,P30). P14, an administrator, said, "I think that families are the experts of their own, particularly even youth." P23, a parent, said, "we have to be part of the language that's controlling and setting the laws and that's... happening at every level of engagement for our families." Roberts [103] argues to shift control of CPS to Black families, specifically. Participants said families should be more involved around how new technologies are used and created (P1,P2,P5,P7,P10,P13,P14,P15,P22,P29). P14 said, "if you're developing anything, [it] needs to be community-led and... family led. " P29 said, "those that are creating [PRMs] should also be diverse and really reflect the communities that will be impacted by it, so that they're thinking... intentionally. " P22 said that PRMs might help make more equitable decisions "if they understood parents more. "

Participants said PRMs evaluating families should at least include data on CPS, reporters, workers, foster parents or homes, agencies, interventions, services, etc (P1,P4,P7,P10,P30). P1 said, "I don't know how you create these tools to measure the right thing if the data that goes in doesn't include specifically who the child protection social worker is, what intervention they received, and at what dosage;... unless you're measuring the other half of the equation, it's hard for me to imagine that you can get a good assessment. " Prior work also suggests including intervention data in PRMs [33]. This is feasible, since is already collected on all parts of the system except for anonymous reporters. However, JMacForFamilies is campaigning for NY State Senate Bill S7326 to require data collection on reporters in New York [62].

Participants said PRMs (and CPS more broadly) should focus on strengths, rather than deficits of families (P1,P3,P4,P12,-P13,P16,P17,P29,P31,P35). By focusing primarily on risk factors and predicting negative outcomes, P29 worried PRMs put families "at a deficit". P35 worried that PRMs do not adequately "take into account the... things [families] may have done or are doing to keep [their] child safe. " Instead, P13 said PRMs should predict "strengths and success. " More broadly in CPS, P14 said, "the narrative that we think about families needs to shift, as well, to one of a strength-based... interaction. " This sentiment is echoed in prior work, as well [57,113].

Participants said PRMs should not use demographics nor zip codes (P3,P12,P15,P26, P27,P28,P29,P30,P33,P36). Participants worried PRMs using zip codes and demographics (which are correlated with race and class) would justify discrimination of poor and Black families (P1,P2,P5,P33). Participants said using demographics was not new to CPS. For example, P12 said, "right now without using data analytics, they're still looking at your age, they're still looking at your zip code." Yet, PRMs justify this practice. Participants said using zip codes and demographics was disciminatory because these factors were irrelevant to parental (un)fitness. P28 said, "it is unfair to say 'because I live in this neighborhood, that must mean I'm a shitty parent'... It's unfair... to say '6 out of 10 of my neighbors had had [a CPS] case, so it's most likely I'm gonna have [a CPS] case'. " P26 said it "makes no sense" to use "your demographics, or past somebody else's history to determine whether you're a fit parent... because life is unpredictable. " Prior work argues people are unpredictable [15].

Participants said PRMs should not automate CPS decisions (P2,P6,P10,P14,P17). P17 said, "[full automation is] too much power, it's too much impact, and 99.9% of the time, [the PRM] fails." P6 pointed out a tension between automation versus worker bias: "I don't... believe that we should just hand the entire decision-making process over to a tool... [But] if we allow a caseworker to override the tool's guidance... then is that just sort of a form of bias in itself?" 15Prior work has also grappled with this tension: some argue humans in the loop often make biased decisions [6,49]. Others argue more automation can worsen disparities and decision quality [26,35,45].

No-tech and Low-tech Alternatives to PRMs

Participants suggested changes they thought would better address many of the problems motivating the use of PRMs, particularly which do not require AI-based technology (low-tech) or require no technology at all (no-tech). 16Participants suggested improving hiring, training, working conditions, and team-based decision-making instead of PRMs. First, participants said improved hiring practices would improve decision-making and alleviate biases, instead of using PRMs (P17,P19,P23,P24,P26,P32). Some said agencies should be more selective about who they hire; P24, a parent, said, "they need to stop hiring workers who just come out of college that don't have no children or have real life experience. " Some also thought hiring more diverse workers could decrease racial biases (P29,P31,P32). 17 Second, participants said CPS agencies should improve supervision, especially of young or inexperienced workers (which is common in CPS [42]) (P10,P16). Third, participants said team-based decision-making (especially diverse teams) could alleviate workers' individual biases (P7,P9,P15,P17,P31,P32). Fourth, participants said agencies should improve worker training (P10,P14,P16,P17,P19,P24). Finally, participants said agencies should improve working conditions, such as giving workers more time to make decisions, reducing caseloads, and increasing pay (P16,P17,P26,P32). This is important, since high case volumes have been a motivation for PRM use. 18 Participants also suggested smaller caseloads would reduce turnover, which would help retain workers who were hired and trained properly and reduce the number of new, inexperienced workers. P16, a retired administrator, said, "I have always found that workers that were well-supported -and whatever that means to them, not as the administration defines-can be very helpful in the longevity and the decreasing of turnover." P19 said PRMs should be unnecessary: "if you're a good social worker, you already know which one of your cases are more high risk and how to prioritize those cases. " P26, a parent, said, "[CPS] staff needs to be trained better, paid better, and maybe if they had happy workers, they care about their job and what they do. "

Participants wanted policy and legislative changes instead of PRMs (P3,P4,P9,P19,P24,P26,P29). Participants said a lot of systemic biases in CPS are caused by laws and policies. For example, P20 said many old laws "harm families, or target low-income Brown and Black families." In order to address systemic biases, participants recommended changing these laws. Participants suggested changing mandated reporting laws (P13,P19,P24). P26 suggested repealing laws and policies, like the Adoption and Safe Families Act (ASFA) [5]. These echo growing movements to repeal ASFA [11] and change mandated reporting laws [62]. P4 also said they want new funded mandates to get resources to communities and address systemic problems.

Participants suggested giving money directly to communities instead of spending it on CPS services or PRMs (P1,P2,P7,-P11,P12,P13,P24,P26,P33). P12 said, "the people making [PRMs]... financially benefit,... where this money could be set to pay for housing and other basic needs. " For example, PRM developers in Allegheny County were paid over $1 million [86]. Beyond development costs, participants also noted ongoing training and maintenance costs. P13 said, "How much it's gonna cost to train... the child protection workers [to use PRMs]... is also money that's being taken away from families." Allegheny County also hired specific employees ("Data Entry Specialists") to help with data entry for their PRM [131].

Participants proposed using diagnostic checklists and practice models instead of PRMs, but others said these low-tech tools had their own problems. Some suggested using diagnostic checklists (e.g. SDM [90]) or practice models (e.g. SofS, SOP [127]) instead of, or alongside, PRMs to alleviate workers' individual biases and improve decision-making (P1,P2,P17,P19). P1 suggested "integrating things like Signs of Safety. There are practice models... that help [workers] explore some very concrete, specific questions that help force them not to just make decisions based on their own hunch. " However, other participants said these low-tech tools had built-in biases (see [116]) and workers frequently manipulate them (against their training) to produce any desired output (P1,P5,P6,P7,P9,P15,P16). 19 Some said diagnostic checklists could be used better if workers were better trained and held accountable to follow the training (P5,P10,P14). Other participants thought tools should spur thought and nudge workers towards good decisions, not predict bad outcomes or give specific recommendations. P17 praised the Columbia-Suicide Severity Rating Scale (C-SSRS) [97,98]: "There are some yes or no answers and it's not about 'Oh, I want to get this kid 5150ed, 'foot_20 it's seeing what is the next step with a *thought*. So if I have information, then I use my *brain*, if I'm a human behind it. And I'm not the only one making this decision: I'm with a team. " Finally, some participants saw PRMs as a repeat of diagnostic checklists. When presented with a list of pros and cons to PRMs, P16, a retired administrator, said, "all these things you have up here are just the same sort of precursor work they did for [SDM] before it came into play. It's no different... and in child welfare things tend to cycle back, probably, you know, 19 P5 said, "I was trained... using Signs of Safety and SOP and saying, 'Well I may see this risk but I'm seeing protective factors that I think mitigate that, so I'm going to override [SDM] and not do that. '... But in practice, that's bullshit. I will override to make a 10-day an IR [Investigative Response] all the time. " decade on, decade off, decade on. So I'm just very curious about what's bringing this up again. " Some participants suggested abolishing the child welfare system and starting anew. However, participants' thoughts on abolition were varied. At the end of one workshop, all four participants (all CPS workers) agreed that abolition would be the best solution (P29,P30,P31,P32). At the same time, a number of impacted parents who were very critical of the system said they did not think it should be abolished, but that it should be heavily reduced and reformed. Views on PRMs and abolition were also interestingly varied. P33 suggested that CPS should be reformed, but that PRMs should be abolished completely. P30 and P31 said to abolish CPS, but not PRMs: "I agree with tearing the system down. I just think that there's a place for the tools." P4 said that regardless of whether or not CPS is reformed or abolished, these are longer term changes and PRMs could help in the short term. See [103] or [108] for more on child welfare abolition.

DISCUSSION

Here, we review novel suggestions and broader themes in Section 5, argue against the use of PRMs in child welfare, compare our study's approach with prior work, and highlight the suggestion to work in solidarity with impacted communities in the future.

Against predictive algorithms in CPS. Our participants gave more novel suggestions and critical feedback than in prior participatory work with impacted communities and workers in CPS [20,26,27,67,68,115]. For example, Brown et al. [20] suggest their participants' "general distrust in the existing system" (which they somewhat vaguely describe as "system-level concerns") led to "low comfort in algorithmic decision-making," and suggested these problems could be improved through "greater transparency and improved communication strategies. " Most of our participants also had "low comfort" in PRMs: They did not want them to be used. In Section 5.1, our participants said PRMs would reify existing tendencies to punish instead of support poor, Black, and other marginalized families, and solidify existing power imbalances in CPS. Even if there are problems with PRMs, proponents argue for their use because they are better than any alternative, i.e. diagnostic checklists or nothing [34]. In Section 5.4, however, participants gave low-and no-tech alternatives to address the problems motivating the use of PRMs: improved hiring, training, and working conditions; law and policy changes; giving money to families instead of CPS; and giving communities control of CPS. Overall, our participants thought PRMs are "doing more harm than good" and could be "replaced by an equally viable low-tech or non-technological approach;" thus, we argue that PRMs should not be used at all [13].

Mitigating harms of PRMs. Our participants also gave suggestions to mitigate the harms of PRMs (likely because they knew the above arguments are unlikely to stop agencies from using them). These suggestions largely differ from standard approaches to "trustworthy" AI. For example, participants did not ask for "greater transparency" around PRMs: They asked for regulations around how PRMs can and cannot be used, better evaluations of PRMs' impacts on communities, and more decisions about PRMs being made by impacted communities. Participants suggested that "improved communication" would not help either: although P14 suggested calling PRM labels "high need" instead of "high risk", many participants said that it matters more who is giving the labels (CPS agencies) and what they are doing with them, e.g. surveillance instead of support.

Agreement between workers and parents. Critiques of PRMs and CPS did not come only from parents, but from workers as well. This is surprising, because some described conflict between parents and workers. P32 said, "many of the white social workers have no knowledge of the suffering that goes on in the lives of the individuals they serve and cannot relate to their struggle. " However, our worker and parent participants often agreed, and workers criticized CPS more than we expected. While this may be a result of self-selection bias, we believe it reveals a subset of CPS workers (not all of them) who work in CPS despite seeing how harmful it is to families (cf. [32]). These workers may be important accomplices for impacted communities organizing for change.

Why is it important to work with impacted stakeholders in child welfare? For one, impacted stakeholders may generate ideas which researchers may not, due to lack of contextual knowledge or differing lived experiences. Many suggestions in Section 5.2 include these kinds of new design ideas. For another, impacted community perspectives are important in their own right, regardless of their value for novel research. Even when participants' suggestions are at odds academic work -e.g. participants suggesting PRMs not use demographics, while prior work [41] suggests using demographics to mitigate disparities in PRMs,-these suggestions are important because they reflect impacted stakeholders' perspectives. The general call to incorporate perspectives of impacted stakeholders into the design process [17,138] is heightened by the fact that the algorithms we focus on are used by governments which are accountable to the public [20,27,57,77,115]. If governments do not participate with impacted communities before they implement new technologies, they risk harming these communities, facing public scrutiny, or losing legitimacy [76,77,96,135]. Arnstein's Ladder of Civic Participation [9] organizes participatory governance into levels of community involvement and empowerment. Lower levels involve consulting impacted communities on specific choices in later stages of development, but restricting communities' power to control whether public projects are implemented at all (which may verge on "pseudo-participation" [92] or even "participationwashing" [118]). Higher levels include empowering communities to negotiate the scope of public projects. Our work lies higher than prior work on Arnstein's Ladder [9] in terms of scope, because we asked participants whether PRMs should be used in the first place, whereas prior work did not [20,26,27,67,68,115]. However, prior work may have been limited in what kinds of choices they put "on the table" for stakeholders, because they worked with CPS agencies, which are either mandated to use, or have already chosen to use, algorithms [36,114]. Yet, by working with CPS agencies, prior work may have more influence over the design and use of algorithms (albeit in constrained ways). In our work, by contrast, we had more freedom to ask participants more basic questions about PRMs because we worked independently from a CPS agency. Yet, CPS agencies have no reason to listen to our suggestions. Thus, by Arnstein's measure [9], our work may not redistribute power to communities as much as prior work, because (by not working with a CPS agency) we do not have much power to change CPS policy on our own. This highlights not only tradeoffs in working with government agencies, but also the importance for researchers to collaborate with workers' and community groups who can apply power to influence agencies, while maintaining independence from agencies.

Work in solidarity with impacted communities. Finally, our participants also suggested that researchers work in solidarity with impacted communities, even to oppose CPS agencies. This may have been overlooked in prior work because they centered public agencies. For example, Brown et al. [20] ask "What can researchers and designers working in partnership with public service agencies do... to raise comfort levels among affected communities?" then answer: "Facilitate... positive relationships between child welfare workers and families. " Yet, if researchers only encourage positive relationships, we may alienate people who have been harmed by CPS and do not want to stay positive. We should follow our participants' suggestion and work with impacted communities as "academic accomplices" [10], whether that means evaluating CPS and workers, getting data in the hands of impacted communities (which is not always easy [2,112]), designing tools to recommend mandated reporters not to report, joining with parents and advocates to fight against CPS agencies, or advocating for (non-technical) systemic changes. As groups like JMacForFamilies [62], Movement for Family Power [99], the upEND Movement [128], and Rise [102] exemplify, impacted communities have been organizing themselves. Our participants suggest we work with them.

A PARTICIPANT EXPERIENCES & DEMOGRAPHICS

In this section, we describe the questions we asked participants about their demographics, participants' responses, and some justification for why we asked participants minimal questions about their child welfare involvement in the demographics section of the survey.

A.1 Demographics Responses by Participant

See Table 3 for all demographics.

A.2 Demographics Questions

We asked participants about their demographics during the presurvey only. Below we include questions and answer options we asked participants: In order to better understand participants' personal experiences with the child welfare system, we: 1) asked participants how familiar with the system they were, 2) asked whether or not they have been subject to a child welfare investigation, and 3) we provided an open response question at the end of the survey and allowed participants to speak about their own experiences during the workshops if they so chose. If participants said they were unfamiliar with the system (which none did), their responses would not have been included in the study. We asked about participants' experiences in this way, rather than asking questions about more intrusive child welfare interactions, like "Have you ever had your children placed in foster care?", because we worried that explicitly asking about more intrusive interactions may have pressured some participants to describe or relive difficult or traumatizing experiences. This aligns with prior work on potential harms of personal disclosure in participatory workshops [55]. We also wanted to allow participants to define which experiences they thought were most relevant to this study within their own terms. There are tradeoffs and limitations to these approaches, however: Because we did not explicitly ask questions about the plethora of ways someone may be impacted by the child welfare system, there may be relevant experiences that additional participants had which they did not disclose to us. Two examples of questions that we did not ask about, but which reflect particularly relevant experiences, include those related to whether participants had experiences of themselves being in foster care or being adopted as youth.