DEE WILSON CONSULTING
Evaluating Differential Response
(Originally published November 2018)
The implementation of two track differential response systems (DR) in more than 20 states has
(arguably) been the most important CPS reform in the United States during the past two decades. In addition to the states currently employing DR (referred to as FAR in Washington State), there have been several states that have abandoned DR, or put DR on hold, following implementation. These states include Florida, Michigan, Illinois and Oregon, among others.
In some states, DR has been met with strong opposition from influential stakeholders such as juvenile court judges (Florida), in part due to early research showing few, if any, benefits for child safety (Michigan and Illinois), or to a widespread concern among policymakers and newly appointed child welfare leaders that DR might jeopardize child safety (Oregon). DR implementation in many states has been (to put it mildly) a rocky road politically, a social milieu that has raised the stakes for scholars who have evaluated DR systems. For the past several years, there has been an intense scholarly controversy regarding DR research marked by claims of some DR critics (see Bartholet, 2016) that positive studies of DR have been influenced by the advocacy goals of proponents, researchers and an influential foundation.
There have been many evaluations of DR, for the most part evaluations of pilots in various states or counties. Fortunately, there have been a couple of well designed multi-state studies published in recent years. A number of studies have compared outcomes such as CPS re-report rates for cases assigned to Assessment Response (AR) vs. Traditional Response (TR), i.e., CPS investigation. Some of the best recent studies have utilized random assignment of screened in CPS re-reports meeting the state's criteria for AR, with some of these reports assigned to AR and others to TR. In this commentary, I discuss findings from randomized trials in Colorado, Illinois and Ohio, a study from a Midwestern state conducted by Loman and Siegel, and findings from multi-state studies conducted by Fluke, Darnell, et al. I also consider critics' reactions to DR research and their fierce opposition to DR. What follows is not a comprehensive review of DR research; rather, it is a selective review intended to draw out salient points and key findings.
Caveats regarding DR research
DR research is afflicted by the same deficiency as other research studies of child protection in the U.S., i.e., dependence on inadequate measures of child safety. Rate of CPS re-report is a flawed measure of child maltreatment, better (by far) than the recurrence measure utilized by state child welfare systems and the federal government, but still inadequate as a stand-alone measure of child safety. In specific cases, a CPS re-report of a family does not necessarily mean that a child has been maltreated, and the absence of re-report may not be an accurate indication of child safety. Furthermore, any stand-alone measure of child safety is inadequate because child safety is a multidimensional concept. A CPS re-report rate is a chronicity measure, not a severity measure. Some of the intense opposition to DR in states I've visited has nothing to do with rates of re-report; it is the result, in the view of professionals, that misuse of DR has contributed to the serious injury of children, including some child maltreatment deaths. It is difficult to know what to make of the concern raised in reviews of child protection practices in Los Angeles County, Massachusetts, Minnesota and Vermont that the child welfare system's use of DR has resulted in serious harm to children, absent careful studies that track trends in child maltreatment deaths and serious abuse/neglect related injuries. In-depth reviews of specific child deaths or near deaths are not adequate to arrive at well founded conclusions, anymore than reviews of a single child death in an investigative track is a sound basis for concluding that a state's CPS system is ineffective or incompetent. Every claim regarding the effectiveness of any aspect of CPS practice and policy based on a single measure of child safety is questionable. The reluctance of child welfare managers to develop and use a severity measure of child safety is due, in my view, to the likelihood that child maltreatment fatality data would be included in such a measure, and could be used to further undermine the political viability of child protection systems. In these circumstances, description of any element of CPS practice and policy as “data driven” or “evidenced based” is a half truth at best. Neither AR or TR should be viewed as evidenced based, given the large variations in DR systems and in states' investigative practices. DR refers to a wide range of policies and practices, not a well defined model of child protection.
DR research of CPS re-report rates
Fuller, et al's, randomized controlled trials of DR implementation in Colorado and Ohio (2014) found much the same CPS re-report rates on families eligible for assessment response in both the AR and TR tracks. However, in Colorado, low and moderate risk families assigned to the AR track “were 18% less likely to to have a subsequent high risk assessment,” according to Fuller, et al.
Fuller's, et al's, study of DR implementation in Illinois found slightly higher re-report rates for families assigned to AR, with families who dropped out of AR or were shifted from AR to TR accounting for most of the difference in re-report rates. Critics of DR have made much of the Illinois study of DR, but the findings are underwhelming for several reasons, in part because of Illinois' idiosyncratic DR system and because re-reports for families initially assigned to AR but actually served by TR were counted as AR re-reports in the study's 'intent to treat' research design.
The best recent studies of DR re-report rates have been done by Fluke, Darnell, et al, using a different methodology. These scholars compared CPS re-report rates at the county level in 6 states (Kentucky, Minnesota, Missouri, North Carolina, Oklahoma and Tennessee) for counties in which the AR utilization rate was above the median for the whole sample (45%) vs. below the median. This study found that higher rates of AR utilization were associated with an 18% lower rate of CPS re-report and with a 37% lower rate of substantiation following a re-report. According to these researchers, “Overall, higher rates of AR were associated with lower rates of re-reports …” However, in a separate study of DR re-report rates in four states, Darnell and Fluke found that there was a tipping point in counties' AR utilization at which the AR advantage in lower re-report rates “flips”, in other words, at some level of greater utilization, adding more high risk reports to AR increases the CPS re-report rate.
Measures of family engagement
In Fuller, et al's, evaluations in Colorado, Illinois and Ohio and in Loman's and Siegel's study,
AR was superior to TR on almost every measure of family engagement:
a) parent satisfaction with the assigned caseworker,
b) feeling respected by the caseworker,
c) caseworkers' listening skills and understanding of family needs,
d) caseworker accessibility,
e) parents' willingness to call CPS, and
f) satisfaction with services received.
In Illinois, more than half (52%) of parents assigned to AR indicated that their family was better off due to CPS intervention vs. less than a third (31%) of parents in the TR track. Parents in AR were more likely to respond that they were a better parent due to CPS intervention than parents who received TR.
Provision of services
In Loman's and Siegel's study of DR in 10 Ohio counties, families assigned to AR received more services in every service category except drug abuse treatment and medical/dental care. This study found “that there was a clear shift under DR toward material services,” and AR families were much more likely than DR families to receive food, clothing, emergency financial assistance and rental assistance. Almost half (48.7%) of AR families received at least one service vs. 36.6% of TR families.
Some state child welfare systems have implemented DR with the expectation of cost neutrality. Other states, like Ohio, have provided additional funding for AR families, a policy decision likely to significantly increase the percentage of low and moderate risk families receiving agency funded services, as well as the initial cost per family of CPS intervention. One of the most important findings in Loman's and Siegel's study is that AR families were modestly less likely to
have two or more subsequent CPS re-reports than TR families. Loman and Siegel maintain that
“FA's reduced the rate at which lower-risk families in the experimental group became chronic families,” a hypothesis that merits further research. If, indeed, AR has greater potential than TR to reduce the development of chronic neglect in (even) a modest percentage of families through higher rates of service provision and more poverty related services, CPS practice and policy in all state systems should be changed to reflect this potential, if not through DR then through greater investment in poverty related services for moderate and low risk families with screened in CPS reports.
In state and county child welfare systems that have implemented DR without additional funding for services or additional caseworkers, DR is likely to lead to a re-prioritization of scarce resources from high risk families to low and moderate risk families. Dedicating resources to low and moderate risk families while inadequately serving high risk families is certain to lead to tensions among advocates and practitioners. Extreme workload pressures and lack of adequate services for families with open CPS cases has had a huge effect on DR implementation around the country, including Minnesota, a state which for many years was in the vanguard of DR implementation. Debates about DR which ignore workload pressures in child protection overlook the elephant in the room. Comparing what occurs in AR in overwhelmed child protection systems to an ideal investigative standard outlined by some DR critics leads to the false assumption that eliminating DR would result in uniformly rigorous CPS investigations. In fact, investigative standards erode and eventually collapse under chronic and extreme workload pressures in all child protection systems.
Is DR part of a family preservation agenda?
Some DR critics (Bartholet, 2016) maintain that DR is part of a family preservation agenda
promoted by influential organizations such as Casey Family Programs. The most recent example of such a connection is Connecticut's 19% reduction in foster care during the 8 years (2011-18) Joette Katz was the state's child welfare director. However, Connecticut's experience with DR is an exception rather than the rule. In Fuller's, et al's, evaluations of DR, there were no statistically significant difference in placement rates between AR and TR tracks in Colorado, Illinois or Ohio. In Loman's and Siegel's study of DR in Ohio, the AR placement rate was 2% lower than the TR placement rate (9.8% vs. 11.8%) during implementation and follow up.
Several state child welfare systems, including Washington's, have adopted DR with the hope of reducing their foster care populations. Policymakers in Washington State were persuaded to
support DR with additional funding because of the expectation that DR would prevent entries into foster care. Washington's use of FAR has not reduced the number of children in foster care, much to the chagrin of some legislators.
DR, with occasional exceptions (e.g., Hawaii), rarely has much of an impact on CPS placement rates. When DR does reduce the placement rate for AR families, the reduction is rarely large enough to have much effect on the size of the state's foster care population. In Loman and Siegel's study of DR in Ohio, the main effect on placement rate was “among the minority of families with a history of child removal,” (23.3% for AR families vs. 32.6% for TR families), a minority of families in the study. DR rarely has much of an effect on the size of a state's foster care population for one main reason: placement rates in AR eligible families are low to begin with, for example no more than 3-4% in some studies. Even if DR resulted in a large percentage decrease in foster placement among AR eligible families, the effect on the number of children in foster care would be minimal, unless large numbers of high risk families are being assigned to the AR track.
Changing the focus of DR evaluations
It is time to move on from the discussion regarding whether DR is safe or unsafe, a positive reform or a misfortune for maltreated children and their parents. DR, like traditional investigative practices, can be done well or poorly. The angry insistence of critics that DR is inherently dangerous has not been confirmed in research, yet there is good reason to pay attention to the concerns of well informed professionals in a number of states regarding misuse of DR.
DR research should refocus on the questions, “What is the optimum use of DR?” and “For what types of families is AR poorly suited?” Darnell and Fluke's research suggest the possibility that an optimum AR utilization rate can be identified, and that intake screening practices can be adjusted to these research findings. It is also possible for states which have chosen not to utilize DR to learn from family engagement practices widely used in DR systems that lead to better family engagement and more positive responses from parents to CPS interventions.
Elizabeth Bartholet, one of the best informed and most articulate DR critics, has asserted that “We also need to do some version of DR but within the framework of the traditional CPS system.” (2016) This comment suggests that a productive conversation between DR critics and proponents may – despite appearances- be possible.
References
Bartholet, E., “Differential Response: A Dangerous Experiment in Child Welfare,” APSAC Advisor, Vol. 28, No. 2, 2016.
Darnell, A. & Fluke, J., “Optimizing Differential Response Thresholds: How Much is Enough?” presentation to Society for Social Work Researchers 18th Annual Conference: Responses for Social Change: Addressing Local and Global Challenges, January 16, 2014.
Fluke, J,. Harlaar, N., Heisler, K., Darnell, A., Brown, B. & Merkel-Holquin, L., “Differential Response and The Safety of Children Reported to Child Protective Services,” APSE Research Brief, Office of the Assistant Secretary for Planning and Evaluation, U.S. Department of Health and Human Services, June 2016.
Fuller, T., Ellis, R. &, Murphy, J., “Examining Outcomes of Differential Response: Results from Three Randomized Controlled Trials,” presentation at 19th annual Conference on Child Abuse and Neglect: Research, Policy and Practice, May 1, 2014.
Hudson, M., “Minnesota's Experience with Differential Response,” APSAC Advisor, Vol. 28, No. 2, 2016.
Hughes, K., “Pioneer Institute: To Ensure Child Safety in Massachusetts, Most Critical Reforms Are to State's Differential Response Program, APSAC Advisor, Vol. 28, No. 2, 2016.
Hughes, R. & Vandervort, F., “Differential Response: A Misrepresentation of Investigation and Case Finding in Child Protective Services,” APSAC Advisor, Vol. 28, No. 2, 2016.
Loman, L.A. & Siegel, G., “Effects of approach and services under differential response on long term child safety and welfare,” Child Abuse and Neglect, 39, 2015.
©Dee Wilson, 2018
To access archives:
deewilsonconsulting.com/writings