While AmericaServes is geared toward helping veterans and their families, other similar service networks have emerged in the US with different focuses: mental health, family and children’s services, or homelessness, for example. The overall aim of service networks is to address the often neglected nature of service provision by integrating medical, social and other services to make it easier for people to access and benefit from them. After all, multifaceted issues require multifaceted interventions.
“These new cross-organizational arrangements are trying to get people to the right benefit plans and services they’re eligible to receive,” he says. Michel SchumatNorthwestern professor of communication studies and associate faculty of the school Institute for Policy Research.
“I grew up without a lot of money,” he continues, “and the people around me often didn’t know where to turn for help. These networks of care systems are about process improvement, to provide the guidance that people need.”
But how well do they carry out their mission?
That’s the overarching question addressed in new research from Shumate and Karen SmilovichKellogg professor of business, along with Northwestern collaborators Zachary Gibson, Mariana Escallon-Barriosand Joshua-Paul Miles—and four authors from Syracuse University. The study was funded by a grant from the Army Research Office.
They worked closely with AmericaServes to better understand the nature of service networks and how to evaluate network success.
The researchers found that different metrics were correlated to measure performance: directing users to the correct service provider was correlated with users actually receiving services, for example. However, they also found that performance measurement in these networks is highly context dependent. Services of high complexity will inevitably underperform on some metrics, so it is not always useful to focus only on which networks are top performers on these dimensions. In fact, if funders make decisions based only on, say, how quickly users are served, they may be incentivizing networks to prioritize issues that are easily solvable.
This is a new area of study, says Smilowitz. “Our study is among the first to leverage real-time data to understand how the work of care systems affects their overall performance.”
Partnership with AmericaServes
The team partnered with AmericaServes for their study because it is a broad network that addresses a range of service requests, including more complex ones.
“My husband is a veteran and both of my grandfathers were veterans. I know the frustration of trying to get benefits and calculate eligibility in the VA system,” says Shumate. “Even the people who work in these networks are finding it difficult to keep up with an evolving set of programs and eligibility requirements.”
Both she and Smilowitz were impressed by AmericaServes’ advanced technology systems, which gave researchers access to every referral and service episode in recent years. They also participated in informal discussions with AmericaServes employees as part of the study. “By interviewing AmericaServes staff we were able to get the context behind the data and interpret it, which led to fantastic insight,” says Smilowitz.
They studied 30 days of service episode data across all 11 AmericaServes networks — more than 1,500 service episodes in total — from early 2020. The researchers focused on three key measures: effectiveness, or whether a particular person seeking services actually receives some form of service; efficiency, measured as the number of days before a person starts receiving care; and accuracy, or whether a service provider to whom a person is referred accepts the referral, meaning they are the right provider for that person’s needs. While previous studies of service networks have examined effectiveness, efficiency and accuracy are newer variables of focus in this research area.
“These metrics really help us understand how things are working in this network right now, such as understanding the trade-offs to achieve efficiency, effectiveness and accuracy,” says Smilowitz.
Complexity matters
Overall, the results showed that the service network performed well, with 72 percent of referrals resulting in a resolution of the expressed need and 88 percent of referrals being accepted by the first provider in an average of just under three days .
The researchers also looked at correlations between effectiveness, efficiency and accuracy. They found that accuracy and efficiency were positively correlated, suggesting that referrals accepted for care on the first try generally resolve the individual’s request. However, effectiveness and efficiency were not correlated, meaning that getting into a system faster does not necessarily lead to a resolution of the request.
At the same time, the researchers saw that context matters a lot—that is, expectations of effectiveness, efficiency, and accuracy must take into account the type of service in question.
For example, many of the services needed are of low complexity—for example, food and clothing. Based on study data, such low-complexity services took an average of only 1.3 days to handle, with 93 percent accuracy and 78 percent efficiency. But more complex services, such as housing and legal services, saw throughput numbers rise to almost 6 days and accuracy and efficiency drop to 71 percent and 57 percent, respectively.
“You might be quick to label a particular network as too efficient or too accurate,” says Smilowitz. “But if you really look at the composition of their service load, it has a significant impact. So you can’t just say “This is a good score for accuracy” or “This is a good score for efficiency” without knowing the context. It was a big moment for us.”
Better policy and coordination
The findings here have important policy and operational implications.
“There are now a growing number of state and federal efforts to create and maintain these kinds of networks,” says Shumate. “They want metrics attached to them, and if they’re primarily based on efficiency and effectiveness, they’ll incentivize low-complexity services. This will incentivize ‘skimming the cream’, where networks focus on the easiest options, such as working for food rather than housing, which is a complex but critical service on which many other things rely.”
Another potential application of the findings, Smilowitz explains, is that “the work can help predict what kinds of needs might arise, and when, among veterans and their families. So we can think about that kind of sequence of services that we should provide, based on that.”
