How Big Data Could Reshape the American Workforce

Lesley Hirsch sits at her computer in a sunny, quiet office at the City University of New York Graduate Center, using Help Wanted Online software to cull through more than 16,000 job boards across the web. Clicking softly on her keyboard, she zooms in, zooms out, pulls up jobs on a national level, then a local level, narrowing her focus to a single wanted ad for a used auto parts salesperson in the Bronx. Next she calls up the number of openings for registered nurses with three to seven years' experience in the metro area. On the day VICE News visited Hirsch, there were 2,000 positions that fit her criteria.

The exact figures aren't important to Hirsch, the director of the New York City Labor Market Information Service, or LMIS. In her eyes, it's the trends that are most useful, and she sees its value primarily as a way of "keeping your finger on the pulse of industry demand."

For example, Hirsch says that of those 2,000 nursing positions, the more interesting number is that half of them require a bachelors' degree, showing the steady march of "credential creep," requiring higher and higher levels of education than were previously needed for a given job.

It's big data in action: Think prediction as pointillism, where we hope that millions of individual actions, taken together, can paint a picture of a larger trend. The more of our lives we live online, the more of our behavior leaves tracks we can trace. Those tracks can then tell us something, for example, about the spread of the flu, or the American wardrobe's long arc toward sweatpants. The hope is that, in the nooks and crannies of our lives and economy that surveys do not or cannot measure, lies an aggregate truth.

Since beginning this work in 2008, Hirsch has found herself on the vanguard of taking a data-driven approach to workforce development; she says the LMIS, a joint project between the City University of New York (CUNY) and New York City, were among the first in the country to use data scraping tools in "a really analytical way." But the thing that most surprised her when she started the LMIS seven years ago was that something like it didn't already exist.

"It was shocking to me," she says. "And it's shocking to me that it's the exception not the rule."

A few subway stops from CUNY is the Grace Institute, a 115-year-old non-profit that provides New York City women with cost-free, six-month job training, where Joan, a Jamaican-born long-time New Yorker, is training for a new career in health care: patient service representative, or PSR. Joan, who is 57 and asked that her last name not be used, has worked in the healthcare sector. But the work had slowed and she knew it was time to make a change.

Joan is part of the first class at Grace that will graduate in the PSR training program. The program started as a collaboration between Hirsch and Grace's executive director, Shari Krull, who together did "a huge deep dive over six months into labor statistics," says Krull. These included scraped data and traditional numbers from government sources.

Together, they looked for a "sweet spot" that would combine Grace's success in training for entry-level administrative jobs with a growing sector that offered room for people in those entry-level jobs to grow too. Ultimately, Grace zeroed in on the healthcare sector and used Hirsch's data to design a new program to train patient service representatives (PSR), with a curriculum that combines the broad business skills training Grace has long taught so well with more specialized instruction on skills like processing insurance information and scanning medical records.

But Krull says the data also led to smaller, more surprising insights. For example, she says, it suggested they calendar their programming so that their graduates enter the job market in months when hiring is booming.

"It totally makes sense," she says. "But you don't think about those things until you start looking at the data."

Big data's move into the world of workforce development is just beginning, as are the debates around its use and its content.

In those long-ago days before social media, workforce development has always relied on government data on labor supply and demand - The Department of Labor's O*Net, for example, is an expansive database that catalogs skills, titles and job descriptions for over 900 occupations, making it, unsurprisingly, a key resource for those in the workforce development field. But while it is "continuously updated," only around 120 of those occupations are actually updated every year. This means that, for some portion of those listed skills and jobs, O*Net's data could be several years out of date. Anthony Carnevale, director of the Georgetown Center on Education and the Workforce, describes government collection of data as "careful, competent and slow," a description that speaks to both its strengths and weaknesses. The data is held to rigorous standards but can take too long to collect and to change with the marketplace, training people in emerging fields or new positions. Workforce development programs still very much rely on those careful, competent, slow resources like O*Net, but they want more — more data, and the more local, the more detailed and the more current, the better.

Enter sites such as LinkedIn. In April, when the White House unveiled its new TechHire initiative, a set of actions meant to encourage training and close the skills gap for tech jobs, LinkedIn was brought on as a prominent partner. Pablo Chavez, LinkedIn's vice president of global public policy and government affairs, said that the company "agreed to provide insights based on data from LinkedIn to the participating cities and regions to help them get an additional useful data point about skills gaps and so forth."

Especially optimistic about this data's potential is Aneesh Chopra, a fellow at the Center for American Progress and formerly the nation's first chief technology officer, under President Obama. While he recognizes the tensions between public and private collaboration, Chopra has a vision for a labor data co-op that could combine traditional government numbers with nimble, real-time data from private sites. He likens the possibilities for personalization and granularity to the promises of the human genome project or, more modestly, to online shopping: When you visit Amazon, he says, it tells you what book or lawn furniture you might like. Why not get to that level of personalization for training and employment, where you could receive recommendations just for you of jobs to apply for or courses to take? While individual sites like Glassdoor might send their own daily job recommendations, what Chopra envisions is on a much grander scale; because it would draw upon a much wider set of data, the results would hopefully be that much richer and more precise.

"In a world in which you now have ubiquitous low cost access to the raw data," Chopra told VICE News, "theoretically, we could and should start to see a far more precise way of understanding labor market demand, labor market supply, and the training programs that could help connect where you are to where you want to be, depending on your ambitions."

Of course, there are pitfalls: LinkedIn has an enormous and growing user base, with almost 350 million members, and therefore a huge set of data about their skills and job histories. Yet when people are deploying the LinkedIn skills tool to endorse their friends for "Using the F-bomb," it's not hard to see what the pitfalls may be when working with self-reported data. Government-collected numbers might be a lot more delayed, but they're also a lot more rigorous. You can't endorsement-bomb someone on the Census.

"We've got all kinds of issues with information now privacy, security," says Carnevale, co-author of a report on this issue. "The one that doesn't get talked about much is quality."

His skepticism is well placed. In Canada last year, the Conservative government found this out the hard way. Using numbers collected via Wanted Analytics — the same software (though, one must emphasize, an entirely different data set) that Hirsch uses to look at New York City— the government claimed there was a rise in job vacancies. However it was quickly pointed out that this supposed rise was in fact due to duplicate or unreliable job postings on the free classified site Kijiji. If you've looked on a site like Kijiji before, you'll know that duplication is really, really common.

LinkedIn's Chavez is aware of the problems associated with self-reported data. "For us, the first step in working with government partners is to be completely transparent about what insights LinkedIn data can, and cannot, provide," he said.

Melinda Mack, executive director of the New York Association of Training and Employment Professionals, jokes that she could go on LinkedIn and declare herself "President of Albany."

But that doesn't diminish Mack's excitement about Hirsch's work, which she calls "invaluable." In her opinion, more access to good, granular data, from the government, or from shops like Hirsch's, will only make their work better. Often, she explains, traditional numbers can be too big or too slow to give a real sense of the needs of local industry, so combining them with real-time data can be crucial to help close those gaps.

"For example," she says, "you get these quarterly reports or monthly reports on jobs. And we often are saying, Great, who has those jobs? Right? Where do they live? What kind of jobs are they? Are they good jobs? Are they shitty jobs?" She laughs, and adds, "I have no clue. I have no clue when I look at that data."

But some worry that big data's sheer volume can give it an aura of truth that makes it difficult to question. "People tend to accept data uncritically," Carnevale told VICE News. "There are a lot of people that use this data and think it's gospel, and it's not."

"There's no such thing as raw data," says Alex Rosenblat, a researcher with Data & Society and co-author of a report on the future of labor warns. "There's always a bias in collection, there's always a bias in how you're going to use it."

She thinks programs like those Chopra envisions could be "really cool," but need to be approached with caution. "The concerns there are similar to other concerns about personalization tools, which is: Will you only be channeled into preconceived notions about your potential based on your data?"

But Mack and others in the field know that data alone is not what gets people to work. As Hirsch said of the algorithms she works with daily, "You really can't expect them to be magic bullets."

"Data can get us 85 percent of the way there," says executive director Krull. "And then you have to talk to employers. […] Because the jobs can be there, but if people can't get through the interviewing process, it doesn't matter."

Watch the first episode of the VICE News series The Business of Life here:

On the morning VICE News visited Grace, students were running through mock interviews with volunteer professionals. The building was full of buoyant, chattering women in sober, business attire. The halls, lined with lockers and inspirational aphorisms ("If you're going through hell, keep going"), hummed with enthusiasm and nerves. One young woman came down to the lobby, grinning that her mock interviewer had actually given her a business card and asked her to stay in touch.

But that data is still making a difference in the lives of women like Joan, the Jamaican-born woman training to become a patient service representative in the health care field. Krull says she sees lots of women like her, who might be "stuck" professionally, even with years of experience. The challenge is to get a handle on the needs of a rapidly changing workforce and then train their students to fill those new needs, which is why Krull says they've looked so much at the labor data.

"Great! You're in healthcare," Krull says. "But what's happening in healthcare so that you can readjust your skills to an area that's booming."

For her part, Joan said she was "a bit worried" about her own mock interviews. "But," she said, "I'm going with the attitude of, "What have I got to lose?"

It's a boldness she attributes to Grace and a reminder of the real but often immeasurable effects of programs driven by hard data. When discussing the change she's undergone and the example she's become to her children, Joan tears up.

"I have a lot of skills and I have a lot to offer, but Grace brought that out of me," she says. "I'm telling you, it's transforming. Being here is transforming."

Photo via Flickr