Workplace Stories Season 4, Skills Odyssey II: Paying for Skills and Much More with "Trustworthy AI"

Guest:

Anshul Sheopuri, Vice President & CTO, IBM Workforce

DETAILS

This week, it’s all about numbers, scale, and achievement. In terms of numbers, how about a Skills-based, AI-enhanced framework that is keeping 250,000 employees happy and appropriately paid? And which saves the company an estimated $100m per year, money avoided by avoiding expensive churn and not paying beyond market rate—even for scare capability? And as for the achievement, the spotlight in this episode is on Anshul Sheopuri, Vice President & CTO, Data & AI At ‘IBM Workforce,’ Big Blue’s immense global HR function, where he’s led the work on using Artificial Intelligence, Machine Learning, and lots of data to improve hiring, compensation and even DEIB policies across the company. So important is this idea of ‘Skills-for-pay’ and ‘Skills as currency’ that he sees it as a ‘silver thread’ unifying people processes and practices … which of course we soon correct to a ‘red thread’! We’ve been looking to meet with Anshul for a long while, and we’re glad we hung in there, as this is an excellent conversation with a true subject matter expert who’s using tech to really make a bunch of positive changes for his colleagues. A really interesting piece of best practices you could start looking at right away is using employee digital footprint to see what their Skills really are. Sadly, Stacia never got the AI help with tonight’s dinner she thought she’d get, but hey. You can’t have it all.

Resources

• Mr Sheopuri’s LinkedIn is here.
• Recent IBM blog Mr. Sheopuri wrote on the issues discussed in the episode here.
• Details on the IBM AI platform, Watson, can be found here.
• The industry initiative around trustworthy AI he mentions in the discussion is here.

Webinar

As with all our seasons, there will be a culminating final live webinar where we will share our conclusions about the show’s findings. As ever, we will share details of that event soon as it is scheduled in early 2022.

Partner

We're also thrilled to be partnering with Chris Pirie, CEO of Learning Futures Group and voice of the Learning Is the New Working podcast. Check them both out.

Season Sponsors

 

 

We are very grateful to our second ‘Skills Odyssey’ sponsors, Visier and Degreed. Visier is a recognized leader in people analytics and workforce planning; with Visier, organizations can answer questions that shape business strategy, provide the impetus for taking action, and drive better business outcomes through workforce optimization. Visier has 11,000 customers in 75 countries, including enterprises like Adobe, BASF, Electronic Arts, McKesson, and more. Degreed is the upscaling platform that connects Learning to opportunities; they integrate everything people use to learn and build their careers, Skills, insights, LMSs, courses, videos, articles, and projects, and match everyone to growth opportunities that fit their unique Skills, roles and goals: learn more about the Degreed platform at degreed.com. We encourage you to show your support for their involvement by checking out both websites—and thanks once again to both organizations.

All three previous seasons of Workplace Stories, as well as our series on Purpose, which was a co-production with the ‘Learning is The New Working’ podcast, along with relevant Show Notes and links, is available here. Find out more about our Workplace Stories podcast helpmate and facilitator Chris and his work here.

Finally, if you like what you hear, please follow Workplace Stories by RedThread Research on your podcast hub of choice—and it wouldn't hurt to give us a 5-star review and share a favorite episode with a friend, as we start to tell more and more of the Workplace Stories that we think matter.

TRANSCRIPT

Five Key Quotes:

All professions and industries are being transformed with digital. And that really means personalization and AI as its centerpiece.

What we've been on is a journey to reconfigure the entire employee lifecycle from the way we hire, the way we grow, the way we promote, the way we learn, the way we pay our employees with Skills—what we call as a ‘silver thread’ that connects the entire employee journey. And AI is enabling the personalization that employees then receive through that journey.

How you should get started on this journey: think about the business problem that you're trying to solve, quantify it as much as possible. Ultimately, the best solutions lie in the intersection of a great business problem, because that's where they’re needed, and a great user problem: does the user really have the need to solve the problem, or is it just a nice to have for the user? Clarifying that is very important.

Using AI across the employee journey, what we see is about a $100 million cost savings per year to the IBM company. And proactive retention, the program that I talked about, is an important component of that. This is a significant return.

Every now and then, I'll get an email from an IBMer who says I was in a role where I was not clear what I was doing, I was looking for another opportunity, and this learning nudge or this job notch helped me realize my career passion. You get goosebumps when you get such a note; it just makes you feel so proud, makes you really feel a sense of purpose and worth. And that's what I think this is about for me—it’s impact at scale, and impact for the entire workforce for IBM, but also the broader industry.

You are listening to Workplace Stories, a podcast by RedThread Research about the near future of work.

Stacia Garr, RedThread:

I'm Stacia Garr, co-founder and principal analyst at RedThread.

Dani Johnson, RedThread:

And I'm Dani Johnson, co-founder and principal analyst at RedThread.

Anshul Sheopuri, IBM:

And I'm Chris Pirie, CEO of The Learning Futures Group.

The team at RedThread Research would like to thank Visier and Degreed for their sponsorship of this podcast, The Skills Odyssey II. Degreed is the upskilling platform that connects learning to opportunities; it integrates everything people use to learn and build their careers—skill insights, LMSs courses, videos, articles, and projects—and match everyone to growth opportunities that fit their unique Skills, roles and goals. Visier is the recognized leader in people analytics and workforce planning; with Visier, organizations can answer questions that shape business strategy, provide the impetus for taking action, and drive better business outcomes through workforce optimization. Visier has 11,000 customers in 75 countries. For more information, you can find links to their websites in the Show Notes for this podcast.

Anshul Sheopuri, IBM:

It all comes down to how do you infer somebody’s Skills, and the way we do that is through digital footprint—we look at things like your resume; your performance management; your performance goals that you've put in the system; your badges that you've got; the learning that you've done; if you're a seller, your sales system, so systems of work; your public Slack, social, public, social, internal comments that you might have had; your internal blogs if you're writing: all those different digital footprints helps u then infer your Skills. And then we would share back with you as an employee: we think your Skills are, you are an expert in this, and this is the reason why we think you're an expert. You can agree, and you can choose to disagree as well. We have about a million or so feedback points from IBMers, but then there's full transparency around the Skills ecosystem. You get to react to it, that helps us give you personalized job recommendations, as an example. It also helps us with better workforce planning, because now we understand depth and can make much more informed assessments around where there are gaps and where there are gluts that we need to solve.

Chris Pirie, The Learning Futures Group:
That's today's guest, Anshul Shapouri, who is vice president and CTO of IBM Workforce. And his use of the phrase ‘Skills ecosystem’ as well as those worked examples that he shared in that clip is a clue to the quality of the great conversation we have coming up. Thanks to his incredible breadth of insight, his depth of experience that he's gained helping lead IBM's comprehensive approach to Skills.

Stay with us, and you'll hear specifically how and why AI has become a critical enabler in the Skills journey, but you'll also learn what factors really contribute to a trustworthy AI strategy. You'll also hear how he sees Skills as the ‘golden thread’ that transforms and links the learning and hiring of a workforce, and even pay processes. He'll give concrete examples and share his insights and learnings and the failures they've had along the way. There's so much experience, learning and advice bundled into these next 50 minutes, I know you're going to find it super valuable, so let's get right into our conversation with Anshul Sheopuri from IBM.

Stacia Garr, RedThread:

Anshul, thank you so much for joining us today and welcome to Workplace Stories; we’re so excited to have you on the podcast today! We’ve obviously been trying to get to this moment, so the whole team has been really working to have this conversation. So thank you for being here!

Anshul Sheopuri, IBM:

Well, thank you for having me; I’m excited to spend this time with you.

Stacia Garr, RedThread:

Well, we’re going to start off with some quick questions to introduce you and your work practice to our listeners, and then we'll go deeper into some areas that we'd love your perspective on. So just to get us started, can you tell us a little bit about IBM—everybody’s heard of IBM, obviously, but can you give us an introduction into what is IBM today?

Anshul Sheopuri, IBM:

I think that's a great question. IBM today is a hybrid cloud and an AI platform company. Let me tell you what that means: you might be making a hotel reservation or an airline booking on your favorite website; you get to engage with a digital agent who answers some questions. That might be powered by AI that IBM may have built. Think about telemedicine; how do you make that happen in today's world? Well, you require infrastructure and secure and hybrid cloud platforms to make that happen? Think about your personal banking experience when you're depositing a check or doing a transfer. Well, that requires a personalized, seamless experience that needs to be secure and rich, and that digital banking experience could be something that IBM provides. So, essentially, IBM today helps enterprises accelerate on their digital transformation journeys using hybrid cloud and AI.

Stacia Garr, RedThread:

And then tell us a bit about yourself, because you're right in the middle of all of this. So what's your title and how would you describe the work that you do?

Anshul Sheopuri, IBM:
So I am a vice president and chief technology officer for IBM's workforce. And it's a super-exciting role; it’s an exciting role because one in, as you probably very well know, these roles didn't exist a decade or so ago. So as technology has transformed and the digital transformation journey I just told you about that IBM is helping lead, that’s something we see in the HR domain as well. And that's my role to be the custodian and the accelerator of our digital transformation journey, making sure that data is used appropriately for the right decisions—ultimately helping IBM have a more skilled, diverse, and engaged workforce. And that's what I make happen through data and digital.

Stacia Garr, RedThread:

In one of our previous conversations, you mentioned that you've also had a little bit of time with something you called Chef Watson, and I think people would love to hear about that. Can you tell us a little bit more?

Anshul Sheopuri, IBM:

I've got to tell you Stacia, upfront, that was the most fun project I've got to work on. I got to work with these great chefs, and I never knew that food could be so, so tasty. Let me tell you about how it happened. So I have worked in AI for the past decade or so in a variety of different industries and domains—financial services, healthcare, digital marketing, retail food, as well as fashion. And, if you were seeing my clothes, you couldn't tell that I worked in fashion, but trust me, I have. And the premise is the same: all professions and industries are being transformed with digital, and that really means personalization and AI as its centerpiece. And that's what we were doing with food, where we were positing that an AI can come up with new recipes that don't exist, that are flavorful, that are surprising, and using the chemistry of food as its basis, as well as a repository of recipes, historically recipes to learn from, that’s what we worked on. And if you think about the implications of something like that in an era of simultaneously food scarcity, but also obesity in some areas, having nutritional food options and implications for flavor and fragrance companies and scientific discovery, their massive implications for a technology like this.

Stacia Garr, RedThread:

Well, I think my family would appreciate it if we had this technology at home, particularly since today happens to be my dinner night, so could we get that over here today? Very cool 🙂

Well thank you, for sharing about that. Let's maybe just jump right in to what we're talking about today, and part of the reason that we were so excited to have you on the podcast season is that IBM has been doing this work on Skills—the focus of this Season—for a really, a very long time. So can you talk to us about why that's the case and what has initially brought about this focus on Skills at IBM?

Anshul Sheopuri, IBM:

Absolutely. And like with all good transformations, this was not something that was cooked up as, “Hey, HR needs to move towards Skills.” It was not an HR imperative; it was a business imperative, as I talked about digital transformation journeys, all companies are going through. Today, when you go and talk to colleagues or friends of mine at an oil and gas company, or a retail store, or an industrial company, they all require design skills, software engineering skills, data analytics skills, security skills: all companies are going through digital transformation journeys. And that's because they're having to do it because their business models are being disrupted, and technology is changing rapidly. So all companies are going through that, and that has implications for Skills that nearly all companies need.

The other big shift that's occurred at the same time, Stacia, is our experience with day-to-day consumer experiences —so the way we hail a cab, the way we watch our favorite movie in the evening, that has really become simplified and personalized. And when you think about how you deliver these consumer grade hyper-personalized experiences, you need AI to do that, that’s what AI gives you. That's the single biggest benefit of AI: giving you very, very personalized and tailored experiences. Now you combine this AI and Skills together, and we realized that five to six years ago that that's the journey that we needed to take internally within IBM for our own employee lifecycle. And so what we've been on is a journey to reconfigure the entire employee lifecycle from the way we hire, the way we grow, the way we promote, the way we learn, the way we pay our employees with Skills, what we call as a ‘silver thread’ that connects the entire employee journey, and AI enabling the personalization that, that employees then receive through that journey.

Stacia Garr, RedThread:

So it sounds like—and correct me if I'm wrong—but it sounds like it began in many ways as a realization that we need this in order to serve what our customers need, where really kind of the future is going, and then given that capability, we absolutely need that for our own folks in order to enable them to meet those needs. Is that a fair interpretation?

Anshul Sheopuri, IBM:

Absolutely, absolutely, that’s how it started. As a company, we felt we needed this internally and our clients were also going to the same transformation journey—they needed it as well.

Stacia Garr, RedThread:

Yeah. Definitely. Let's talk a little bit, then, about some of the details of this. I think you gave us kind of a nice high-level sketch of what's happened, but let's talk a little bit about the specific application. Can you maybe start at the beginning? You mentioned, overall what, where it was, but like really, how does this get started, because a lot of folks are really at the journey, even maybe a sophisticated HR leader like yourself, who says, we need to bring this in, but I don't know who to talk to, how to make this work. And obviously you were different in that you had the consumer side where you already focused, but what do these conversations look like? And then what are the major milestones to know that you've made some progress?

Anshul Sheopuri, IBM:

The first place that we started, Stacia, several years ago, was learning. We realized that the way people learn was changing dramatically, and Skills was—people don't learn just for the sake of learning, they’re learning to acquire new Skills. What that resulted in is, just think about how you watch a movie, on your favorite video streaming platform—that’s how we want IBMers to learn. So having personalized learning recommendations focused to you based on other learning that people like you do, based on the Skills you have, based on the Skills aspirations that you have. We started service learning recommendations with that nice tile-like I like video streaming experience. And today that platform that we have called Your Learning is widely adopted within IBM—98% of IBMers use it every quarter. It has a net promoter score of 58, and breaking that down a bit, net promoter score is a very high bar, nine and 10 are your advocates, seven and eight on a 10-point scale are neutral, and six in below detractors and NPS measures and percentage of advocates minus detractors, so you really need to have a really high percentage of advocates to get such a high score.

And so that's where we started, because that's where we saw the need at the beginning. The other thing that we also realized, Stacia, is this is not just about a new digital experience; this is about transforming the way people work, the way IBMers work, the way HR practitioners work. So when you get these digital nudges… you may have a bot, now you have to engage much more digitally first as a user. And therefore, we needed to really understand user needs, go through the design thinking with our sponsor users, co-create with them, and we needed to understand where their workflows were and how we need them in their workflow. We needed to also reinvent the way HR teams work. So learning consultants before, their job was administer courses, show up and administer courses, make sure that X percent of IBM must show up in a certain class; it was really administration and compliance. We needed to shift the way our teams work; we needed more content creation and curation, we needed design Skills, we needed cloud Skills, we needed data analytics Skills to ensure that we understand the efficacy of learning on Skills. Are people picking up new Skills? Are we deploying those individuals to our client accounts, is value being realized to clients? So a shift in mindset and Skills and roles was needed within the HR function also to make that happen. And then we've seen the at pattern play out in different domains, not just learning, but in the way we hire, pay, grow et cetera as well.

Dani Johnson, RedThread:

I’d love to follow up with a couple of questions there, Anshul, if that's okay? The first one is, why did you start with learning? And the second one is, I'm talking to a lot of organizations that are finally to this point—it sounds like you're seven or eight years ahead!—they’re finally to this point, but they're having trouble getting buy-in from not just the organization, but the learning team. How did you do that?

Anshul Sheopuri, IBM:

So why did we start with learning? When we poll IBMers as to what their highest need is, and today, as we look at what is the single biggest driver of engagement as well as retention, the single biggest driver, unequivocally, in all our surveys and all our analytics turns out to be career growth and clarity of career growth. And if you think about learning, that's the fundamental core pillar of how we help IBMers get that career growth. Since then, since we started with learning, we've now connected the dots between the different experiences of career. So learning, mentorship, Skills, job opportunities: all those are different point experiences that make up a great career experience together. And so I gave you the example of Your Learning and it's adoption, but since then we've connected the dots. So let me just make up an example: You are somebody who is interested in Design Skills; you take a digital learning class; maybe you're interested in a mentor, who's a designer. And a nudge service and recommends to you a mentor or that skill that you'd like to pick up. You also, once you've been mentored, you've now taken a few more classes, you’ve gotten some badges in design, a job recommendation surfaces up within IBM for a job in that area. So connecting all these dots in a single digital experience, and today thousands of IBMers find jobs within IBM based on job recommendations that we surface to them in the same experience. So connecting the dots across this journey was also very important.

Then to your second question around buy-in; I do think it's an important place to start if you focus on the business problem and the user problem. This is not about a technology play alone; this is a play around what is the problem we're trying to solve for the business in order to grow revenue or to improve profit, whatever your business objectives might be, what Skills do you need, and what Skills do you have, and how do you achieve that gap, so what value does this experience add to your business performance? I think it's an important characteristic to start from that conversation, versus just a HR productivity conversation. Obviously there are benefits also from an efficiency standpoint, when you embark on a digital journey and you could have productivity gains as well as we've had in some many places.

Chris Pirie, The Learning Futures Group:

You mentioned something really interesting—you talked about this transformation, Anshul, and obviously technology is very much at the center of it, but you also talked about mindsets and nudges, and these are words that you associate with cultural change as well as a technology change. Can you say something about how those two things work together—how did you lead employees through the cultural change that was needed as well?

Anshul Sheopuri, IBM:

There are many facets of it, right? One is you need leadership at the top to be clear, direct, visionary, and be exemplar users of the transformation. This is not something that you're in a vacuum, “Hey, this is a good use of technology but now it's up to you to use it.” Showing up, being visible, being clear that we expect and are asking our employees to learn. So today we say you should be spending at least 40 hours a year learning. And on average, we see that number is 70-plus every year. So being clear, visible at the top from a leadership standpoint, I think, is an important piece of it.

Second, I would say is Skills, emphasizing Skills development as a fundamental currency that employees have. It's not just something that's intangible, it's something we have conversations about through the year as part of career conversations, it’s part of our performance management system, it’s the currency through which we operate—and it’s just not a word. It's something that's so important for our clients because ultimately we're in the business of helping our clients accelerate their digital transformation journeys. And you do that through technology and through expertise. So Skills therefore become the currency for us internally, but also with our clients.

Chris Pirie, The Learning Futures Group:

That’s really fascinating, because I don't think we've spoken to anybody yet who's taken this idea of Skills and used it in the context of speaking to customers and prospects. So is that language of Skills being used in how customers engage with IBM?

Anshul Sheopuri, IBM:

Absolutely. And see, we're a B2B company, so we work with enterprises, so therefore is so much more visible to our clients versus if you're a B2C company and therefore it's so much more clear what Skills are needed, and what Skills we need to have. The final thing I'd say is to think about this change as design-led versus technology-first. So this is really about what you want, and making sure that it's something that has pull versus push and having a change management mindset that is a lot more about exhibiting the right behaviors in our conversations, the growth mindset, being courageous and having that, what are users really asking for, and playing that back to them in terms of these digital experiences.

Stacia Garr, RedThread:

Let's continue with the, you called it a silver thread. I'm going to call it a ‘red thread.’

Anshul Sheopuri, IBM:

No pun intended!

Dani Johnson, RedThread:

None at all!

Stacia Garr, RedThread:

Talk a little bit about hiring, because that is one of the other areas that you're using Skills very heavily, and we obviously know a lot of folks are focused in this area right now particularly given the tight talent market. So can you talk to us about how you're using Skills in your overall hiring approach?

Anshul Sheopuri, IBM:

Absolutely. And what I would just anchor around first, Stacia, is that we try to make sure that Skills is not something that—it’s a silver thread, but is in conjunction with a few other important ways in which we apply Skills and that's around selection and trustworthy AI. So it's about Skills, but also making sure we're making fair and unbiased decisions through all these journey points so at the end of it, you're getting the right quality of hires, but also diverse candidates at the back end. And that permeates the entire hiring process: the way we source candidates—we want to make sure the universities that we're looking at, as an example, have the right Skills in AI, in cloud and security, in blockchain and the other areas that are important to us, but are also diverse. They have the right candidate profiles, so we're getting the right pools of candidates into the pipelines, and we look at internal and external data in order to make that happen.

We're also very cognizant that in some areas, you may not have a diverse slate. Then, we should be helping recruiters by giving them nudges, “Hey, did you consider this candidate, the slate is not diverse—maybe you want to go back and look at the slate, and maybe explore a couple of other candidate pools?” And we might give them some examples of that. So this notion of in the workflow nudges, because, people all have the right intent, but we're all busy, and so in the workflow reminding recruiters, “Hey, the slate may not be as diverse as you would like, go and have a look at it,” helps the recruiter be much more effective and efficient.

Stacia Garr, RedThread:

I want to jump in there, because in the previous episode that we recorded, we talked to someone else about this problem, and he made a really good point, which was that, historically, we’ve thought about talent acquisition and recruiting as a filtering problem, but instead we need to be thinking about it as a matching problem. So if we have the technology and particularly someone like IBM does with the AI to understand, okay, like these candidates who on the face of it, they're a, let's say an 80% match, but they score higher in some of these other areas—whether it's diversity or maybe it's some personal characteristics like learning, agility or something like that, but maybe some of the hard Skills that they have are not quite as matched as some of these other candidates, but instead being able to say, okay, like on balance, maybe there's a match, a slightly different match, with these folks than these folks, and so these folks who might be more diverse, or like I said, different learning characteristics or whatever, they are more likely to get matched and not filtered out. So are you all thinking about it in this way? Because it's not just a nudge to a recruiter—it’s actually fundamentally using the tech to potentially identify people who may not have been identified in the past?

Anshul Sheopuri, IBM:

I think that's really well said, Stacia. So there's a portion about where you go into source pools of candidates. Now, when you come to candidates, you've got to make sure that you're looking at it holistically. And this is not about, these are people who have been my buddies or people who I know and therefore they should, they should go into the pool, but this notion around learning agility, maybe you have some diverse experiences, and this is an important one where you really look at the slate and make sure that the slate has candidates who have diverse experiences. And we see the statistic in different places, Stacia, when you ask hiring managers or managers in general, do you think your decisions are fully unbiased? Most people will not say no themselves—they recognize it and most people recognize them, that they have unconscious bias that seeps into their decision-making process. So reminding them to focus on Skills when you're making the decision, focus on learning agility, when you're making the decision versus focusing on just a university background or people like you. I think those are also important reminders to build into the system. And then there are process aspects as well, which I couldn't underscore more: do you even have a diverse slate of interviewers who are interviewing the candidates or not, because people who are, who have a certain profile would, may tend to gravitate towards similar individuals. So I think thinking of this as a total system versus just a technology and thinking about the different touchpoints also is very important.

And then, having these closed-looped feedback systems. So we look at the quality of our candidates, and that number has been increasing 10% year over year in 2020. And, and that's an important sort of barometer for us to ensure that the actions that we're taking along the lines of what I sort of described is having the right results and implications. So having this closed -oop feedback system is also important because we do see areas, right with any good feedback system. Things might work well at the aggregate, but you will have pockets of opportunity, so it also helps us then zone in on pockets of opportunity where we have scope for growth.

Stacia Garr, RedThread:

You mentioned just a few moments ago, this phrase ‘trustworthy AI,’ and I know that has a certain meaning for you all at IBM, and I think we're going to talk about it a little bit more. Would you mind just sharing with folks what that means to you all?

Anshul Sheopuri, IBM:

Maybe the best way to sometimes describe what a phrase means is through an example, right, Stacia, and so if for your audience and listeners, you're probably hearing a lot of noise around AI. Some of that is, firstly, AI has been an acronym and, and a concept that is not at all new, I mean, and it might be new in the news media, but the phrase was first coined as way back as 1956, so it's about 70 years old, right? It's gone through, it's been through its periods of hibernation and renewal. And really what has accelerated AI, and really made it much more pervasive in the last few years, has been data, compute, and the combination of both of them. But really in the last couple of years this is further accelerated with digital transformation journeys. But now that it's so pervasive, we've got to make sure that AI is resulting in fair and unbiased decisions, and there are examples in, in the popular media, in areas like image processing in criminal justice systems, et cetera, where sometimes AI can have unintended consequences. So for us, making sure AI is trustworthy is not just sort of a checkbox at the backend, but it's sort of a systematic method that you apply at the front end. And it has five elements: transparency—so making sure that the AI is something that is visible to the users, think of it like nutrition labels; when you go to buy your favorite beverage, you see how many carbs, proteins, calories that it has, you can make an informed choice about it. So today, AI is not pervasively that much transparent it's, it's more of a black box, but we'd like to see more and more of it move towards this transparency part of it, and that's something we're very passionate about, so we have fact sheets that we published to our users—these are nutrition labels that users can then see to understand where is AI good, where is it not good.

Explainability is a second pillar. So this is about, we make this recommendation to you, but why are we making this recommendation, so you understand the why behind it. Third fairness is the accuracy, or the percentage of recommendations we're making to you similar across different disparate groups—men versus women, underrepresented minorities versus others. So fairness is an important criteria, and so the ability to identify and mitigate bias in the process—again, upfront—robustness, having the right governance around privacy and business control points. Those are the five pillars that we see—transparency, explainability, fairness, robustness, and privacy—that are core to our principles of trustworthy AI, and those that we embed across these different lifecycle ones.

Stacia Garr, RedThread:

Thank you for that. You mentioned that sometimes the best way to understand something is through an example. So can you maybe help us understand, what does that look like in hiring, for example?

Anshul Sheopuri, IBM:

Maybe I can start with another example that might be of interest to your users as well, which is pay—which is an area which is probably closest to many people. And I want to give you this example more than anything, just to illustrate the importance of trustworthy AI, even in areas that might be deemed as sensitive as pay, right? So today at IBM, we give managers pay recommendations. These are recommendations that help them make informed salary choices. Now I just want to be fully clear, though: the decision is always with the manager. The AI is there to give recommendations, but the manager has the ultimate accountability, and that's an important characteristic of AI—we want it to be augmented AI, it augments your ability to make a decision.

And think about it from the perspective of your own perspective and your experience when you did a pay negotiation as an employee with your manager; it’s not comfortable, you may not have all the information, you’re not clear what parameters the other individual is using to make, come back with some counter offers. And oftentimes, people feel these are ad hoc, at times biased conversations that may be occurring, so the whole point of this was to move towards an objective trustworthy AI based pay recommendation system where it's Skills-based pay, so it’s clear what is the basis of the pay recommendations—it’s Skills based pay. We look at four dimensions, Skills, performance, competitiveness, and potentials, but Skills is a major angle to it. And when you think about Skills, how do we evaluate Skills in the context of pay? We think about it in the context of Skills value in the marketplace, so for the particular skill, let's take an example, data science/Hadoop: are we seeing the new hire pays in that to be of a certain level, is there voluntary attrition in the market at a certain level, is there a long lead time to hire those Skills, is there a high offer, eject ratio? All those are different markers of scarcity of that skill in the marketplace and of value in the market perceives associated with that Skill. So the transparency around the why of a certain skill being more scarce or less scarce, and sharing the approach back to this fact sheet, this nutrition label back with users, that's an important element.

Explainability: if a recommendation is high pay or low pay, explaining why—maybe the skill is not scarce, maybe the competitiveness is you are very competitive or maybe you're not competitive and therefore you get a higher paid recommendation, so explaining the why behind the what, right, is something that managers today see. This is deployed across IBM, tens of thousands of managers see it and make pay recommendations and pay decisions for our 250,000-plus IBMers based on this AI fairness. The ability to identify and mitigate any bias, we do that upfront. And then finally, as I was sharing in the learning examples, the way we work in compensation has also changed. So our compensation professionals, five years ago, their job was to ensure that managers completed pay planning in the system; it was administrative—today, they look at are the investments going in the right areas that are strategic for IBM. So they need different kinds of analytics, tools, different Skills and roles are emerging in that space. And then finally, privacy employee data, it needs to be on a need-to-know basis, and those who need to know it, get to see it, making sure that that occurs as well. And what we've seen, the proof of the pudding is in the eating, right? So as we've rolled this out to managers, what we find is when managers follow these pay recommendations, attrition is reduced by a third of when they don't. So that's good for our population. It’s good for the business as well.

Dani Johnson, RedThread:

So I love this. I’m going to return to a question that Chris asked earlier about culture, because I'm imagining that this may have been a little bit disruptive to the way that you did things in the past. How are you equipping your managers to help build the culture around these pay-for-Skills?

Anshul Sheopuri, IBM:

I think you’re spot on. This doesn't occur in a vacuum, which is why I started with this entire sort of transformation journey story that I shared with Skills as a currency. And so it's just a natural extension; if we had started with this, it would be artificial, right? We needed to embed this in different parts of the journey, and pay was just a natural extension of it. So one starting with that mindset of Skills, by the time we actually reached this point of applying pay-for-Skills, Skills were already known, understood to be the currency in the way we operate.

I mean, the other benefit we do have is we have a rich legacy of innovation dating all the way back to helping put a man on the Moon, helping the United States with standing up Social Security, just important, ‘little’ important things along the way. So we have a culture of innovation, and we are a tech company, so I think that also certainly helps, but I think ultimately this boils down to being very clear, explaining the why: we are all on a digital transformation journey, we need to pay for Skills because managers see that in their interactions with clients—they see the importance of Skills in the marketplace, and clients are asking for it. And so we need to make sure we're investing in the right areas. Our ability to articulate the ‘why’ I think has been probably the best driver of adoption. .

Dani Johnson, RedThread:

So is it a ‘this is why, and’ sort of discussion that managers have with folks? Like, you’re not going to get a huge pay because your Skills are fairly common within the organization, or we see a decreasing need for them and you have this opportunity to develop the ones that we are looking for. Is that kind of how you're approaching it?

Anshul Sheopuri, IBM:

I would say ‘yes, and,’ right? 🙂 So this is not an isolated Skills conversation: the Skills conversations are current through the year: there’s a career conversation around Skills that you have, these are the Skills that you have, let me, let me hear about your career aspirations. Okay, you want to, you want to deepen your skill in this area, you want to perhaps be considered for this job. Now these are the Skills that you have, those are the Skills that you may not have you want to develop. Therefore, by the time you reach the ‘pay’ conversation, it's a natural extension of the ‘career’ conversation; it’s not a new conversation that you're having.

Stacia Garr, RedThread:

You did a wonderful job of answering my question of what is trustworthy AI and how does it apply for pay: I just want to make sure that we pull out one of the things that was most remarkable for me was, how this actually works in terms of some of the processes you've put around it, particularly around this retention program that you have in place, where people may be underpaid or you make an adjustment to make sure that that's not the case. Can you talk a little bit about that specific angle of this example?

Anshul Sheopuri, IBM:

Sure. One of the things that we realized is that, even though we're going through the period that's called the Great Resignation, so we do certainly see attrition as, and in the talent market overall, as an important dynamic to understand. We are ultimately given that we are the Skills-as-a-currency business, talent is our currency, so making sure we have the right talent in the right places is so important for us. And the way we do that is making sure we are giving people the best career experience. Because as I said, that's the right driver, that’s the number one driver of retention, but also making sure our people are paid competitively. And at times you might see the market moving faster in certain areas, not moving as fast in other areas, and making sure we are proactively investing dollars in individuals through base-pay adjustments upfront rather than lose them, which is then a loss to IBM, because it, once somebody leaves by the time you get them back, you've lost productivity and all those other costs.

So the cost of attrition is also higher. From an IBM standpoint, it's actually much more financially prudent for us to invest in people upfront rather than to lose them and then bear those bear that cost of attrition. It's also better for the IBMer, because then we're making sure that they're paid more appropriately. So this is a program that we've invested in for, I would say the past six or seven years, Stacia, as an enterprise-wide program, and what we find is that 95% of the recommendations are accepted by managers in order to invest in employees proactively, and it all boils down to where do we see the highest attrition risk and highest cost of attrition in our key strategic Skills that we're going after which are important for the business.

Stacia Garr, RedThread:

And if I remember right, one of the things that at least struck me was that when those recommendations are made to managers, they can't take that money and apply it to somebody else—it’s pre-approved by Finance, it says, we're going to give Dani a 10% raise and you can't go and give that to Chris instead if you want to, but you can press the button and it will just happen; there's no negotiating, no haggling, just this what it is.

Anshul Sheopuri, IBM:

That's right, because by looking at the market data, we have an understanding of what's competitive pay in that context, and so it takes away that degree of attrition risk, especially when we know that there's attrition risk in a certain area, it takes away that need for that negotiation that you just described and apply the same bar across the key employee populations.

Chris Pirie, The Learning Futures Group:

It’s interesting, isn't it, because the managers—they don't have insight, they don't see the inequities that are happening at a macro scale, and so this nudge, I think, is a really great example of how you're bringing insights to somebody to help them do their day-to-day work, be a good manager, bringing insights they could never possibly have, because the visibility you have more broadly.

Anshul Sheopuri, IBM:

That's right. So if you think about a manager who is in California who has six or seven employees, they may not see the Skills across tens of thousands of employees, understand their attrition patterns, understand other dynamics associated with them. By surfacing all these data points together, we're helping the manager make much more data-driven decisions.

Chris Pirie, The Learning Futures Group:

Yes, I think it's a really nice example of technology automating somebody's ability to do their job well by providing them with tools and insights that they could never have in their ordinary day-to-day context. So I like that; I like that example a lot.

Stacia Garr, RedThread:

I just wanted to make sure, also, you mentioned in a previous call the return that you all have received on this and I just think that's a really remarkable number. Could you share what that was on this whole program?

Anshul Sheopuri, IBM:

Sure. So across our AI, across the employee journey, what we see is about a $100 million cost savings per year to the IBM company. And proactive retention, the program that I talked about, is an important component of that, so $100 million a year is a significant return to the company. What I would also say is, which is why I was sharing with you upfront, Dani, when you asked this question, how should you get started on this journey: think about the business problem that you're trying to solve, quantify it as much as possible, and think about the user problem, because ultimately the best solutions lie in the intersection of a great business problem, because that's where they’re needed, and a great user problem: does the user really have the need to solve the problem, or is it just a nice to have for the user? Clarifying that is also very important.

Stacia Garr, RedThread:

I want to move us on real quickly to workforce planning, which I think is interesting in particular because in terms of the automated recommendations, or the way that you use the AI, it's a bit different for workforce planning than it is for pay. So can you talk a little bit about what those distinctions are and how you're using the Skills data in that context?

Anshul Sheopuri, IBM:

The way we think about Skills is along three dimensions: capacity, depth, and value or scarcity, . those are three dimensions. What I talked about in the context of pay is in the context of value or scarcity, and I gave you some examples of markers that help us get there. In the context of workforce planning, it's really about capacity and depth. So if you think about how we made decisions a decade ago, we would ask how many headcount is sitting in Germany: today, we ask how many data scientists with Python Skills are in Berlin, in a certain location, right, who are experts in the technology and not just experienced. So the nature of questions that are, are much more fine grained. If we can answer these questions, we can look at the capacity side of it and the demand side of it and understand where demand-supply, mismatches, and solve for them.

It all comes down to how do you infer some of these Skills, and the way we do that is through digital footprint. We look at things like your resume; your performance management; your performance goals that you've put in the system; your badges that you've gotten; the learning that you've done; if you're a seller, your sales systems, so systems of work; your public Slack, social, public, social, internal comments that you might have had; your internal blogs that you're writing. All those different digital footprint helps us then inform your Skills, and then we would share back with you as an employee, we think your Skills are, you are an expert in this, and this is the reason why we think your expert. You can agree and you can choose to disagree as well. We have about a million or so feedback points from IBMers on their Skills, and the accuracy that ranges between 85%-plus—obviously higher in some pockets and lower in other areas—but then there's full transparency around the Skills ecosystem. You get to react to it; that helps us give you personalized job recommendations, as an example. It also helps us with better workforce planning, because now we understand depth and can make much more informed assessments around where there are gaps and where there are gluts that we need to solve for.

Chris Pirie, The Learning Futures Group:

This is really interesting to see how that plays out and how the individual gets to see the disclosure, and also, I like the idea of inferring from people's digital activities what their Skills are. I’m sort of interested in this idea of depth, how you measure it; is there a scale, is there transparency around what you mean by depth in terms of any skillset?

Anshul Sheopuri, IBM:

Yes, this is an important one, and it's not an obvious sort of answer—it's an evolving field, I would say, and we ourselves have gotten much more accurate over the past five years. Think of it more as an internal relative scale: you have five categories, and when you think about lines drawn across the board between them. Because the other thing that's important to realize is the line is not static; as the market evolves, what an expert in Python has changed, two months ago versus to today, that line itself might have moved in terms of market expectations. So our focus is more on transparency—exposing that data back to IBMers, telling them where they are. And I would focus less on the data, and more about, think about the conversation that that enables with your manager, because then you can have that conversation around, all right, what can I do to improve my Skills? Maybe I need to pick up some of these digital credentials and learning, and maybe there are obviously 15% of cases where the data itself is wrong, which is completely fine because it helps us then understand at aggregate levels where we have gaps and gluts in the system.

Chris Pirie, The Learning Futures Group:

Interesting stuff.

Stacia Garr, RedThread:

I think one of the points I was trying to get to is, is you're using that depth based on the digital for a point of folks for workforce planning, but that is not what is driving the pay decisions or the pay recommendations. I just want to make that clear.

Anshul Sheopuri, IBM:

Absolutely, and thank you for clarifying that Stacia as well, that the depth is really for, for workforce planning. The other thing that I would say here is what was also interesting is we did a small study and when you ask two experts on a blind profile of an individual and you ask them to assess their Skills on the same five-point dimension, experts disagree quite a bit as well. Does that surprise you? So one of the important criteria we look at is our accuracy as defined as agreement with IBM and feedback. How does that compare with expert disagreement? And that's an important barometer for us, because we want the accuracy of the system to be greater than human accuracy in this context. So for users who are or audience members who are embarking on this journey, that's something I would also encourage them to look at.

Stacia Garr, RedThread:

I want to spend just one moment or two on the tech stack. I know that we're running tight on time, but there's been a lot of discussion we've had on tech stack, so maybe the first and most direct question is, have you all developed all of the tech that underlies this internally, or are you using any external systems?

Anshul Sheopuri, IBM:

All internally is what develops, as you can imagine. We do work with clients to offer this to several of our clients as well. I think of the tech stack as having three core elements: one is an underlying data lake, this is where the data is collected, harmonized, governance, making sure the data is clean, controlled, curated, all that good stuff—good data hygiene. That’s on a hybrid cloud platform, so it's secure because as I described some of this data, it's in very different diverse places, right? Second is think about AI services—our ability to understand what people are seeing, interpret that with the right intents, the fact that deep learning might, somebody might be using the phrase ‘deep learning’ interchangeably with another phrase, which is similar to that and the ability to look at those connection points, infer, intent language and form from connections. The third piece is process and experience, so sharing that data back with IBMers, but also in the content of for taxonomy, looking at that much more holistically from a process standpoint, and so that front-end user experience and ability to collect feedback, and then I've tried to avoid jargon through this conversation, but I’ll throw in one, one jargon phrase, just for completeness, what we call semi-supervised learning: as we get feedback from IBMers that should improve the accuracy of the machine learning, so that's what we call supervised learning, but in some areas we may not have enough feedback and so the ability to look at supervised unsupervised together, do extrapolations and leverage that together. And we use a semi-supervised learning approach in order to make that happen.

Stacia Garr, RedThread:

Wonderful. All right. Well, I think we could ask a ton more questions on tech, but we're not going to for today. I just want to ask you, as you reflect back on this whole journey, what are some of the critical lessons you've learned through this work?

Anshul Sheopuri, IBM:

Just to be consistent, and I want to make sure that I help your users with this point again and again, which is: it is so important to start with a business problem and the user’s problem. In fact, for somebody whose title is chief technology officer, for me to say this two or three times, which is don't start with data, don't start with technology.

Now, it is important once you have a validated business problem with a validated user problem, to make sure that we're setting yourself up with a solid technology foundation. And if I was to reflect on our lessons learned—and one of the things that we could have done better, I could have done better—is sometimes we've taken some shortcuts which have then resulted in technology debt that we've had to then solve for at the back end. So invest more in your data platform. Some of it is just evolution of technology: when we started this five, six years ago, the data platforms themselves were not that robust, and so making sure you have end-to-end processes around governance. Ability to connect people data to financial data, to client data, to marketing data, because ultimately you're not making these decisions in vacuum; you’re making them also to understand implications and businesses. Those are some of the lessons that I would encourage you all to be aware of.

There are a couple of places where I would say where we had some quick fails, and have battle scars to go with that. Those are areas where we were not design-led, so I can't emphasize more the importance of having 20, 30 sponsor users who are continuously giving you feedback, whom you're testing your prototypes with, whom you're really making sure that you’re solving for their real user problem—the need that they have versus a want that they might express in an afternoon over coffee. Really synthesizing out the need is so important to ensure that you're creating the right cultural context and you have low adoption risk.

Stacia Garr, RedThread:

That's wonderful. I like that: quick fails! Let’s talk just for a moment about the future; you are in a place where you can see it, maybe because you're a little farther ahead—what do you see as being in the future? What will we be doing in five years?

Anshul Sheopuri, IBM:

I'll give you two answers to that question—one is for the broader industry, and one is for IBM. What I personally see as the opportunity for the broader industry; I believe we're still at the nascency of applying AI in a manner that I would like to see it being applied from an industry point of view, which is why in December, IBM, along with about a dozen other companies, large brands, got together and launched what we're calling the Data and Trust Alliance, to launch the very first set of standards for trustworthy AI in HR. It's very similar to what I just described as what we're doing internally, but I'm super passionate about seeing in five years all vendors, all companies, or a large portion of them, apply these same standards in all employee lifecycle decisions. That's just so critical to ensure we have the right quality decisions, but also diverse outcomes for the larger workforce, and that's something I'm super passionate about and I see as the future—it’s going to happen. There's a lot of regulation in this space as well that's coming, but I think it takes a village to move an industry, and that's what I see happening a lot.

From an IBM lens, I see even greater integration of the consumer-grade experience: think about your personal experiences, and we're seeing some of that confluence beginning to occur. Five years ago, we had disparate video watching experiences from social experiences, from hailing a cab experience. With many of the consumer trends that are emerging, there’s much more confluence across those areas that you'll begin to see more and more. And I think you will see the same within the employees as well, so how you, across different business and HR workflows, I think you'll see a lot greater confluence in that; those lines will diminish even more and more. And I think that's a fantastic opportunity for all of us to really drive business transformation, not just HR transformation.

Stacia Garr, RedThread:

Definitely. Final question—very quick—which is one we ask everybody, which is the Purpose question. So why do you do the work you do?

Anshul Sheopuri, IBM:

Great question. Every now and then, Stacia, I'll get an email from an IBMer who—it doesn't happen that often, but when I do—and that email says, “I was in a role where I was not clear what I was doing, I was looking for another opportunity, and this learning nudge or this job notch helped me realize my career passion.” You get goosebumps when you get such a note; it just makes you feel so proud, makes you really feel a sense of purpose and worth. And that's what I think this is about for me. It's impact at scale, and impact for the entire workforce, for IBM, but also the broader industry.

Stacia Garr, RedThread:

Wonderful. I think, for us, in a different way, that impact on people's careers and the opportunity to help them do their work better and fulfil their greater purpose is what we do, too. So thank you. Thank you for that.

Anshul Sheopuri, IBM:

Thank you for the time.

Chris Pirie, The Learning Futures Group:

Wonderful, great conversation. Thanks.

Thanks for listening to this episode of Workplace Stories. Dani and Stacia, how can our listeners get more involved in the podcast?

Dani Johnson, RedThread:

Well, they can subscribe and rate us on the podcast platform of their choice.

Stacia Garr, RedThread:

They can also share this, or their favorite, episode with a colleague or a friend.

Dani Johnson, RedThread:

You can check out the beautiful, handcrafted transcripts at redthreadresearch.com/podcast and see what else we have to offer as far as research goes.

Chris Pirie, The Learning Futures Group:

Or, Stacia, they could…

Stacia Garr, RedThread:

Consider joining the conversation and community by joining our RedThread membership.

Chris Pirie, The Learning Futures Group:

A big thanks to our guests on all our podcast Seasons for sharing their insights and thoughts.

We should thank our sponsor, of course, for making all this possible— of course, we should thank our beloved listeners. Thank you.

Thank you.