Events

What are some of the obvious and less obvious risks of choosing and implementing D&I technology? In our recent study with Mercer, we examined the emerging market for D&I tech. As part of our exploration, we needed to take a step back to understand the potential gains or pitfalls that come when well-intentioned companies use technology and AI to solve endemic people challenges.

Here is an excerpt from that report which breaks down some of the risks and payoffs of implementing D&I tech:

What are the benefits and risks of D&I technology?

While there are many potential benefits of D&I technology, the most apparent one is the opportunity to create consistent, scalable practices that can identify or mitigate biases across organizations, often in real-time. Many people-related decisions leave a lot of room for bias, particularly when it comes to an assessment of a person’s skills, behaviors, or value (e.g., for hiring, performance evaluation, promotion, or compensation).

Much of the technology on the market today is designed to change the processes that enable bias or identify that bias exists. Another benefit customers see in D&I technology is the increased understanding of the current state of diversity and inclusion throughout the organization. With greater visibility, leaders can better measure and monitor the impact of D&I initiatives.

Benefits:

• Implementing more consistent, less-biased, and scalable people decision-making processes
  Increasing the understanding of the current state of diversity and inclusion across the entire organization, using both traditional and new metrics
• Measuring and monitoring the impact of efforts designed to improve D&I outcomes
• Raising awareness of bias occurring in real-time and at the individual level and enabling a range of people to act on it
• Enabling action at individual levels by making new, appropriate information available to employees at different levels within the organization
• Signaling broadly the importance of a diverse and inclusive culture to the organization

Risks:

• Implementing technology that itself may have bias due to the data sets on which the algorithms are trained or the lack of diversity of technologists creating it
• Creating legal risk if problems are identified and the organization fails to act
• Enabling the perception that the technology will solve bias problems, not that people are responsible for solving them
• Reducing people’s sense of empowerment to make critical people decisions
• Implementing technology or processes that are disconnected from other people processes or technologies
• Enabling employee perceptions of “big brother” monitoring, an over-focus on “political correctness,” or “reverse-discrimination”

“Metrics aren’t always immediately useful.”

Susie Lee, Degreed

One of the reasons so many L&D functions struggle with learning measurement and learning impact is that they have no consistent data. In fact, according to Brandon Hall, only 51% of companies say that they are effective or very effective at measuring formal learning. And even fewer are effective when it comes to measuring informal (19%) and experiential (29%) learning¹.

While these statistics focus on a more traditional way of viewing learning and development, the fact that only 51% of organizations are effectively measuring formal learning – which, by the way, they have complete control over and for which they have complete access to the data – is telling. By and large, L&D functions do not have a data culture. But they could have one.

How should they start? Leaders told us that they needed to overcome two major challenges in order to get consistent data: patience and standardization.

Be patient – it’s a virtue

Most L&D functions either really struggle to collect data and information on a regular basis, don’t do it at all. At least a part of this struggle stems from the practice of focusing on one-time measurements. When L&D functions focus on calculating the ROI or learner satisfaction associated with one course or initiative, the tendency is to collect only the information needed to serve that one purpose.

This focus on point-in-time results means that longitudinal data, interactions, and correlations are hard to come by in many organizations. Interactions and correlations over time provide ongoing insights about what is happening and why. Without consistent L&D data, it is difficult, if not impossible, to understand the impact employee development is having on organizational goals. Understanding this impact is the first step in being able to make intentional decisions about where to go next.

Collecting data over time can be challenging, and the fact that data and metrics may not be immediately useful can add to that challenge. But establishing continuous collection goes a long way in building a data and metrics culture.

In our conversations with leaders, three pieces of advice for how to consistently collect data stood out:

• Start where you are and keep at it. Metrics aren’t always immediately useful – it often takes time, dedication, and investment. But the results are worth it. Several leaders emphasized the importance of starting where you are and building capability as you go along. No need to boil the ocean, but you must be consistent. Rachel Hutchinson, director of L&D at Hilti, said her team collects data for 6 months on any given initiative or change: 4 months to get results and 2 extra months to make sure it wasn’t a blip in the data.
• Consider continuous data feeds. This bears repeating: L&D functions should think in terms of continuous data feeds instead of static reports or one-off calculations of metrics for two reasons. First, continuous data feeds provide the most recent data, giving L&D functions the ability to adjust to conditions more quickly. Secondly, it’s only slightly more difficult to set up a data feed than ask for a one-time data dump. Working with IT and other business functions to set up feeds will ultimately save the entire organization time and effort.
• Automate data collection. In circumstances where it is necessary to collect data (rather than using other sources), automate it! Learning leaders we spoke with do this by planning for surveys, evaluations, and feedback as a part of the design process for any initiative, utilizing scheduling software or investing in measurement software that helps you do it (Watershed, forMetris, etc.).

Standardization

The other part of the consistency story is data and metrics standardization. Why? In order to consistently monitor and make good decisions, data and metrics need to be correct and comparable.

Standardization also ensures that L&D data and metrics are consumable by other business functions, by central data analytics teams, and by other technologies and systems. L&D functions should start by identifying any existing data standards their organization may have and adopting them.

That said, our interviews with L&D leaders indicated that the first challenge was often standardizing data and metrics within their own department. They talked about three types of standardization:

• Consistent formats. If data will be compared over time or with data from different systems or functions, using consistent formats is key. One leader mentioned the first time she realized the importance of this was in trying to pull a dataset for a given timeframe. Since she worked for a global organization, dates were stored differently depending on where in the world they were. While technology is getting smarter and better able to correct some of the challenge, identifying standards for data formats upfront can save a lot of headache down the line.
• Consistent scales and data types. In situations where data needs to be collected, particularly through surveys or evaluations, consistent scales and data types should be used. Three-point scales vs. 4-point scales vs. open-ended answers all affect how easily data can be compared. One leader told us of the struggle he had of simply trying to get the entire organization to use one form so that course evaluation information was standard and could be compared.
• Consistent collection methods. How and when you ask questions, who you ask them to, what words you use to ask them, who is asking them, and the format in which they are asked, can have an effect on responses. More evolved L&D functions are standardizing some of these things – sometimes with the use of technology – too ensure that bias doesn’t creep into their data and to ensure consistency.

Unfortunately, developing a data and metrics culture within L&D functions is most likely not second nature: it takes work and investment. And it’s often not the sexy part of what we do. But we think that this culture and the ability to consistently collect and analyze data is the first nut L&D functions need to crack. As organizations begin to collect and assess information regularly, they will better understand how employee development is affecting the organization and their options for having impact will increase.

Why are we seeing more attention on D&I right now? When we began our recent study with Mercer, we recognized there were many factors driving the emerging market for D&I technology solutions. Here is an excerpt from that study with some of our thoughts on why D&I tech is a market that is gaining so much momentum.

There are numerous trends driving the increased attention on the D&I conversation, not least of which is the changing racial and ethnic mix of the U.S. population. Image 1, below, shows the projected growth of ethnic diversity among younger Americans through 2065.

People between ages 15 and 24 make up close to 20% of the world’s population. Further, by 2025, millennials (those born between 1980 and 1996) are expected to comprise three-quarters of the global workforce. Younger and increasingly diverse populations often bring with them evolving expectations and a willingness to bring D&I to the forefront of societal conversations.

Figure 1: Changing Face of America,1965-2065 (% of the total population) | Source: Pew Research Center 2015 report, "Modern Immigration Wave Brings 59 Million to US, Driving Poplation Growth and Change Through 2065" | Note: Whites, black, and Asians include single-race non-hispanics. Asians include Pacific Islanders. Hispanics can be of any race.

In addition, workplaces are becoming more multicultural with global talent moving across countries and positions. Non-traditional forms of work continue to gain popularity, such as freelancing, virtual work, and short-term project-based assignments.

There is also a shortage of talent that is especially acute in knowledge industries. The financial and business services industries expect a shortage of 10.7 million candidates by 2030, which will continue to fuel this upward trend in global talent interconnectedness. These workplace changes in demographics, non-traditional workforces, and talent shortages are strong forces pushing diversity and inclusion to center stage.

The amplified attention on D&I is also due to its increasingly well-documented relationship to business outcomes. Research shows that more diverse and inclusive organizations outperform those that are not. A survey of 1,700 organizations across eight countries found that organizations with above-average total diversity had both 19% higher innovation revenues and 9% higher margins.

Therefore, organizational leaders are increasingly seeing D&I as critical to achieving financial goals. These trends, accelerated by the rise of #MeToo in October 2017, created a seismic shift in the discussion around sexual harassment that has spilled over into other diversity and inclusion topics such as gender identity racism, ableism, sexual orientation, national origin, age, veteran status, religion, and more. For example, 56% of millennials believe that “business leaders have a greater responsibility to speak out on social issues now than in years past.”

This growing and collective frustration has increased the desire for a new approach to diversity and inclusion.

We’re about two-thirds of the way through our research process for our people analytics technology (PAT) study, so thought we’d provide three initial insights on what we’ve learned to date. (Please note, this research is ongoing.)

And now, without further delay, let’s move on to what we’re learning…

1. It's all about the Benjamins. We’ve known for a while that the PAT market was receiving significant interest from the investment community – a point that was underscored by the $45 million Series D investment in Visier in 2017. However, our analysis of the market shows that interest has continued with great enthusiasm across the last few years. For example, in 2018, Humu raised $30 million and Culture Amp raised $40 million, among others. Earlier this year, Peakon raised $35 million in its series B, and Perceptyx just announced today a strategic investment from TCV, a private equity firm. New players, like Cultivate, are also off to the races with fair-sized seed rounds. There’s also been significant investment in people analytics technologies as a result of people analytics vendor acquisitions where the acquirer intends to invest significantly. Prime examples of this are Glint (acquired by LinkedIn/Microsoft, but had raised $80 million beforehand), Shape Analytics (acquired by Reflektive, which itself received a $60 million series C investment in 2018), and of course, Qualtrics (acquired by SAP in late 2018). And finally, there is Medallia, which just had its IPO last month, which provided it with plenty of cash to invest.
2. Mo’ money… All that money appears to be making an impact, if you ascribe to the idea that more investment should eventually equal more revenue. Now, we aren’t going to spoil the surprise of our research by giving overall market growth numbers here, but we will tell you that nearly all vendors are reporting accelerating revenue growth, with 2019 looking to be the best year yet. In addition to the significant investment mentioned above, additional reasons for this revenue growth appear to be a healthy economy, broader corporate investment in people analytics teams, and people analytics teams’ increased sophistication (and therefore ability to use some of these tools).
3. …Mo’ (user) problems? But there are some challenges ahead. As these vendors grow, they are looking to expand their user bases (the old “land and expand” approach) within existing customers. However, people analytics tech isn’t like a lot of other tech. First, not everyone should have access to the insights available within people analytics technologies – at least not carte blanche. Second, there are a wide variety of users within any given organization who have varying levels of data maturity, which requires different levels of sophistication and customization of dashboards, data structures, and data analysis tools and capabilities. Third, most of the vendors’ technologies were originally built to cater either to really sophisticated people analytics teams or to less data savvy HR business partners.

This combination of issues is resulting in challenges for these vendors as they try to expand their user bases. Essentially, we are seeing:

• Really sophisticated technologies trying to simplify aspects of their tech for a less data-savvy user (but the technology still being too difficult) OR
• Less sophisticated technologies struggling to make their technology adequately sophisticated for people analytics teams to justify the investment (versus those teams building it themselves)

As a result, we’re seeing instances of product/market fit when it comes to secondary users (the ones to whom these vendors must expand their offerings). Vendors are aware of this challenge (in some of our practitioner interviews, less-than-expected end-user adoption rates have come up as a real issue), but the path forward for many of them is not totally clear. We expect we will see some vendors shift their target users or potentially break up their offerings into more discrete packages to make it easier to build for and sell to different users. We see this as being one of the critical issues many PAT vendors will have to solve for in the coming 18 months.

We are analyzing our people analytics technology vendor data, and will be uncovering a lot more insights in the coming weeks. If you’re able to join us at PAFOW Philadelphia, September 5 & 6 for the sneak peek of the results, we strongly encourage you to do so! Otherwise, stay tuned here for more information on this research.

We just had a wonderful preliminary meeting with members who will be joining us in Washington D.C. for our collectives on Building Skills for the Near Future. In all, around 40 leaders joined and participated in an open exchange of ideas.

Below is the mind map of that discussion. Click in the box to explore! As always, we'd love your comments or thoughts!

On July 31, 2019 we conducted the second roundtable in our Performance Management study. We sincerely appreciate the group of thoughtful leaders (you know who you are) that came together to review some of the initial numbers from our survey and provide their insights and experience

The highlights from that hour-long web-chat are succinctly outlined in the mind map below. Big branches represent main discussions, smaller branches represent some of the responses and detail.

Click in the graphic to make it bigger, move it around, etc.

We'd like to thank the learning leaders who took part in this discussion! We think collaboration makes the research better, and we learned a lot from this roundtable.
In July 2019 we conducted our second Learning Tech Ecosystems Roundtable. The turnout was excellent and we were able to have a great discussion regarding the strategy of thinking through a learning tech ecosystem.

The hour-long web-chat resulted in a really rich discussion, the highlights of which are shown on the mindmap below. Big branches represent the main topics.

Click the graphic to get a bigger view.

Learning Tech Ecosystem Roundtable #2 Mindmap

Source: RedThread Research, 2019

One of the main questions we had going into the Learning Tech Ecosystem Study was, “what are learning leaders considering when they create learning tech ecosystems?” While each organization is doing different things, several threads have emerged:

• User experience
• Skills!
• Data integration
• Sustainability
• Ecosystems for everyone

Experience

Number 1 answer? Survey says: Experience! A lot of the hype from the articles we reviewed in our literature review and almost every interview we conducted mentioned making the learning experience and supporting tech a little less Minesweeper and a little more Fortnite.

Employee experience, the ease of access, and usability are front and center in ecosystem decisions. Learning leaders shared the common desire use technology that allows them to make learning available to the employee and make it accessible in the moment of need to ensure a seamless experience.

The main reasons for this? 1) Establishing a clear signal through the noise to help employees understand what is truly important and beneficial to them; and 2) Matching the experience employees get internally with that they can get externally – hoping to engage employees and nudge them in ways that helps them personally and the organization as a whole.

Skills!

L&D leaders also highlighted the importance of shifting their efforts from providing role-based learning to skill and capabilities-based learning. As up-skilling and re-skilling become must-have conversations and development goal for leaders, democratizing learning was a goal that we heard frequently in our conversations.

This fundamentally changes the type of ecosystems that organizations need. They need to be more flexible, more integrated with the work employees are doing, and more adaptable. Instead of completing a curriculum for a role, organizations are encouraging employees to develop skills, and then using technology to help them determine where those skills can help them.

Data Integration

L&D leaders are also beginning to think much more seriously about data. In our interviews, almost every learning leader mentioned the usefulness of data – not just to help them do their own jobs better, but to provide more information to the larger organization and to individual employees by seamlessly integrating data from different functions: talent, staffing, recruiting, career, performance, and experience.

While this was one of the largest challenges that L&D leaders raised, they were also thoughtful in their responses when we asked them how they were addressing it. Many are going beyond the analytics provided by any one tool and instead finding ways to consolidate that information. Answers ranged from using an ‘anchor’ technology as the system of record and integrating data feeds from other tools to using Power BI or simple Tableau dashboards to gather, crunch, and present metrics.

"The current state of the market is requiring CLOs to become CIOs."

Interviewed Learning Leader

Sustainability

Another point that leaders surfaced was this idea of sustainability in two ways: the viability of the individual technologies in the ecosystem long-term, and the completeness and effectiveness of the ecosystem as a whole.

Several leaders were cautious of using newer technologies because of the current consolidation of the learning tech market, and it is difficult to know if they’ll be around in a year. One leader shared a few questions he asks himself when vetting tech:

• How long has the organization been in business?
• Are venture capitalists betting on the technology?
• Do their other customers have similar challenges to ours, and does the vendor partner well?
• Are the companies young and nimble enough to react?

While it’s never guaranteed that vendors of any sort have long-term sustainability, asking the right questions upfront can save a lot of headache down the road.

Many leaders also mentioned their need to have their ecosystems grow and morph along with their organizations and its needs. We found that leaders with this need are often open to more experimentation, but also much more insistent that vendors play nicely together. In fact, one leader is approaching this by conducting joint sessions with several vendors in their ecosystem. This leader brings these vendors in together to help them understand their company needs and asks them to work together on solutions that will meet those needs.

Ecosystems for Everyone

Finally, an unexpected concern of many leaders is designing an ecosystem that equally benefits the deskless employees, those located in rural environments, and the hesitant adopter.

Deskless workers currently make up about 80% of the global worker population,¹ and yet, many learning technologies overlook them: from not addressing them at all (a misstep) to assuming that making something mobile and responsive solves all problems (it doesn’t). Leaders we talked to emphasized the differences in audiences and identifying technologies that will work for all, or bifurcating the ecosystem so that it will serve all.

Really great tech ideas can be derailed over the simplest things: bandwidth for instance. As one of our roundtable attendees put it, “It’s hard trying to make a seamless experience especially for those employees who are not even online.” Leaders are in need of solutions that allow their deskless employees (and those in more rural parts of the world) to access learning on their personal mobile devices on their own time instead of locking it behind a firewall.¹

Additionally, with several generations (sometimes up to 5)² working together in a company, the rate of adopting new technologies can vary, ranging from enthusiasm to reluctance. A learning leader we spoke to is tackling this by instituting a blended approach of technology enabled content being delivered by an instructor in a classroom setting.

"It's hard trying to make a seamless experience especially for employees who are not even online."

Interviewed Learning Leader

What’s happening in the world of D&I since February 2019?

Since we published our report early this year, the space and the market has continued to gain momentum. As we had predicted, existing bias within algorithms are coming front and center in the D&I space.¹ In May, the U.S congress proposed a law that would hold companies accountable for algorithms that result in discrimination and would require them to regularly evaluate their tools for accuracy, fairness, bias and discrimination.²

As we predicted, existing bias within algorithms are coming front and center in the D&I space.

Bias in artificial intelligence (AI) has become a larger issue as more of it enters our lives. In our broader society, one of the most notable examples is that of gender bias in virtual assistants – that most of these assistants have female names and voices (think Alexa, Siri, etc.) and have submissive personalities. In response, UNESCO recently published a report that points to the importance of digital skills among girls and women and the lack of diversity in the technology sector.³ To challenge these biases, Virtue, the creative agency owned by publisher Vice, in collaboration with Copenhagen Pride, Equal AI, Koalition Interactive and thirtysoundsgood, developed a genderless virtual assistant called “Q”.⁴ As part of their #BiasCorrect campaign, Catalyst, along with Burns Group, launched a Slack plug-in that alerts the user when biased language is being used against women.⁵

We heard from a number of new D&I tech vendors after we published our February report.

As technology becomes more advanced and enters into new fields, companies are starting to find unexpected and unforeseen D&I use cases for technology not designed for that particular purpose. One such example is Microsoft, which recently piloted their fingerprint biometrics-enabled cards for social benefits in Mexico’s Sonora state. This technology proved to be extremely popular with the senior population. While the initial objective was to improve safety and security in card related transactions, the elderly population reported they were able to use it with much greater ease. The technology will allow Microsoft to securely, transparently and reliably move funds to the people who might otherwise struggle to access them.⁶

What have we done in the last six months on D&I tech?

On our end, we have had a lot of conversations about diversity and inclusion and technology. For example, we shared our most recent update on D&I technology at UNLEASH 2019 in Las Vegas and in countless other conversations. In addition, we just launched our new one-page D&I tech infographic, to share our findings more succinctly. Further, we spoke about the power of networks and technology for career women at the ASU/GSV summit and launched a related study on Women, Networks, and Technology. You can see both our review of literature and some of the initial findings from our interviews; the final report will be published later this summer.

Technology providers with engagement platforms are also beginning to highlight the role such tools can play in promoting diversity and inclusion programs in companies through recognition, sharing, and feedback.

What new technology have we seen?

As we predicted, we heard from a number of new D&I tech vendors after we published the D&I tech report. We want to provide you with a round-up of the new technology providers and also update our interactive market map. In the sections below, we review the new technology we’ve seen since launching the report.

Acquire

As we stated in the original report, talent acquisition has the largest percentage of D&I technology solutions focused on sourcing and selecting diverse candidates. We identified the following new vendors in this space:

• Incluzion is a talent acquisition marketplace that provides companies with a way to recruit, hire and pay diverse freelance talent.
• Mom Source Network offers virtual networking to connect moms to others facing similar challenges and to connect those who are looking to return to the workforce with women currently working.

Talent acquisition has the largest percentage of D&I technology solutions focused on sourcing and selecting diverse candidates.

Develop & Advance

Our study on women, networks and technology led us to discover several vendors in the mentoring/career management space that are focused on career development for women and minority groups:

• Everwise, a mentoring solution, helps companies with development content and curriculum for women leaders.
• River offers mentoring software and services to organizations, helping them support their employee development initiatives such as D&I and Leadership Development through structured formal mentoring programs.
• InstaViser offers cloud-based software that integrates with existing CRM software and other management platforms, helping corporates, academic institutions and non-profits manage and scale their mentorship programs. Features include smart matching, specific diversity mentoring, scheduling, video and audio conferencing, and more.

Image 1: Screenshot of InstaViser’s technology | Source: InstaViser, 2019.

Engage & Retain

We touched upon the role technology, such as virtual reality, can play in sexual harassment training and in allowing anonymous reporting in our study. Over the course of the past few months, we came across several additional vendors that are developing technologies that help report and address issues of workplace harassment:

• Bravely provides a platform for confidential conversations with coaches, allowing marginalized groups and minorities within the organizations to speak up without the fear of retribution and to talk about issues they face at work.
• Project Callisto provides a platform, called Callisto Expansion, that allows survivors of professional sexual coercion to securely and anonymously store information about their perpetrator. It connects survivors to attorneys who can help them understand their options and available actions.
• tEQuitable helps companies reduce misconduct and proactively prevent bias, discrimination, and harassment. It offers employees a safe place to navigate issues ranging from micro-aggressions/micro-inequities to overt discrimination. The product also provides the company with data on behavioral trends, identifies systemic culture issues, and recommendations for remediation.

Image 2: Screenshot of tEQuitable’s Technology | Source: tEQuitable, 2019.

• Vault allows users to report sexual harassment in a confidential and safe way. These reports can be kept private until the user is ready to share them with the case manager. The platform also allows other employees to weigh in on an incident at the same time in a structured manner and the reports are time-stamped.
• Another vendor, Woices, also launched a mobile app that allows users to report incidents of harassment and share them anonymously. While still in its initial launch phase, the solution will be able to use AI to identify patterns in reporting and make recommendations based on insights.

Image 3: Screenshot of Woices Technology | Source: Woices, 2019.

In our search, we also came across platforms that allow women to communicate and share openly:

• Elpha is a community where women in tech talk candidly online. One of their recent initiatives focuses on identifying top companies for women, and the results based on ratings and feedback submitted by over 1,000 of their members which will be published on their website.

Image 4: Screenshot of Elpha’s technology | Source: Elpha, 2019.

• Similarly, while not a technology solution, Panda provides a platform for women to connect and network across the globe. Once part of their directory, members can network and exchange ideas through it.

Technology providers with engagement platforms are also beginning to highlight the role such tools can play in promoting diversity and inclusion programs in companies through recognition, sharing, and feedback:

• Inspirus, a Sodexo Group company, promotes diversity and inclusion initiatives through engagement, recognition, micro-learning and events. The Inspirus Employee Engagement platform allows employees to participate in D&I training courses and track their progress, recognize inclusive behaviors, and promote awareness of D&I-focused events.

Image 5: Screenshot of Inspirus Technology | Source: Inspirus, 2019.

Analyz

• Affirmity, focused on analytics and pay equity in the diversity and inclusion space, launched its ERG platform early this year to help organizations manage and scale their employee resource groups. It also offers an ERG mobile app to employees.
• Paygaps.com is a cloud based platform that simplifies the gender pay gap reporting and data analysis process.

Image 6: Screenshot of Affirmity’s Technology | Source: Affirmity, 2019.

Organizational Network Analysis (ONA) is emerging as a growing field that organizations are increasingly looking to use to identify opportunities for greater inclusion, understanding gender differences in behavior, identifying high-potentials, and building more effective relationships with underrepresented populations:

• Humanyze measures collaboration in organizations with advanced analytics, ONA, and behavioral science to increase the speed and accuracy of operational decisions.
• Another ONA vendor, Innovisor helps companies break down collaboration barriers by revealing gender issues and biases.
• Finally, Polinode provides a platform designed to collect, analyze and visualize data on relationships within organizations.

Image 7: Screenshots of Polinode’s Technology | Source: Polinode, 2019.

Market Map

We have also updated our interactive D&I technology market map and moved it to the RedThread website. You can access the new interactive map of the vendors here, as well as the original, comprehensive report here, in which we explore in depth:

ONA is emerging as a growing field that organizations are increasingly looking to use to identify opportunities for greater inclusion.

• The history and advent of this inflection point for D&I
• Detailed market observations and analysis on the D&I technology market
• The trajectory, acceleration and shape of the D&I technology market
• Potential risks and benefits of D&I technology
• Specific types of technology solutions
• Detailed vendor landscape and product offerings
• Case studies and customer perspectives
• Predictions for D&I technology over the next 18 months

Conclusion

As is evident from the number of new additions of technologies over the past few months, the D&I technology market continues to grow rapidly. This has been equally matched by the growing concerns over exacerbating existing bias as well the legal risks associated with them. Incidents of data privacy breaches such as those at Facebook and Google have brought issues of data security and ethics to the forefront, making employees and organizations wary of adopting new technologies. The number of lawsuits alleging sexual harassment have been on the rise as well, as stated by the U.S EEOC’s October 2018 report on sexual harassment.⁷ Interestingly, there has been a drop in the number of high-profile accusations, which may reflect a change in organizations approach to handling such cases (potentially responding proactively versus reactively).⁸ As the D&I technology market grows, the human component remains a crucial element. Leading practices, challenges faced, and lessons learned will be crucial sources as organizations look for ways to effectively manage these technologies.

Finally, we will be updating our D&I technology tool on a regular basis as we come across new vendors in this space. If you are a technology vendor in the D&I space and think you should be included in our tool, but are not, reach out to us here.

Appendix A: Table of Referenced Vendors and Capabilities

Six decades and not much change

We have been talking about learning measurement for more than six decades. Ever since Kirkpatrick came up with his 4-level model, learning & development (L&D) functions have been trying to understand the relationship between what they do and business outcomes. Since that time, over 20 have been developed to help L&D functions support their claim to impact.

Lately, we have become really interested in a variation of that question: How does what and how we measure affect the organization? What resulted was a surprise. While we approached it with a “there’s got to be a right way” mindset, what we have found is quite different. In fact, through this study, we have made two large realizations:

Realization 1: There has never been a more important time to get learning impact right.

Let’s start with a short history lesson. At the beginning of the first industrial revolution, we started using assembly lines to create products. These lines were very inefficient and fraught with problems: people learned only on the job, no one had a clear idea of what their roles were, and the workloads were incredibly uneven.

Then along came a guy named Frederick Winslow Taylor. Taylor was a mechanical engineer who applied engineering principles to the lines in order to clean them up and make the more efficient.

One of the first things he did was to establish processes and divide up the work. Then he trained employees to do that work in efficient ways. He, in essence, turned a chaotic malfunctioning system involving both individuals and equipment into one cohesive well-functioning machine.

This made Taylor famous. He wrote a book on these principles called The Principles of Scientific Management and is considered one of the world’s first management consultants.

As industry progressed, these principles were applied to types of work beyond assembly lines. They helped organizations grow and develop by structuring their workforce and ensuring that everyone knew what their role was and how to do it most efficiently.

Today, organizations continue to define roles, add people, train employees the ‘correct’ way, eliminate wasted time and effort, and gain efficiencies using these principles. This focus on efficiency has worked for a really long time. The US, for example, had fairly consistent gains in productivity from when we started measuring in 1947 through 2007.

However, like any system or machine, it’s impossible to make it infinitely efficient. At some point, any system will reach its optimal efficiency and additional effort won’t yield very big gains. We’re seeing that now in the market.

Image 1: US Productivity is Falling Off – Possibly Because of a Focus on Efficiency | Source: Data from the US Bureau of Labor Statistics

Image 1 shows trend lines from 1947 to 2007 (blue line), trend lines from 2001 through 2007 (orange), and the actual productivity gains from 2007 to 2017. As you can see, our actual productivity growth is the lowest it has been since we started measuring.

So efficiency is not getting us as far as it used to. And we need to start rethinking our reliance on it for business gains. Interestingly, more evolved organizations already are. A few months ago, we stumbled upon some research done by IBM. This study compares business focuses of industry outperformers (shown in orange in Image 2) to all other companies in that industry (shown in gray). The results are telling.

Image 2: Significant Outperformers Focus on Different things | Source: IBM Global Study

While the rest of the industry is focused on improving operational efficiency, outperformers focus on items that are inherently messy and inefficient, such as developing new products and services, expanding into new markets, and developing new distribution channels. And they need workforces who can think outside of the proverbial black box, take calculated risks, and think critically.

So what does this have to do with L&D and its impact? For starters, we’re dealing with a completely different world. When Taylor basically invented the L&D department, he did so to ensure that people were being trained to do a certain role in the most efficient way possible.

But L&D’s traditional methods and measures are not going to get it done. Most organizations cannot wait (and most likely don’t care) for L&D functions to calculate an ROI on a program or initiative to determine if it was effective and then adjust; it’s too late by then. Organizations need L&D functions to react more quickly and with flexibility – which requires a radically different mindset than the one currently en vogue.

We think that L&D functions everywhere need a chakabuku – a swift, spiritual kick to the head that alters their reality forever.¹ L&D functions desperately need to rethink their measurement strategies and understand the implications that they have – not just for employee development, but for organizational performance as well.

Realization 2: Learning Impact is Hard.

We were hoping that our research efforts would point to definitive things that all organizations should do. What we found was just the opposite. We talked to over 40 really smart leaders doing really innovative things when it came to how they were showing and having impact. But none of them were doing the exact same things.

Some of these leaders identified strongly with the idea of ROI. Some lived and died by Kirkpatrick, Phillips, or some other model. Some focused heavily on drawing a straight line from what they did to business results. But all of them were having at least some modicum of success. As it turns out, how learning impacts an organization is specific to that organization, its goals, and its resources.

Fortunately, patterns exist. While metrics and tactics should be aligned to the organization, business goals, and employee development needs, our literature review, roundtables, and interviews led us to 7 patterns that more evolved organizations follow for moving beyond simply showing impact to actually having impact.

Image 3: Learning Impact Patterns Found in More Evolved L&D Functions | Source: RedThread Reserach, 2019

Over the next 7 weeks, we’ll be sharing these patterns, examples of what organizations are doing, and questions you can ask yourself about your own learning impact practices.

Next article in series: Learning Impact: Tying to Business Goals.