Events

One of the main questions we had going into the Learning Tech Ecosystem Study was, “what are learning leaders considering when they create learning tech ecosystems?” While each organization is doing different things, several threads have emerged:

• User experience
• Skills!
• Data integration
• Sustainability
• Ecosystems for everyone

Experience

Number 1 answer? Survey says: Experience! A lot of the hype from the articles we reviewed in our literature review and almost every interview we conducted mentioned making the learning experience and supporting tech a little less Minesweeper and a little more Fortnite.

Employee experience, the ease of access, and usability are front and center in ecosystem decisions. Learning leaders shared the common desire use technology that allows them to make learning available to the employee and make it accessible in the moment of need to ensure a seamless experience.

The main reasons for this? 1) Establishing a clear signal through the noise to help employees understand what is truly important and beneficial to them; and 2) Matching the experience employees get internally with that they can get externally – hoping to engage employees and nudge them in ways that helps them personally and the organization as a whole.

Skills!

L&D leaders also highlighted the importance of shifting their efforts from providing role-based learning to skill and capabilities-based learning. As up-skilling and re-skilling become must-have conversations and development goal for leaders, democratizing learning was a goal that we heard frequently in our conversations.

This fundamentally changes the type of ecosystems that organizations need. They need to be more flexible, more integrated with the work employees are doing, and more adaptable. Instead of completing a curriculum for a role, organizations are encouraging employees to develop skills, and then using technology to help them determine where those skills can help them.

Data Integration

L&D leaders are also beginning to think much more seriously about data. In our interviews, almost every learning leader mentioned the usefulness of data – not just to help them do their own jobs better, but to provide more information to the larger organization and to individual employees by seamlessly integrating data from different functions: talent, staffing, recruiting, career, performance, and experience.

While this was one of the largest challenges that L&D leaders raised, they were also thoughtful in their responses when we asked them how they were addressing it. Many are going beyond the analytics provided by any one tool and instead finding ways to consolidate that information. Answers ranged from using an ‘anchor’ technology as the system of record and integrating data feeds from other tools to using Power BI or simple Tableau dashboards to gather, crunch, and present metrics.

"The current state of the market is requiring CLOs to become CIOs."

Interviewed Learning Leader

Sustainability

Another point that leaders surfaced was this idea of sustainability in two ways: the viability of the individual technologies in the ecosystem long-term, and the completeness and effectiveness of the ecosystem as a whole.

Several leaders were cautious of using newer technologies because of the current consolidation of the learning tech market, and it is difficult to know if they’ll be around in a year. One leader shared a few questions he asks himself when vetting tech:

• How long has the organization been in business?
• Are venture capitalists betting on the technology?
• Do their other customers have similar challenges to ours, and does the vendor partner well?
• Are the companies young and nimble enough to react?

While it’s never guaranteed that vendors of any sort have long-term sustainability, asking the right questions upfront can save a lot of headache down the road.

Many leaders also mentioned their need to have their ecosystems grow and morph along with their organizations and its needs. We found that leaders with this need are often open to more experimentation, but also much more insistent that vendors play nicely together. In fact, one leader is approaching this by conducting joint sessions with several vendors in their ecosystem. This leader brings these vendors in together to help them understand their company needs and asks them to work together on solutions that will meet those needs.

Ecosystems for Everyone

Finally, an unexpected concern of many leaders is designing an ecosystem that equally benefits the deskless employees, those located in rural environments, and the hesitant adopter.

Deskless workers currently make up about 80% of the global worker population,¹ and yet, many learning technologies overlook them: from not addressing them at all (a misstep) to assuming that making something mobile and responsive solves all problems (it doesn’t). Leaders we talked to emphasized the differences in audiences and identifying technologies that will work for all, or bifurcating the ecosystem so that it will serve all.

Really great tech ideas can be derailed over the simplest things: bandwidth for instance. As one of our roundtable attendees put it, “It’s hard trying to make a seamless experience especially for those employees who are not even online.” Leaders are in need of solutions that allow their deskless employees (and those in more rural parts of the world) to access learning on their personal mobile devices on their own time instead of locking it behind a firewall.¹

Additionally, with several generations (sometimes up to 5)² working together in a company, the rate of adopting new technologies can vary, ranging from enthusiasm to reluctance. A learning leader we spoke to is tackling this by instituting a blended approach of technology enabled content being delivered by an instructor in a classroom setting.

"It's hard trying to make a seamless experience especially for employees who are not even online."

Interviewed Learning Leader

What’s happening in the world of D&I since February 2019?

Since we published our report early this year, the space and the market has continued to gain momentum. As we had predicted, existing bias within algorithms are coming front and center in the D&I space.¹ In May, the U.S congress proposed a law that would hold companies accountable for algorithms that result in discrimination and would require them to regularly evaluate their tools for accuracy, fairness, bias and discrimination.²

As we predicted, existing bias within algorithms are coming front and center in the D&I space.

Bias in artificial intelligence (AI) has become a larger issue as more of it enters our lives. In our broader society, one of the most notable examples is that of gender bias in virtual assistants – that most of these assistants have female names and voices (think Alexa, Siri, etc.) and have submissive personalities. In response, UNESCO recently published a report that points to the importance of digital skills among girls and women and the lack of diversity in the technology sector.³ To challenge these biases, Virtue, the creative agency owned by publisher Vice, in collaboration with Copenhagen Pride, Equal AI, Koalition Interactive and thirtysoundsgood, developed a genderless virtual assistant called “Q”.⁴ As part of their #BiasCorrect campaign, Catalyst, along with Burns Group, launched a Slack plug-in that alerts the user when biased language is being used against women.⁵

We heard from a number of new D&I tech vendors after we published our February report.

As technology becomes more advanced and enters into new fields, companies are starting to find unexpected and unforeseen D&I use cases for technology not designed for that particular purpose. One such example is Microsoft, which recently piloted their fingerprint biometrics-enabled cards for social benefits in Mexico’s Sonora state. This technology proved to be extremely popular with the senior population. While the initial objective was to improve safety and security in card related transactions, the elderly population reported they were able to use it with much greater ease. The technology will allow Microsoft to securely, transparently and reliably move funds to the people who might otherwise struggle to access them.⁶

What have we done in the last six months on D&I tech?

On our end, we have had a lot of conversations about diversity and inclusion and technology. For example, we shared our most recent update on D&I technology at UNLEASH 2019 in Las Vegas and in countless other conversations. In addition, we just launched our new one-page D&I tech infographic, to share our findings more succinctly. Further, we spoke about the power of networks and technology for career women at the ASU/GSV summit and launched a related study on Women, Networks, and Technology. You can see both our review of literature and some of the initial findings from our interviews; the final report will be published later this summer.

Technology providers with engagement platforms are also beginning to highlight the role such tools can play in promoting diversity and inclusion programs in companies through recognition, sharing, and feedback.

What new technology have we seen?

As we predicted, we heard from a number of new D&I tech vendors after we published the D&I tech report. We want to provide you with a round-up of the new technology providers and also update our interactive market map. In the sections below, we review the new technology we’ve seen since launching the report.

Acquire

As we stated in the original report, talent acquisition has the largest percentage of D&I technology solutions focused on sourcing and selecting diverse candidates. We identified the following new vendors in this space:

• Incluzion is a talent acquisition marketplace that provides companies with a way to recruit, hire and pay diverse freelance talent.
• Mom Source Network offers virtual networking to connect moms to others facing similar challenges and to connect those who are looking to return to the workforce with women currently working.

Talent acquisition has the largest percentage of D&I technology solutions focused on sourcing and selecting diverse candidates.

Develop & Advance

Our study on women, networks and technology led us to discover several vendors in the mentoring/career management space that are focused on career development for women and minority groups:

• Everwise, a mentoring solution, helps companies with development content and curriculum for women leaders.
• River offers mentoring software and services to organizations, helping them support their employee development initiatives such as D&I and Leadership Development through structured formal mentoring programs.
• InstaViser offers cloud-based software that integrates with existing CRM software and other management platforms, helping corporates, academic institutions and non-profits manage and scale their mentorship programs. Features include smart matching, specific diversity mentoring, scheduling, video and audio conferencing, and more.

Image 1: Screenshot of InstaViser’s technology | Source: InstaViser, 2019.

Engage & Retain

We touched upon the role technology, such as virtual reality, can play in sexual harassment training and in allowing anonymous reporting in our study. Over the course of the past few months, we came across several additional vendors that are developing technologies that help report and address issues of workplace harassment:

• Bravely provides a platform for confidential conversations with coaches, allowing marginalized groups and minorities within the organizations to speak up without the fear of retribution and to talk about issues they face at work.
• Project Callisto provides a platform, called Callisto Expansion, that allows survivors of professional sexual coercion to securely and anonymously store information about their perpetrator. It connects survivors to attorneys who can help them understand their options and available actions.
• tEQuitable helps companies reduce misconduct and proactively prevent bias, discrimination, and harassment. It offers employees a safe place to navigate issues ranging from micro-aggressions/micro-inequities to overt discrimination. The product also provides the company with data on behavioral trends, identifies systemic culture issues, and recommendations for remediation.

Image 2: Screenshot of tEQuitable’s Technology | Source: tEQuitable, 2019.

• Vault allows users to report sexual harassment in a confidential and safe way. These reports can be kept private until the user is ready to share them with the case manager. The platform also allows other employees to weigh in on an incident at the same time in a structured manner and the reports are time-stamped.
• Another vendor, Woices, also launched a mobile app that allows users to report incidents of harassment and share them anonymously. While still in its initial launch phase, the solution will be able to use AI to identify patterns in reporting and make recommendations based on insights.

Image 3: Screenshot of Woices Technology | Source: Woices, 2019.

In our search, we also came across platforms that allow women to communicate and share openly:

• Elpha is a community where women in tech talk candidly online. One of their recent initiatives focuses on identifying top companies for women, and the results based on ratings and feedback submitted by over 1,000 of their members which will be published on their website.

Image 4: Screenshot of Elpha’s technology | Source: Elpha, 2019.

• Similarly, while not a technology solution, Panda provides a platform for women to connect and network across the globe. Once part of their directory, members can network and exchange ideas through it.

Technology providers with engagement platforms are also beginning to highlight the role such tools can play in promoting diversity and inclusion programs in companies through recognition, sharing, and feedback:

• Inspirus, a Sodexo Group company, promotes diversity and inclusion initiatives through engagement, recognition, micro-learning and events. The Inspirus Employee Engagement platform allows employees to participate in D&I training courses and track their progress, recognize inclusive behaviors, and promote awareness of D&I-focused events.

Image 5: Screenshot of Inspirus Technology | Source: Inspirus, 2019.

Analyz

• Affirmity, focused on analytics and pay equity in the diversity and inclusion space, launched its ERG platform early this year to help organizations manage and scale their employee resource groups. It also offers an ERG mobile app to employees.
• Paygaps.com is a cloud based platform that simplifies the gender pay gap reporting and data analysis process.

Image 6: Screenshot of Affirmity’s Technology | Source: Affirmity, 2019.

Organizational Network Analysis (ONA) is emerging as a growing field that organizations are increasingly looking to use to identify opportunities for greater inclusion, understanding gender differences in behavior, identifying high-potentials, and building more effective relationships with underrepresented populations:

• Humanyze measures collaboration in organizations with advanced analytics, ONA, and behavioral science to increase the speed and accuracy of operational decisions.
• Another ONA vendor, Innovisor helps companies break down collaboration barriers by revealing gender issues and biases.
• Finally, Polinode provides a platform designed to collect, analyze and visualize data on relationships within organizations.

Image 7: Screenshots of Polinode’s Technology | Source: Polinode, 2019.

Market Map

We have also updated our interactive D&I technology market map and moved it to the RedThread website. You can access the new interactive map of the vendors here, as well as the original, comprehensive report here, in which we explore in depth:

ONA is emerging as a growing field that organizations are increasingly looking to use to identify opportunities for greater inclusion.

• The history and advent of this inflection point for D&I
• Detailed market observations and analysis on the D&I technology market
• The trajectory, acceleration and shape of the D&I technology market
• Potential risks and benefits of D&I technology
• Specific types of technology solutions
• Detailed vendor landscape and product offerings
• Case studies and customer perspectives
• Predictions for D&I technology over the next 18 months

Conclusion

As is evident from the number of new additions of technologies over the past few months, the D&I technology market continues to grow rapidly. This has been equally matched by the growing concerns over exacerbating existing bias as well the legal risks associated with them. Incidents of data privacy breaches such as those at Facebook and Google have brought issues of data security and ethics to the forefront, making employees and organizations wary of adopting new technologies. The number of lawsuits alleging sexual harassment have been on the rise as well, as stated by the U.S EEOC’s October 2018 report on sexual harassment.⁷ Interestingly, there has been a drop in the number of high-profile accusations, which may reflect a change in organizations approach to handling such cases (potentially responding proactively versus reactively).⁸ As the D&I technology market grows, the human component remains a crucial element. Leading practices, challenges faced, and lessons learned will be crucial sources as organizations look for ways to effectively manage these technologies.

Finally, we will be updating our D&I technology tool on a regular basis as we come across new vendors in this space. If you are a technology vendor in the D&I space and think you should be included in our tool, but are not, reach out to us here.

Appendix A: Table of Referenced Vendors and Capabilities

Introduction

In the spring of 2019, when summer still felt like a distant dream, we conducted an online poll and more than 20 interviews with people analytics practitioners and solution providers to understand their current challenges and what they most wanted to know about the people analytics technology market.

We did this so we could better understand what our overall research – including our online survey and literature review – on people analytics technology should focus on. We figured you’re likely curious to know about what we learned, so we’ve summarized our findings in this article, specifically focusing on

• Poll results
• Interviews: what we heard
• Overall impressions
• Our next steps

Poll results

Our poll revealed some interesting insights into what is on top of minds when folks think of people analytics technology. We asked respondents (vendors, practitioners, and others with an interest in the field) to tell us what they think we should 1) cover in our research, 2) not cover in our research, and 3) any other feedback or ideas they wanted to share. The findings reaffirmed our suspicions: the top issue is better understanding the solution categories and capabilities for people analytics technologies.

Interviews: What we heard

Capabilities and needs – mo' solutions, mo' problems?

As our poll reflects, more than 70 percent of respondents indicated they want more clarity on people analytics technologies’ capabilities and a framework for thinking about the different solution providers. This sentiment was echoed in our interviews: when we asked folks to provide a way to categorize the market, we received as many different frameworks as interviews!

Our conversations revealed that while some group people analytics solution providers into categories of data storage, analysis, and visualizations, others think of them in terms of solutions that perform transactional/reporting versus analytics functions. Still others separated solutions into pure-HR players versus ones that cater to business intelligence capabilities.

Ultimately, both technology providers and organization leaders recognize the opportunity to work together to identify and use technology in a meaningful and purpose-driven manner. To do so, buyers and users should have a realistic and accurate understanding of the technology.

A good starting point for the users could be clarity, consistency and agreement on how they define people analytics and success within the organization. Buyers could also work with the vendors more, especially at the beginning of implementations, to put in place processes to ensure insights and findings from using the technology are shared broadly in an actionable way.

Similarly, vendors need a clear understanding of the business and talent challenges as well as the objectives of the organization. One of the vendors we spoke to stated,

“ typically start by saying that they have a data problem…and are trying to wrangle the data etc. Once we talk to them though, the problems that they are trying to solve for are business problems, such as turn over issue and are trying to wrangle the data to figure out what is driving the turnover.”

This points to the need for customers to spend time clarifying the business problems they are trying to solve and for vendors to spend time ensuring they understand those problems before implementing a solution.

Data Integration – Connecting the Dots

A topic that frequently popped up in our interviews was the opportunities and challenges of data integration from different sources. Some of the key questions we heard from practitioners during our interviews included

1. Are there people analytics tools that can help practitioners integrate their data and allow for data interoperability?
2. Which technologies speak to each other and can be used as an integrated suite?
3. Do the existing tools require data be within one system or should it be siloed in different systems?

While some of these concerns tie in with the issue of capabilities that we touched upon earlier, it is clear that data integration is a major challenge  Practitioners we spoke to mentioned that a growing number of vendors claim to offer multiple services and solutions for all things people analytics, but fail to provide that. For example, one person stated,

“ really wish the technology vendors would stop promising a one-stop shop for all your analytics need…These technologies require a lot of thoughtfulness in implementation to have them be useful.”

Further, other practitioners indicated a frustration with vendors who have incremental additional charges for additional data sources. While no one likes being “nickled and dimed,” these practitioners were especially sensitive, as they felt that for the extra time, effort, and cost to integrate these additional data, they could build the analysis capabilities themselves for roughly the same cost, and with additional flexibility.

Data Security and Ethics – Collect, Measure, Share?

A recurring concern expressed by practitioners and vendors alike centered around the issue of compliance and ethics in data collection, use, and sharing — almost a third of those who responded to our poll listed issues of ethics, access, privacy, ownership, and security as a top issue they want to better understand. Vendors we spoke to also brought up issues of data democratization such as how do businesses view their position in terms of data transparency and what should they be doing with it.

One of the vendors we spoke to expressed concerns about not keeping employees in the loop with regards to the data collected on them.  As tools that monitor and collect passive data (which can often be sensitive ) become more popular, managing and sharing that data with employees is increasingly becoming an issue of concern and was frequently brought up in our conversations.

Another vendor we spoke to referred to the recent data privacy breach incidents at Facebook and Google and their impact on the conversation about data sources by bringing it to the forefront.

Some of the concerns raised by people analytics practitioners and vendors regarding this issue include these questions:

1. How transparent are the solution providers regarding the data they are collecting and providing access to among the various levels within the customer's organization?
2. What are some of the decisions made by solution providers to help customers protect their employees?
3. How much of the data do practitioners tell their employees they are collecting?
4. How involved and informed are the business leaders about the data that people analytics practitioners have?

Implementation and End-User Value – Will the Real People Analytics Solution Please Stand Up?

While there is a lot of marketing about the value of people analysis technology, practitioners want to understand – ideally from other practitioners – others' implementation experiences and how quickly and efficiently those solutions delivered value.

In particular, practitioners would like to see more real customer case studies and to have opportunities to discuss the “how-to” and “lessons learned” with other practitioners.  This will allow them to align their expectations and objectives and achieve greater value from these solutions.

In addition, practitioners would like more information about whom the technology is designed to serve. For example, they reported that they will often hear that a solution can be made accessible to HR Business Partners, only to find that it is too technical or difficult for someone who is not especially data-savvy. By having a more realistic understanding about whom the system is really designed for, people analytics practitioners will be better able to understand and highlight the value of specific technologies.

Vendors are also interested in gaining greater clarity on who the organization leaders see as the end-user of the product and the expectations and practical ways organizations intend to use their solutions. They want to understand from practitioners not only what they want, but for whom, and why they want it. As one of the interviewees we spoke to put it,

“ require a level of sophistication from HR folks to go into the tool and identify the questions they should be asking….The people analytics team, the HRBPs, or the communications… have such varied levels of maturity.”

And while upskilling and building data capabilities for the wider section of users is crucial, it should be matched by a clear identification of the primary user.

Overall Impressions

Both the conversations and poll results are reflective of our findings from the literature review: the people analytics technology landscape is a rapidly growing space with great potential. However, there remains an urgent need for greater clarity and knowledge sharing on issues of various technology capabilities, ethics and security, data integration, and transparency about real use cases on scaling insights for business value and custom solutions that fit business needs.

Our Next Steps

We recently closed our online vendor survey (and thank the more than 35 vendors who participated!). We are analyzing that data and holding vendor briefings over the course of summer. We will also be conducting a second round of interviews with practitioners during the summer which, along with the survey results, will inform our final findings. These findings will be presented in the fall at PAFOW Philadelphia conference, with the final report published later in the fall.

Why is employee experience so sizzling?

Employee experience (EX) is hot. Don’t believe us? Just check out how it has trended on Google over the last decade.

While the term “EX” just sounds smart and snappy, there are more substantive reasons for the increased focus on employee experience, such as changing demographics,¹ a hypercompetitive talent market,² and business’ ravenous need for innovation.³ The heightened emphasis on EX is such that, in a global 2018 study of five hundred CHROs, 83% of organizational leaders emphasized a positive employee experience as crucial to organizational success.⁴

All this EX talk led us to some “burning” questions like, what exactly is this secret “employee experience” hot sauce? And if your organization chooses to focus on it, how do you approach it? And will it live up to all the hype?

To get our answers, we read the most relevant published works on employee experience, ranging from as early as 2011 to as recently as a few months ago (though the bulk of the articles were from the past three years). We also looked at related topics, such as employee engagement, customer satisfaction, and technology. In all, we examined over 60 scholarly and business articles, reports, blogs, and books.

This article gives you the highlights of what we learned and suggests publications for further reading. Here’s what you’ll find below:

• What we saw
• What caught our attention
• Additional literature we recommend

What we saw:

We identified six primary themes within the literature we reviewed:

1. Employee experience: A combination of employees’ perceptions and interactions
2. Employee experience and engagement: Related but different
3. Symbiosis: At the heart of employee experience
4. Multi-method listening: Key to understanding employee experience
5. HR: Not the only one responsible for employee experience
6. Technology: Clarifying and enhancing employee experience

1. Employee experience: A combination of employees’ perceptions and interactions

It turns out, lots of folks have different employee experience “recipes” – meaning there’s no universal definition for employee experience. Instead, we found a range of definitions, such as:

• How difficult it is to get work done and how people are expected to behave⁵
• How the work environment and work habits enable employees to perform their jobs⁶
• How employees interact with the physical environment, social connections, and work tasks⁷

We found the most convincing definitions, though, to be those that took a macro-level perspective and argued that employee experience originates in employees’ perceptions and interactions.^{8, 9} These perceptions are subjective interpretations of what employees encounter, observe, or feel in their interactions with the organization.¹⁰ Interactions are further described as the intersection of cultural, technological, and physical elements of the work environment with employees’ expectations, needs, and wants.¹¹

2. Employee experience and engagement: Related but different

Employee experience and engagement are often used as synonyms, but our review of research-based articles underscores that these are different (though related) concepts.

For example, engagement and employee experience differ in their top-down vs. bottom-up approach.^{12, 13} Employee engagement is seen as a top-down concept because organizational leaders choose strategies that impact engagement scores. On the other hand, employee experience is seen as a bottom-up concept because it focuses on those perceptions and interactions employees have – not just on the measured engagement score.

Another difference between employee experience and employee engagement is that experience is described as the cause, whereas engagement is described as the effect.¹⁴ The combination of what organizations do, say, and give (experience) influences what employees see, feel, and hear (engagement).

These perspectives bring the employee front and center. They highlight the influential role that organizations play in how employees behave and approach their work.

3. Symbiosis: At the heart of employee experience

Just like honeybees and flowers, a positive employee experience results in a mutually beneficial relationship for both employees and organizations. This relationship is based on the impact that a positive employee experience has on engagement:

• Organizations with transformational leaders and a supportive work environment – both aspects of employee experience – have better employee engagement, based on a comprehensive review of 214 studies.¹⁵
• Employees with a high level of engagement are healthier and feel an overall sense of wellbeing. In contrast, employees with a low level of engagement experience more illnesses, stress, and burnout.

While this is great news for employees, they are not the only ones that benefit. While there is plenty of research that indicates a positive relationship between employee engagement and business outcomes, there is also research that indicates organizations receive business value directly from a positive employee experience. Specifically, companies with a strong employee experience have twice the innovation and customer satisfaction and have higher profits than organizations with a weaker employee experience.¹⁶

4. Multi-method listening: Key to understanding employee experience

More organizations are starting to recognize the dynamic nature of employee experience and the need to account for spontaneous and deliberate interactions.¹⁷ A dynamic employee experience evolves when employees interact with the organization in organic ways or through specific touchpoints.

To view the entire employee experience picture, organizational leaders increasingly want effective ways to measure both constant and sporadic interactions. Thus, a multi-method listening approach is gaining popularity.¹⁸ In addition to pulse or touch-point triggered surveys, organizations are also exploring the use of wearable technology devices and social networking applications to collect frequent data from various sources.^{19, 20}

5. HR: Not the only one responsible for employee experience

Employee experience is often viewed as the primary responsibility of human resources or talent management functions. However, that viewpoint is changing.

HR does not own most aspects of employee experience, whether you measure it via “touchpoints” (42%) or “moments of truth” (78%).²¹ Instead, business managers or other non-HR support functions own them.

Thus, some authors recommend building a cross-functional coalition where members can approach employee experience from different, yet complementary angles based on their expertise.²² This cross-functional coalition includes shared accountability across HR, marketing, IT, finance, and facilities to design a positive employee experience.

6. Technology: Clarifying and enhancing employee experience

Technology provides an opportunity to better understand employee experience. It clarifies what employees want, value, and need. In fact, most CHROs (57%) believe that technology enables them to prioritize budget and time investments on employee experience.²³

Organizational leaders also recognize that technology enables them to analyze, automate, and collaborate when designing a positive employee experience. For example, more technology vendors are offering artificial intelligence capabilities that provide a deeper view of employee experience than ever before.²⁴

There is also an increased focus on providing a personalized digital experience that resembles what employees encounter as consumers. Employees want technology that enables them to communicate and connect with others in the organization.²⁵ They also want technology that anticipates their needs and streamlines administrative processes. A large global survey highlights that most workers prefer digital as opposed to personal interactions to handle HR tasks.²⁶

But there is a large gap between the technology that employees encounter at work and the one they encounter as consumers. To address it, organizations want to offer digital solutions that enable collaboration, anticipate employees’ needs, and appeal to individual preferences.

What caught our attention:

Of the literature we reviewed, we found the articles below particularly intriguing and useful. We learned from their perspectives and encourage you to do the same.

Maximizing the Employee Experience: How Changing Workforce Dynamics are Impacting Today’s Workplace

Dr. Brad Harrington

"Taking a systems-oriented approach as the foundation of an organization’s people strategy will require more cross-functional collaboration with HR’s various functions and much greater alignment and integration with organizational leaders."

Highlights:

• Summarizes the significant workplace trends that influence organizations trying to positively impact employees' experience.
• Describes the importance of developing and establishing an integrated people strategy that considers the organization’s mission, values, and workforce expectations.
• Suggests that using a seven-step change management model may help in maximizing a positive employee experience.

Building Business Value with Employee Experience

Kristine Dery and Ina Sebastian

“In our research, companies with great employee experience  (i.e., low work complexity, and strong behavioral norms for collaboration, creativity, and empowerment) were more innovative and profitable and had higher levels of customer satisfaction.”

Highlights:

• Discusses findings from MIT Sloan Center for Information Systems Research’s recent global survey on employee experience.
• Defines employee experience according to work complexity and behavioral norms.
• Connects a positive employee experience to innovation, customer satisfaction, and profitability.
• Provides recommendations on how to build a positive employee experience through digital capabilities and leadership.

Employee Experience vs. Engagement, and 3 Things You Should Start Thinking About Now

David Sturt and Todd Nordstrom

“As leaders, employee experience is something we all should start looking at seriously… because, like it nor not, there’s a silent revolution taking place with employees all over the world. And, if we do our best, we will see those engagement scores move in the right direction.”

Highlights:

• Discusses differences between employee experience and employee engagement.
• Provides examples of ways in which organizations have tried to foster a positive employee experience.
• Offers suggestions to build a positive employee experience by focusing on organizational culture, flexibility, and storytelling.

With CX, Engaged Employees Mean Everything

Bruce Temkin

“In companies that lead in CX, 75% of employees are highly or moderately engaged.”

Highlights:

• Connects customer experience to organizational culture and internal processes.
• Substantiates the notion that a positive customer experience is linked to highly or moderately engaged employees.
• Recommends specific actions to increase engagement via the “Five I’s of Employee Engagement.”

Designing Employee Experience: How a Unifying Approach Can Enhance Engagement and Productivity

Eric Lesser, Janet Mertens, Maria-Paz Barrientos, and Meredith Singer

"Applying a holistic, iterative design approach to change can help ensure that employees see improvements relevant to their work and can set their expectations for continuous intervention."

Highlights:

• Identifies factors that shape employee experience such as work-based relationships, the design of employees’ physical work environment, and the tools employees use in their work-related activities.
• Emphasizes the importance of personalization, transparency, simplification, authenticity, and organizational responsiveness as ways for organizations to enhance employee experience.
• Offers suggestions to design a positive employee experience through using analytics, understanding and investing in crucial touchpoints, and building a coalition.

How AI Can Help Redesign The Employee Experience

Forbes Insights

“The hope isn’t simply for AI to ensure smarter, faster hiring, but to improve the entire lifecycle of the employee experience, from job application to exit interview. In other words, can AI make workers happier?”

Highlights:

• Identifies ways in which AI can create a positive employee experience.
• Mentions specific tech vendors that gather data and provide insights into an organization’s employee engagement, manager effectiveness, communication, talent development, and organizational culture.
• Describes how AI technologies can provide a “consumerized” employee experience to address HR needs.

Additional Recommended Reads:

1. “The Employee Experience in 2019,” Charong Chow, SocialChorus, 2019.
2. “The Heart of the Matter: Emotions in the Employee Experience,” Christina Zurek, ITA Group, 2019.
3. “Employee Experience: Enabling Your Future Workforce Strategy,” Kristine Dery, Nick van der Meulen, Ina Sebastian, MIT Center for Information Systems Research, 2018.
4. “The New CHRO Agenda: Employee Experiences Drive Business Value,” Pat Wadors, ServiceNow, 2019.
5. “Employee Experience Reimagined,” Michael Liley, Patricia Feliciano, and Alex Laurs, Accenture, 2017.
6. “2018 Employee Experience Report,” Toluna Group & Udemy, 2018.

While we have identified the articles above as being the most crucial for readers to review, we read many others. If you’d like a full list of the articles we covered, please do not hesitate to reach out to us at [email protected].

Footnotes

 

Did you know that employee experience can affect the experience of your customers? The following slides highlight initial findings from our Employee Experience / Customer Experience study.

Red threadresearch excx_medallia experience_final from Dani Johnson

Introduction

It’s old news that gender diversity matters.¹ Yet, while the discussion around and evidence for gender diversity in organizations has grown, the number of senior women – especially women of color² – in organizations has not.

For years, organizations have tried approaches such as mentoring and sponsorship programs for women and providing women-focused employee resource groups. Given that representation numbers remain stubbornly low, we clearly need a different approach.

We think at least part of the solution lies in better understanding employees’ professional connections. Networks, and the informal information they facilitate, are one of the primary ways people learn about career advancement and development opportunities. Research³ suggests that women and men's networks – and the information within them – are different, thus networks could vastly influence opportunities for career development and advancement. Understanding these connections between people – who knows whom and why – could help organizations understand why some employees rise and why others do not.

We looked at approximately 50 academic and business articles, reports, and books to better understand network dynamics, how they differ by gender, and the potential opportunity for technology to democratize or accelerate women's effective use of their networks. This short article will summarize the literature we reviewed in terms of the following:

• What we saw
• What we didn't see
• What we learned

What we saw:

This literature review focused on research covering the fundamental differences between the networks of men and women and the resulting different outcomes. We also reviewed literature on the approaches organizations can use to help women create networks that support their advancement. The findings from this review fell into three major themes:

1. Professional networks are different between men and women
2. Differences in networks result in different outcomes
3. Specific network characteristics matter for advancement

Professional networks are different for men and women

Men and women have different networks, at least partly due to traditional social dynamics⁴. Traditionally, women have been primarily responsible for life at home and, therefore, created more ties with the community, family, and friends. Likewise, men have traditionally been responsible for the financial stability of the family and created more professionally-focused networks.

While these traditional social dynamics have helped shape professional networks, other variables at play in organizations likely have a more immediate impact on the structure and composition of professional networks.

For example, men tend to hold a larger percentage of leadership positions and to be promoted more often than women. As a result, when men move up the ranks in organizations, they are more likely to be surrounded by other men (rather than women) in similar leadership or influential roles. As information and job opportunities tend to be shared within networks, men continue to have better opportunities due to their participation in these networks, perpetuating the gender imbalance at higher organizational levels.

Women, on the other hand, are, generally speaking, less likely to be promoted into senior roles. Those that are promoted tend to find themselves as part of a male-dominated professional network (only one in five C-suite executives is a woman⁵).  Women who do not move into leadership roles often sit in less influential, lower-status positions in organizations.⁶ These positions do not afford women the same connections into higher-status networks, curtailing access to information and sponsorship for better development and career opportunities.

Differences in networks result in different outcomes

While on the surface network differences may not seem like that big of a deal, those differences affect how individuals move around and up through an organization.  There is a tendency for professional networks to be closed, meaning individuals in one group are all connected to each other and the individuals within those networks have access to and share the same information. Therefore, new and unique information often does not flow into the network.

Closed networks tend to impact women differently than they impact men.⁷ For men, as they are more likely to be in networks with others in higher-status roles, information that is shared is likely to be important or strategic, which benefits them. Whereas for women, who are less likely to be in high-power networks, closed networks result in less important, less strategic information sharing.

Specific network characteristics matter for advancement

The literature indicates there are several things individuals should consider when developing networks to facilitate career growth. While these practices are not unique to women’s networks, being aware of these characteristics will enable women to better leverage their professional networks for advancement.

First, women need a small “inner group” of connections who can provide immediate support. Individuals benefit from being a central figure in a close-knit group. The literature indicates that these inner groups are crucial because they create a sense of communality and can tap into a woman’s shared experience.

Second, women need to become “brokers” – people connected to fundamentally different networks that can provide them with new and diverse sources of information.⁸ An individual who can position themselves as a broker between two high-powered groups, or who is directly connected to a broker, has a higher likelihood of accessing unique information and opportunities. The literature also hints that while this is an acceptable and even coveted position for men, women acting as brokers may experience some backlash (which could impact performance⁹).

This brings us to the third characteristic. To counteract backlash, women may also need to become “energizers” in their network. An energizer is someone that engages with others in a way that builds trust, instills a sense of purpose, and fosters an environment of psychological safety where people are not worried that they will be judged or dismissed. Because of this, energizers are presented with new opportunities and information more often than non-energizers. Research¹⁰ suggests that being seen as an energizer is a critical factor in advancement.

What we didn't see:

There were also a couple of thing we expected to see that just didn't show up:

What causes differences

The literature offered little insight into the actual mechanisms that cause differences in networks of men and women. So little has been written on the topic that we found ourselves accessing research in tangential fields to understand how and why these gaps may occur. This gap likely impedes organizational efforts to develop innovative and effective interventions.

Likewise, most of the research geared toward practitioners largely focuses on “fixing” women’s networks through mentoring, sponsorship, and Employee Resource Groups (ERGs). On the surface, these solutions seem like great ideas; however, because they are often not intentionally managed or designed, they may serve as a forum for the shared experience of women rather than a resource for helping women build networks and advance careers.

How technology can help

There was also little research that discussed using technology as a part of the solution to directly address the advancement of women. In fact, we feel that the gaps in current research have impacted the current technology solutions available. While there are technologies with network analysis capabilities, many – if not all – offer only descriptive reports on the structure and makeup of various networks. Descriptive statistics are a good start, but don’t point organizations toward potential drivers of these differences or suggest solutions.

What caught our attention:

In the process of putting this literature review together, we came across several articles that simplified complex ideas, offered unique solutions, or expanded our understanding of the topic.

The first three articles provide insight into social networks, how they impact aspects of our lives including work, and how different groups of women may face very different realities in the workplace. The final two articles are more academic and provide insight into characteristics of professional networks and their consequences for women. We’ve also included a bonus article that highlight network practices that are critical to the success of women in organizations. Finally, a list of additional readings is provided to offer further insight on the topics covered in this literature review.

Social Network Theory: Explaining Society

Tobias Stone

"…understanding the underlying mechanisms of social networks allows you to use them more effectively. Also, this descriptive analysis can explain some of the weirder things happening in our lives."

Highlights:

• Provides a high-level overview of social networks
• Breaks down basic aspects of social network theory and related concepts
• Explains how social networks impact almost every aspect of our lives

This three-part series (provided via text and/or audio) gives an interesting look at social networks and dives into how they impact our world – from happiness to politics. It also asks the question as to whether technologies that tap into our networks are friend or foe.

Mentorship, Sponsorship, and Networks: The Power and Value of Professional Connections

Center for Women and Business at Bentley University

"…when it comes to professional networks, women and men find themselves in very different positions."

Highlights:

• Provides a brief overview of important concepts and the benefits of mentorship and sponsorship to individuals and organizations
• Outlines a framework for understanding networking
• Offers insight on how to establish a mentorship culture for organizations and how to navigate networks for individuals

Not only is this a great review of the current research on sponsorship and mentoring, but it also brings in a network-based perspective. In addition, the list of resources and articles provided is a robust set of materials to help you further understand this topic.

Connections that Count: The Informal Networks of Women of Color in the United States

Catalyst / Katherine Giscombe

"Depending on the work environment, it may be difficult to form relationships at all with dissimilar colleagues, which would then lead women of color to turn to similar colleagues or people from outside the work organization for advice."

Highlights:

• Dives into the increased complexities of professional networks for women of color
• Discusses different network strategies women of color may use when dealing with barriers to organizational advancement
• Provides recommendations for organizations to think past a "one-size fits all" mentality when addressing the professional networks of women

This article highlights the complex network challenges that women of color face. It forces the reader to see that solutions designed for women do not always apply to all women and solutions should not advance one group of women over others.

Do Women Suffer from Network Closure?

Mark Lutter

"Females … have a higher risk of career failure than do their male colleagues when affiliated in cohesive networks, but women have better survival chances when embedded in open, diverse structures."

Highlights:

• Discusses the importance and impact of social capital, along with the two main ways that social capital is built (or borrowed) – cohesive and weak ties
• Illustrates differences in female and male networks and the practical impact that has on women's careers

This is an examination of how women's networks impact career advancement and is an example of what other research is starting to uncover.

To Land Top Jobs, Women Need Different Types of Networks than Men

Kellogg Insight

"While it's true that highly connected women tend to land better jobs, the most successful women also have something they cannot get through ‘beers with the boys’…"

Highlights:

• Summarizes the findings of a recent empirical investigation¹¹ on the impact of differences between men and women’s networks among MBA students
• Illustrates that women need a fundamentally different network to find career success
• Discusses the difference in business networking between men and women

If you can get your hands on the original empirical article that this summary is based on, we highly recommend it. However, if you cannot, this overview hits all the major points and interviews one of the researchers to add context.

Bonus: Invisible Network Drivers of Women’s Success: How Successful People Manage Collaborative Overload

Inga Carboni, Rob Cross, Aaron Page and Andrew Parker

“If the ultimate goal of gender diversity efforts is to build organizations in which employees work together and evolve as professionals without regard to gender, then let’s focus on the relationships themselves.”

• Suggests that understanding gender diversity requires organizations to think beyond implicit bias
• Presents the BEST model to help organizations understand the critical networking practices that distinguish high-performing women, including boundary-spanning, efficiency, stickiness, and trust
• Provides a number of tips for organizations to put their findings into practice

Additional Reading:

• • “What’s in the ‘old boys’ network? Accessing social capital in gendered and racialized networks,” S. McDonald, 2011.

• “The role of organizational network analysis in people analytics,” David Green, 2018.
• “Women and the labyrinth of leadership,” A. Eagly and L. Carli, 2007.
• “Why women build less effective networks than men: The role of structural exclusion and personal hesitation,” E. Greguletz, M. Diehl, and K. Kreutzer, Human Relations, 2018.
• “Taking gender into account: Theory and design for women’s leadership development programs,” R. Ely, H. Ibarra, and D. Kolb, INSEAD, 2011.
• “Gender, brokerage and performance: A construal approach,” R. Brands, Academy of Management, 2019.caut

It has been a tad shy of a decade since organizations began redesigning their approaches to performance management (PM). While previous approaches focused mainly on top-down, annual, staid processes, organizations now have to go beyond simply managing performance: they need to enable it. More specifically, organizations need practices that motivate, engage, and develop employees through a more collaborative, dynamic, and personalized process.

Yet, with all the changes made to PM, people are still unhappy, unmotivated, and disengaged. In addition, practices hailed as innovated and forward-thinking haven’t shown themselves to be the “cure-all” they may have been touted as.

So, the question remains – Where are we in the great PM experiment?

We have partnered with Glint to answer this question and looked at 40 academic and business articles, reports, and books for this literature review.

What we saw in the literature

Not surprisingly, the literature shows a general consensus that the traditional annual PM process isn’t enough. As it turns out, most traditional processes don’t drive engagement, often don’t encourage development, and don’t focus on the employee experience. While the research recognizes that some of the more traditional aspects of PM are still very necessary, organizations have experimented with ways to better engage employees in the process.

And experiment they have. We found literature on everything from completely ditching the performance review to facilitating continuous conversations, to adaptable goals. And while the sentiment is right, we think that there is an overemphasis in the literature on the process (i.e., the nuts and bolts of how we conduct performance management and its individual components), and an under-emphasis on the changing relationship between employee and employer (manager and up) and what that means for  performance. Specifically, four general themes emerged:

Traditional approaches are no longer appropriate

The traditional model of performance management – the one introduced in the mid 1990s and discussed ad nauseum as an evil necessity – is unlikely to yield the results organizations want because it doesn’t focus on feedback as an informal, real-time way to engage and develop talent. It also disregards the importance of developing and maintaining a relationship between manager and employee, and instead, rigidly sticks to a standardized cycle that is not aligned with how and when work gets accomplished or when feedback is needed.

A “one-size-fits-all” approach to performance management can’t handle the highly dynamic and customized world we live and work in because it doesn’t take into account the work type nor the people in the organization.

Because of this, organizations are shifting performance away from a complex, top-down system to a mechanism that can enhance employee experience – turning something that happens to the employee into something that happens for the employee.

Ratings aren't the problem – we are

Over and over again, authors, particularly in more recent publications, urge caution about removing ratings. In fact, some authors say that organizations have reached premature conclusions about ratings: instead of fixing them, they’re buying into the myth that they serve no purpose and should be removed.

However, the literature also indicates that removing ratings does not exonerate organizations from providing feedback or evaluation; and it may make it harder. Evaluation is necessary to provide meaningful and personalized feedback so employees can improve.

A bigger problem appears to be managers’ inability or unwillingness to diagnose and confront performance problems¹. This is particularly challenging in situations where ratings have been removed. And as organizations remove ratings, they need to rely more heavily on more frequent feedback in order to guide employees. Unfortunately, many organizations aren’t sure that their managers are either willing or able to have those feedback discussions². Which brings us to our third theme.

Relationships are increasingly important

As organizations have replaced annual performance reviews with more regular feedback conversations, there has been a necessary renewed focus on the relationships in organizations – particularly those between employees and managers.

To do this effectively, organizations have to trust their managers to move past just managing projects to truly managing people. This has expanded the role of managers from beyond simply assigning work to one that also includes motivating and engaging team members and holding individuals accountable.

The literature also addresses the growing trend of peer reviews – the practice of employees providing open feedback to each other. This has prompted organizations to begin to think about how best to create a “culture of feedback” where everyone is able to provide quality feedback.

Fairness matters

Finally, fairness. As organizations adopt more frequent feedback and more open conversations, they also need to think through how they create an environment of trust and fairness. More specifically, employees need to feel that the feedback they get is credible and fair – regardless of whether it comes from a peer or a manager.

The literature points out that in most PM processes, subtle forms of bias exist, and these biases can create different outcomes for different groups. For example, similarity bias may subtly influence a manager to provide slightly higher ratings for someone more like them (i.e., same likes/dislikes, same gender and race, similar background). These biases are particularly relevant when talking about the relationship between manager and employee. Most PM systems are not yet set up to protect against them.

Interestingly, removing ratings doesn’t remove the potential for bias – it can actually increase it. When organizations remove ratings, they often replace them with fairly ambiguous criteria for evaluation which allow for much broader interpretation. This broader interpretation can lead to perceptions of unfairness and violate norms of trust.

Articles that caught our eye

Of the literature we reviewed, several pieces stood out to us. Each of the following pieces explored ideas that we found useful and interesting. We found them helpful in expanding the way we have been thinking about PM, its challenges, and its possible solutions.

Performance Management: A Marriage Between Practice and Science – Just Say “I Do”

Paul E. Levy, Steven T. Tseng, Christopher C. Rosen and Sarah B. Lueke

“Spoiler alert: the fix is not to blindly get rid of ratings.”

This chapter discusses recent criticisms of traditional PM practices and reviews them in light of academic research. In an effort to reduce the gap between practice and science in PM, the chapter highlights what organizations can do to improve their PM practices and where scholars should focus their research efforts.

Highlights:

• Argues that practitioners are driving the criticism of PM and that the gap between science and practice needs to be addressed
• Suggests that solutions to address criticisms of PM should come from both a practical and research-based point of view
• Advocates that removing ratings should be the rare exception and not the general rule

"Re-Engineering Performance Management"

Ben Wigert and Jim Harter / Gallup, Inc.

“Performance management has buckled because organizations have prioritized measurement over development.”

This report presents research on why traditional practices are not working, insights on how to improve them, and expectations that today’s employees have for their employing organizations.  The authors recommend that organizations should create a culture of performance development by establishing expectations, continually coaching, and creating accountability.

Highlights:

• Presents the research behind why traditional PM is not effective in today’s organizations
• Discusses the changing nature of what employees expect from their organizations and how organizations can think through what (if any) changes are necessary in their approach to PM

"Straight Talk About Employee Evaluation and Performance Management"

Lucia Rahilly, Bryan Hancock and Bill Schaninger / McKinsey

“…there is still no substitute for the direct feedback and coaching that happens day in and day out…”

This podcast, with transcription provided, discusses recent research by McKinsey on what drives effective PM. The discussion focuses on the role of the manager to engage in quality performance and development conversations with direct reports, the need for some sort of evaluative component, and the finding that perceptions of fairness impact the degree to which PM is seen as effective.

Highlights:

• Discusses the current trends in PM and the necessary reliance on the ability of managers to provide coaching and feedback
• Explains that people still want to know how they’re performing and that some sort of evaluative component is likely necessary
• Illustrates the importance of perceptions of fairness in the PM approach

"3 Biases that Hijack Performance Reviews and How to Address Them"

Beth Jones, Khalil Smith, and David Rock

“…not all biases make us actively malicious. The key is how we manage our biases.”

The article discusses bias from a neuroscience perspective, highlighting that bias is our brain’s constant search for efficiency. While bias is not inherently bad, it can lead to negative outcomes if left unexplored. The authors discuss three biases – expedience, distance, and similarity – and how managers and organizations can mitigate their impact on performance appraisal.

Highlights:

• States that bias negatively impacts performance appraisal and briefly discusses the impact of three prevalent biases
• Provides high-level suggestions on how to mitigate the influence of these biases in performance appraisal

"Putting the System into Performance Management Systems: A Review and Agenda for Performance Management Research"

Deidra J. Schleicher, Heidi M. Baumann, David W. Sullivan, Paul E. Levy, Darel C. Hargrove and Brenda A. Barros-Rivera

"…much work is yet to be done in developing a body of scientific knowledge about performance management systems that can better inform practice."

While this article isn’t particularly provocative or stirring (it's why we put it last), it does provide a foundational summary of current PM research, which is helpful in understanding more progressive and innovative perspectives. The authors present a model of PM and summarize research from 1980 to 2017. Based on this review, they provide recommendations for future research in PM. This article is great for leaders and practitioners that want to geek out on the history of PM.

Highlights:

• Presents a model of PM to organize components of PM and to integrate perspectives
• Suggests that there are only seven core tasks involved in PM
• Illustrates the importance and value of both formal and informal components of PM
• Builds a case that we’ve excluded the examination of important variables in PM
• Argues that more research is still needed on PM

Overall Impressions

The literature on PM is vast and varied, and there are many, many smart people with different perspectives. We’re pretty sure no one perspective is the “right” perspective. That said, we’re starting to see large-scale agreement for the notion that traditional, top-down, annual-driven PM is less likely to reign supreme in the workforce of the future. With this shift, we think we’ll also see an increased emphasis on the role of relationships in organizations, the expectations of managers, and the importance of trust and fairness in PM approaches.

Introduction

If you have been following our Learning Impact project, you know that the main premise of this research is that we’re evaluating “learning” in organizations all wrong. Therefore, in conducting a fairly in-depth review of existing literature on the topic, we were not at all surprised about the state of learning impact – only somewhat disappointed.

We looked at over 50 academic and business articles, reports, and books for this literature review, which has given us a decent understanding of the known world of evaluating learning. This short article will summarize:

• What we saw
• What we learned
• Overall impressions

Word cloud of the learning impact literature: Most prevalent words in literature reviewed.

What we saw

What we hoped to see in the literature were new ideas – different ways of defining impact for the different conditions we find ourselves in. And while we did see some, the majority of what we read can be described as same. Same trends and themes based on the same models with little variation. We have highlighted four of those themes below.

Models, models, models!

Much of the literature focused on established evaluation models. These articles generally fell into three main categories: use cases, suggested improvements, or critiques.

By far, the most common model addressed in the research is Kirkpatrick, although, given the number of articles written on how to use it effectively, adaptation is still a challenge. It was frequently called the “industry standard” (including by Kirkpatrick Partners themselves).¹

Articles on Kirkpatrick appeared to be fairly passionate, either for or against. While many authors doubled down on it, we also ran across several articles, like this one or this one, that offer what we consider to be fair critiques of the model, including that there is a lack of empirical evidence for existing learning evaluation models and their tie to business results.

Other models, including Phillips’ Chain of Impact, Kaufman’s 5 Levels of Evaluation, Brinkerhoff’s Success Case Method, and Anderson’s Return on Expectation, among others, were also explored. In total, we looked at over 20 models. They are summarized in the table below.

Evaluation Model Summary

 

Model	Year	Incredibly simplified steps	Read more
Kirkpatrick 4 Levels of Training Evaluation Kirkpatrick	1976	Termed “the industry standard” by many of the articles we read, Kirkpatrick’s four levels are used widely to determine learning effectiveness. Reaction Learning Behavior Results	“The Kirkpatrick Model”
Kaufman’s 5 levels of Evaluation Kaufman, Keller, Watkins	1994	Kaufman’s model adapts Kirkpatrick’s original model to include 5 levels and is used to evaluate a program from the employee’s perspective. Input and process Acquisition Application Organization Output Societal outcomes	“What Works and What Doesn’t: Evaluation Beyond Kirkpatrick”
Success Case Method Robert Brinkerhoff	2006	Particularly effective in assessing important or innovative programs. It focuses on looking at extremes: most successful and least successful  cases, and examining them in detail. Needs assessment Program plan and design Program operation and implementation Learning Usage and endurance of training Payoff	“Success Case Method”
Chain of Impact Jack Phillips	1973	Adapts the Kirkpatrick model by adding a fifth step: ROI. Purpose is to translate business impact of learning into monetary terms so that it can be compared more readily. Evaluation Reaction Learning Behavior Results ROI	ROI Institute
Value of Learning Model Valerie Anderson	2007	Consists of a three-stage cycle applied at the organizational level. One of the few models that does not necessarily use the course or initiative as the unit of measurement. Anderson also introduced the term, “Return on Expectation” as a part of her work. Determine current alignment against strategic priorities Use a range of methods to assess and evaluate the contribution of learning including learning function measures, ROE – expectation, ROI, and Benchmark and capacity measures Establish the most relevant approaches for your org	“A new model of value and evaluation: A new model of value and evaluation”
CIPP Model Daniel Stufflebeam	1973	Framework was designed as a way of linking evaluation to program decision-making (i.e., making decisions about what happens to the program). Has a use case for resource allocation and/or cost-cutting measures. Utilizes the following areas: Context Input Product or results Analyze	“The CIPP Evaluation Model: How to Evaluate for Improvement and Accountability”
UCLA Model Marvin Alkins Dale Woolley	1969	Five kinds or need areas of evaluation – each designed to provide and report information useful for making judgments relative to the categories: Systems assessment Program planning Program Implementation Program Improvement Program certification	“A Model for Educational Evaluation”
Discrepancy Model	1966	Used in situations where there is an understanding that a program does not exist in a vacuum, but instead within a complex organizational structure. Program Cycle Framework: Design Installation Process Product Cost-benefit	“ABCs of Evaluation”
Goal Free Evaluation Michael Scriven	1991	Focuses on actual outcomes of a program rather than only those goals that are identified. Scriven believed that goals of a particular program should not be taken as a given. Goals and objectives Processes and Activities Outcomes	“The ABCs of Evaluation”
LTEM: Learning Transfer Evaluation Model Will Thalheimer	2018	Designed to help organizations get feedback to build more effective learning interventions and validate results. Tier 8: Effects of Transfer Tier 7: Transfer Tier 6: Task Competence Tier 5: Decision-making Competence Tier 4: Knowledge Tier 3: Learner Perceptions Tier 2: Activity Tier 1: Attendance	“The learning-transfer evaluation model: Sending messages to enable learning effectiveness.”

Justification as the goal

Much of the literature reviewed focused on utilizing learning evaluation, measurement, and analytics to either prove L&D’s worth to the organization or to validate L&D’s choices and budget. Words and phrases like “justify” and “show value” were used often.

Interestingly, according to David Wentworth at Brandon Hall Group, the pressure to defend L&D’s decisions and actions appears to be coming from the L&D function itself (44%), rather than other areas of the business (36%),² which means, while business leaders may not be explicitly asking for “proof”, L&D departments most likely feel the need to quantify employee development in order to have that proverbial seat at the table.
Literature also focused heavily on Return on Investment, or ROI. How-to articles and research in this space continues to attempt to tie the outcomes of a specific program or initiative to financial business results.

Course-focused

Almost all of the literature we reviewed utilized the ‘course’ or ‘program’ as the unit of measurement. While several models address the need to take into account environment and other variables, they appear to do so in order to either control the entire experience or be able to isolate the “learning” from everything else.

To date, we have not been able to find any literature that addresses evaluating or measuring continuous learning as we understand it (i.e., individuals utilizing the environment and all resources available to them to continuously develop and improve). We feel that this is a shortfall of the current research and should be addressed.

Finally, the research focused heavily on learning from the L&D function’s point of view. Few appear to be looking at the field of learning evaluation/measurement/analytics from a holistic viewpoint. We expected to see more literature addressing L&D’s role in delivering the business strategy, or at lease providing information to other functions that could be helpful to them in making decisions.

Aged ideas

While we do not disparage any of the great work that has been done in the area of learning measurement and evaluation, many of the models and constructs are over 50 years old, and many of the ideas are equally as old.

On the whole, the literature on learning measurement and evaluation failed to take into account that the world has shifted – from the attitudes of our employees to the tools available to develop them to the opportunities we have to measure. Many articles focused on shoe-horning many of the new challenges L&D functions face into old constructs and models.
We realize that this last finding may spark a heated conversation – to which we say, GOOD! It’s time to have that conversation.

5 articles you should read

Of the literature we reviewed, several pieces stood out to us. Each of the following authors and their work contained information that we found useful and mind-changing. We learned from their perspectives and encourage you to do the same.

Article 1:  Making an Impact³

Laura Overton and Dr. Genny Dixon at Towards Maturity

“96% are looking to improve the way they gather and analyze data on learning impact. However, only 17% are doing it.”

Highlights:

• Points out areas where L&D functions measure and what is important to them
• Provides an interesting discussion on evidence to understanding impact
• Shows compelling data about benefits of those who measure vs. those who guess
• Gives some good hints for getting started

Towards Maturity’s 2016 report provides some interesting statistics about the world of learning metrics / measurement / analytics / evaluation. This article provides a sound platform for continued research on learning measurement and evaluation, as it provides a good summary of how learning leaders are currently thinking about the space.

Article 2:  Human Capital Analytics @Work⁴

Patti Phillips and Rebecca Ray at The Conference Board

“Aspirational organizations use analytics to justify actions. Experienced organizations build on what they learned at the aspirational level and use analytics to guide actions.”

Highlights:

• Outlines an analytics maturity model for organizations to gauge their evolution when it comes to using data.
• Provides a good discussion on “converting data to money”, or utilizing data to provide a comparison of cost savings
• Identifies four key elements to help organizations make analytics work: Frameworks, Process Models, Guiding Principles, and Capability, all of which should be considered when putting together a learning strategy
• Recounts some good examples and case studies

This article broadens the discussion about learning measurement to people analytics in general – something that L&D functions should be considering as they revamp their measurement and evaluation methods.

Article 3: The Learning-Transfer Evaluation Model⁵

Will Thalheimer at Work-Learning Research, Inc.

“For too long, many in the learning profession have used these irrelevant metrics as indicators of learning.”

Highlights:

• Honest (yet biting) assessment of the current 4-level models and their success to this point in time
• Section about the messages that measurement can send
• Discussion on measuring negative impact, as well as positive impact
• Introduction of the first new model for learning evaluation in about 10 years

Will addresses several points that have evolved our thinking. On top of that, Will is a witty writer who is easy to read and downright entertaining.

Article 4: Making data analytics work for you – instead of the other way around⁶

Helen Mayhew, Tamin Saleh, and Simon Williams

“Insights often live at the boundaries.”

Highlights:

• Emphasizes the importance of focusing on “purpose-driven” data, or data that will help you meet your specific purpose.
• Introduces the idea that large differences can come from exploiting and amplifying incrementally small improvements.
• States that incomplete information is not useless and should not be treated as garbage – it has value and can be essential in helping people connect the dots
• Provides a good discussion on using feedback loops instead of feedback lines

This article addresses data analytics in general, but provides several applicable points that L&D departments can incorporate.

Article 5: Leading with Next-Generation Key Performance Indicators⁷

Michael Schrage and David Kiron

“Measurement Leaders look to KPIs to help them lead — to find new growth opportunities for their company and new ways to motivate and inspire their teams.”

Highlights:

• Provides a decent discussion on Key Performance Indicators and what they currently mean in organizations
• Points to Chief Marketing Officers and their increasing accountability for growth-oriented objectives (we think CLOs and L&D in general are close behind).
• Has an excellent discussion on leading versus lagging indicators, and the importance of both in “measuring”.
• Recounts several good case studies that helped us think differently about what a KPI is and how it can be used

We found this article eye-opening. While it is not geared specifically to “learning”, it provides several, adaptable ideas that we feel will be important for next-generation learning measurement and evaluation.

Bonus: 4 Measurement Strategies That Create the Right Incentives for Learning⁸

Grovo

“As human beings, we are compelled to adapt our behavior to the metrics we are held against.”

Highlights:

• Has a great discussion on how even the act of measuring learning can be a motivation.
• Introduces several non-traditional ways to “measure” learning
• Makes the point that measurement strategy should be a part of the learning strategy, not as a way to measure its effectiveness.

Yes, we know it’s a blog, and yes, we realize it was written by a vendor. But this piece made some interesting points – particularly, how what we measure impacts the business.

Overall Impressions

If we were to sum up all we read into a short statement, it would be this: L&D has a long way to go. That said, we are also hopeful. As L&D functions further integrate into the rest of the business, as tools for analytics and measurement get better, and as we begin to define new models that incorporate new ways of learning and new environmental variables, we can imagine a world, in the not too distant future, where we finally – after more than 50 years of trying – maybe crack this nut.

We would love to hear what you think – what did we miss? What else should we be looking at? Comment below.

In this Research:

Diversity and inclusion is not a new idea for today's corporations, but over the last 18 months, the slow D&I burn has turned into a flashpoint, in part due to the #MeToo moment. Leaders across organizations are asking: "How can we systematically challenge the status quo, and build a more diverse and inclusive workforce?"

It is upon this foundational question that technology companies have begun to construct dozens of new and innovative ideas to support equity, diversity and inclusion in the workplace—recognizing that new technological capabilities, paired with this increased urgency, represents an opportunity to address D&I challenges in novel ways.

Back in March 2018, I posted to LinkedIn what I thought would be a rather quickly forgotten question: What technology had others seen that focused on improving diversity and inclusion (D&I) in companies? The response was huge, with lots of people I'd never met sharing how their company was using technology to tackle diversity and inclusion in ways that I'd not even dreamed of. Clearly, something big was happening – so the question turned into a quest to understand this new market.

We've ended the first 2 phases of that quest with the publication of our research on D&I tech, Diversity and Inclusion Technology: The Rise of a Transformative Market, which we, RedThread Research, have completed in partnership with Mercer.

Let me take a step back and tell you why I was even asking the question. Years ago, I'd asked folks what vendors they used to help with D&I. Most people just scratched their heads, and said, “Huh? I don’t understand what you mean.” So, I went about my merry way working on a study that ultimately focused on D&I practices, with no technology component.

Post #MeToo. Post many public D&I missteps that cost executives their jobs and companies their stock prices. I thought, surely, now, there must be technology focused on this space. But I just hadn’t read that much about it.

I started talking to a lot of people about this topic and found that it resonated with many of them. One of those people was Carole Jackson, a former colleague and current Principal at Mercer, focused on their When Women Thrive research. We found a shared passion for this topic and we agreed to partner on this research to bring a heightened understanding of the D&I technology market to both vendors and customers.

So, what began as my simple question ended up turning into a quest to find as many technology vendors focused on D&I as possible – and document who they are and what they do. Why? Three reasons:

1. This market is exploding with new vendors – Our study has nearly 100 in it (and that's in just this 1st phase of the research) and many of them have only started within the last 3 years. Given this, organizational leaders need to better understand the innovative technology solutions available, and technology vendors need to see where opportunity for new products and solutions exists.
2. D&I technology has the potential to be a disruptor – Structural biases hide in our processes and behaviors and, applied correctly, D&I technology can enable scalable, consistent treatment of people decisions while also alerting users to previously hidden patterns of bias. That said, our glasses are not so rosy as to blind us to the potential limitations and even detrimental impacts of D&I tech.
3. Too little information is available on the market – The folks over at Gartner have written a report on this topic, but not everyone can access that. Further, focusing on the question of “If There’s Too Much Diversity Tech?” doesn’t give folks insight into the range and capabilities of D&I tech. We wanted to do an in-depth study that would help vendors and buyers truly understand the market.

To that end, our study answers 5 questions:

1. What is D&I technology?
2. Why are D&I technologies coming to market right now?
3. What are the benefits and potential risks?
4. What types of D&I technologies exist?
5. Who are some of the players in the different D&I technology categories?

This report is a both qualitative and quantitative study that summarizes the D&I tech market landscape, based on a vendor and customer survey, customer interviews, and the feedback we received. It also includes an interactive market map tool that allows readers to quickly understand which vendors are in the market.

THANK YOU! To everyone – practitioners and vendors alike – for participating in this research! We hope you'll continue to be part of the D&I tech conversation going forward!