Cultivate is an individualized AI-based coach for your team members that uses data from an employee’s chats, emails, and calendars to help them become better leaders. This data gives us the ability to passively understand information about digital relationships. One type of information we can measure is metadata, like number of one-on-one meetings, amount of after hours messaging, and communication frequency. In fact, there are plenty of tools available that can parse metadata from these channels and offer similar metrics. You might find them in Microsoft’s Viva Insights feature or Google’s Time Insights for your calendar.
What is metadata, exactly? You can think of metadata as data about other data. So for an email, for example, the metadata would include the sender of the email, any recipients, and the time the email was sent. With that information, you can know if people are chatting after hours, who is initiating messages with whom, and how often two people are communicating.
Interestingly, a recent paper found that workers have gotten more silo-ed as they moved to remote work. They find that people build stronger relationships with a select group of coworkers but have fewer casual conversations and transient relationships. They measured relationship strength purely from metadata, which sparked a thought for us—aren’t relationships defined much more by the substance of conversation, and not only the frequency of conversation? Not everyone is best friends with their doorman. So we set out to measure exactly how much information may be missing by only looking at metadata.
The ability to go beyond the simple counts and frequencies that metadata can provide is what makes Cultivate unique and effective as a coaching and analytics tool. Cultivate does this by analyzing the content of these messages for metrics such as asking for opinions, giving recognition, requesting feedback, and many others which cannot be gathered from the metadata alone.
To effectively measure different aspects of managerial relationships, we created four new variables which we call communicative, informative, responsive, and tone. Communicative is restricted to metadata, but the other three variables use message content to unlock different elements of the digital relationship between a manager and her direct report.
- Communicative: This is our metadata variable. It includes the number of hours a manager and direct report spend in meetings together, the number of events they share, how long they spend communicating, and how dense those communications are (e.g. number of messages and sentences exchanged across all emails and chats).
- Informative: This variable looks at the richness of information shared between a manager and direct report, giving a deeper dive into the content of their communications. It includes instances of praise, informing doubt, sharing opinions and requesting feedback.
- Responsive: This variable measures the attentiveness in a relationship between a manager and direct report. It includes how quickly they respond to one another when information is requested, how much information they share in those instances, and how often they request ad hoc meetings.
- Tone: This variable examines how a manager communicates with her direct reports. It includes measures of politeness and kindness in the relationship and how closely those variables are matched between the manager and her direct reports.
The following analysis shows results for just one manager and her nine direct reports, but we have run this on a number of different managers and can attest that similar patterns are seen throughout. In the first chart, we rank her communication to all nine direct reports on each of the four variables. Here we aim to show that message content provides relationship insights above and beyond what you can glean from metadata alone.
In fact, that is exactly what we find. While there is some consistency, especially amongst the information-related variables, you can see that those ranks do differ across these categories. For example, if you follow the green line, it seems that the manager communicates the most with user w_T and that those conversations include the most information and happen the most quickly. However, that relationship ranks 7th out of 9th on tone. This may indicate that this is a close relationship and they don’t feel the need to be extra polite or kind in their communications with one another, or it could signal something more problematic. As with all of our analyses, we provide information without passing judgment, so this simply presents an opportunity for the manager to assess the relevant relationship and make sure it is healthy.
Now let’s compare this user to zzh, the purple line. They have similar ranks on communicative and informative but diverge significantly on responsive and tone, indicating that we are indeed measuring something different with message content than metadata alone can provide. Although the manager may delay responses more with zzh than with W_t and hold fewer ad hoc meetings, when they do speak, their communications are more kind and more polite. This manager has a different relationship with these two direct reports, which looking solely at the metadata would not tell us.
While these aggregate findings are interesting alone, we wanted to also examine how communication patterns might change over time. We analyzed communications from May through October 2021 to study how relationships fluctuate month to month. We did analyses for both rankings—examining how the relationships change relative to the rest of the team—and standardized values—examining how the relationships change in absolute terms. Below we look at the rankings again, but over the course of six months.
While the static rankings show that this manager has different relationships with zzh and w_T, you can see here that the character of those differing relationships evolves over time. In fact, if you only look at the relationships in the month of May, zzh and w_T are within one or two ranks of each other in every category. You could conclude that in May, their relationships with their manager were very similar. However, throughout the course of six months, those relationships change significantly. In September and October we see the divergence that appears in the static rankings above. Again we find that by looking at four different categories within managerial relationships over time, we can learn a lot more about the variances within relationships than we can with metadata alone.
While rankings allow us to compare relationships to some degree, they cannot tell the whole story either. In comparing zzh and w_T, we can conclude that since w_T is almost always ranked higher in both communicative and informative than zzh, their manager communicates more with w_T and shares more information in those communications. However, we have no context for how different those patterns of communication are, absolutely. The manager could send w_T 100 messages a day and send zzh only 10, or it could be 100 and 99. While either scenario would rank w_T ahead of zzh, the first obviously communicates something very different than the second. In order to learn as much as possible about these relationships, we looked at the standardized values of these metrics over time as well.
As with the rankings, there is variability across our measurements, and relationships do change over time. Unsurprisingly, the relationship between the manager and user w_T appears at the top of most of these graphs. The interesting piece with w_T is looking at how those variables change over time. While we know w_T stays at the top of the rankings in both, we see that volume and quality of communication took a dip in the late summer. Another interesting trend is that overall tone seems to decrease over the course of these six months. Again we do not share this information to make any judgments, but because it can teach a manager otherwise unknowable specifics about their relationships with each of their direct reports, over and above what metadata offers, helping the manager build relationship intelligence.
This data is invaluable at the individual manager level and fits nicely into our AI coaching platform. In previous studies, we have proven the value of the content-based analyses, showing time and again that those digital behaviors correlate most highly to manager performance. However, these individual analyses do not tell a cohesive story about relationship health within a company or a team at an aggregate level. For our next post, we will analyze company-wide relationships to see how much variation there is within companies across these four communication categories and how those relationships vary company by company.
Rachel is the Senior People Scientist at Cultivate. As a psychologist, she’s always been interested in people: how we think, grow, evolve, and interact. She is excited to help Cultivate users interpret their behaviors through a scientific, research-based lens.
