How do we investigate and act on Engagement Index results?

Originally published Jun 29, 2015 in JiveWorks. This experience was what pushed me to create my own Engagement Index using Jive DES data, which led me to much deeper understanding of all of the data that was being lumped in as “engagement,” thus prompting me to define engagement differently than Jive had and eventually abandon the concept of an overall Engagement Index in favor of more precise and meaningful measures. And my question led Jive to change the data they included in their Engagement Index, making it more useful for those who rely on it.

I’ve been an advocate for actionable analytics for a long time, and I have been pleased to see and hear Jive emphasizing the term when discussing their objectives for reporting enhancements. I’d like some help in understanding how to act on the new Engagement Index report. I have some concerns. For convenient reference, I have bolded my questions.

Here is what I see for my company today, June 29 2015:

WTF Happened in May?

My first concern is how to explain the huge drop in engagement in May. If I publish this report and share it with my stakeholders, that’s certainly the first thing they will ask. And I have no idea how to answer them. Help, Jive! You created this report, now teach us how to use it. How can I find out what changed in May?

Is the Report Correct?

Not to put to fine a point on it, but I’ve had problems with the integrity of the data on Jive’s reports before, so the first thing I’d like is a way to validate that the report is accurate. I’d like to drill down on this report by business unit or location, to try to see if there’s any noticeable anomaly, but there’s no filtering option. (While I’m at it, I’d like to be able to select the time range and so not have those empty months show on the report, but for some reason only the end date can be modified, not the start date.)

So, I looked for another report to help me understand this one and the best I could see was the Daily Activity report for the same time period:

I’d have expected to see a similar drop off in daily activity in May, but there is no such aberration. Views are by far the dominant activity, so if they aren’t changed, then why has the Engagement Index changed so drastically? And then there’s another oddity: when we upgraded to Jive Cloud we had a huge spike in activity as people checked out all the new features (they hadn’t seen an upgrade in years…). So, why is the Engagement Index really low in November, too? (Maybe it only has data from November 8 forward, since we upgraded on November 7?)

Here’s what the Daily Activity looks like without Views, since it’s starting to look like maybe they aren’t part of the Engagement Index:

I sure don’t see anything conclusive here that helps me really explain the Engagement Index results. Maybe May is a bit lower in this report, but nothing drastically lower the way the Engagement Index shows.

So if I can’t find any corresponding change in activity levels, how about on the user side? I looked at the User Adoption chart:

Since the active, participating and contributing users are measured over the previous 30 days, I included all of June that was available, just to be sure there wasn’t some drastic change that took place in May. Nothing jumps out at me from this data to explain why May on the Engagement Index is so low.

What Else Can I Look At?

I guess I’m going to have to dig into the Jive DES to try to find some answers. Any suggestions on where to start with the Jive DES data? I guess the first step will be to see if I can calculate the Engagement Index myself from that data and see if it matches what this report displays. Assuming that I find the report is accurate, once I have built my own calculation, then I should be able to drill down into the data to try to see where the changes are taking place. Something big seems to have happened somewhere…

Any suggestions? Hoping for help from analytics stars such as Udit ShahDirk McNealy or Claire Flanagan.

Looking in Rear View Mirror vs. Looking at the Road Ahead through the Windshield

Oh, one more thing… it would be really useful to get an idea how June’s numbers look, but that’s not possible with this report. If I select a date past June 1 but before the end of June (because June has not ended), then the report shows results through May. Rear-view mirrors are useful for certain tasks, but it’s also really helpful to get a look ahead at where the numbers are going so you can take corrective action as needed (back to that actionable bit again). Is there a way the report could let us see what’s developing during the month?

This issue was resolved on Aug 6, 2015.

Here’s an update on this issue from Jive Support:

Another quick update – It looks like the Modify User activity was the dominating factor for your drop in Engagement Index. Our Analytics team calculated your engagement index for the past 6 months with the Modify User activity removed entirely, May and June look comparable to other months in that view.

We are still working to find what may have caused this shift in the Modify User activity. In our internal Jive instance we see a similar shift in this particular activity, but starting in June instead of May. I will let you know once I have more information on what could have caused this shift.

And then, this update which sounds promising to me:

After further discussion, it has been decided that the User Modify event is not an event that we wish to track for the Engagement Index, as it is unlikely to be related to actual user engagement. A bug has been filed to remove this item from the Engagement Index calculation, JIVE-60554. I don’t currently have an ETA for when this bug will be addressed. As I mentioned above, this one activity seems to have been the driving factor for your shift in Engagement Index.

Sounds like a satisfactory resolution to me and I look forward to seeing this bug resolved so the Engagement Index will become meaningful (I hope).

Nonetheless, I still think there is a need to provide more ways for us to dig into, and therefore understand and explain, the data underlying the Engagement Index report. That need remains outstanding.

Leave a Reply

Your email address will not be published. Required fields are marked *