Monitoring and Analyzing Player Mastery, Confidence and Engagement in Drive

Drive monitors and reports on three critical player attributes that work together to provide you with information that influence a player’s performance in Drive: confidence, engagement, and mastery. Mastery and engagement can be monitored from the author’s dashboard, as shown in Figure 1, as well as within various reports available from the Reports section of the authoring site. Confidence and mastery are shown on a variety of reports related to topics and learning objectives. (Figure 2) All three attributes are showcased on the player progress report. (Figure 3)

drivekb-mastery2-1

Figure 1. Player mastery and Player Engagement can easily be monitored from author’s dashboard.

drivekb-mastery2-2

Figure 2. A variety of Drive reports available via the Reports section of the authoring site showcase confidence and mastery levels, enabling you to see where learners are performing well and where they need additional support or more time in Drive.

drivekb-mastery2-3

Figure 3. The Player Progress report shows you engagement, mastery, and confidence for every registered player.

The three attributes are interrelated so understanding each of them can help a manager coach employees or a game author to coach the managers whose employees are participating in the Drive game. Let’s look at each one.

Mastery

Mastery ratings indicate how well players can perform the learning objectives specified in a Drive game. Players always start with a “low” mastery rating. The speed with which they move to a “high” rating depends on how well they perform within the mini-games. They need to successfully master all the content within the games, which typically requires 3 to 5 plays of each mini-game at a minimum. (Exact number will depend on how many mistakes they make and how much content your games include.)

  • Because Drive, like all the Knowledge Guru apps, is designed to provided learners with spaced repetition/retrieval practice, the goal is for them to acquire mastery over time – a span of a few weeks rather than within a single session or two. Players’ performance mastery rating will adjust based on their most recent scores on any game they play.

What to analyze related to mastery

A learner’s manager can look at these mastery ratings and gauge how well a learner can perform essential skills or demonstrate essential knowledge in his or her role.

  • Mastery ratings that are low indicate that the player has not achieved any of the learning objectives within the game.
  • Mastery ratings that are moderate indicate that progress toward achievement is underway. By looking at a player’s report, the manager can see which objectives the player is closest to mastering and which ones the player needs to spend more time with.
  • Mastery ratings that are high indicate the learner has successfully achieved all the learning objectives.

Confidence

Mastery is only one part of the performance picture. A learner can have high mastery over content but feel low confidence in using this knowledge or demonstrating a skill. Conversely, a learner can be overly-confident and believe their skill and knowledge set is higher than it actually is. The confidence rating within Drive is designed to help learners and their managers spot discrepancies between confidence and mastery.

Drive has learners self-assess their confidence levels on each learning objective that is part of the Drive game. They rate themselves on a scale of 0 (no confidence) to 9 (high confidence). As learners play the various mini-games that are part of a Drive game, Drive compares their performance to their confidence ratings. If they perform better or worse than their confidence rating would warrant, Drive prompts learners to re-rate themselves to a more accurate place.

A player’s confidence ratings, as well as their actual performance within the games, influences the game content that player sees on each login to the game. Drive prioritizes games and content within games in this order:

  1. Low confidence, low performance
  2. High confidence, low performance
  3. Low confidence, high performance

What to analyze related to confidence

  • Is confidence significantly higher or lower than mastery over learning objectives? If there is a mismatch is it due to lack of mastery (which could be addressed by more game play) or is it due to over-confidence with actual mastery lower than the desired target level? When confidence is too high, a conversation needs to occur with that learner to get better alignment between perceived skills and actual skills.
  • Is confidence increasing over time? Can you see a desirable uptick in confidence levels as mastery gains occur?
  • Does any individual or group have low confidence but high mastery? If yes, a manager may want to have a conversation about what is keeping confidence ratings low.

Engagement

Engagement ratings tell you how frequently your learners are interacting with Drive. There are four categories of engagement. Here’s what each category means:

  • Highly Engaged = Accessing 3x/week (total of 15 minutes’ time)
  • Moderately Engaged = Accessing 1x/week (total of 5 minutes’ time)
  • Minimally Engaged = Accessing 1x/two weeks
  • No Engagement = No access for more than 2 weeks’ time

Once players achieve mastery, most are not likely to continue interacting with the game. After a player’s mastery level peaks, that player’s engagement will likely fall to “no engagement.” Your focus is to monitor engagement right after you release a Drive game and make sure it stays in the “moderate” to “high” categories.

What to analyze related to engagement

  • Is your engagement level matching mastery levels? Ideally, if mastery is low to mid-level, you want engagement levels to be high or at least moderate.
  • If engagement and mastery are both low, what do you need to do to encourage play? What might be hindering participation and execution? Are players finding value in the Drive experience? If not, why not?

Helpful Tip

Make sure automatic email reminders are enabled. To check, go to Implement in the left-hand navigation pane. Select Drive Access. The drop down menu under Automatic Email Reminders should say “Reminders Enabled.” With the auto emails enabled, Knowledge Guru will automatically send reminders to players after extended periods of inactivity.

drivekb-mastery2-4

When is a game “done?”

Learners and their managers can consider a Drive game complete for a learner when the learner’s performance mastery AND confidence are both high. A typical learner can achieve high ratings on both metrics after encountering all the game content, which should require several instances of play. Depending on the number of performance objectives in the overall game experience and the content included within a single mini-game, this should require 45 to 60 minutes of total playing time, which translates into 9 to 12 separate days of logging into the game (5 minutes per day). Users can continue to access the experience after achieving high mastery. They are never cut off from using it as long as a valid subscription exists.

How to Build Custom Reports in Knowledge Guru Drive

Each Knowledge Guru app allows you to customize your reports to include the information you want and filter out what you don’t.

Quick Steps for Building a Custom Report in Drive

  1. Expand the Reports option within left-hand navigation pane, and select Custom Report Builder.
  2. Select all the items you would like to include in your custom player report.
  3. Choose the orientation/format for your report from the drop-down menu.
  4. Click GENERATE REPORT.

Click through the slideshow below to see the steps in action.

If you want to track something specific besides the default fields already in the custom report builder, you can customize unique fields. For example, maybe you want to track high scores by job title or compare performance across different departments. To do this, you can change the field that you run reports on by creating custom fields. You can create up to three different custom fields.

Quick Steps for Creating Custom Fields

  1. Expand the Customize option within left-hand navigation pane, and select Registration Fields.
  2. Type in the Field Names you would like to report on (i.e. location, job title, department, etc.)
  3. Insert the appropriate Field Values below each Field Name.
  4. Click SAVE VALUES.
  5. Expand the Reports option within left-hand navigation pane, and select Custom Report Builder.
  6. You will now see your new Custom Field Names you can use to run customized reports.

Once you’ve finalized the items you want to include in your custom report, you can download your report as either a PDF or CSV file.

Click through the slideshow below to see the steps in action.

How to Track Player Progress in Knowledge Guru Drive

Knowledge Guru Drive includes a variety of reports you can use to track player progress. The authoring tool describes your report options, so this article will not detail all options. Instead, we will focus on the types of information you might want to gather – and which report is the right one to use.

  • Log into your game's admin tool at kguru.co/login. Use your email address and password.

Reports Available for Drive 

  1. If you want to verify that people are staying engaged and mastering the content, access the Drive Overview report. Here you can verify the number of players you have in the game and what their progress is within the game. You can also see quick stats and charts on players’ confidence and engagement levels. This is a useful report to showcase to managers and stakeholders who want to see how a game is going at a glance.
  2. If you want to know how a specific player is performing in the game, access the Player Progress Report.
  3. If you want to see overall player performance across topics (how well are people doing as a big group), access the Topics and Objectives Report.
  4. If you want to compare performance across groups, you can access up to three different reports. Each of these reports will mirror whatever name you created for your first, second, and third custom user registration fields. Example: The default name for Field 1 is Location. So if you did not change this field name and you set up multiple location options, then you can pull a report called Location Performance vs Overall that lets you see how people performed in specific locations. For more information on creating custom user registration fields, click here.
  5. The Inactive Players Report provides a list of players who have become disengaged based on time away from playing. (Note: This report does not include players who have achieved Mastery.)
  6. The Players with Mastery Report provides a list of players who have achieved Mastery status in the game. The ultimate goal is for all players to achieve Mastery.

PDF or CSV? 

Each report can be downloaded as a PDF or as a CSV file. You can click EXPORT next to the report you want to export on the View Reports screen, or click EXPORT REPORT on the top right of your screen when you’re viewing a report. Choose the CSV option if you think you might want to do further manipulation of the data. By downloading as a CSV, you can then re-save the report as an Excel spreadsheet, reformat it to your taste, and then sort the data to suit your needs.

Custom Report Builder Option

You may have a need for some very specific data – and not care about other data. You can use the Custom Report Builder to choose what information you include/exclude from a report.

Send Automatic Reports

You can send reports to your stakeholders automatically with the Auto Report Sender. Learn how here.

How to Send Automatic Reports in Knowledge Guru Drive

Need instructions for a Legend or Quest game?

 

Let managers know how their players are doing by scheduling automatic reports that can be sent directly to their inbox. You choose which reports you want to send and how often you want to send them. You also decide what information is included in the reports each recipient receives.

Quick Steps for Sending Automatic Reports

  1. Expand the REPORTS option within left-hand navigation pane, and select AUTOMATED REPORTS.
  2. Click CREATE NEW REPORT.
  3. Type a report name in the NAME OF REPORT box.
  4. Check the box next to each report you would like to automatically send.
  5. Use the FILTER dropdown(s) next to each registration field you created to only send reports for a specific group of players.
  6. Use the FREQUENCY drop-down to decide how often reports will be automatically sent.
  7. (Optional) select an expiration date for your automatic report. Format: mm/dd/yyyy.
  8. Type the email addresses of the report recipient(s) into the RECIPIENTS box. Separate each email with a comma.
  9. Click CREATE REPORT.

If you no longer want to send your automatic report, click the red REMOVE button next to the report you wish to cancel.

Click through the slideshow below to see the steps in action.