Desirable difficulties such as retrieval practice (testing) and spacing (distributed studying) are shown to improve long-term learning. Despite their knowledge about the benefits of retrieval practice, learners struggle with application. We propose a mechanism of embedding desirable difficulties in learning sessions called "retrieval-based teaching." We define it as asking learners many ungraded, granular questions during sessions. We hypothesized that this method could motivate learners to (1) study more and (2) increase the spacing of their studying. We tested these two hypotheses through a quasi-experiment in an introductory programming course. We compared 684 learners' granular activities with an interactive eBook between the discussion sections where the intervention was implemented and the control sections. Over four terms, there were a total of 17 graduate-level facilitators who led the discussion sections. Each term, there were five discussion sections, each led by a distinct facilitator. Only one of the five per term implemented the treatment in their section(s) by dedicating most of the session time to retrieval-based teaching. Our analysis of these data collected over four consecutive terms shows that retrieval-based teaching motivated learners to space their studying over an average of 3.78 more days, but it did not significantly increase the amount they studied. Learners in the treatment group earned an average of 2.36 percentage points higher in course scores. Our mediation analysis indicates that spacing was the main factor in increasing the treated learners' scores.

Prior literature suggests that computer science education (CSE) was less affected by the pandemic than other disciplines. However, it is unclear how the pandemic affected the quality and quantity of learners' studying in CSE. We measure the impact of the pandemic on the amount and spacing of learners' studying in a large introductory computer science course. Spacing is defined as the distribution of studying over multiple sessions, which is shown to improve long-term learning. Using multiple regression models, we analyzed the total number of learners' interactions with the eBook and the number of days they used it, as a proxy for studying amount and spacing, respectively. We compared two sequential winter terms of the course, one during (Winter 2021) and one prior to the pandemic (Winter 2020). After controlling for possible confounders, the results show that learners had 1,345.87 fewer eBook interactions and distributed their studying on 2.36 fewer days during the pandemic when compared to the previous term prior to the pandemic. We also compared four terms prior to the pandemic (Fall and Winter of 2018 and 2019) to two terms during the pandemic (Fall 2020 and Winter 2021). We found, on average, learners had 3,376.30 fewer interactions with the eBook and studied the eBook on 16.35 fewer days during the pandemic. Contrary to prior studies, our results indicate that the pandemic negatively affected the amount and spacing of studying in an introductory computer science course, which may have a negative impact on learning outcomes.

Spacing and procrastination are often thought of as opposites. It is possible, however, for a learner to space their studying by doing something every day throughout the term and still procrastinate by waiting until late in the term to increase their amount of studying. To analyze the relationship between spacing and procrastination, we examined 674 learners' interactions with a course eBook over four terms of an introductory programming course. We measured each learner's term-level spacing as the number of days they interacted with the eBook, and each learner's term-level procrastination as the average delay from the start of the term for all their eBook interactions. Surprisingly, there was a small, yet positive, correlation between the two measures. Which, then, matters for course performance: studying over more days or studying earlier in the term? When controlling for total amount of studying, as well as a number of background and demographic characteristics in an SEM analysis, we find a strong positive effect of spacing but no significant effect of procrastination on final assessment scores.

Extensive prior research shows that spacing - the distribution of studying over multiple sessions - significantly improves long-term learning in many disciplines. However, in computer science education, it is unclear if 1) spacing is effective in an incentivized, non-imposed setting and 2) when incentivized, female and male learners space their studying differently. To investigate these research questions, we examined how learners in an introductory computer science course (378 female and 310 male) spaced their studying. A retrieval practice tool in the course (worth 5% of the course score) incentivized learners to space their studying by awarding a point per day of usage. To measure how much each learner spaced, we examined their interactions with the course eBook, which served as their primary learning resource. Specifically, when comparing two learners with the same background and demographic characteristics, the same measure of course easiness, and the same amount of content studied, we considered the learner who distributed their studying over more days to be the one who spaced more. Using this definition, our structural equation modeling (SEM) results show that, 1) on average, learners who spaced their studying over 14.516 more days (one standard deviation) got 2.25% higher final assessment scores; and 2) female learners spaced their studying over 4.331 more days than their male counterparts. These results suggest that, in an introductory computer science course, incentivized spacing is effective. Notably, when compared to their male counterparts, female learners both exhibited more spacing and obtained higher final assessment scores through spacing.

Generating multiple-choice questions is known to improve learners' critical thinking and deep learning. Visualizing relationships between concepts enhances meaningful learning, learners' ability to relate new concepts to previously learned concepts. We designed and deployed a collaborative learning process through which learners generate multiple-choice questions and represent the prerequisite knowledge structure between questions as visual links in a shared map, using a variation of Concept Maps that we call "QMap." We conducted a four-month study with 19 undergraduate learners. Learners sustained voluntary contributions, creating 992 good questions, and drawing 1,255 meaningful links between the questions. Through analyzing self-reports, observations, and usage data, we report on the technical and social design features that led learners to sustain their motivation.

Retrieval practice, spacing, and interleaving are known to enhance long-term learning and transfer, but reduce short-term performance. It can be difficult to get both learners and facilitators to use these techniques since they perceive them as impeding initial learning. We leveraged user experience design and research techniques, including survey and participant observation, to improve the design of a practice tool during a term of use in a large introductory Python programming course. In this paper, we describe the design features that made the tool effective for learning as well as motivating. These include requiring spacing by giving credit for each day that a learner answered a minimum number of questions, adapting a spaced repetition algorithm to schedule topics rather than specific questions, providing a visual representation of the evolving schedule in order to support meta-cognition, and providing several gameful design elements. To assess effectiveness, we estimated a regression model: each hour spent using the practice tool over the course of a term was associated with an increase in final assessment scores of 1.04%, even after controlling for many potential confounds. To assess motivation, we report on the amount of practice tool use: 62 of the 193 learners (32%) voluntarily used the tool more than the required 45 days. This provides evidence that the design of the tool successfully overcame the typically negative perceptions of retrieval practice, spacing, and interleaving.

In an introductory Python programming course intended for non-majors with little prior CS experience, with 85 male and 108 female learners, we were able to capture electronic traces of learners' studying and problem-solving. There was no significant difference in final assessment scores by gender but we found that female learners spent 12.1 more hours studying over the term while male learners on average earned 2.7 more points per hour of solving problem set questions over the first half of the term. We were able to capture their learning behavior because learners studied using the Runestone interactive textbook and completed weekly problem sets in the same platform for the first half of the term. We analyzed these logs to determine three quantities for each learner. One is study time, as measured by total use of Runestone outside of weekly assignments. The second is speed, as measured by the number of points learners earned per hour working on problem sets. The third is earliness, as measured by how far before the deadlines they worked on weekly assignments. We conclude that male learners were faster at completing problem sets early in the term but that female learners found an alternative pathway to success.

The Runestone ebook platform is open source, extensible, and already serves over 25,000 learners a day. The site currently hosts 18 free ebooks for computing courses. Facilitators can create a custom course from any of the existing ebooks on the site and can have their learners register for that custom course. Facilitators can create assignments from the existing material in each ebook, evaluate assignments, and visualize learner progress. Facilitators can even create new content for assignments. The Runestone ebooks contain instructional material and a variety of practice problem types with immediate feedback. One of the practice types, Parsons problems, is also adaptive, which means that the difficulty of the problem is based on the learner’s performance. Learner interaction is recorded and can be analyzed. This paper presents the history of Runestone, describes the interactive features, summarizes the previous research studies, and provides detail on the recorded data. Interaction data can be shared with other learning environments through the Learning Tools Interoperability Standard (LTI).