In 21 of 119 Penn State AI tutor chats, reasoning never reached the homework
Penn State's Suman Saha won a two-year, $199,009 NSF grant to test ALAASKA, an AI tutor that gives hints instead of answers. In its first classroom pilot, 21 of 119 chats showed reasoning missing from submitted work. Grading only final answers may miss real thinking.
This story was produced through MarketScale. See how Education Technology teams put it to work with Executive Thought Leadership.
Key takeaways
In about one of every six tutoring chats, reasoning appeared in the conversation but not in the student’s submitted answer, suggesting final products alone may miss parts of the problem-solving process.
Penn State's study compares an AI tutor against equivalent coursework, which isolates the tutor's conversational support from AI use in general.
Free workspace
Turn your Education Technology expertise into content.
Record interviews, organize footage, and write with AI on a free trial of the MarketScale platform for qualifying companies. No demo required, no credit card.
Twenty-one out of 119. That's how many student conversations with an AI tutor at Penn State held evidence of reasoning that was missing from the answer the student turned in. The figure comes from an early analysis of the tool's first classroom run in spring 2026.
The tool is called ALAASKA (Adaptive Learning for All through AI-Powered Student Knowledge Assessment). Its lead developer, Suman Saha, an associate teaching professor of computer science and engineering, now has federal money to test it under controlled conditions. In a November 13 Q&A with Penn State News reporter Sarah Small, he described a two-year, $199,009 U.S. National Science Foundation Research Initiation in Engineering Formation grant. NSF's award record, No. 2605386, lists Stephanie Cutler and Ibukun Osunbunmi of Penn State's Leonhard Center for the Enhancement of Engineering Education as co-principal investigators.
Simple division turns 21 of 119 into a rate of about one in six. In about one of every six tutoring chats, the thinking happened and the submitted answer never showed it. Saha said the finding suggests instructors may learn more by examining the problem-solving process, not only the final product.
A tutor built to hold back the answer
Saha started building ALAASKA in 2025. It grew out of earlier classroom work with short AI-generated quizzes, flashcards, summaries and videos. The platform reads a student's responses as they work, estimates where that student's understanding stands, and serves the next useful thing: a targeted explanation, a question, a hint or a quick knowledge check. It does not solve the problem. It points to the next step and waits.
Saha said students often liked how it broke hard problems into manageable pieces without giving away the answer. His reading of the 21-of-119 result is cautious. It suggests instructors may learn more by looking at the problem-solving process, not only the finished product.
ALAASKA by the numbers
Penn State News; U.S. National Science Foundation
One design choice matters a lot for anyone thinking about rollout. According to the NSF award abstract, ALAASKA adapts in real time without extensive existing student data or repeated diagnostic testing. Saha made the same point: it doesn't need a history of a student before it can help. For operators, that could make it quicker to put into a new course.
What the two-section test isolates
The grant pays for a comparison inside a large Programming Language Concepts course at Penn State. One section uses ALAASKA. The other completes equivalent activities and reflections through the course learning management system. Both sections share aligned syllabi, assignments, rubrics and grading policies. Saha said the setup tests whether the tutor's conversational support improves learning, as opposed to AI use in general, and it doesn't assume students in the comparison section stay away from outside AI tools.
That last condition is the realistic part. For buyers, a comparison built that way could show what the tutor's conversational support adds, separate from AI use in general.
Conversational microlearning, as Saha describes it, is AI that offers questions, hints and feedback in short exchanges to guide a student's reasoning instead of handing over finished answers. The NSF-funded study asks whether that format builds three habits tied to an engineer's professional formation: value-driven reasoning, persistence and reflection.
The data plan is broad. NSF's abstract lists pre- and post-surveys, course performance measures, assignment and exam grades, platform interaction logs and learning management system traces on the quantitative side. On the qualitative side, the team will code student reflections and selected conversations for how students describe their reasoning, effort, ethical choices and use of feedback. The work draws on Self-Regulated Learning theory and the ABCE AI Literacy Framework, which covers affective, behavioral, cognitive and ethical dimensions.
The early figure has limits. It came from one upper-level course, and it doesn't say whether those 21 students understood the material and just wrote thin answers, or whether the reasoning depended on the tutor's prompting. Results from the funded comparison aren't available yet.
Employers want proof of judgment too
The question ALAASKA is built around, how to see reasoning when AI can produce the output, is the same one engineering employers are asking about new graduates. In an October 29 Semiconductor Engineering report, Liz Allan found universities reworking assessment around oral exams, in-class quizzes, labs and projects. ChipAgents founder and CEO William Wang told the outlet that engineers now carry responsibility for validating AI-generated work and catching incorrect assumptions. Andy Nightingale, vice president of product management and marketing at Arteris, described the workflow he wants graduates to bring: specify the problem, generate alternatives, validate assumptions, test results and document decisions. Axiomise CEO Ashish Darbari pointed to the design review, where an engineer has to explain their reasoning on the spot.
Educators outside the U.S. are making the same argument. Writing for Times Higher Education Campus on November 11, Samer Al Kork of Al Akhawayn University in Morocco argued that when a student can generate code in seconds, the output says little about what they understand. What counts is how they reached a decision and whether they can check and explain it. Saurabh Mehta, chief academic officer at Vidyalankar Institute of Technology in Mumbai, said in an October episode of the institute's podcast that assessment weight should move from the outcome to the process. He said he has tried planting bugs in problems to see whether students can find them, and that VIT has formed a committee to work on a new way of assessment.
Penn State wants to learn whether patterns in student-AI conversations can help instructors recognize when persistence and reflection are emerging and when students may need more support. If they can, that could give instructors in large courses a view of the process without an oral exam for every student. Saha frames large courses as the main use case, since that's where instructors can't give everyone individual support at every moment.
Oral defenses still test something a transcript can't. Al Kork and Darbari both stress explaining a technical choice live, to a client or a review panel. A chat log shows a student reasoning with a patient tutor. It doesn't show whether that student can hold up when a customer pushes back. For programs weighing one method against the other, that suggests the two could measure different things.
Checks for a chat-based tutor pilot
For engineering departments weighing AI tutors in high-enrollment courses, the Penn State design points to three checks. First, check whether a pilot compares the tutor against equivalent non-AI activities with matched rubrics and grading. Without that, a gain could just reflect AI use in general. Second, ask who reads the interaction logs and what rubric defines reasoning. NSF's award abstract lists refined instruments and coding rubrics for studying AI-supported learning among the expected outcomes. Third, find out whether the tool needs historical learner data before it helps, since that decides how fast it can go into a new course.
Ask any AI tutoring vendor how often student chats show reasoning that never reaches the submitted work, and how they measure it. Penn State's 21 of 119, from a single course, is an early reference point for that conversation.
Penn State plans to share design guidelines, assessment instruments and methods for analyzing student-AI conversations through faculty-development work with the Leonhard Center. NSF's abstract adds conference publications and workshops, and Saha hopes to extend the approach to more courses, Penn State campuses and partner institutions. Exam grades are among the data the study will collect. Whether reasoning seen in the chats carries over to exams is the result to watch for.
Sources
Featured companies
Your experts belong here
Every story in MarketScale Education Technology starts with a company putting its implementation leads, instructional designers, and district partners on the record. Buyers are already reading this topic. The only question is whose experts they find.
Procurement teams read long before they ever call, and your implementers get to answer their questions first.
About the author
The MarketScale Newsroom reports on the companies, technologies, and trends shaping 16 B2B industries. It turns primary sources and expert commentary into clear, useful coverage for the people doing the work.