# In 21 of 119 Penn State AI tutor chats, reasoning never reached the homework

By MarketScale Newsroom · Published 2026-11-21 · Education Technology on MarketScale
Canonical: https://www.marketscale.com/industries/education-technology/in-21-of-119-penn-state-ai-tutor-chats-reasoning-never-reached-the-homework

> An NSF-funded study will test whether ALAASKA's hint-first chats beat standard coursework, after a pilot found reasoning that never reached the homework.

## Key points

- In about one of every six tutoring chats, reasoning appeared in the conversation but not in the student’s submitted answer, suggesting final products alone may miss parts of the problem-solving process.
- Penn State's study compares an AI tutor against equivalent coursework, which isolates the tutor's conversational support from AI use in general.

Twenty-one out of 119. That's how many student conversations with an AI tutor at Penn State held evidence of reasoning that was missing from the answer the student turned in. The figure comes from an early analysis of the tool's first classroom run in spring 2026.

The tool is called ALAASKA (Adaptive Learning for All through AI-Powered Student Knowledge Assessment). Its lead developer, Suman Saha, an associate teaching professor of computer science and engineering, now has federal money to test it under controlled conditions. In a November 13 Q&A with Penn State News reporter Sarah Small, he described a two-year, $199,009 U.S. National Science Foundation Research Initiation in Engineering Formation grant. NSF's award record, No. 2605386, lists Stephanie Cutler and Ibukun Osunbunmi of Penn State's Leonhard Center for the Enhancement of Engineering Education as co-principal investigators.

Simple division turns 21 of 119 into a rate of about one in six. In about one of every six tutoring chats, the thinking happened and the submitted answer never showed it. Saha said the finding suggests instructors may learn more by examining the problem-solving process, not only the final product.

## A tutor built to hold back the answer

Saha started building ALAASKA in 2025. It grew out of earlier classroom work with short AI-generated quizzes, flashcards, summaries and videos. The platform reads a student's responses as they work, estimates where that student's understanding stands, and serves the next useful thing: a targeted explanation, a question, a hint or a quick knowledge check. It does not solve the problem. It points to the next step and waits.

Saha said students often liked how it broke hard problems into manageable pieces without giving away the answer. His reading of the 21-of-119 result is cautious. It suggests instructors may learn more by looking at the problem-solving process, not only the finished product.

Block Field

One design choice matters a lot for anyone thinking about rollout. According to the NSF award abstract, ALAASKA adapts in real time without extensive existing student data or repeated diagnostic testing. Saha made the same point: it doesn't need a history of a student before it can help. For operators, that could make it quicker to put into a new course.

## What the two-section test isolates

The grant pays for a comparison inside a large Programming Language Concepts course at Penn State. One section uses ALAASKA. The other completes equivalent activities and reflections through the course learning management system. Both sections share aligned syllabi, assignments, rubrics and grading policies. Saha said the setup tests whether the tutor's conversational support improves learning, as opposed to AI use in general, and it doesn't assume students in the comparison section stay away from outside AI tools.

That last condition is the realistic part. For buyers, a comparison built that way could show what the tutor's conversational support adds, separate from AI use in general.

Block Field

The data plan is broad. NSF's abstract lists pre- and post-surveys, course performance measures, assignment and exam grades, platform interaction logs and learning management system traces on the quantitative side. On the qualitative side, the team will code student reflections and selected conversations for how students describe their reasoning, effort, ethical choices and use of feedback. The work draws on Self-Regulated Learning theory and the ABCE AI Literacy Framework, which covers affective, behavioral, cognitive and ethical dimensions.

The early figure has limits. It came from one upper-level course, and it doesn't say whether those 21 students understood the material and just wrote thin answers, or whether the reasoning depended on the tutor's prompting. Results from the funded comparison aren't available yet.

## Employers want proof of judgment too

The question ALAASKA is built around, how to see reasoning when AI can produce the output, is the same one engineering employers are asking about new graduates. In an October 29 Semiconductor Engineering report, Liz Allan found universities reworking assessment around oral exams, in-class quizzes, labs and projects. ChipAgents founder and CEO William Wang told the outlet that engineers now carry responsibility for validating AI-generated work and catching incorrect assumptions. Andy Nightingale, vice president of product management and marketing at Arteris, described the workflow he wants graduates to bring: specify the problem, generate alternatives, validate assumptions, test results and document decisions. Axiomise CEO Ashish Darbari pointed to the design review, where an engineer has to explain their reasoning on the spot.

Educators outside the U.S. are making the same argument. Writing for Times Higher Education Campus on November 11, Samer Al Kork of Al Akhawayn University in Morocco argued that when a student can generate code in seconds, the output says little about what they understand. What counts is how they reached a decision and whether they can check and explain it. Saurabh Mehta, chief academic officer at Vidyalankar Institute of Technology in Mumbai, said in an October episode of the institute's podcast that assessment weight should move from the outcome to the process. He said he has tried planting bugs in problems to see whether students can find them, and that VIT has formed a committee to work on a new way of assessment.

Penn State wants to learn whether patterns in student-AI conversations can help instructors recognize when persistence and reflection are emerging and when students may need more support. If they can, that could give instructors in large courses a view of the process without an oral exam for every student. Saha frames large courses as the main use case, since that's where instructors can't give everyone individual support at every moment.

Oral defenses still test something a transcript can't. Al Kork and Darbari both stress explaining a technical choice live, to a client or a review panel. A chat log shows a student reasoning with a patient tutor. It doesn't show whether that student can hold up when a customer pushes back. For programs weighing one method against the other, that suggests the two could measure different things.

## Checks for a chat-based tutor pilot

For engineering departments weighing AI tutors in high-enrollment courses, the Penn State design points to three checks. First, check whether a pilot compares the tutor against equivalent non-AI activities with matched rubrics and grading. Without that, a gain could just reflect AI use in general. Second, ask who reads the interaction logs and what rubric defines reasoning. NSF's award abstract lists refined instruments and coding rubrics for studying AI-supported learning among the expected outcomes. Third, find out whether the tool needs historical learner data before it helps, since that decides how fast it can go into a new course.

Block Field

Penn State plans to share design guidelines, assessment instruments and methods for analyzing student-AI conversations through faculty-development work with the Leonhard Center. NSF's abstract adds conference publications and workshops, and Saha hopes to extend the approach to more courses, Penn State campuses and partner institutions. Exam grades are among the data the study will collect. Whether reasoning seen in the chats carries over to exams is the result to watch for.

## Sources

- [Q&A: Leveraging artificial intelligence to propel engineering education](https://www.psu.edu/news/engineering/story/qa-leveraging-artificial-intelligence-propel-engineering-education)
- [Award Details - NSF Award Search](https://www.nsf.gov/awardsearch/show-award/?AWD_ID=2605386)
- [AI Raises The Bar For Tomorrow's Engineers](https://semiengineering.com/ai-raises-the-bar-for-tomorrows-engineers/)
- [What should engineering education look like in the age of AI?](https://www.timeshighereducation.com/campus/what-should-engineering-education-look-age-ai)
- [Engineering Education in the Age of AI: Rethinking Learning, Assessments ...](https://www.youtube.com/watch?v=z-Ghf5prRX4)

Tags: Penn State, ALAASKA, National Science Foundation, Suman Saha, AI tutoring, engineering assessment

---
Source: MarketScale, https://www.marketscale.com/industries/education-technology/in-21-of-119-penn-state-ai-tutor-chats-reasoning-never-reached-the-homework. Published for AI indexing and citation; cite the canonical URL. Site guide for agents: https://www.marketscale.com/llms.txt
