The Wild West of AI Grading – AI Series Part 2

As a follow up to my previous post on AI, I tested various AI grading options. Below are my reviews of the ones I have tried so far. If I test others, I will update this post.

The first 2 tests involved a graduate assignment to write a plan for collaboration as a special education teacher. The students were given an outline to expand along with a rubric.

Test 1: MagicSchool.AI, a freemium (free, individual paid, and school licenses) website with a variety of tools to assist teachers. They also have a student version.

With the “Student Work Feedback” tool, I uploaded the grading rubric for the assignment and a student’s assignment. It was fairly quick and easy. Select the grade level (university, in this case), the assignment description, focus of feedback (the rubric), and the student work. Below is the form and the response one assignment received.

MagicSchool.AI’s response, based on the rubric I uploaded:

Feedback on Plan for Collaboration Assignment

Areas of Strength:

• Thoroughly addressed collaboration with general education teachers and related service providers
• Demonstrated a clear understanding of the importance of communication and clear expectations in partnerships
• Engaged with class readings to support claims and decisions

Areas for Growth:

• Incomplete plan for collaboration with administration, lacking specific strategies and details
• Limited engagement with creating successful partnerships with parents/families, especially in cases of language barriers and difficult communication
• The plan for the IEP process could benefit from more emphasis on valuing student and family/parent input

General Feedback on Writing Mechanics:

• Ensure consistent formatting and spacing throughout the assignment
• Be mindful of spelling and grammar errors, such as missing spaces and incomplete sentences

My thoughts: I think MagicSchool.AI has some amazing tools and I often share it with others. However, I found the grading AI to be lacking, at least on this assignment. The student had a detailed plan for collaboration and I did not find any spelling or grammar errors (minus incomplete sentences, which were allowed as the assignment was an expanded outline format). I appreciated that it saved the results in my MagicSchool.AI account for later reference.

Test 2: Brisk Teaching, a freemium (free for individuals, pricing available for schools/districts) Chrome extension that I used within Google Docs. The feedback was closer to how I graded the students. I used the same assignment and same student as Test 1.

Brisk Teaching’s feedback, based on the rubric I uploaded:

Collaboration with School Community:

– Your plan for collaboration with the school community is well-articulated, with clear strategies for engaging instructional aides, co-teachers, related service providers, administration, and community agencies.

– The “Classroom Vision” page is an excellent tool for setting expectations with instructional aides, aligning with the rubric’s call for a “complete and thorough plan.”

– Your approach to sharing Student Learning Profiles with general education teachers is commendable, ensuring alignment with the Jozwik reading on the importance of team understanding.

– The open phone/door policy and leveraging existing relationships of service providers demonstrate a strong commitment to collaboration, reflecting the “mutual trust” principle from the readings.

My thoughts: The Brisk teaching extension was easy to use and I found the feedback to closely mirror that of my own. The downside is that it did not provide any scoring from the rubric, just narrative feedback.

Test 3: SchoolAI.com

As I have used SchoolAI to create assignments, I decided to explore its AI grader, Essay Grading Assistant, found under the Assistants menu, for an assignment on Assistive Technology (AT), specifically Augmentative and Alternative Communication (AAC).

SchoolAI’s grading based on the rubric I uploaded:

Based on the SPED 450 AAC Notes provided and comparing them against the AAC Notes Rubric, here’s a detailed grading and feedback:

**1. Basics (4 Points):** The notes offer a comprehensive understanding of AAC, distinguishing between unaided and aided systems, and outlining the importance of AAC for individuals with communication disorders. The exploration of AAC’s augmentative and alternative roles is insightful.

**Feedback:** Your foundational coverage of AAC is exceptional. The differentiation between augmentative and alternative communication provides a clear basis for understanding the spectrum of AAC applications. The mention of specific AAC tools, like the Avaz AAC and its expressive voice options, adds valuable depth to this section.

**2. IEP Documentation (3 Points):** The notes detail the process of selecting AAC tools through feature matching and the importance of documenting AAC in the IEP. However, more specific examples of IEP documentation, including how AAC goals are formulated and monitored, could enhance this section.

**Feedback:** You’ve outlined a robust approach to AAC selection and IEP documentation. To further strengthen this area, consider integrating examples of AAC goals within IEPs and how these goals are evaluated over time.

**3. Examples (4 Points):** The document provides a wide range of examples for AAC tools across emergent, entry, intermediate, and advanced levels. This thoroughness in showcasing AAC equipment tailored to different proficiency levels is commendable.

**Feedback:** Your examples across proficiency levels are particularly strong, offering tangible insights into how AAC tools can be matched to individual needs. This practical approach helps clarify the real-world application of AAC technologies.

**4. General Notes (4 Points):** The notes reflect a deep engagement with AAC, emphasizing the importance of partnerships and the tailored selection of AAC tools. The consideration of both low-tech and high-tech solutions underscores a holistic view of AAC.

**Feedback:** Your holistic perspective on AAC, considering both unaided and aided systems, enriches the notes. The emphasis on partnerships and personalized AAC tool selection demonstrates an understanding of the nuanced needs of individuals using AAC.

**5. Quality (4 Points):** The document is well-organized, demonstrating a thoughtful consideration of AAC’s complexities and the practicalities of implementing AAC strategies. The attention to detail and the clear, accessible language used throughout the notes indicate a high level of effort and understanding.

**Feedback:** The quality and organization of your notes are outstanding. Your ability to convey complex information in an accessible manner is a testament to your deep understanding of AAC and your commitment to supporting individuals with communication disorders.

**Total Score: 19/20**

Overall, your AAC notes are comprehensive and insightful, providing a strong foundation in AAC basics, thoughtful consideration of AAC’s role in IEP documentation, and a wide range of practical examples. The depth of detail and the clear, structured presentation make these notes an exceptional resource for understanding and applying AAC strategies.

My thoughts: Since this wasn’t an essay, I wasn’t sure how it would respond, and those fears were unfounded. It not only gave detailed feedback, it also provided a number score for each of the 4 areas and a total score. For each student, it was within about 1 point of the score I planned on giving. When it differed, I took a second look to see if I missed something. Each time, I went with my score over the AI, but I appreciated the detailed narrative feedback to provide to the students, saving me time by generating the comments for me to edit and personalize. As with MagicSchool.AI, it saved the results to my account.

In summary, both Brisk and SchoolAI performed well and I would use both again.

What AI graders have you tested? I would love to know what others have experienced. Feel free to reply in the comments below.