Testing Open Source AI on Real SAT, GRE, and MCAT Questions

The useful test for open-source AI study help is not whether a model looks impressive on a leaderboard. It is what happens when a student pastes in a real released SAT, GRE, or MCAT-style question, asks for an answer explanation, and then has to decide whether to trust it.

For this round of testing, I used released or official sample-style questions from the College Board, ETS, and AAMC question ecosystems, then compared how five open models handled the work: DeepSeek-R1, Qwen3 in 8B and 32B sizes, Gemma 3 12B, Phi-4-mini, and Mistral Small 3.1. The goal was narrow: not “is open-source AI good,” but “can this model help a test-taker review real exam questions without adding bad explanations to the pile?”

The short answer: small local models are useful study assistants, but they are not safe answer checkers for multi-step exam work. Qwen3-8B is the most realistic “normal laptop” option and can help with basic SAT math explanations, GRE reading review, and flashcard drafting. Phi-4-mini is even easier to run, including on CPU-only machines, but its science hallucinations make it risky for MCAT answer verification. The larger reasoning models, especially DeepSeek-R1 and Qwen3-32B, are the first open models in this test that felt genuinely useful on harder SAT math, GRE quant, and MCAT passage reasoning.

How I Tested the Models

Each model received the same kind of student prompt: solve the question, explain the reasoning, and identify the final answer. For math questions, I checked whether the model’s steps actually led to the answer, not merely whether it guessed the right option. For verbal and reading questions, I looked for whether the explanation used evidence from the passage instead of inventing outside context. For MCAT-style science passages, I paid attention to whether the model stayed inside the passage, respected units and experimental setup, and avoided adding biological or chemical facts that were not given.

That matters because the failure mode is different from a normal wrong answer. A student can usually spot “I don’t know.” A polished explanation with one bad algebra move or one invented science claim is harder to detect, especially late at night after a long practice set.

I did not treat AIME, MMLU-Pro, GPQA, or vendor benchmark claims as exam-prep proof. They are useful signals, but they are not the same task as explaining a College Board math item, an ETS quant comparison, or an AAMC-style passage question to a student who is trying to learn from a miss.

SAT Math: Qwen3-8B Is Usable, Larger Reasoning Models Are Better

SAT Math was the friendliest section for open models. The questions are usually short, the relevant information is visible in the prompt, and many items reward clean algebra rather than broad factual knowledge. On easier and medium-difficulty items, Qwen3-8B often produced explanations a student could actually use: define the variable, isolate the expression, substitute carefully, and check the requested quantity.

That makes Qwen3-8B the first model I would suggest to a student who wants to experiment locally. SiliconFlow’s education guide says Qwen3-8B can run on a 16GB laptop through Ollama and handles basic SAT math and GRE reading comprehension at about 85% of ChatGPT accuracy, while still failing on multi-step quantitative reasoning.[1] That matched the feel of the SAT portion: good enough to explain many missed questions, not good enough to become the answer key.

The larger reasoning models were noticeably more reliable when a question required two or three linked moves. DeepSeek-R1 and Qwen3-32B were better at holding the target expression in mind, avoiding premature arithmetic, and noticing when a question asked for a transformed value rather than the obvious intermediate result. SiliconFlow’s math-focused comparison reports DeepSeek-R1 as achieving o1-comparable math reasoning at $2.18 per million output tokens, compared with GPT-5 Pro at $10 per million output tokens.[2] That is vendor-published data, so it should not be read as a guarantee for SAT performance, but it explains why the model deserves a serious look for math-heavy study help.

The practical workflow is simple: use official SAT questions first, ask the model to explain only after you have attempted the problem, then compare its final answer with the official key. If the explanation clarifies the missed step, keep it. If it disagrees with the answer key, the model loses. The official key does not need to debate the chatbot.

For students building a math review loop, the model is best used as a second-pass explainer alongside a real practice plan, not as a replacement for one. The same principle applies to the SAT Math Practice Guide 2026: first isolate the tested skill, then review the error, then add targeted reps.

GRE Quant: Small Models Break Where Students Most Need Help

GRE Quant exposed the weakness of the smaller models more sharply than SAT Math. The issue was not that Qwen3-8B, Phi-4-mini, or Gemma 3 12B could never do algebra. The issue was that they could sound confident while mishandling the exact parts of GRE quant that make students miss questions: comparison logic, hidden constraints, rate relationships, and multi-step word problems.

A typical failure looked like this: the model translated the first sentence correctly, performed one reasonable operation, then treated an intermediate value as the final answer. In a quantitative comparison item, that is especially dangerous because the correct response may depend on whether the relationship is always true, sometimes true, or not determined. A fluent paragraph does not rescue a broken comparison.

DeepSeek-R1 was the strongest open model in this part of the test. It was better at slowing down, checking cases, and resisting the urge to choose a quantity before proving the relationship. Qwen3-32B also performed well enough to be useful for review, especially when prompted to show assumptions and test edge cases. Neither should be treated as a private GRE tutor with perfect judgment, but both were far more helpful than the small local models for explaining why a tempting shortcut fails.

If you are studying for GRE Quant on a normal laptop, this creates an awkward split. The model you can run easily is not the model I would trust most. Qwen3-8B is fine for generating extra practice prompts, rephrasing an explanation, or walking through a straightforward arithmetic review. For official quant questions you actually missed, use a larger reasoning model if available, and still verify against an official or trusted explanation.

GRE Verbal: Less Dramatic, Still Not Automatic

GRE Verbal was less punishing than GRE Quant because the models did not have to maintain a long calculation chain. Qwen3-8B and Mistral Small 3.1 were often useful for restating dense passage sentences, separating a main claim from supporting evidence, and explaining why an answer choice was too broad or too strong.

The danger was over-explanation. When a model disliked an answer choice, it sometimes supplied a reason that sounded plausible but was not anchored tightly enough in the passage. For GRE reading comprehension, that is not a minor style issue. The exam rewards textual discipline. A model that adds a motive, tone, or implication the passage does not support is training the wrong habit.

The better prompt was not “explain this question.” It was: “Quote or paraphrase the exact part of the passage that supports the correct answer, then explain why each wrong answer goes beyond, contradicts, or misses that evidence.” That prompt reduced the amount of free-floating literary commentary and made the answer easier to audit.

MCAT Science Passages: Reasoning Helps, Hallucination Hurts

MCAT-style science passages are where open models become both exciting and irritating. The larger reasoning models were genuinely helpful when the question depended on interpreting a graph, tracking an experimental condition, or connecting a passage result to a simple equation. DeepSeek-R1 and Qwen3-32B were more willing to work from the passage instead of reaching immediately for memorized science facts.

That distinction matters. Many MCAT questions are not asking, “Do you know a fact?” They are asking whether you can use a fact under the constraints of a passage. A model that dumps outside biochemistry can make a student feel as if the question required more memorization than it actually did. That is bad review.

Phi-4-mini was the model I would keep away from MCAT answer checking. Hugging Face describes Phi-4-mini as a 3.8B-parameter model under the MIT license that can run on CPU-only machines, which is exactly the kind of access students need.[3] The problem is that accessibility does not equal reliability. In science-heavy prompts, it was too willing to produce confident factual claims that should have been qualified or checked.

For MCAT review, I would use Phi-4-mini for low-stakes tasks: turn my own notes into flashcards, summarize a passage after I have already reviewed it, or quiz me on definitions I provide. I would not ask it to decide whether an answer choice is scientifically correct. That is where charming wrongness becomes expensive.

Vision-capable open models add another wrinkle. SiliconFlow’s education guide lists Qwen2.5-VL-7B-Instruct at $0.05 per million tokens and describes it as able to analyze charts, diagrams, and handwritten notes.[1] That is relevant for MCAT passages with figures, but it should be tested against the actual figure and official answer, not treated as proof that the model understands every experimental graph.

What You Can Realistically Run on Student Hardware

Hardware is not a side detail. It decides whether “free open-source AI” is actually available to a student or only available to someone with a workstation.

Qwen3-8B is the practical local default because it can run on a 16GB laptop via Ollama.[1] That does not make it the most accurate model in the test. It makes it the model a large number of students can install, try, and abandon without renting a GPU or learning an entire developer workflow.

Phi-4-mini goes even further on access because CPU-only use is realistic.[3] For students with older machines, that is not a small thing. The tradeoff is that the model belongs in the “study assistant” bucket, not the “answer verifier” bucket.

DeepSeek-R1 and very large context models are different. DeepSeek-R1 may be one of the best open choices for hard reasoning, but the full model is not a normal laptop install. Llama 4 Scout is another example of the gap between model capability and student practicality: Latitude’s March 2026 comparison describes its 10-million-token context window, but also notes that it requires more than 48GB of VRAM, making it unrealistic for most personal laptops.[4]

That leaves most students with two honest choices: run a smaller model locally and limit what you trust it to do, or use a larger open model through a hosted service and treat the cost as part of your prep budget.

Which Model I Would Use for Each Study Task

The cleanest way to choose is not by brand loyalty. Choose by consequence. If the model is wrong, what damage does it do?

For answer verification, model size and reasoning quality matter. For flashcards, summaries, and study scheduling, the stakes are lower because the student can inspect and edit the output before using it. That is where small local models earn their keep.

For SAT students, the best use is targeted review after an official practice set. If a score report or error log shows that linear equations, functions, or data analysis are driving the score gap, a local model can generate explanations and extra drills around that weakness. That fits naturally with the SAT Score-Gap Method, where the model helps with review but official questions define the target.

For AP or semester-long prep, the safest role is even more limited: use the model to organize review, explain teacher-provided material, and quiz from your notes. A tool can fit inside a structured plan like a 16-week AP study plan with an AI tutor, but it should not become the source of record for facts that will be tested.

Where Paid Tools Still Have the Advantage

The case for open models is strongest on access and cost. MIT Sloan summarized research from Nagle and Yue in 2025 finding that open models averaged 89.6% of closed-model performance at $0.23 per million tokens versus $1.86 per million tokens, an approximately 7:1 cost ratio.[5] For students who need frequent explanations, that cost difference is real.

But 89.6% of closed-model performance is not the same as 89.6% accuracy on your MCAT passages or GRE quant set. It is an aggregate performance comparison, not an exam-specific guarantee. Paid tools still tend to win on convenience, polished interfaces, multimodal handling, and fewer setup decisions. They may also be less frustrating for students who do not want to learn local inference, quantization, context limits, or model selection.

That does not make paid tools automatically safer. A closed model can also make mistakes. The difference is that many students are more likely to over-trust a polished paid interface than a terminal window running locally. The same rule applies either way: official questions, official answer keys, and trusted explanations anchor the study process.

The Prompt That Reduced the Most Bad Explanations

The best prompt was not fancy. It simply forced the model to expose its work and stay inside the exam task:

Solve this as an exam question.

1. State the final answer first.
2. Then show the reasoning step by step.
3. For every calculation, show the equation or substitution.
4. For reading or science passages, cite the exact passage detail that supports the answer.
5. If the question cannot be answered from the information given, say so instead of adding outside assumptions.
6. At the end, list one common trap in this question.

For GRE Quant, I would add: “Check whether the relationship must always be true.” For MCAT passages, I would add: “Do not use outside science facts unless the question explicitly requires background knowledge.” For SAT Math, I would add: “Make sure the final answer matches what the question asks for, not just an intermediate variable.”

These prompts do not make a weak model strong. They make errors easier to see. That is the actual win.

A Usable Decision Rule

If you are testing open source AI models for student study help, start by deciding whether the task is high-stakes or low-stakes.

• High-stakes: checking answers, explaining missed official questions, interpreting MCAT science, solving GRE Quant. Use DeepSeek-R1 or Qwen3-32B if you can, and verify the output.
• Medium-stakes: SAT Math explanations, GRE reading review, passage summaries. Qwen3-8B can be useful, especially when the official answer is available.
• Low-stakes: flashcards, study schedules, note cleanup, simple quizzes from your own material. Phi-4-mini, Qwen3-8B, Gemma 3 12B, and Mistral Small 3.1 can all help.
• Unsafe use: letting any model, open or paid, become the final authority over an official exam question.

Small local models under roughly 8B parameters are not reliable answer checkers for multi-step exam work. Larger reasoning models can be useful for targeted SAT math and MCAT science review when prompted carefully and checked against official sources. The student who benefits most is not the one who asks the model to replace practice. It is the one who uses it to make practice review faster, clearer, and less lonely without surrendering the answer key.

References

1. Best Open Source LLM For Education & Tutoring In 2026, SiliconFlow, 2026, https://www.siliconflow.com/articles/en/best-open-source-LLM-for-education-tutoring
2. Best Open Source LLM For Math In 2026, SiliconFlow, 2026, https://www.siliconflow.com/articles/en/best-open-source-LLM-for-math
3. Best Open Source Models to Run Locally in 2026, Hugging Face, May 2026, https://huggingface.co/blog/daya-shankar/open-source-llm-models-to-run-locally
4. LLMs for Education: Domain-Specific Model Comparison, Latitude, March 2026, https://latitude.so/blog/llms-for-education-domain-specific-model-comparison
5. AI open models have benefits, so why aren’t they more widely used?, MIT Sloan, 2025, https://mitsloan.mit.edu/ideas-made-to-matter/ai-open-models-have-benefits-so-why-arent-they-more-widely-used