_____ _ ____
| ____|_ __(_) ___ / ___| _ __ ___ _ __ ___ ___ _ __
| _| | '__| |/ __| \___ \| '_ \ / _ \ '_ \ / __/ _ \ '__|
| |___| | | | (__ ___) | |_) | __/ | | | (_| __/ |
|_____|_| |_|\___| |____/| .__/ \___|_| |_|\___\___|_|
|_|
COMP 388 at Loyola is the special-topics CS course, whatever the professor wants to teach that semester. The Spring 2026 version was on large language models, which lines up with the TLA+ and fine-tuning work at ai4fm. This repo holds two of its assignments.
The first assignment was the "hello world" of loading a Hugging Face model and running it. The script (llm_prompt.py) loads gpt2 via transformers, takes a prompt off the command line, and does manual token-by-token generation instead of calling .generate(). The reason for that is annoying: on an M1 Mac, calling .generate() would crash with a bus error half the time, and so would single-word prompts. The workaround is forcing CPU-only, setting PYTORCH_ENABLE_MPS_FALLBACK=1, and hand-rolling a sampling loop just to get reliable runs.
The assignment then asked for a "simple evaluation," done two ways:
evaluate_wiki_tf.py). GPT-2 got 6/10. Every one it got wrong was a false statement that it confidently called true.evaluate_boolq.py). Both runs got 45.0%. Identical. Looking at the outputs, GPT-2 was basically answering "yes" to everything, so the passage didn't matter.Not a deep result, but it is a small model failing in a clearly characterizable way rather than just being bad.
The second assignment was the more interesting one. The task was to pick a small open model that ships both a base and an instruction-tuned variant, compare them, and then try few-shot prompting on the base model. The models are Qwen2.5-0.5B and Qwen2.5-0.5B-Instruct: same architecture, same tokenizer, only difference is the instruction-tuning step.
The base-vs-chat comparison was textbook: the base model treats your prompt like a document to continue ("What is the capital of France?" → it generates more questions or bullet points), and the chat model just answers "The capital of France is Paris." Fine.
The evaluation was on SNLI (natural language inference), 100 examples, greedy decoding:
| Configuration | Accuracy | |--------------------------|----------| | Chat model (zero-shot) | 61% | | Base model (zero-shot) | 64% | | Base model (few-shot) | 61% |
The base model beating the chat model zero-shot was not the expected outcome, and few-shot made it worse, not better. The likely reading is that at 0.5B the instruction tuning overfits to chatty response shapes that hurt on a constrained-label task like NLI, and that the few-shot prompts inflate the context enough that the small model loses the plot. The few-shot run also hit CPU and memory ceilings, because the in-context examples triple the prompt length; a GPU or a quantized model would have finished cleanly.
It's coursework, not a product. But the HW2 result, a small instruction-tuned model getting beaten by its base on a classification task, is the kind of thing worth keeping in mind when picking models for the ai4fm fine-tuning work. The repo is honest about the constraints: pinned to Python 3.11, macOS M1, CPU-only, hardcoded paths because the assignment template wanted it that way.