← Back to blog

What to record when comparing local models

Speed is only part of the picture. Keep the task, settings and outputs consistent to make a comparison useful.

Blue neural network connections; conceptual illustration

Conceptual illustration · AI generated

Start with a small set of tasks

Choose tasks you would actually use: summarize a Turkish passage, explain a short piece of code or answer a question about your document. Give each model the same input and keep the original outputs.

Record the setup alongside the result

Write down the model, file variant, runtime and version, device memory, context length and generation settings. Separate loading time from answer generation time.

Look beyond speed

Did the answer finish the task? Did it invent information, miss an important detail or produce awkward Turkish? Showing both successful and unsuccessful outputs gives readers more context than a single score.

Scope of this note

This is a checklist for planning a comparison, not a claim that a particular model was tested on hardware. A results article should include actual outputs and measurement conditions.

© detayz · blog