Comparing Llama Models for Python Function Generation

Testing two models, Llama 3.1 (8B) and Llama 3.2 (3.2B), on their ability to generate a Python function that computes the height of an object using its image size in pixels and the object’s distance.

Experimental Setup

Both models were run locally on a PC equipped with:

* CPU: Intel i7-8700
* RAM: 16GB
* GPU: Nvidia Quadro P2200

The models operated in generation-only mode, with no feedback or fine-tuning. Each attempt to solve the task involved the following process:

1. Generating Code: The task prompt was structured as a docstring:

Create a Python function `object_height` that calculates object height 
based on the image height in pixels.

Input parameters:
- focal length `F` in mm, default 35 mm
- image height `H` in pixels, default 1152
- pixel size `p` in μm, default 3.45 μm
- object distance `D`, default 6.5 m

Returns:
The object height in mm as an absolute float value.

Raises:
`RuntimeError` if input parameters are invalid (negative or zero).
Extracting the Function Code: The generated response was parsed to isolate the function implementation.

2. Running the Code: The function code was executed locally.

3. Testing the Function: The function was validated with:
* Default parameters.
* Various valid inputs.
* Invalid parameters to test error handling.

The task was considered successfully solved only if the function passed all tests.

Evaluation Metrics

To measure the effectiveness of each model, three key metrics were considered:
* Number of Attempts: how many attempts were required to produce a correct solution.
* Time per Attempt: the time taken to generate and process the function code in each iteration.
* Total Time to Solve the Task: the cumulative time from the start of the process until a correct solution was achieved.

Writing python function with code-specific models

My goal is to create a local AI agent that understands my projects' context and generates working code efficiently. While general-purpose LLMs are impressive, code-specific models should be better suited for this task.

I tested the following models:
- CodeLlama (13B parameters),
- CodeStral (22B parameters),
- Qwen2.5-Coder (14B parameters),
- DeepSeek-Coder-v2 (16B parameters)

For the testing environment, I spun up a server with a Tesla T4 GPU (16GB VRAM). This hardware constraint helped narrow down my model selection.

I gave each model the same test: generate a function that calculates an object's size using known image parameters and distance, as described earlier. To improve code extraction from the responses, I implemented a more robust parsing pattern:

match = re.search(
    r"(?:`{3}\w*|\[PYTHON\])\n(.+?)(?:`{3}|\[\/PYTHON\])",
    out_str,
    flags=re.DOTALL,
)

Number of Attempts

The results show significant variance between the models' performance. CodeStral emerged as the clear leader, requiring a median of only 5 attempts to complete the task. In stark contrast, Alibaba's Qwen2.5 needed around 110 attempts (22 times more!) to achieve the same result.

Comparing these results with previous tests using general-purpose models, only CodeStral and CodeLlama showed improved performance. This suggests that being a code-specific model doesn't automatically guarantee better efficiency.

Compare Phi4 with CodeLlama and CodeStral

After discovering the news about Phi4 - 'state-of-the-art open small size model from Microsoft' I couldn't walk by and tested it in generating mode comparing with my previous favorites: CodeLlama and CodeStral.

Here are my results:
- smallest number of attempts to generate working code - 3.5 (median);
- however, highest attempt time - 30.5 seconds (median) on Tesla T4 GPU;
- total task solving time - 1.9 minutes (median), slightly higher than others.

All three distributions of total solving time are non-normal. Despite close medians, Kruskal-Wallis test (α=0.05) shows significant differences. Mann-Whitney U (α=0.05) test revealed:
- CodeLlama vs CodeStral: no significant difference;
- CodeLlama vs Phi4: Phi4 has significantly higher values;
- CodeStral vs Phi4: no significant difference.

These results from Phi4, a general-purpose model, are impressive. It's a strong candidate to join CodeLlama and CodeStral in further research.

Keep a Cool Head - Tuning the Models' Temperature

The temperature parameter controls how diverse a model's outputs can be. Lower temperature values make the model more deterministic, causing it to focus on the most likely responses for better accuracy. I conducted experiments with this parameter on three selected models (Phi4, Codestral, and Codellama), which revealed some interesting patterns.

Each model was tasked with generating code of the object_height function 30 times to pass a specific test. A clear trend emerged across all models: a lower temperature of 0.2 consistently delivered the best performance in generating test function code. This finding was statistically validated using the Kruskal-Wallis test (alpha = 0.05), both with and without the results from this temperature group. The conclusion is clear - for code generation, it's best to keep the model's head cool.

The Phi4 model's performance at temperature 0.2 is particularly impressive - generating working code on the first attempt in 17 out of 30 trials (over 55%) - an outstanding result!

The second insight comes from examining the default temperature values across different models, which we can infer from the graph. The general-purpose Phi4 operates with a default temperature around 0.5-0.7, allowing for more creative responses across various scenarios. Interestingly, the code-specific Codestral model has a default temperature of about 0.2 - a setting that aligns well with its specialized purpose. Perhaps surprisingly, the Codellama model runs with a higher default temperature of around 0.5-0.7, despite its code-focused nature.

These findings highlight that tuning the temperature parameter is a crucial step in optimizing code generation performance. The time invested in such experiments is clearly worthwhile, as it can significantly impact the model's effectiveness in generating correct code.