LLM Parameters: A Comprehensive Guide for Developers

Large Language Models (LLMs), a concept barely known by the general public until 2019, are now ubiquitous in almost every aspect of digital life, transforming how we communicate, work, learn, and even create, powering everything from search engines and virtual coding assistants to customer service and content creation. Trained on massive amounts of diverse human writing, LLMs are extremely skilled at pattern recognition in language and are gaining use cases across all domains.

Most commercial business-to-consumer LLM products, such as ChatGPT, Claude, Gemini, and DeepSeek are very generalist models; they aim to perform reasonably well across a vast range of different tasks, offering flexibility and the appeal of the chatbot to everyday users. On the other hand, fine-tuning the underlying models' parameters can improve performance on specific tasks and boost speed and efficiency by eliminating unnecessary generalization.

What Are LLM Parameters?

There are many similar but distinct "dials" a developer can adjust while customizing an LLM for specific tasks. One can fine-tune the model by modifying either what the model learns, how the model learns, or how the model behaves during inference time.

• What: Further training a pre-trained model on new, domain-relevant data will help the model perform better on domain-related tasks.
• How: Modifying the model's hyperparameters, which affect the model's training configuration, can improve how well a model learns.
• Behavior: Modifying the model's inference-time parameters, which affects the text the model generates, can help tailor the responses to desired behavior.

Key Inference Parameters

Modifying inference (or "runtime") parameters does not change the underlying model, it changes the quality of responses on the generation side. These modifications can typically only be made via a company's API and are not available for customization within the chatbot interface. Here are examples of common runtime parameters:

• Temperature: Often a proxy for "creativity," the temperature controls the randomness of the model's output on a scale of 0-1 or 0-2. Low temperatures yield conservative, predictable responses while high temperatures yield more diverse responses. Extremely high values can cause the output to become erratic and nonsensical.
  • The system temperature for the ChatGPT chatbots is estimated online to be around 0.7 on the 0-2 scale, though this is not publicly available.
• Top-p: Also known as "nucleus sampling," Top-p limits the pool of candidate words in the generated text by setting a threshold for which words are considered, excluding low-probability words. Since Top-p also controls the randomness of the response, OpenAI's Documentation recommends not changing both temperature and Top-p at the same time; however as with most parameter tuning, it can be useful to experiment and test which parameter combinations are right for your purpose!
• Max tokens: This is the maximum number of tokens in the generated response. Note that changing max tokens will not necessarily affect the brevity in style; rather the response will terminate once hitting the max token output.
• Presence penalty: This parameter "punishes" (penalizes) the reuse of tokens that have already appeared in the generated text, which can encourage the model to introduce new words or topics.
• Frequency penalty: The frequency penalty is very similar to the presence penalty, but differs by scaling the reduction in likelihood for reappearance by the frequency of the repeated token.
  • In other words, the frequency penalty increases the penalty on tokens that have been used more often, whereas the presence penalty applies a flat penalty to any token which has already appeared, regardless of frequency.
• Stop sequences: Stop sequences are specific tokens or strings which trigger the model to stop producing further output following the sequence. These can be used to end responses at a logical stopping point, such as when a new line is generated or punctuation is included.

Hyperparameter Tuning Techniques

Hyperparameters affect *how* the LLM learns by modifying aspects of the underlying architecture and training process. First, we will look at some common hyperparameters, then we will discuss hyperparameter tuning techniques.

• Number of layers: Increasing the depth of a neural network can allow it to learn more complex relationships, but this added complexity also increases computation time and resources as well as risks overfitting.
• Number of training epochs: The number of training epochs is the number of times the full set of training data is passed through the neural network. More training epochs can yield better performance, but too many epochs can lead to overfitting.
• Learning rate: The learning rate controls how quickly the model updates its weights during training. Increasing this value can speed up training but may overshoot the optimal solution, while decreasing this value can help stabilize convergence but slow down training.
  • Fixed and adaptive learning rate schedules, which vary the learning rate over time, can be implemented to further improve training time and performance.
• Batch size: The batch size is how many training samples are passed through the network at one time. Larger batches may provide a smoother gradient estimate leading to better generalization but have a higher memory requirement, while smaller batches can yield noisier gradient estimates by updating the weights more frequently but have a lower memory requirement.
• Dropout rate: "Dropout" refers to the process of deactivating a neuron during training. The dropout rate is the probability of each neuron being deactivated for each given training sample, so if the dropout rate is set to 0.3, then for each training sample, approximately 30% of the neurons are deactivated. Lower dropout rates can lead to faster convergence but risk overfitting, while higher dropout rates can slow down training but help the model generalize better by essentially training ensemble of subnetworks with shared weights.

Notice that for each hyperparameter, there are pro and cons of both increasing and decreasing the values. The next natural question to ask is: How can a developer optimize these hyperparameters for their specific use case?

• Grid search: A grid search tries all possible hyperparameter value combinations from a defined set. This type of search is exhaustive, so if completed it is guaranteed to find the best combination, but it can be infeasible due to time or computational resources.
• Random search: A random search randomly samples from the hyperparameter space, which can be more efficient, especially when some hyperparameters have a larger effect than others. However, it is not guaranteed to find the best hyperparameter configuration.
• Bayesian optimization: In typical Bayes fashion, Bayesian optimization uses performance of the previous runs to predict what range of hyperparameters to explore next.
  • Other more sophisticated hyperparameter search techniques have also emerged, including evolutionary optimization utilizing evolutionary algorithms and methods which iteratively discard unlikely hyperparameter configurations to reduce the search space.
• Manual tuning: Also common in practice is manually tuning hyperparameters iteratively based on observations of the model's inference performance. In hybrid with more systematic searching, a developer may opt to explore the hyperparameter space more broadly, then manually tune on a more granular level. The drawback of manual tuning is that it relies on human judgement, which can be detrimental depending on the model's use case.

Training models across large hyperparameter spaces can be very resource-intensive. To address this, cross-validation is a widely used technique involving partitioning the dataset into multiple subsets, training the model on some subsets, and validating it on the remaining ones. This process is repeated several times, varying which subsets are used for training and validation.

Optimizing Model Size and Performance

Optimizing the size of the model is crucial for performance. Larger models, such as GPT-4, perform well for most tasks due to the massive number of learned parameters (weights) and vast corpus of training data. However, these models are extremely resource and time-intensive to train and typically have much longer inference times. Luckily, for most specialized tasks, the full breadth of a large model is not necessary, and through careful fine-tuning, hyperparameter tuning, and runtime configuration setting, a developer can often achieve desired performance using a much more compact model.

Earlier, we discussed one aspect of model size, the number of layers. Listed below are some other common techniques to reduce the size of the model.

• Pruning: This process involves removing less significant neurons from the model, which reduces its complexity, thus reducing the size and inference time. Neuron significance can be measured in many different ways; some examples include the magnitude of the weight, the activation levels, and sophisticated algorithms such as neuron importance score propagation (NISP).
• Quantization: Quantization reduces the model size by decreasing the numerical precision of the model's weights and activations, by converting more precise numbers (like float32) to less precise numbers (like int8). This can reduce computation time without, in many cases, losing much accuracy.
• Knowledge distillation: In this process, a smaller "student" model is trained to replicate the behavior of a larger "teacher" model. This can increase speed and enhance generalization, but can also lead to quality reduction and information loss.

Just as with other parameter tuning, optimizing the model size while retaining performance involves experimentation with various values to find the right balance.

What Next?

Knowing what attributes of a model you can customize is only the first step! The next step is the fun part: experimenting with different parameters and configurations yourself! Mastering the customization of LLMs requires a deep understanding of the impact of these parameter choices, but once these skills are acquired and grown, developers can unlock the full potential of these models while also maintaining resource and time efficiency.