Parameter Estimation

The whole idea of bias and variance comes into play from the exercise of estimating some hidden parameter. By hidden I mean we do not have direct access to it. Such a parameter can just be seen as a figment of imagination, but serves a crucial purpose in fulfilling some larger objective. To motivate the discussion, suppose I am a medical professional and I am interested in designing a diagnostic test for type-II diabetes. My understanding of bodily functions informs me the fasting blood glucose level of a diabetic patient may be higher than the ones without. Thus, I could potentially use this as a diagnostic test. So I put my head down and start designing my experiments with the objective of comparing

In my previous blog, I discussed the implication of statistical bias and variance. We had a biased coin and were interested in estimating the probability() that it lands on a head. We were aware of the real value of , estimated this value as by sampling data and applying statistical methods to it. We referred to the sampled data as the training set and fitted our model to this set. The end goal of this exercise was to demonstrate the variation of the quality of fit based on the variation in the training set. The variation was performed along two dimensions:

1. Resampling the training set for a fixed number of tosses. and
2. Resampling the training set for a different number of tosses.

As such, for a given number of tosses, one can evaluate the quality of fit based on the spread and the mean of for different training sets. We referred to the former as the variance and the latter as the bias. As we increased the number of samples in the dataset, the spread reduced and the mean got closer to the actual value of . Moreover, the variance and the bias was directly related to the capacity of the model. Higher the capacity, higher the variance and lower the bias. This is often the kind of picture of ML modeling painted in popular media.

However, it is rarely the case that an ML practitioner can cleanly decompose the modeling exercise into its elemental constituents. In reality, there are a number of factors present that hinder this kind of analysis. Consequently, additional concepts must be introduced in our framework. In this article, we will take a look at what these hindering factors are. To limit the scope of this article, we will only deal with a few of them. Some of the factors that elicit a need for extending bias-variance trade-off are discussed in the subsequent section.

Lack of Explicit Target Function

Machine Learning is inherently a function approximation problem. At an overarching level, ML problems can be defined in three different categories - supervised learning, unsupervised learning, and reinforcement learning. There are another set of learning paradigms, but they combine elements of the above. But in all of them, the central problem can reduced to that of function approximation. In supervised learning, the practitioner is given an input dataset and a set containing corresponding output values. They are tasked to learn a function estimating the relationship between input and output. In unsupervised learning, they are given a dataset and tasked to learn a function describing the underlying structure of the dataset. In reinforcement learning, they are tasked to estimate a function that guide an agent to take certain actions in an environment that maximizes its long term reward.

Why does function approximation limit the applicability of bias-variance trade-off? Typically, the ground truth function is not explicitly available for comparison. As such, there is a need for using proxy methods to estimate the quality of the fit. This is typically done by splitting the training set in different parts, withholding one of them and estimating the performance on it. This practice emphasizes the quality of the signal in the dataset. Applying bias-variance trade-off to inadequate or poor signal in the training set will cause the practitioner to misunderstand the nature of the problem. For example, the collected data may imply that accuracy is maximized for linear regression when the actual underlying function is quadratic. Sample complexity is a useful concept as a remedy. The general idea behind sample complexity is to get an estimate of the number of samples needed to reach an accuracy within acceptable range.

Algorithmic Design Choices

Another issue with bias-variance trade-off is that it is a poor tool to reason about a model’s performance. For example, based on bias-variance discussion alone, it is not clear why convolutional neural networks should be used for image processing over fully connected neural networks. For this, the concept of inductive bias comes in handy. The concept is rooted in the fact that an ML algorithm typically considers a family of function, also known as hypothesis class to fit the data. During training, it selects the function that closely represents the target function during training. The propensity of the algorithm to consider this subset is referred to as inductive bias.

It is important to note there a subtle difference between inductive bias and the statistical bias we mentioned previously. The statistical bias refers to the difference between the target function and the estimated function. On the other hand inductive bias refers to the family of functions considered by the algorithm. Thus it is independent of the target function. This separation between the target function and the hypothesis class lead to several benefits in designing a learning system. For example, the inductive bias can be used to characterize algorithms. The practitioner can then select the algorithms for further analysis based on the suitability of their inductive bias to capture the target function.

Another benefit of inductive bias over statistical bias is it is tangible and intuitive. One can introduce various modeling constructs to induce a certain bias. For example, decision trees class overall considers a hypothesis class consisting of all possible decision trees. ID3 algorithm induces bias towards shorter trees by introducing maximum information gain principle. This type of inductive bias is also referred to as preference bias. The DecisionTreeClassifier in Scikit-Learn also comes with a number of hyperparameters to restrict the class of trees considered for learning. For example, one can restrict the maximum depth of the trees by setting max_depth variable. This type of bias is known as restriction bias and it trims the original hypothesis class to a smaller class of hypothesis. Another beautiful example of inductive bias is that of convolutional layers along with the max pooling layer. They are specifically designed to induce a bias towards recognizing visual features than fully connected layers.

Training Dynamics

The effect of solution evolution on learning cannot be explored with bias-variance trade-off. As noted earlier, ML algorithms start by considering a hypothesis class . It then subsequently shrinks this class towards a function that closely represents the target function to be captured. Inductive bias helps one understand the class of functions considered and preferred over the course of solution evolution. However, it only informs a general picture of the algorithm. The specific dynamics of the solution evolution as the model is fitted to a given data are unknown. For example, the empirical evidence suggests the learning rate schedule in neural network training can cause a dramatic effect on the neural network training. But the exact effect on the dynamics of the learning dynamics cannot be explored with the tools at hand. Luckily, in the context of neural network training, a powerful method called Neural Tangent Kernel is available. The method essential transforms the analysis from parameter space to a convex function space.

Regimes Beyond Bias-Variance Trade-off

References

[1] Hastie, T., Tibshirani, R., Friedman, J. H., & Friedman, J. H. (2009). The elements of statistical learning: data mining, inference, and prediction (Vol. 2, pp. 1-758). New York: springer. [2] Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019). Reconciling modern machine-learning practice and the classical bias–variance trade-off. Proceedings of the National Academy of Sciences, 116(32), 15849-15854. [3] Mitchell, T. M. (2007). Machine learning (Vol. 1). New York: McGraw-hill.

Introduction

Following our discussion on the mind-projection fallacy in my previous article, I now want to delve into what happens when we model a simple scenario. In this article, I take you through a fictitious example of modelling a prediction problem. Given a coin, our task is to estimate the probability that it will land on heads when tossed. Initially, we have no idea about this coin’s behavior, but we aim to uncover its characteristics through experimentation, guided by statistical learning methods.

In the next section, I discuss what kind of models are appropriate to estimate this probability. Specifically, we explore two simple models, each with a different “capacity”—a concept I will explain later. The subsequent section presents the experiments I performed to contrast these models. Here, I introduce two different data generating processes—an approximation of the game used to collect the data—which are later utilized to fit the models. Following this, I present the results of these experiments and then engage in a discussion contrasting the applicability of these models. This journey will help us understand the nuances of model bias and variance, and how these elements influence our findings.

Methodology

First and foremost, we need a model to represent the coin’s behavior. More specifically, we aim to capture certain aspects of the data-generating process. You may recall that the process of repeated coin tosses is also known as Bernoulli trials. In this case, we assume each coin toss is independent and that each toss results in heads with a probability denoted as ‘p’. Thus, we can estimate ‘p’ by performing a number of trials and computing the value that maximizes the probability of observing the number of heads we actually get.

Before we go further, it is crucial to understand the concept of probability here. The probability represents a degree of plausibility, measured in the range of 0 and 1. This degree of plausibility is a mathematical tool and may not represent reality itself. After all, the outcome of a coin toss is binary: it is either heads or tails. In theory, we might be able to predict the exact outcome of the toss by running a sophisticated multi-physics simulation that models the dynamics of the coin toss considering factors like air resistance and gravity. However, these efforts might be overly complex. Instead, we start with a simpler model based on our intuition of the data-generating process.

Now, let us dive into generating samples. Suppose we perform trials, resulting in m heads and n tails. It can be shown the fraction of trial in which the coin lands on heads is the maximum likelihood estimate(MLE) of . In layman’s terms, this is the value of that maximizes the likelihood of observing the outcomes we actually did, given the data model.

The likelihood calculation unfolds as follows: we represent the outcome of trial as . The probability that we observe such outcomes, conditioned on the value of is:

Notice the right-hand side decomposes the conditional events. This is valid under the assumption the trials are independently distributed as we noted previously. Now expanding the right-hand side part of the equation is easy:

Now, let us take a step further and consider a more complex model. This model, unlike the Bernoulli trials, doesn’t assume each trial to be independent. Instead, it incorporates the outcome of the previous trial into the prediction for the next one. This is known as the Markov property. In simple terms, it is like remembering the outcome of the last toss when predicting the next one. For instance, if the coin lands heads up, it is more likely to come up with heads in the next trial with a probability . If it lands tails up, the probability it lands heads in the next trial is . This is akin to saying that the coin has some ‘memory’ of the last outcome which influences the next.

This model’s complexity - or ‘capacity’ - is higher than the simple Bernoulli trial. In the context of machine learning, the capacity of a model refers to the complexity of the functions it can learn. A model with a higher capacity can learn more complex patterns, but it is also more prone to overfitting, which is the trap of modeling the random noise in the data rather than the underlying pattern. The figure below illustrates the state diagrams for both models, one without the Markov property and one with it. The added complexity of the Markov model is visible in its state diagram as asymmetric transitions heads vs tails, representing the influence of the previous trial on the next one.

To compute the likelihood for this model, we need to approach the problem differently. We are now required to estimate both and . Note,the one with Markov property is generalization of the one without. By setting , we recover Bernoulli trial. The likelihood is formulated as follows:

We divide our sequence of trials into pairs of subsequent trials to simplify the likelihood computation. We can have four types of such pairs - , , , and . Suppose we have , , and number of occurrences of such pairs. Then the above formulation reduces to the following expression:

This might seem complex, but it is just a mathematical way to account for the varying probabilities based on the outcome of the previous toss. The MLE values for and can be calculated using calculus, as we did before. However, due to the complexity of the model, we will estimate these values numerically by performing a grid search.

By comparing the performance of these two models - the simpler Bernoulli trial and the more complex Markov model - we can begin to understand the trade-offs between model complexity and accuracy, a concept central to the understanding of model bias and variance. When a model is too simple to capture the nuances of the data, it might have a high bias, leading to inaccurate predictions. This is often the case with the Bernoulli model, which assumes that each coin toss is independent of the others. On the other hand, a more complex model, like the Markov model, can capture more detailed patterns in the data, reducing bias. However, with increased complexity comes the risk of overfitting, which occurs when a model adapts too closely to the training data and performs poorly on unseen data. This is a manifestation of high variance.

In the following sections, we will delve deeper into these concepts, empirically contrasting the performance of these models and discussing how this simple coin toss scenario can shed light on the intricacies of model bias and variance. Stay tuned for the exciting exploration ahead!

Experiments

In total, we carry out four distinct experiments. We generate two types of datasets assuming Bernoulli trials without the Markov property and another assuming Bernoulli trials with the Markov property. We simulated these datasets by assuming and .We arbitrarily selected these values to ensure the coin is biased in both cases, and to ensure that the data generated with the Markov property represents a more complex process than the data without it.

We vary the number of samples per dataset from 10 to 100 in steps of 10. For each of these settings, we generate 100 datasets. This allows us to accurately compute the mean values of and . These experiments will provide us with a clearer understanding of how the chosen models perform under different conditions.

Results and Discussion

The results of our experiments are plotted in the figures below. Figure 2 presents computed for the data generated with Bernoulli coin.The dashed line represents the true value of p, while the solid lines represent the mean values of p estimated by Bernoulli and Markov model. The shaded region represents the standard deviation for both the models. Notice both models can accurately estimate the true value of p, as indicated by the proximity of the mean value to the true value. However, the estimation by the Markov model has higher variance than the Bernoulli model. This is because the Bernoulli model has fewer parameters to capture the underlying data generation process. On the other hand, Markov model has an extra parameter which may overfit to the superficial irregularities causing the estimated p to have higher variance. Figure 4 further corroborates our claim. Notice the model fits a non-zero value to for all experiments.

Figure 3 presents the results of fitting our models to the data generated for the Markov coin. Since Bernoulli model assumes , it fails to account for the Markov property and ends up estimating a higher value for than its true value. It is interesting to note the estimated value is higher instead of lower. Why? In contrast, Markov model accurately captures the true value of by accounting for the Markov property. We can see in Figure 4 that the model accurately predicts the value of .

Finally, notice the spread in all figures decreases as we increase the number of samples per dataset. This trend aligns with our expectations, as a larger dataset allows the model to capture more of the underlying pattern, thereby improving its precision. This is a clear demonstration of the bias-variance tradeoff: as we increase our sample size, our model’s variance decreases, leading to more reliable and precise estimates. It also demonstrates the bias-variance tradeoff for a model occurs in the context of a data generating process. We can comment on the capacity of a model on its own, but a discussion on bias and variance requires the context of a data generating process.

Conclusion

In this study, we explored the fundamental concepts of modelling using two basic datasets. These datasets were generated using a biased coin, incorporating both independent Bernoulli trials and a Markov property. We applied two models of varying capacity to these datasets, aiming to estimate the probability of obtaining a ‘heads’ outcome in a coin toss. Our findings revealed that the Bernoulli Model, which has lower capacity, requires fewer samples than the more complex Markov model to accurately estimate the probability for the Bernoulli coin. However, as we increased the number of samples per dataset, both models demonstrated improved precision.

Interestingly, the higher capacity of the Markov model enabled it to accurately estimate the probability for the Markov coin, while the Bernoulli model fell short in capturing the influence of the previous state. We contend that this is due to the Markov model’s higher capacity, which allows it to overfit to minor irregularities in the data, necessitating more samples to mitigate this effect. Simpler models like the Bernoulli model can readily ignore these irregularities due to its inherent assumptions about the data-generating process. However, this simplicity can lead to inaccuracies if the model fails to account for certain aspects of the data generation process, as we observed with the Markov coin.

This study’s findings illustrate a classic case of the bias-variance trade-off in modeling[1]. It is crucial to note that while much of the popular literature on bias-variance trade-off attributes the difference between prediction and ground truth to the model capacity, this argument overlooks the critical role of the data generating process. A model with greater capacity does not necessarily exhibit low bias. Also, modern machine learning theory suggests more capacious model can increase both the accuracy and the precision at the same time[2].

In conclusion, while statistical bias is a significant factor, it is just one part of the larger narrative. Other forms of bias exist in machine learning algorithms that are equally important to consider[3]. I plan to delve into these in subsequent articles, expanding our understanding of bias in machine learning.

References

[1] Hastie, T., Tibshirani, R., Friedman, J. H., & Friedman, J. H. (2009). The elements of statistical learning: data mining, inference, and prediction (Vol. 2, pp. 1-758). New York: springer. [2] Belkin, M., Hsu, D., Ma, S., & Mandal, S. (2019). Reconciling modern machine-learning practice and the classical bias–variance trade-off. Proceedings of the National Academy of Sciences, 116(32), 15849-15854. [3] Mitchell, T. M. (2007). Machine learning (Vol. 1). New York: McGraw-hill.