This article was originally written in Chinese and translated into English by AI.
Lesson three in the AI for Beginners tutorial series.
Introduction
This lesson emphasizes the idea that “a machine-learning method = model + strategy + algorithm” and helps readers develop a deeper understanding of all three elements.
Main Content
In the first lesson, What Is AI?, we learned what artificial intelligence and machine learning are.
Review – machine learning: machine learning is the study of learning algorithms from data.
In the second lesson, Basic Machine-Learning Terminology, we learned the basic terms and workflow of machine learning.
Review – the machine-learning workflow: collect and organize data, preprocess the data (feature engineering), train the algorithm, test the algorithm, and apply the algorithm.
After two introductory lessons, can we finally begin studying machine-learning algorithms?
Sorry, not yet.
Before formally studying them, we must establish the following idea:
A machine-learning method = model + strategy + algorithm
The fundamentals of machine-learning methods remain the same despite their many forms. Once you understand “model + strategy + algorithm” in depth, you no longer need to apply classical machine-learning algorithms mechanically. Instead, you can use their basic ideas flexibly and design a distinctive algorithm for a distinctive problem.
Unfortunately, terminology in machine learning and AI is frequently misused or mixed together. The same word can have different meanings in different machine-learning literature.
I therefore hope readers will not merely memorize the three words “model,” “strategy,” and “algorithm.” Instead, understand that most machine-learning methods can be divided into these three parts, and develop a deep understanding of each one.
1. Model
1.1 Definition of a Model
Definition: a model is the conditional probability distribution or decision function to be learned.
This definition may feel obscure to a beginner. This lesson will approach the concept from another angle; later articles will explain conditional probability distributions and decision functions.
An informal but accessible explanation is that a model is a way to express the relationship between features and labels, or among features.
In supervised learning, we want to understand the relationship between the independent variable X and dependent variable y. Given X—the features—what method should we use to predict y—the label? That method is the model. In unsupervised learning, the model describes relationships among the independent variables X. For explanations of supervised and unsupervised learning, see Basic Machine-Learning Terminology.
The following discussion focuses mainly on supervised learning.
Consider the problem of predicting Beijing house prices. Suppose we know a home’s floor area and its distance from the city center and want to estimate its price.
The simplest prediction method has the form y = w1*x1 + w2*x2 + b. This is a model. Here, y is the dependent variable representing the house price; x1 and x2 are the independent variables representing distance and floor area; and w1, w2, and b are model parameters.
More specifically, we call a function that makes predictions from a linear combination of features a linear model.
y = -5*x1 + 6*x2 + 30 is one specific linear model, while y = -7*x1 + 2*x2 + 10 is another.
Instead of a linear model, we could use conditional statements to form a decision-tree model:
if the house is less than 5 km from the city center:
price = 100,000 * floor area
elif the house is 5–10 km from the city center:
price = 80,000 * floor area
else:
price = 60,000 * floor area
Later lessons will introduce more models.
1.2 Two Different Concepts: General and Specific
Strictly speaking, machine learning has two different concepts that are both called a model. In everyday use, however, the two senses are mixed together without causing much difficulty.
One sense is general, with parameters that have not yet been determined. “A linear model,” for example, generally refers to:
The values of parameters b, w1, w2, and so on are undetermined. The set of every possible parameter value is the model’s hypothesis space.
The other sense refers to a specific set of parameter values, usually learned from data. Suppose, for example, that after studying data for hundreds of homes in Beijing, we find that the linear model that best matches reality is “house price = 8 * floor area – 10 * distance from the city center in kilometers + 20.” Every parameter now has a specific value.
When we use the word “model” later, context will tell us whether it refers to a class such as linear models or decision-tree models, or to one specific formula. It will generally refer to a specific model.
We can therefore see that, in supervised learning, a model is a method for predicting y from X. In unsupervised learning, it expresses relationships among features. More precisely, “model” can refer either to a general class of results—such as the set of all decision trees or all linear models—or to a specific rule learned from data, such as a linear model whose parameters have been determined.
2. Strategy
Earlier, we said that a model’s hypothesis space is the set of every possible parameter value.
For the house-price problem above, suppose we choose the linear model:
y = w1*x1 + w2*x2 + b
The parameters w1, w2, and b have infinitely many possible values.
With so many possibilities, how can we judge whether a model is good at predicting house prices? What criterion should we use to select the best one? This is the second part of a machine-learning method: the strategy.
2.1 Loss Functions
A loss function is a function that measures the quality of a model’s predictions.
For regression problems, the most common criterion—or loss function—for evaluating a model is the squared loss:
In other words, it is the square of the difference between the model’s prediction for a sample and the sample’s true value.
The formula above applies to one sample. Across all samples, it becomes:
Another option is absolute loss, although it is not commonly used:
Here is an intuitive way to understand a loss function. Suppose we predict house prices from floor area. The loss function measures the lengths of the red lines in the image below. We want the loss to be as small as possible—that is, the red lines should be as short as possible—so that the predictions are as close as possible to the true values.
Common loss functions for classification problems include log loss.
Later tutorials will introduce many different loss functions.
2.2 Strategy
For the house-price problem, suppose we use squared loss: the square of the difference between the true value and the model’s prediction. A smaller loss indicates a better model.
Our strategy is therefore to minimize the squared loss.
How to minimize it is the problem addressed by the algorithm.
Sometimes a larger value of an objective function indicates a better model. In that case, our strategy is to maximize the function.
It is worth noting that a loss function has the following two properties in most cases:
- Its values are continuous.
- It is differentiable with respect to the model parameters.
In classification, prediction accuracy is a criterion for evaluating a model, but it is not a loss function because it is discontinuous and not differentiable with respect to the model parameters. If we have 100 samples, for example, accuracy can take only the finite values 0%, 1%, and so on through 100%, so it is not continuous.
Absolute loss is a loss function, but it is not commonly used because it is not differentiable everywhere with respect to the parameters, making some problems difficult to solve.
3. Algorithm
In the preceding discussion, we selected a model—that is, a way to express the relationship between features and labels, whether linear, conditional, or otherwise. We also selected a strategy, the criterion for evaluating models. The next step is to use an algorithm to determine specific values for the model parameters: to select the optimal model from the hypothesis space.
An algorithm is the specific computational method used to learn a model.
Algorithms fall into two categories: those that can be solved directly and those that require iterative solutions.
Later lessons will cover specific algorithms.
Summary and Further Reading
- A machine-learning method = model + strategy + algorithm.
- A model is a way to express the relationship between features and labels, or among features—the conditional probability distribution or decision function to be learned.
- A strategy is the criterion used to evaluate a model. It is often continuous and differentiable with respect to the model parameters.
- An algorithm is the specific computational method used to learn a model.
Recommended Reading
- The three elements of statistical learning: Section 1.3 of Hang Li’s Statistical Learning Methods
Author: M小白
To be continued...