8. Calculate Ridge Regression Coefficients Using Matrices#
8.1. Introduction#
In the previous experiment, we learned about the Ridge Regression and LASSO Regression methods and conducted practical training on both methods using scikit-learn. In this challenge, we will attempt to directly calculate the Ridge Regression coefficient \(w\) using Python and compare the results with those obtained by scikit-learn.
8.2. Key Points#
-
Calculate Ridge Regression coefficients using Python
-
Calculate Ridge Regression coefficients using scikit-learn
8.3. Calculate Ridge Regression Coefficients Using Python#
In the previous lesson, we already knew the vector expression of Ridge Regression:
And the analytical solution of this vector expression:
Exercise 8.1
Challenge: Referring to formula (2), complete the Python function to calculate the Ridge Regression coefficient \(w\).
Hint: Use
np.eye()
to generate the identity matrix and note that the
formula involves matrix multiplication.
import numpy as np
def ridge_regression(X, Y, alpha):
"""
参数:
X -- 自变量数据矩阵
Y -- 因变量数据矩阵
alpha -- lamda 参数
返回:
W -- 岭回归系数
"""
### 代码开始 ### (≈ 3 行代码)
### 代码结束 ###
return W
Solution to Exercise 8.1
import numpy as np
def ridge_regression(X, Y, alpha):
"""
Parameters:
X -- Independent variable data matrix
Y -- Dependent variable data matrix
alpha -- lamda parameter
Returns:
W -- Ridge regression coefficients
"""
### START CODE HERE ### (≈ 3 lines of code)
XTX = X.T * X
reg = XTX + alpha * np.eye(np.shape(X)[1])
W = reg.I * (X.T * Y)
### END CODE HERE ###
return W
Next, we generate the test data:
np.random.seed(10) # 设置随机数种子
X = np.matrix(np.random.randint(5, size=(10, 10)))
Y = np.matrix(np.random.randint(10, size=(10,1 )))
alpha = 0.5
Run the tests
Calculate the value of the Ridge Regression coefficient \(w\):
ridge_regression(X, Y, alpha).T
Expected Output
matrix([[ 1.42278923, 2.20583559, -0.6391644, 0.64022529, -0.44014758,
1.66307858, -0.83879894, -0.25611354, -0.06951638, -2.56882017]])
8.4. Calculate Ridge Regression Coefficients Using scikit-learn#
In the above challenge, you have learned to calculate the Ridge Regression coefficient \(w\) using Python. Next, let’s see if the results are consistent with those calculated by scikit-learn.
Exercise 8.2
Challenge: Calculate the Ridge Regression coefficient \(w\) using scikit-learn.
Hint: Add the
fit_intercept=False
parameter to the Ridge regression model to cancel the
intercept.
from sklearn.linear_model import Ridge
def ridge_model(X, Y, alpha):
"""
参数:
X -- 自变量数据矩阵
Y -- 因变量数据矩阵
alpha -- lamda 参数
返回:
W -- 岭回归系数
"""
### 代码开始 ### (≈ 3 行代码)
### 代码结束 ###
return W
Solution to Exercise 8.2
from sklearn.linear_model import Ridge
def ridge_model(X, Y, alpha):
"""
Parameters:
X -- independent variable data matrix
Y -- dependent variable data matrix
alpha -- lambda parameter
Returns:
W -- Ridge regression coefficients
"""
### START CODE HERE ### (≈ 3 lines of code)
model = Ridge(alpha, fit_intercept=False)
model.fit(X,Y)
W = model.coef_
### END CODE HERE ###
return W
Run the tests
Calculate the value of the Ridge Regression coefficient \(w\):
ridge_model(X, Y, alpha)
Expected output
matrix([[ 1.42278923, 2.20583559, -0.6391644, 0.64022529, -0.44014758,
1.66307858, -0.83879894, -0.25611354, -0.06951638, -2.56882017]])
As we can see, as expected, the values of the \(w\) coefficients calculated by the two methods are exactly the same.