1、TensorFlow提供自动求导机制

• Variable对象（是可以被训练的对象）：

tf.Variable(initial_value,dtype) initial_value可以指定参数的初始值（可以是数字、Python列表、ndarry对象、Tensor对象），是对Tensor对象进一步的封装，在模型训练过程中自动记录梯度信息，由算法自动优化，在机器学习中作为模型参数

In [4]: tf.Variable(3)  #TensorFlow默认是32位整数/浮点数
Out[4]: <tf.Variable 'Variable:0' shape=() dtype=int32, numpy=3>

In [5]: tf.Variable([1,2])  #等价于tf.Variable(np.array([1,2]))
Out[5]: <tf.Variable 'Variable:0' shape=(2,) dtype=int32, numpy=array([1, 2])>
#指定为浮点数，一般不建议64位浮点数
In [6]: tf.Variable([1,2],dtype=tf.float64)
Out[6]: <tf.Variable 'Variable:0' shape=(2,) dtype=float64, numpy=array([1., 2.])>

• 可以用.numpy()来读取其值：

x=tf.Variable([1,2])
x.numpy()
Out[10]: array([1, 2])

• trainable属性：表示是不是可以被训练的变量，是ResourceVariable来实现的：

In [11]: x.trainable   #不是Tensor的属性
Out[11]: True
In [12]: type(x)
Out[12]: tensorflow.python.ops.resource_variable_ops.ResourceVariable

• 赋值： #不是Tensor的属性

对象.assign() #修改值
对象.assign_add() #加法
对象.assign_sub() #减法

In [16]: x=tf.Variable([1,1])
         x.assign([3,4])
Out[16]: <tf.Variable 'UnreadVariable' shape=(2,) dtype=int32, numpy=array([3, 4])>
#加法
In [17]: x.assign_add([1,1])
Out[17]: <tf.Variable 'UnreadVariable' shape=(2,) dtype=int32, numpy=array([4, 5])>
#减法
In [18]: x.assign_sub([0,1])
Out[18]: <tf.Variable 'UnreadVariable' shape=(2,) dtype=int32, numpy=array([4, 4])>
#判断是Tensor还是Variable
In [19]: isinstance(x,tf.Tensor),isinstance(x,tf.Variable)
Out[19]: (False, True)

2、自动求导机制：

2.1、GradientTape：（是一个上下文管理器对象）

with GradientTape() as tape: #tape是自己定义的一个名称
函数表达式
grad=tape.gradient(函数,自变量) #表示被求导的函数和被求导的自变量，grad是返回值

• 例如求y=x²在x=3时的导数（dy_dx=6，此时y=9）

import tensorflow as tf
x=tf.Variable(3.)
with tf.GradientTape() as tape:
    y=tf.square(x)
    
dy_dx=tape.gradient(y,x)
print(y,'\n',dy_dx)
#输出：
tf.Tensor(9.0, shape=(), dtype=float32) 
tf.Tensor(6.0, shape=(), dtype=float32)

• GradientTape(persistent=False, watch_accessed_variables=True)

persistent默认为False，表示求导一次后就销毁了，如果为True，则可以多次求导，然后需要调用“del tape”手动释放。

import numpy as np
import tensorflow as tf
x=tf.Variable(3.)
with tf.GradientTape(persistent=True) as tape:
    y=tf.square(x)    #y=x²
    z=pow(x,3)        #y=x³
    
dy_dx=tape.gradient(y,x)
dz_dx=tape.gradient(z,x)   #若persistent=False，则这里会报错
print("当x=",x.numpy(),"时：\n",
      "y=",y.numpy()," y'=",dy_dx.numpy(),' ',
      "z=",z.numpy()," z'=",dz_dx.numpy())
del tape     #手动释放tape
#输出：
当x= 3.0 时：
 y= 9.0  y'= 6.0   z= 27.0  z'= 27.0

2.2、 watch_accessed_variables、tape.wahtch：监视可训练变量

表示自动监视所有的可训练变量，默认为True，若修改为False则不能监视可训练对象：

import tensorflow as tf
x=tf.Variable(3.)
with tf.GradientTape(watch_accessed_variables=False) as tape:
    y=tf.square(x)
    
dy_dx=tape.gradient(y,x)
print(dy_dx)
#输出为None
None

此时，可以手动添加对变量（也可以是非可训练变量）的监视：tape.wahtch(变量)：

import tensorflow as tf
x=tf.Variable(3.)   #若x=tf.constant(3.)，则x是Tensor对象而不是可训练对象，也可以用watch监视
with tf.GradientTape(watch_accessed_variables=False) as tape:
    tape.watch(x)
    y=tf.square(x)
dy_dx=tape.gradient(y,x)   #自动求导
print(dy_dx)
#输出：
tf.Tensor(6.0, shape=(), dtype=float32)

2.3、二阶导数：两次GradientTape

#f(x,y)=x²+2y²+1
import tensorflow as tf
x=tf.Variable(3.)
y=tf.Variable(4.)
with tf.GradientTape(persistent=True) as tape2:
    with tf.GradientTape(persistent=True) as tape1:
        f=tf.square(x)+2*tf.square(y)+1
    first_grads=tape1.gradient(f,[x,y])  #first_grads：函数f在点（3,4）的一阶偏导数
second_grads=[tape2.gradient(first_grads,[x,y])]
    
print(f,"\n",first_grads,"\n",second_grads)
del tape1
del tape2
#输出：
tf.Tensor(42.0, shape=(), dtype=float32) 
[<tf.Tensor: shape=(), dtype=float32, numpy=6.0>, <tf.Tensor: shape=(), dtype=float32, numpy=16.0>] 
[[<tf.Tensor: shape=(), dtype=float32, numpy=2.0>, <tf.Tensor: shape=(), dtype=float32, numpy=4.0>]]

2.4、对向量求偏导

import tensorflow as tf
x=tf.Variable([1.,2.,3.])
y=tf.Variable([4.,5.,6.])
with tf.GradientTape(persistent=True) as tape:
    f=tf.square(x)+2*tf.square(y)+1
df_dx,df_dy=tape.gradient(f,[x,y]) 
    
print(f,"\n",df_dx,"\n",df_dy)
#输出：
tf.Tensor([34. 55. 82.], shape=(3,), dtype=float32) 
tf.Tensor([2. 4. 6.], shape=(3,), dtype=float32) 
tf.Tensor([16. 20. 24.], shape=(3,), dtype=float32)

3、代码实现

3.1、TensorFlow实现医一元线性回归

import tensorflow as tf
import numpy as np
x=np.array([137.97,104.50,100.00,124.32,79.20,99.00,124.00,114.00,
            106.69,138.05,53.75,46.91,68.00,63.02,81.26,86.21])
y=np.array([145.00,110.00,93.00,116.00,65.32,104.00,118.00,91.00,
            62.00,133.00,51.00,45.00,78.50,69.65,75.69,95.30])
learn_rate=0.0001   #超参数——学习率
iter=10             #迭代次数
display_step=1      #设置每迭代1次输出结果，方便查看

np.random.seed(612)
w=tf.Variable(np.random.randn()) 
b=tf.Variable(np.random.randn())
#注意上面randn()参数为空，返回一个数字（类型float），
#TensorFlow在创建Variable类型时如果是浮点数则默认是用32位的
#也可以指定dtype=tf.float64(numpy默认是用64位的)
mse=[]              #保存每次迭代后的损失
for i in range(0,iter+1):
    with tf.GradientTape() as tape:
        pred=w*x+b
        Loss=0.5*tf.reduce_mean(tf.square(y-pred))
    mse.append(Loss)
    
    dL_dw,dL_db=tape.gradient(Loss,[w,b])  #自动获取偏导数
    w.assign_sub(learn_rate*dL_dw)         #自动更新模型参数w、b
    b.assign_sub(learn_rate*dL_db)
    
    if i%display_step==0:
        print("i:%i,Loss:%f,w:%f,b:%f"%(i,Loss,w.numpy(),b.numpy()))
   
输出：
i:0,Loss:4749.362305,w:0.946047,b:-1.153577
i:1,Loss:89.861862,w:0.957843,b:-1.153412
i:2,Loss:89.157501,w:0.957987,b:-1.153359
i:3,Loss:89.157379,w:0.957988,b:-1.153308
i:4,Loss:89.157364,w:0.957988,b:-1.153257
i:5,Loss:89.157318,w:0.957987,b:-1.153206
i:6,Loss:89.157280,w:0.957987,b:-1.153155
i:7,Loss:89.157265,w:0.957986,b:-1.153104
i:8,Loss:89.157219,w:0.957986,b:-1.153052
i:9,Loss:89.157211,w:0.957985,b:-1.153001
i:10,Loss:89.157196,w:0.957985,b:-1.152950

3.2、实现多元线性回归

• NumPy实现：（参考笔记3——梯度下降法求解多元线性回归（numpy实现））
• TensorFlow实现：

import tensorflow as tf
import numpy as np
area=np.array([137.97,104.50,100.00,124.32,79.20,99.00,124.00,114.00,
               106.69,138.05,53.75,46.91,68.00,63.02,81.26,86.21])
room=np.array([3,2,2,3,1,2,3,2,2,3,1,1,1,1,2,2])
price=np.array([145.00,110.00,93.00,116.00,65.32,104.00,118.00,91.00,
                62.00,133.00,51.00,45.00,78.50,69.65,75.69,95.30])

x0=np.ones(len(area))                           #样品数量len(area)
x1=(area-area.min())/(area.max()-area.min())    #归一化，最小的数映射为0，最大为1
x2=(room-room.min())/(room.max()-room.min())
X=np.stack((x0,x1,x2),axis=1)                   #堆叠为16行3列的二维数组，分别与w1,,w2,w3相乘，w1*x1+w2*x2+w3
Y=price.reshape(-1,1)

learn_rate=0.2                                  #超参数——学习率
iter=50                                         #迭代次数
display_step=10                                 #设置每迭代10次输出结果，方便查看
np.random.seed(612)
W=tf.Variable(np.random.rand(3,1))              #随机生成初值,3行1列，建议指定为float32
mse=[]

for i in range(0,iter+1):
    with tf.GradientTape() as tape:             #上下午管理器——检测可训练变量
        PRED=tf.matmul(X,W)                     #预测值
        Loss=0.5*tf.reduce_mean(tf.square(Y-PRED))#损失
    mse.append(Loss)                            #将损失记录在mse中
    
    dL_dW=tape.gradient(Loss,W)                 #求Loss对W的偏微分
    W.assign_sub(learn_rate*dL_dW)              #梯度更新W值
    
    if i%display_step==0:
        print("i:%i,Loss:%f" %(i,Loss))

输出：
i:0,Loss:4426.049598
i:10,Loss:84.522900
i:20,Loss:81.569366
i:30,Loss:81.101425
i:40,Loss:80.832718
i:50,Loss:80.674744

3.3、误差：

在训练集上的误差——训练误差（training error）；在新样品上的误差——泛化误差（generalization error）
训练误差小、泛化误差大——过拟合（overfitting）
学习不足——欠拟合（underfitting）

3.4、波士顿房价数据集一元线性回归

import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
boston_housing=tf.keras.datasets.boston_housing
#train_x404行13列，test_x是102行13列，y均是是房价
(train_x,train_y),(test_x,test_y)=boston_housing.load_data()
x_train=train_x[:,5]            #取出第五列，房间数
y_train=train_y                 #为了好看
x_test=test_x[:,5]              # x_test.shape=(102,)是一维数组
y_test=test_y

learn_rate=0.04                                  #超参数——学习率
iter=2000                                         #迭代次数
display_step=200                                 #设置每迭代10次输出结果，方便查看
np.random.seed(612)
w=tf.Variable(np.random.randn())              #随机生成初值,3行1列，为float32（因为是一个浮点数，numpy是64位的）
b=tf.Variable(np.random.randn())
mse_train=[]
mse_test=[]

for i in range(0,iter+1):
    with tf.GradientTape() as tape:
        pred_train=w*x_train+b
        loss_train=0.5*tf.reduce_mean(tf.square(y_train-pred_train))
        
        pred_test=w*x_test+b
        loss_test=0.5*tf.reduce_mean(tf.square(y_test-pred_test))
        
    mse_train.append(loss_train)
    mse_test.append(loss_test)

    dL_dw,dL_db=tape.gradient(loss_train,[w,b])
    w.assign_sub(learn_rate*dL_dw)
    b.assign_sub(learn_rate*dL_db)

    if i%display_step==0:
        print("i:%i, Train Loss:%f, Test Loss:%f"%(i,loss_train,loss_test))

#可视化输出
plt.figure(figsize=(15,10))
#绘制训练集的房间数~房价的散点图、模型预测的房间数~预测房价的直线
plt.subplot(221)
plt.scatter(x_train,y_train,color="blue",label="data")
plt.plot(x_train,pred_train,color="red",label="model")
plt.plot(loc="upper left")
#绘制训练误差和测试误差
plt.subplot(222)
plt.plot(mse_train,color='blue',linewidth=3,label="train loss")
plt.plot(mse_test,color="red",linewidth=1.5,label="test loss")
plt.legend(loc="upper right")
#绘制所有训练集的实际房价和预测房价
plt.subplot(223)
plt.plot(y_train,color='blue',marker="o",label="true price")
plt.plot(pred_train,color="red",marker=".",label="predict")
plt.legend(loc="upper right")
#测试集实际房价和预测房价
plt.subplot(224)
plt.plot(y_test,color="blue",marker="o",label="true_price")
plt.plot(pred_test,color="red",marker=".",label="predict")
plt.legend()

plt.show()

输出：
i:0, Train Loss:321.837585, Test Loss:337.568634
i:200, Train Loss:28.122614, Test Loss:26.237764
i:400, Train Loss:27.144739, Test Loss:25.099327
i:600, Train Loss:26.341949, Test Loss:24.141077
i:800, Train Loss:25.682899, Test Loss:23.332981
i:1000, Train Loss:25.141851, Test Loss:22.650158
i:1200, Train Loss:24.697672, Test Loss:22.072006
i:1400, Train Loss:24.333025, Test Loss:21.581436
i:1600, Train Loss:24.033665, Test Loss:21.164261
i:1800, Train Loss:23.787905, Test Loss:20.808697
i:2000, Train Loss:23.586145, Test Loss:20.504938

3.5、波士顿房价数据集多元线性回归

import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
boston_housing=tf.keras.datasets.boston_housing
#train_x404行13列，test_x是102行13列，y均是是房价
(train_x,train_y),(test_x,test_y)=boston_housing.load_data()
#归一化，train_x.min(axis=0)表示训练集中第1,、2、3...13列的最小值组成的数组
x_train=(train_x-train_x.min(axis=0))/(train_x.max(axis=0)-train_x.min(axis=0))
y_train=train_y                 #为了好看

x_test=(test_x-test_x.min(axis=0))/(test_x.max(axis=0)-test_x.min(axis=0))
y_test=test_y

x0_train=np.ones(len(train_x)).reshape(-1,1)    #len(train_x)=404
x0_test=np.ones(len(test_x)).reshape(-1,1)      #len(test_x)=102

#生成所需要的矩阵，concat堆叠成矩阵，类型转化函数cast转为float32
#X_train.shape=TensorShape([404, 14]), 第1列为1，其他13列为归一化后13个属性对应的结果
#X_test.shape=TensorShape([102, 14])
X_train=tf.cast(tf.concat([x0_train,x_train],axis=1),tf.float32)
X_test=tf.cast(tf.concat([x0_test,x_test],axis=1),tf.float32)

#将房价转化为列向量
Y_train=tf.constant(y_train.reshape(-1,1),tf.float32)
Y_test=tf.constant(y_test.reshape(-1,1),tf.float32)


learn_rate=0.01                                  #超参数——学习率
iter=8000                                         #迭代次数
display_step=500                                 #设置每迭代10次输出结果，方便查看
np.random.seed(612)
W=tf.Variable(np.random.randn(14,1),dtype=tf.float32)              #随机生成初值,3行1列，为float32（因为是一个浮点数，numpy是64位的）
mse_train=[]
mse_test=[]

for i in range(0,iter+1):
    with tf.GradientTape() as tape:
        PRED_train=tf.matmul(X_train,W) #迭代结束后，PRED_train是对训练集的房价预测值
        Loss_train=0.5*tf.reduce_mean(tf.square(Y_train-PRED_train))
        
        PRED_test=tf.matmul(X_test,W) #迭代结束后，PRED_test是对测试集的房价预测值
        Loss_test=0.5*tf.reduce_mean(tf.square(Y_test-PRED_test))
        
    mse_train.append(Loss_train) #存放每一次迭代损失值
    mse_test.append(Loss_test)

    dL_dW=tape.gradient(Loss_train,W)
    W.assign_sub(learn_rate*dL_dW)

    if i%display_step==0:
        print("i:%i, Train Loss:%f, Test Loss:%f"%(i,Loss_train,Loss_test))

#可视化输出
plt.figure(figsize=(20,4))
#绘制训练集和测试集的“误差~迭代次数”关系
plt.subplot(131)
plt.ylabel("MSE")
plt.plot(mse_train,color="blue",linewidth=3)
plt.plot(mse_test,color="red",linewidth=1.5)#为了线段不重叠，线宽要比上一个窄
#制所有训练集的实际房价和预测房价
plt.subplot(132)
plt.plot(y_train,color='blue',marker="o",label="true price")
plt.plot(PRED_train,color="red",marker=".",label="predict")
plt.legend()
plt.ylabel("Price")
#测试集实际房价和预测房价
plt.subplot(133)
plt.plot(y_test,color="blue",marker="o",label="true price")
plt.plot(PRED_test,color="red",marker=".",label="predict")
plt.legend()
plt.ylabel("Price")

plt.show()

输出：（训练集误差不断减少，测试集在i=3000增加，出现过拟合）
i:0, Train Loss:263.193451, Test Loss:276.994110
i:500, Train Loss:26.911524, Test Loss:26.827427
i:1000, Train Loss:21.887274, Test Loss:22.039745
i:1500, Train Loss:19.030268, Test Loss:20.212139
i:2000, Train Loss:17.118929, Test Loss:19.456861
i:2500, Train Loss:15.797001, Test Loss:19.260979
i:3000, Train Loss:14.858858, Test Loss:19.365536
i:3500, Train Loss:14.177205, Test Loss:19.623528
i:4000, Train Loss:13.671043, Test Loss:19.949770
i:4500, Train Loss:13.287543, Test Loss:20.295107
i:5000, Train Loss:12.991437, Test Loss:20.631872
i:5500, Train Loss:12.758676, Test Loss:20.945164
i:6000, Train Loss:12.572536, Test Loss:21.227776
i:6500, Train Loss:12.421191, Test Loss:21.477076
i:7000, Train Loss:12.296155, Test Loss:21.693033
i:7500, Train Loss:12.191257, Test Loss:21.877159
i:8000, Train Loss:12.101960, Test Loss:22.031694

• 可以修改迭代次数3000，和显示间隔100：

iter=3000                                         #迭代次数
display_step=100                                 #设置每迭代10次输出结果，方便查看
#输出：
i:0, Train Loss:263.193451, Test Loss:276.994110
i:100, Train Loss:44.476345, Test Loss:47.471569
i:200, Train Loss:36.176552, Test Loss:37.562954
i:300, Train Loss:31.584028, Test Loss:32.202713
i:400, Train Loss:28.789461, Test Loss:28.952513
i:500, Train Loss:26.911524, Test Loss:26.827427
i:600, Train Loss:25.520697, Test Loss:25.333918
i:700, Train Loss:24.405626, Test Loss:24.216917
i:800, Train Loss:23.460527, Test Loss:23.340536
i:900, Train Loss:22.630890, Test Loss:22.629452
i:1000, Train Loss:21.887274, Test Loss:22.039745
i:1100, Train Loss:21.212658, Test Loss:21.544201
i:1200, Train Loss:20.596289, Test Loss:21.124844
i:1300, Train Loss:20.030684, Test Loss:20.769012
i:1400, Train Loss:19.510202, Test Loss:20.467239
i:1500, Train Loss:19.030268, Test Loss:20.212139
i:1600, Train Loss:18.587011, Test Loss:19.997719
i:1700, Train Loss:18.177067, Test Loss:19.818951
i:1800, Train Loss:17.797461, Test Loss:19.671591
i:1900, Train Loss:17.445543, Test Loss:19.551962
i:2000, Train Loss:17.118929, Test Loss:19.456861
i:2100, Train Loss:16.815464, Test Loss:19.383459
i:2200, Train Loss:16.533215, Test Loss:19.329269
i:2300, Train Loss:16.270424, Test Loss:19.292068
i:2400, Train Loss:16.025497, Test Loss:19.269894
i:2500, Train Loss:15.797001, Test Loss:19.260979
i:2600, Train Loss:15.583610, Test Loss:19.263735
i:2700, Train Loss:15.384138, Test Loss:19.276777
i:2800, Train Loss:15.197500, Test Loss:19.298813
i:2900, Train Loss:15.022709, Test Loss:19.328751
i:3000, Train Loss:14.858858, Test Loss:19.365536