<jupyter_start><jupyter_text>人工智能高阶人才培训班>  第三课：深度学习基础设施与工具(1) ![](img/tensorflow.jpeg)<jupyter_code>### 我们在本次实践中，将会使用到的知识点如下：
* 1. numpy,pandas,matplotlib基础科学计算库
* 2. tensorflow，keras深度学习库
* 3. 线性回归，逻辑回归，神经网络等<jupyter_output><empty_output><jupyter_text>Part 1 TensorFlow基础  1.1 TensorFlow的安装 ```python Current stable release for CPU-onlypip install tensorflow==2.0.0 Preview nightly build for CPU-only (unstable)pip install tf-nightly Install TensorFlow 2.0 GPU versionpip install tensorflow-gpu==2.0.0```  1.2查看版本<jupyter_code>import tensorflow as tf
tf.__version__<jupyter_output><empty_output><jupyter_text>1.3 Hello World<jupyter_code># Create a constant op
# This op is added as a node to the default graph
hello = tf.constant("Hello, TensorFlow!")
#
print(hello)<jupyter_output>tf.Tensor(b'Hello, TensorFlow!', shape=(), dtype=string)<jupyter_text>> b’String’  ‘b’ indicates Bytes literals.  http://stackoverflow.com/questions/6269765/<jupyter_code>x = [[2.]]
m = tf.square(x)
print(m)<jupyter_output>tf.Tensor([[4.]], shape=(1, 1), dtype=float32)<jupyter_text>调用`.numpy`方法可以实现tensor和ndarray之间转换<jupyter_code>m.numpy()
a = tf.constant([[1, 2],
                 [3, 4]])

b = tf.constant([[2, 1],
                 [3, 4]])

ab = tf.matmul(a, b)

print('a * b = \n', ab.numpy())<jupyter_output>a * b = 
 [[ 8  9]
 [18 19]]<jupyter_text>1.4 EveryThing is Tensor> 在人工智能领域只有一种类型数据，它的名字叫做『Tensor』 ![](img/tensor_rank.png)  1. 标量（0D 张量）仅包含一个数字的张量叫作标量（scalar，也叫标量张量、零维张量、0D 张量）。<jupyter_code>import numpy as np
x = np.array(888)
x
x.ndim<jupyter_output><empty_output><jupyter_text>2. 向量(1D 张量)数字组成的数组叫作向量（vector）或一维张量（1D 张量）<jupyter_code>x = np.array([12, 3, 6, 14, 7])
print(x)
print(x.ndim)<jupyter_output>[12  3  6 14  7]
1<jupyter_text>3. 矩阵(2D 张量)向量组成的数组叫作矩阵（matrix）或二维张量（2D 张量）。<jupyter_code>x = np.array([
    [5, 78, 2, 34, 0],
    [6, 79, 3, 35, 1], 
    [7, 80, 4, 36, 2]])
print(x.ndim)<jupyter_output>2<jupyter_text>4. 3D张量与更高维张量将多个矩阵组合成一个新的数组，可以得到一个 3D 张量，你可以将其直观地理解为数字组成的立方体。<jupyter_code>x = np.array([[[5, 78, 2, 34, 0],
               [6, 79, 3, 35, 1], 
               [7, 80, 4, 36, 2]], 
              [[5, 78, 2, 34, 0], 
               [6, 79, 3, 35, 1], 
               [7, 80, 4, 36, 2]],
              [[5, 78, 2, 34, 0],
               [6, 79, 3, 35, 1], 
               [7, 80, 4, 36, 2]]])
print(x.ndim)
print(x.shape)<jupyter_output>3
(3, 3, 5)<jupyter_text>张量是由以下三个关键属性来定义的:  轴的个数（阶）。例如，3D 张量有 3 个轴，矩阵有 2 个轴。这在 Numpy 等 Python 库中 也叫张量的 `ndim`。 形状。这是一个整数元组，表示张量沿每个轴的维度大小（元素个数）。例如，前面矩阵示例的形状为(3, 5)，3D 张量示例的形状为(3, 3, 5)。向量的形状只包含一个元素，比如 (5,)，而标量的形状为空，即 ()。 数据类型（在 Python 库中通常叫作 dtype）。这是张量中所包含数据的类型，例如，张量的类型可以是 float32、uint8、float64 等。在极少数情况下，你可能会遇到字符（char）张量。注意，Numpy（以及大多数其他库）中不存在字符串张量，因为张量存储在预先分配的连续内存段中，而字符串的长度是可变的，无法用这种方式存储。  现实世界中的数据张量- 向量数据：2D 张量，形状为 (samples, features)。- 时间序列数据或序列数据：3D 张量，形状为 (samples, timesteps, features)。 - 图像：4D张量，形状为(samples, height, width, channels)或(samples, channels, height, width)。- 视频：5D张量，形状为(samples, frames, height, width, channels)或(samples, frames, channels, height, width)。  向量数据这是最常见的数据。对于这种数据集，每个数据点都被编码为一个向量，因此一个数据批量就被编码为 2D 张量（即向量组成的数组），其中第一个轴是样本轴，第二个轴是特征轴。我们来看两个例子。 - 人口统计数据集，其中包括每个人的年龄、邮编和收入。每个人可以表示为包含 3 个值 的向量，而整个数据集包含 100 000 个人，因此可以存储在形状为 (100000, 3) 的 2D 张量中。- 文本文档数据集，我们将每个文档表示为每个单词在其中出现的次数（字典中包含 20 000 个常见单词）。每个文档可以被编码为包含 20 000 个值的向量（每个值对应于 字典中每个单词的出现次数），整个数据集包含 500 个文档，因此可以存储在形状为 (500, 20000) 的张量中。  时间序列数据或序列数据![111111](img/time_serial_data.png)**举例**：- 股票价格数据集。每一分钟，我们将股票的当前价格、前一分钟的最高价格和前一分钟 的最低价格保存下来。因此每分钟被编码为一个 3D 向量，整个交易日被编码为一个形 状为(390, 3)的 2D 张量（一个交易日有 390 分钟），而 250 天的数据则可以保存在一 个形状为(250, 390, 3)的 3D 张量中。这里每个样本是一天的股票数据。- 推文数据集。我们将每条推文编码为 280 个字符组成的序列，而每个字符又来自于 128 个字符组成的字母表。在这种情况下，每个字符可以被编码为大小为 128 的二进制向量 （只有在该字符对应的索引位置取值为 1，其他元素都为 0）。那么每条推文可以被编码 为一个形状为 (280, 128) 的 2D 张量，而包含 100 万条推文的数据集则可以存储在一 个形状为 (1000000, 280, 128) 的张量中。  图像数据![](img/image_data.png)**举例**：如果图像大小为 256×256，那么 128 张灰度图像组成的批 量可以保存在一个形状为(128, 256, 256, 1)的张量中，而 128 张彩色图像组成的批量则可以保存在一个形状为(128, 256, 256, 3)的张量中。<jupyter_code>from sklearn.datasets import load_digits
digits = load_digits()
print("images shape: %s" % str(digits.images.shape))
print("targets shape: %s" % str(digits.target.shape))
img = digits.images[0]
import matplotlib.pyplot as plt
%matplotlib inline
plt.matshow(img, cmap=plt.cm.Greys);
img.shape
img<jupyter_output><empty_output><jupyter_text>**数据批量的概念（Batch Size）**<jupyter_code>batch01 = digits.images[:128]
batch02 = digits.images[128:256]
from PIL import Image
color_img = Image.open("img/cxy-logo.png")
plt.imshow(color_img)
import numpy as np
color_img = np.array(color_img)
color_img.shape<jupyter_output><empty_output><jupyter_text>视频数据视频可以看作一系列帧， 每一帧都是一张彩色图像。由于每一帧都可以保存在一个形状为 (height, width, color_ depth) 的 3D 张量中，因此一系列帧可以保存在一个形状为 (frames, height, width, color_depth) 的 4D 张量中，而不同视频组成的批量则可以保存在一个 5D 张量中，其形状为 (samples, frames, height, width, color_depth)。 **举例**：一个以每秒 4 帧采样的 60 秒视频片段，视频尺寸为 144×256，这个视频共有 240 帧。4 个这样的视频片段组成的批量将保存在形状为(4, 240, 144, 256, 3) 的张量中。总共有 106 168 320 个值！如果张量的数据类型（dtype）是 float32，每个值都是 32 位，那么这个张量共有 405MB。  TensorFlow中的数据表示 > tf.constant creates a tf.Tensor from a fixed value> you can read more here:>  https://www.tensorflow.org/api_docs/python/tf/constant<jupyter_code># 0-D tensor (scalar)
t_0d_py = 4

# 1-D tensor (vector)
t_1d_py = [1, 2, 3]

# 2-D tensor (matrix)
t_2d_py = [[1, 2], 
           [3, 4], 
           [5, 6]]

# 3-D tensor
t_3d_py = [[[0, 0], [0, 1], [0, 2]],
           [[1, 0], [1, 1], [1, 2]],
           [[2, 0], [2, 1], [2, 2]]]

python_defined_tensors = [t_0d_py, t_1d_py, t_2d_py, t_3d_py]

for pdt in python_defined_tensors:
    print(tf.constant(pdt))<jupyter_output>tf.Tensor(4, shape=(), dtype=int32)
tf.Tensor([1 2 3], shape=(3,), dtype=int32)
tf.Tensor(
[[1 2]
 [3 4]
 [5 6]], shape=(3, 2), dtype=int32)
tf.Tensor(
[[[0 0]
  [0 1]
  [0 2]]

 [[1 0]
  [1 1]
  [1 2]]

 [[2 0]
  [2 1]
  [2 2]]], shape=(3, 3, 2), dtype=int32)<jupyter_text>联想：NumPy Arraystensorflow中的`Tensor`和`numpy.array`定义的矩阵是等价的<jupyter_code>import numpy as np
# 0-D tensor (scalar)
t_0d_np = np.array(4, dtype=np.int32)

# 1-D tensor (vector)
t_1d_np = np.array([1, 2, 3], dtype=np.int64)

# 2-D tensor (matrix)
t_2d_np = np.array([[1, 2], 
                    [3, 4], 
                    [5, 6]],
                   dtype=np.float32)

# 3-D tensor
t_3d_np = np.array([[[0, 0], [0, 1], [0, 2]],
                    [[1, 0], [1, 1], [1, 2]],
                    [[2, 0], [2, 1], [2, 2]]],
                   dtype=np.int32)

numpy_defined_tensors = [t_0d_np, t_1d_np, t_2d_np, t_3d_np]

for ndt in numpy_defined_tensors:
    print(tf.constant(ndt))<jupyter_output>tf.Tensor(4, shape=(), dtype=int32)
tf.Tensor([1 2 3], shape=(3,), dtype=int64)
tf.Tensor(
[[1. 2.]
 [3. 4.]
 [5. 6.]], shape=(3, 2), dtype=float32)
tf.Tensor(
[[[0 0]
  [0 1]
  [0 2]]

 [[1 0]
  [1 1]
  [1 2]]

 [[2 0]
  [2 1]
  [2 2]]], shape=(3, 3, 2), dtype=int32)<jupyter_text>关于数据类型通常，建议在TensorFlow中使用`np.array`（或`np.asarray`）手动定义张量的值。这样做的主要原因是，您可以指定希望使用其表示值的确切数据类型（`dtype`）。TensorFlow与NumPy紧密集成，大多数TensorFlow数据类型具有对应的NumPy `dtype`。TensorFlow中数据类型 | 对应的NumPy中数据类型 | 描述--- | --- | ---`tf.float32` | `np.float32` | 32 bit floating point.`tf.float64` | `np.float64` | 64 bit floating point.`tf.int8` | `np.int8` | 8 bit signed integer.`tf.int16` | `np.int16` | 16 bit signed integer.`tf.int32` | `np.int32` | 32 bit signed integer.`tf.int64` | `np.int64` | 64 bit signed integer.`tf.uint8` | `np.uint8` | 8 bit unsigned integer.`tf.string` | N/A | String type, as byte array`tf.bool` | `np.bool` | Boolean.`tf.complex64` | `np.complex64` | Complex number made of two 32 bit floating point numbers: real and imaginary parts.`tf.qint8` | N/A | 8 bit signed integer used in quantized Ops.`tf.qint32` | N/A | 32 bit signed integer used in quantized Ops.`tf.quint8` | N/A | 8 bit unsigned integer used in quantized Ops.<jupyter_code># Just to show that they are equivalent
(tf.float32 == np.float32 and
 tf.float64 == np.float64 and
 tf.int8 == np.int8 and
 tf.int16 == np.int16 and
 tf.int32 == np.int32 and
 tf.int64 == np.int64 and
 tf.uint8 == np.uint8 and
 tf.bool == np.bool and
 tf.complex64 == np.complex64)<jupyter_output><ipython-input-405-20f5057ff7a2>:9: DeprecationWarning: `np.bool` is a deprecated alias for the builtin `bool`. To silence this warning, use `bool` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.bool_` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  tf.bool == np.bool and<jupyter_text>> 特别地，`np.array()`中不支持string类型，当需要定义string类型输入时，使用python的列表。<jupyter_code>tf_string_tensor = [b"first", b"second", b"third"]<jupyter_output><empty_output><jupyter_text>常量与变量*   `tf.constant`, 创建一个常量张量，常量中的值是不可变的。 *   `tf.Variable `, 此方法封装了一个可变张量，稍后可以使用assign对其进行更改 创建一个常量<jupyter_code>a = tf.constant([[2,3]])
print(a)<jupyter_output>tf.Tensor([[2 3]], shape=(1, 2), dtype=int32)<jupyter_text>常量无法被修改<jupyter_code>try:
  a.assign(2.)
except:
  print('Exception raised ')
v = tf.Variable(5.)

print('Old value for v =', v.numpy())
v.assign(2.)
print('New value for v =', v.numpy())<jupyter_output>Old value for v = 5.0
New value for v = 2.0<jupyter_text>你可以像python变量一样，进行快速的递增、递减操作<jupyter_code>v.assign(2.)
print('value     : ', v.numpy())
print('increment : ', v.assign_add(1.).numpy())
print('decrement : ', v.assign_sub(1.).numpy())
print('name  : ', v.name)
print('type  : ', v.dtype)
print('shape : ', v.shape)
print('device: ', v.device)<jupyter_output>name  :  Variable:0
type  :  <dtype: 'float32'>
shape :  ()
device:  /job:localhost/replica:0/task:0/device:CPU:0<jupyter_text>扩展最常用的常量定义，比如定义全一或者全零的矩阵,使用`tf.ones`和`tf.zeros`<jupyter_code>print(tf.ones(shape=(2, 1)))
print(tf.zeros(shape=(2, 1)))<jupyter_output>tf.Tensor(
[[1.]
 [1.]], shape=(2, 1), dtype=float32)
tf.Tensor(
[[0.]
 [0.]], shape=(2, 1), dtype=float32)<jupyter_text>定义随机的常量数组，通过随机函数中的一些常用分布<jupyter_code>tf.random.normal(shape=(2, 2), mean=0., stddev=1.)
tf.random.uniform(shape=(2, 2), minval=0, maxval=10, dtype='int32')<jupyter_output><empty_output><jupyter_text>1.5 计算图 TensorFlow API中的主要构造是“计算图”。计算图是说明通过多个操作的数据流的简便方法。这是一个基本示例，显示了将两个数字相加的计算图：箭头表示流经图形的数据（在这种情况下为数字7、3和10），而节点表示某种形式的计算（在此情况下为加法）。 我们可以将这些计算中的多个链接在一起，以形成更复杂的数据转换：此处，数字3和7被发送到加法运算和乘法运算。然后将这两者的输出发送到减法运算。有几种方法可以说明上述操作。* 我们可以将其显示为单线数学方程式，使用括号显示操作顺序：$$output = \left( 3 + 7 \right) - \left( 3 \times 7 \right) = -11$$* 我们还可以将单独的操作定义为数学函数：$$\begin{split}a(x, y) &= x + y \\b(x, y) &= x \times y \\c(x, y) &= a(x, y) - b(x, y) \\ \\c(3, 7) &= a(3, 7) - b(3, 7) = -11\end{split}$$* 通过程序描述如下<jupyter_code>def add(x, y):
    return x + y
   
def multiply(x, y):
    return x * y
    
def subtract(x, y):
    return x - y
    
x = 3
y = 7
a = add(x, y)
b = multiply(x, y)
c = subtract(a, b)

c<jupyter_output><empty_output><jupyter_text>数据的使用不是一次性的，在整个计算中可以多次重用：该图类似于先前的模型，但是现在我们通过在最后添加3来重用3这个输入。代码可能看起来像这样：<jupyter_code>x = 3
y = 7
a = add(x, y)
b = multiply(x, y)
c = subtract(a, b)
d = add(c, x)

d
node1 = tf.constant(3.0, tf.float32)
node2 = tf.constant(4.0) # also tf.float32 implicitly
node3 = tf.add(node1, node2)
print("node1:", node1, "\n", "node2:", node2)
print("node3: ", node3)<jupyter_output>node1: tf.Tensor(3.0, shape=(), dtype=float32) 
 node2: tf.Tensor(4.0, shape=(), dtype=float32)
node3:  tf.Tensor(7.0, shape=(), dtype=float32)<jupyter_text>![Graph](https://www.tensorflow.org/images/getting_started_adder.png)  [TensorFlow 2.0] 使用`GradientTape` 计算梯度> 常量需要使用`tape.watch()`来追踪，变量默认记录相关梯度<jupyter_code>a = tf.random.normal(shape=(2, 2))
b = tf.random.normal(shape=(2, 2))

with tf.GradientTape() as tape:
  tape.watch(a)  # Start recording the history of operations applied to `a`
  c = tf.sqrt(tf.square(a) + tf.square(b))  # Do some math using `a`
  # What's the gradient of `c` with respect to `a`?
  dc_da = tape.gradient(c, a)
  print(dc_da)
a = tf.Variable(a)

with tf.GradientTape() as tape:
  c = tf.sqrt(tf.square(a) + tf.square(b))
  dc_da = tape.gradient(c, a)
  print(dc_da)<jupyter_output>tf.Tensor(
[[ 0.28811598 -0.06045575]
 [ 0.7964895   0.22431341]], shape=(2, 2), dtype=float32)<jupyter_text>Lab01: 线性回归<jupyter_code>from __future__ import absolute_import, division, print_function
import tensorflow as tf
import numpy as np
rng = np.random<jupyter_output><empty_output><jupyter_text>定义超参值<jupyter_code># Parameters.
learning_rate = 0.01
training_steps = 1000
display_step = 50<jupyter_output><empty_output><jupyter_text>训练数据<jupyter_code># Training Data.
X = np.array([3.3,4.4,5.5,6.71,6.93,4.168,9.779,6.182,7.59,2.167,
              7.042,10.791,5.313,7.997,5.654,9.27,3.1])
Y = np.array([1.7,2.76,2.09,3.19,1.694,1.573,3.366,2.596,2.53,1.221,
              2.827,3.465,1.65,2.904,2.42,2.94,1.3])
n_samples = X.shape[0]<jupyter_output><empty_output><jupyter_text>定义变量和函数<jupyter_code># Weight and Bias, initialized randomly.
W = tf.Variable(rng.randn(), name="weight")
b = tf.Variable(rng.randn(), name="bias")

# Linear regression (Wx + b).
def linear_regression(x):
    return W * x + b

# Loss Function: Mean square error.
def mean_square(y_pred, y_true):
    #return tf.reduce_sum(tf.pow(y_pred-y_true, 2)) / (2 * n_samples)
    return tf.reduce_mean(tf.square(y_pred - y_true))
# Stochastic Gradient Descent Optimizer.
optimizer = tf.optimizers.SGD(learning_rate)
# Optimization process. 
def one_step():
    # Wrap computation inside a GradientTape for automatic differentiation.
    with tf.GradientTape() as g:
        pred = linear_regression(X)
        loss = mean_square(pred, Y)

    # Compute gradients.
    gradients = g.gradient(loss, [W, b])
    
    # Update W and b following gradients.
    optimizer.apply_gradients(zip(gradients, [W, b]))
import time
t0 = time.time()
# Run training for the given number of steps.
for step in range(1, training_steps + 1):
    # Run the optimization to update W and b values.
    one_step()
    
    if step % display_step == 0:
        pred = linear_regression(X)
        loss = mean_square(pred, Y)
        print("step: %i, loss: %f, W: %f, b: %f" % (step, loss, W.numpy(), b.numpy()))
t_end = time.time() - t0
print('Time per epoch: %.3f s' % (t_end / 20,))<jupyter_output>step: 50, loss: 0.266430, W: 0.386140, b: -0.154782
step: 100, loss: 0.242139, W: 0.370748, b: -0.045658
step: 150, loss: 0.223090, W: 0.357117, b: 0.050978
step: 200, loss: 0.208152, W: 0.345046, b: 0.136556
step: 250, loss: 0.196436, W: 0.334357, b: 0.212341
step: 300, loss: 0.187249, W: 0.324890, b: 0.279453
step: 350, loss: 0.180044, W: 0.316507, b: 0.338885
step: 400, loss: 0.174393, W: 0.309084, b: 0.391516
step: 450, loss: 0.169962, W: 0.302509, b: 0.438124
step: 500, loss: 0.166487, W: 0.296688, b: 0.479398
step: 550, loss: 0.163762, W: 0.291532, b: 0.515950
step: 600, loss: 0.161625, W: 0.286966, b: 0.548318
step: 650, loss: 0.159949, W: 0.282923, b: 0.576982
step: 700, loss: 0.158635, W: 0.279343, b: 0.602367
step: 750, loss: 0.157604, W: 0.276172, b: 0.624846
step: 800, loss: 0.156796, W: 0.273364, b: 0.644753
step: 850, loss: 0.156162, W: 0.270877, b: 0.662382
step: 900, loss: 0.155664, W: 0.268675, b: 0.677993
step: 950, loss: 0.155275, W: 0.266725, b: 0.691818
step: 1000,[...]<jupyter_text>使用修饰符`@tf.function` 变为静态图<jupyter_code>@tf.function
# Optimization process. 
def one_step():
    # Wrap computation inside a GradientTape for automatic differentiation.
    with tf.GradientTape() as g:
        pred = linear_regression(X)
        loss = mean_square(pred, Y) 
    # Compute gradients.
    gradients = g.gradient(loss, [W, b])
    
    # Update W and b following gradients.
    optimizer.apply_gradients(zip(gradients, [W, b]))
    
# Weight and Bias, initialized randomly.
W = tf.Variable(rng.randn(), name="weight")
b = tf.Variable(rng.randn(), name="bias")
# Variables for plotting cost function
W_history = []
cost_history = []

import time
t0 = time.time()
# Run training for the given number of steps.
for step in range(1, training_steps + 1):
    # Run the optimization to update W and b values.
    one_step()
    
    if step % display_step == 0:
        pred = linear_regression(X)
        loss = mean_square(pred, Y)
        print("step: %i, loss: %f, W: %f, b: %f" % (step, loss, W.numpy(), b.numpy()))
        # Recorde W and b    
        W_history.append(W.numpy())
        cost_history.append(loss.numpy())   
t_end = time.time() - t0
print('Time per epoch: %.3f s' % (t_end / 20,))<jupyter_output>step: 50, loss: 0.306112, W: 0.408061, b: -0.310189
step: 100, loss: 0.273259, W: 0.390160, b: -0.183281
step: 150, loss: 0.247495, W: 0.374308, b: -0.070895
step: 200, loss: 0.227290, W: 0.360270, b: 0.028629
step: 250, loss: 0.211445, W: 0.347838, b: 0.116765
step: 300, loss: 0.199019, W: 0.336829, b: 0.194814
step: 350, loss: 0.189274, W: 0.327080, b: 0.263932
step: 400, loss: 0.181632, W: 0.318446, b: 0.325140
step: 450, loss: 0.175639, W: 0.310801, b: 0.379344
step: 500, loss: 0.170939, W: 0.304030, b: 0.427345
step: 550, loss: 0.167253, W: 0.298034, b: 0.469853
step: 600, loss: 0.164363, W: 0.292724, b: 0.507496
step: 650, loss: 0.162096, W: 0.288022, b: 0.540832
step: 700, loss: 0.160319, W: 0.283858, b: 0.570353
step: 750, loss: 0.158924, W: 0.280171, b: 0.596496
step: 800, loss: 0.157831, W: 0.276905, b: 0.619647
step: 850, loss: 0.156974, W: 0.274013, b: 0.640149
step: 900, loss: 0.156301, W: 0.271452, b: 0.658304
step: 950, loss: 0.155774, W: 0.269185, b: 0.674382
step: 1000[...]<jupyter_text>查看曲线拟合情况<jupyter_code>import matplotlib.pyplot as plt
%matplotlib inline

# Graphic display
plt.plot(X, Y, 'ro', label='Original data')
plt.plot(X, np.array(W * X + b), label='Fitted line')
plt.legend()
plt.show()<jupyter_output><empty_output><jupyter_text>了解参数W是如何被优化的<jupyter_code># Show the cost function
plt.plot(W_history,cost_history)
plt.show()<jupyter_output><empty_output><jupyter_text>分析优化过程   Lab02：多元线性回归<jupyter_code># Lab Multi-variable linear regression
import tensorflow as tf

X = tf.constant([[73., 80., 75.],
          [93., 88., 93.],
          [89., 91., 90.],
          [96., 98., 100.],
          [73., 66., 70.]])
Y = tf.constant([[152.],
          [185.],
          [180.],
          [196.],
          [142.]])
# Parameters.
learning_rate = 1e-5
training_steps = 1000
display_step = 50

N = X.shape[0]<jupyter_output><empty_output><jupyter_text>> **注意：**此处如果继续使用1e-2的学习率，容易出现意想不到的结果，思考一下为什么？<jupyter_code>W = tf.Variable(tf.random.uniform(shape=(3, 1)), name='weight')
b = tf.Variable(tf.random.uniform(shape=(1,)), name='bias')
# Linear regression (Wx + b).
def multi_linear_regression(X):
    return tf.matmul(X, W) + b

# Mean square error.
def mean_square(y_pred, y_true):
    return tf.reduce_mean(tf.square(y_pred - y_true))

# Stochastic Gradient Descent Optimizer.
optimizer = tf.optimizers.SGD(learning_rate)
# Optimization process. 
def run_optimization():
    # Wrap computation inside a GradientTape for automatic differentiation.
    with tf.GradientTape() as g:
        pred = multi_linear_regression(X)
        pred = tf.squeeze(pred,axis=1)
        loss = mean_square(pred, Y)
    # Compute gradients.
    gradients = g.gradient(loss, [W, b])
    
    # Update W and b following gradients.
    optimizer.apply_gradients(zip(gradients, [W, b]))
# Run training for the given number of steps.
for step in range(1, training_steps + 1):
    # Run the optimization to update W and b values.
    run_optimization()
    
    if step % display_step == 0:
        pred = multi_linear_regression(X)
        loss = mean_square(pred, Y)
        print("step: %i, loss: %f" % (step, loss))<jupyter_output>step: 50, loss: 14.107248
step: 100, loss: 13.769302
step: 150, loss: 13.443494
step: 200, loss: 13.129648
step: 250, loss: 12.827313
step: 300, loss: 12.536352
step: 350, loss: 12.256269
step: 400, loss: 11.986787
step: 450, loss: 11.727706
step: 500, loss: 11.478773
step: 550, loss: 11.239454
step: 600, loss: 11.009799
step: 650, loss: 10.789246
step: 700, loss: 10.577768
step: 750, loss: 10.375127
step: 800, loss: 10.180838
step: 850, loss: 9.994935
step: 900, loss: 9.817021
step: 950, loss: 9.646853
step: 1000, loss: 9.484459<jupyter_text>Lab03：逻辑回归（二分类）<jupyter_code>import pandas as pd
import tensorflow as tf
import numpy as np

df = pd.read_csv('data/data-03-diabetes.csv', header=None, delimiter=',')
df.head()
X = df[[0,1,2,3,4,5,6,7]].values
Y = df[8].values
X.shape,Y.shape
X = tf.constant(X,dtype='float32')
Y = tf.constant(Y,dtype='float32')
# Parameters.
learning_rate = 1e-2
training_steps = 10000
display_step = 500

N = X.shape[0]

num_features = X.shape[1]
num_classes = 1
W = tf.Variable(tf.random.normal((num_features, num_classes)), name='weight')
b = tf.Variable(tf.random.normal([num_classes]), name='bias')
# Linear regression (Wx + b).
def logits_regression(x):
    return tf.sigmoid(tf.add(tf.matmul(x,W),b))

# loss/error.
def loss_f(y_pred, y_true):
    return -tf.reduce_mean(y_true * tf.math.log(y_pred) + (1 - y_true) *
                       tf.math.log(1 - y_pred))

# Stochastic Gradient Descent Optimizer.
optimizer = tf.optimizers.SGD(learning_rate)


def accuracy(y_pred, y_true):
    # Accuracy computation
    # True if hypothesis>0.5 else False
    predicted = tf.cast(y_pred > 0.5, dtype=tf.float32)
    accuracy = tf.reduce_mean(tf.cast(tf.equal(predicted, y_true), dtype=tf.float32))
    return accuracy
# Optimization process. 
def run_optimization():
    # Wrap computation inside a GradientTape for automatic differentiation.
    with tf.GradientTape() as g:
        pred = logits_regression(X)
        pred = tf.squeeze(pred,axis=1)
        loss = loss_f(pred, Y)
    # Compute gradients.
    gradients = g.gradient(loss, [W, b])
    
    # Update W and b following gradients.
    optimizer.apply_gradients(zip(gradients, [W, b]))
# Run training for the given number of steps.
for step in range(1, training_steps + 1):
    # Run the optimization to update W and b values.
    run_optimization()
    
    if step % display_step == 0:
        pred =logits_regression(X)
        pred = tf.squeeze(pred,axis=1)
        loss = loss_f(pred, Y)
        acc = accuracy(pred, Y)
        print("step: %i, loss: %f, accuracy: %f" % (step, loss, acc))<jupyter_output>step: 500, loss: 0.471938, accuracy: 0.769433
step: 1000, loss: 0.471928, accuracy: 0.769433
step: 1500, loss: 0.471918, accuracy: 0.769433
step: 2000, loss: 0.471908, accuracy: 0.769433
step: 2500, loss: 0.471899, accuracy: 0.769433
step: 3000, loss: 0.471890, accuracy: 0.769433
step: 3500, loss: 0.471882, accuracy: 0.769433
step: 4000, loss: 0.471874, accuracy: 0.769433
step: 4500, loss: 0.471866, accuracy: 0.768116
step: 5000, loss: 0.471859, accuracy: 0.768116
step: 5500, loss: 0.471851, accuracy: 0.768116
step: 6000, loss: 0.471844, accuracy: 0.768116
step: 6500, loss: 0.471838, accuracy: 0.768116
step: 7000, loss: 0.471832, accuracy: 0.768116
step: 7500, loss: 0.471826, accuracy: 0.768116
step: 8000, loss: 0.471820, accuracy: 0.768116
step: 8500, loss: 0.471814, accuracy: 0.768116
step: 9000, loss: 0.471809, accuracy: 0.768116
step: 9500, loss: 0.471804, accuracy: 0.768116
step: 10000, loss: 0.471799, accuracy: 0.768116<jupyter_text>Lab04：多分类问题<jupyter_code>import pandas as pd
import tensorflow as tf
import numpy as np

df = pd.read_csv('data/data-04-zoo.csv', header=None, delimiter=',', 
                 skiprows=19,dtype=np.float32)<jupyter_output><empty_output><jupyter_text>可以查看csv头文件了解数据属性<jupyter_code>df.head()
X = df[[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15]].values
Y = df[16].values
Y = tf.constant(Y,dtype='int32')
X.shape, Y.shape
# Parameters.
learning_rate = 1e-3
training_steps = 1000
display_step = 50

N = X.shape[0]

num_features = X.shape[1]
num_classes = 7 # 0-6
W = tf.Variable(tf.random.normal((num_features, num_classes)), name='weight')
b = tf.Variable(tf.random.normal([num_classes]), name='bias')
# Logistic regression (Wx + b).
def logistic_regression(x):
    # Apply softmax to normalize the logits to a probability distribution.
    return tf.nn.softmax(tf.matmul(x, W) + b)

# Cross-Entropy loss function.
def cross_entropy(y_pred, y_true):
    # Encode label to a one hot vector.
    y_true = tf.one_hot(y_true, depth=num_classes,dtype='float32')
    # Clip prediction values to avoid log(0) error.
    y_pred = tf.clip_by_value(y_pred, 1e-9, 1.)
    # Compute cross-entropy.
    return tf.reduce_mean(-tf.reduce_sum(y_true * tf.math.log(y_pred)))

# Accuracy metric.
def accuracy(y_pred, y_true):
    # Predicted class is the index of highest score in prediction vector (i.e. argmax).
    correct_prediction = tf.equal(tf.argmax(y_pred, 1), tf.cast(y_true, tf.int64))
    return tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

# Stochastic gradient descent optimizer.
optimizer = tf.optimizers.SGD(learning_rate)
# Optimization process. 
def run_optimization(x, y):
    # Wrap computation inside a GradientTape for automatic differentiation.
    with tf.GradientTape() as g:
        pred = logistic_regression(x)
        loss = cross_entropy(pred, y)

    # Compute gradients.
    gradients = g.gradient(loss, [W, b])
    
    # Update W and b following gradients.
    optimizer.apply_gradients(zip(gradients, [W, b]))
# Run training for the given number of steps.
for step in range(1, training_steps + 1):
    # Run the optimization to update W and b values.
    run_optimization(X, Y)
    
    if step % display_step == 0:
        pred = logistic_regression(X)
        loss = cross_entropy(pred, Y)
        acc = accuracy(pred, Y)
        print("step: %i, loss: %f, accuracy: %f" % (step, loss, acc))<jupyter_output>step: 50, loss: 3.669946, accuracy: 1.000000
step: 100, loss: 3.612689, accuracy: 1.000000
step: 150, loss: 3.557250, accuracy: 1.000000
step: 200, loss: 3.503543, accuracy: 1.000000
step: 250, loss: 3.451489, accuracy: 1.000000
step: 300, loss: 3.401008, accuracy: 1.000000
step: 350, loss: 3.352032, accuracy: 1.000000
step: 400, loss: 3.304497, accuracy: 1.000000
step: 450, loss: 3.258330, accuracy: 1.000000
step: 500, loss: 3.213479, accuracy: 1.000000
step: 550, loss: 3.169891, accuracy: 1.000000
step: 600, loss: 3.127503, accuracy: 1.000000
step: 650, loss: 3.086274, accuracy: 1.000000
step: 700, loss: 3.046155, accuracy: 1.000000
step: 750, loss: 3.007096, accuracy: 1.000000
step: 800, loss: 2.969064, accuracy: 1.000000
step: 850, loss: 2.932007, accuracy: 1.000000
step: 900, loss: 2.895896, accuracy: 1.000000
step: 950, loss: 2.860691, accuracy: 1.000000
step: 1000, loss: 2.826364, accuracy: 1.000000<jupyter_text>Lab05 神经网络初探<jupyter_code>import tensorflow as tf
import matplotlib.pyplot as plt
import random<jupyter_output><empty_output><jupyter_text>Softmax 多分类实现<jupyter_code>from __future__ import absolute_import, division, print_function

import tensorflow as tf
import numpy as np
# MNIST dataset parameters.
num_classes = 10 # 0 to 9 digits
num_features = 784 # 28*28

# Training parameters.
learning_rate = 0.01
training_steps = 1000
batch_size = 256
display_step = 50
# Prepare MNIST data.
from tensorflow.keras.datasets import mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()
# Convert to float32.
x_train, x_test = np.array(x_train, np.float32), np.array(x_test, np.float32)
# Flatten images to 1-D vector of 784 features (28*28).
x_train, x_test = x_train.reshape([-1, num_features]), x_test.reshape([-1, num_features])
# Normalize images value from [0, 255] to [0, 1].
x_train, x_test = x_train / 255., x_test / 255.
# Use tf.data API to shuffle and batch data.
train_data = tf.data.Dataset.from_tensor_slices((x_train, y_train))
train_data = train_data.repeat().shuffle(5000).batch(batch_size).prefetch(1)
# Weight of shape [784, 10], the 28*28 image features, and total number of classes.
W = tf.Variable(tf.ones([num_features, num_classes]), name="weight")
# Bias of shape [10], the total number of classes.
b = tf.Variable(tf.zeros([num_classes]), name="bias")

# Logistic regression (Wx + b).
def logistic_regression(x):
    # Apply softmax to normalize the logits to a probability distribution.
    return tf.nn.softmax(tf.matmul(x, W) + b)

# Cross-Entropy loss function.
def cross_entropy(y_pred, y_true):
    # Encode label to a one hot vector.
    y_true = tf.one_hot(y_true, depth=num_classes)
    # Clip prediction values to avoid log(0) error.
    y_pred = tf.clip_by_value(y_pred, 1e-9, 1.)
    # Compute cross-entropy.
    return tf.reduce_mean(-tf.reduce_sum(y_true * tf.math.log(y_pred)))

# Accuracy metric.
def accuracy(y_pred, y_true):
    # Predicted class is the index of highest score in prediction vector (i.e. argmax).
    correct_prediction = tf.equal(tf.argmax(y_pred, 1), tf.cast(y_true, tf.int64))
    return tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

# Stochastic gradient descent optimizer.
optimizer = tf.optimizers.SGD(learning_rate)
# Optimization process. 
def run_optimization(x, y):
    # Wrap computation inside a GradientTape for automatic differentiation.
    with tf.GradientTape() as g:
        pred = logistic_regression(x)
        loss = cross_entropy(pred, y)

    # Compute gradients.
    gradients = g.gradient(loss, [W, b])
    
    # Update W and b following gradients.
    optimizer.apply_gradients(zip(gradients, [W, b]))
# Run training for the given number of steps.
for step, (batch_x, batch_y) in enumerate(train_data.take(training_steps), 1):
    # Run the optimization to update W and b values.
    run_optimization(batch_x, batch_y)
    
    if step % display_step == 0:
        pred = logistic_regression(batch_x)
        loss = cross_entropy(pred, batch_y)
        acc = accuracy(pred, batch_y)
        print("step: %i, loss: %f, accuracy: %f" % (step, loss, acc))
# Test model on validation set.
pred = logistic_regression(x_test)
print("Test Accuracy: %f" % accuracy(pred, y_test))
# Visualize predictions.
import matplotlib.pyplot as plt
# Predict 5 images from validation set.
n_images = 5
test_images = x_test[:n_images]
predictions = logistic_regression(test_images)

# Display image and model prediction.
for i in range(n_images):
    plt.imshow(np.reshape(test_images[i], [28, 28]), cmap='gray')
    plt.show()
    print("Model prediction: %i" % np.argmax(predictions.numpy()[i]))<jupyter_output><empty_output><jupyter_text>多层神经网络<jupyter_code>from __future__ import absolute_import, division, print_function

import tensorflow as tf
import numpy as np
# MNIST dataset parameters.
num_classes = 10 # total classes (0-9 digits).
num_features = 784 # data features (img shape: 28*28).

# Training parameters.
learning_rate = 0.001
training_steps = 3000
batch_size = 256
display_step = 100

# Network parameters.
n_hidden_1 = 128 # 1st layer number of neurons.
n_hidden_2 = 256 # 2nd layer number of neurons.
# Prepare MNIST data.
from tensorflow.keras.datasets import mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()
# Convert to float32.
x_train, x_test = np.array(x_train, np.float32), np.array(x_test, np.float32)
# Flatten images to 1-D vector of 784 features (28*28).
x_train, x_test = x_train.reshape([-1, num_features]), x_test.reshape([-1, num_features])
# Normalize images value from [0, 255] to [0, 1].
x_train, x_test = x_train / 255., x_test / 255.
# Use tf.data API to shuffle and batch data.
train_data = tf.data.Dataset.from_tensor_slices((x_train, y_train))
train_data = train_data.repeat().shuffle(5000).batch(batch_size).prefetch(1)
# Store layers weight & bias

# A random value generator to initialize weights.
random_normal = tf.initializers.RandomNormal()

weights = {
    'h1': tf.Variable(random_normal([num_features, n_hidden_1])),
    'h2': tf.Variable(random_normal([n_hidden_1, n_hidden_2])),
    'out': tf.Variable(random_normal([n_hidden_2, num_classes]))
}
biases = {
    'b1': tf.Variable(tf.zeros([n_hidden_1])),
    'b2': tf.Variable(tf.zeros([n_hidden_2])),
    'out': tf.Variable(tf.zeros([num_classes]))
}
# Create model.
def neural_net(x):
    # Hidden fully connected layer with 128 neurons.
    layer_1 = tf.add(tf.matmul(x, weights['h1']), biases['b1'])
    # Apply sigmoid to layer_1 output for non-linearity.
    layer_1 = tf.nn.sigmoid(layer_1)
    
    # Hidden fully connected layer with 256 neurons.
    layer_2 = tf.add(tf.matmul(layer_1, weights['h2']), biases['b2'])
    # Apply sigmoid to layer_2 output for non-linearity.
    layer_2 = tf.nn.sigmoid(layer_2)
    
    # Output fully connected layer with a neuron for each class.
    out_layer = tf.matmul(layer_2, weights['out']) + biases['out']
    # Apply softmax to normalize the logits to a probability distribution.
    return tf.nn.softmax(out_layer)
# Cross-Entropy loss function.
def cross_entropy(y_pred, y_true):
    # Encode label to a one hot vector.
    y_true = tf.one_hot(y_true, depth=num_classes)
    # Clip prediction values to avoid log(0) error.
    y_pred = tf.clip_by_value(y_pred, 1e-9, 1.)
    # Compute cross-entropy.
    return tf.reduce_mean(-tf.reduce_sum(y_true * tf.math.log(y_pred)))

# Accuracy metric.
def accuracy(y_pred, y_true):
    # Predicted class is the index of highest score in prediction vector (i.e. argmax).
    correct_prediction = tf.equal(tf.argmax(y_pred, 1), tf.cast(y_true, tf.int64))
    return tf.reduce_mean(tf.cast(correct_prediction, tf.float32), axis=-1)

# Stochastic gradient descent optimizer.
optimizer = tf.optimizers.SGD(learning_rate)
# Optimization process. 
def run_optimization(x, y):
    # Wrap computation inside a GradientTape for automatic differentiation.
    with tf.GradientTape() as g:
        pred = neural_net(x)
        loss = cross_entropy(pred, y)
        
    # Variables to update, i.e. trainable variables.
    trainable_variables = list(weights.values()) + list(biases.values())

    # Compute gradients.
    gradients = g.gradient(loss, trainable_variables)
    
    # Update W and b following gradients.
    optimizer.apply_gradients(zip(gradients, trainable_variables))
type(weights.values)
# Run training for the given number of steps.
for step, (batch_x, batch_y) in enumerate(train_data.take(training_steps), 1):
    # Run the optimization to update W and b values.
    run_optimization(batch_x, batch_y)
    
    if step % display_step == 0:
        pred = neural_net(batch_x)
        loss = cross_entropy(pred, batch_y)
        acc = accuracy(pred, batch_y)
        print("step: %i, loss: %f, accuracy: %f" % (step, loss, acc))
# Test model on validation set.
pred = neural_net(x_test)
print("Test Accuracy: %f" % accuracy(pred, y_test))
# Visualize predictions.
import matplotlib.pyplot as plt

# Predict 5 images from validation set.
n_images = 5
test_images = x_test[:n_images]
predictions = neural_net(test_images)

# Display image and model prediction.
for i in range(n_images):
    plt.imshow(np.reshape(test_images[i], [28, 28]), cmap='gray')
    plt.show()
    print("Model prediction: %i" % np.argmax(predictions.numpy()[i]))<jupyter_output><empty_output><jupyter_text>Keras实现<jupyter_code>from tensorflow import keras
from tensorflow.keras.datasets import mnist
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Dropout
batch_size = 128
num_classes = 10
epochs = 12
# ==============================================================================
# prepare data
(X_train, y_train), (X_test, y_test) = mnist.load_data()

X_train = X_train.reshape(60000, 784)
X_test = X_test.reshape(10000, 784)
X_train = X_train.astype('float32')
X_test = X_test.astype('float32')
X_train /= 255
X_test /= 255
print(X_train.shape[0], 'train samples')
print(X_test.shape[0], 'test samples')

y_train = keras.utils.to_categorical(y_train, num_classes)
y_test = keras.utils.to_categorical(y_test, num_classes)
X_train.shape
# ==============================================================================
# build model
model = keras.Sequential([
    keras.layers.Dense(128, activation='relu',input_shape=(28*28,)),
    keras.layers.Dense(num_classes, activation='softmax')
])

model.compile(loss='categorical_crossentropy',
              optimizer='adam', metrics=['accuracy'])

history = model.fit(X_train, y_train,
                    batch_size=batch_size,
                    epochs=epochs,
                    verbose=1)
# ==============================================================================
# predict
score = model.evaluate(X_test, y_test, batch_size=batch_size)
print('\nTest loss:', score[0])
print('Test accuracy:', score[1])<jupyter_output>79/79 [==============================] - 0s 856us/step - loss: 0.0709 - accuracy: 0.9781

Test loss: 0.07088800519704819
Test accuracy: 0.9781000018119812
{'fraction_non_alphanumeric': 0.15424410806667943, 'fraction_numerical': 0.08893329318698164, 'mean_word_length': 4.336546888694128, 'pattern_counts': {'":': 0, '<': 138, '<?xml version=': 0, '>': 149, 'https://': 3, 'lorem ipsum': 0, 'www.': 2, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '39', 'dir': 'jupyter-structured-clean-dedup', 'id': '76759', 'license': 'MIT', 'n_tokens_mistral': 17669, 'n_tokens_neox': 15768, 'n_words': 3424, 'path': 'Day03-TensorFlow/01.tensorflow_fundation_tutorials.ipynb', 'repo_name': 'shaiic/ai-lab', 'size': '36533'}