Econ 425T / Biostat 203B
Display system information for reproducibility.
import IPython
print(IPython.sys_info()){'commit_hash': 'add5877a4',
'commit_source': 'installation',
'default_encoding': 'utf-8',
'ipython_path': '/Users/huazhou/opt/anaconda3/lib/python3.9/site-packages/IPython',
'ipython_version': '8.8.0',
'os_name': 'posix',
'platform': 'macOS-10.16-x86_64-i386-64bit',
'sys_executable': '/Users/huazhou/opt/anaconda3/bin/python3',
'sys_platform': 'darwin',
'sys_version': '3.9.12 (main, Apr 5 2022, 01:56:13) \n[Clang 12.0.0 ]'}sessionInfo()R version 4.2.2 (2022-10-31)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Big Sur ... 10.16
Matrix products: default
BLAS: /Library/Frameworks/R.framework/Versions/4.2/Resources/lib/libRblas.0.dylib
LAPACK: /Library/Frameworks/R.framework/Versions/4.2/Resources/lib/libRlapack.dylib
locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8
attached base packages:
[1] stats graphics grDevices utils datasets methods base
loaded via a namespace (and not attached):
[1] Rcpp_1.0.9 here_1.0.1 lattice_0.20-45 png_0.1-8
[5] withr_2.5.0 rprojroot_2.0.3 digest_0.6.29 grid_4.2.2
[9] jsonlite_1.8.0 magrittr_2.0.3 evaluate_0.15 rlang_1.0.6
[13] stringi_1.7.8 cli_3.4.1 rstudioapi_0.13 Matrix_1.5-1
[17] reticulate_1.27 rmarkdown_2.14 tools_4.2.2 stringr_1.4.0
[21] htmlwidgets_1.6.1 xfun_0.31 yaml_2.3.5 fastmap_1.1.0
[25] compiler_4.2.2 htmltools_0.5.4 knitr_1.39 Load some libraries.
# Load the pandas library
import pandas as pd
# Load numpy for array manipulation
import numpy as np
# Load seaborn plotting library
import seaborn as sns
import matplotlib.pyplot as plt
# Set font sizes in plots
sns.set(font_scale = 1.2)
# Display all columns
pd.set_option('display.max_columns', None)
# Load Tensorflow and Keras
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layerslibrary(keras)
library(jpeg)In this example, we train a CNN (convolution neural network) on the CIFAR100 data set. Achieve testing accuracy 44.5% after 30 epochs. Random guess would have an accuracy of about 1%.
The CIFAR100 database is a large database of \(32 \times 32\) color images that is commonly used for training and testing machine learning algorithms.
50,000 training images, 10,000 testing images.
Acquire data:
# Load the data and split it between train and test sets
(x_train, y_train), (x_test, y_test) = keras.datasets.cifar100.load_data()
# Training set
x_train.shape(50000, 32, 32, 3)y_train.shape
# Test set(50000, 1)x_test.shape(10000, 32, 32, 3)y_test.shape(10000, 1)cifar100 <- dataset_cifar100()
x_train <- cifar100$train$x
y_train <- cifar100$train$y
x_test <- cifar100$test$x
y_test <- cifar100$test$yTraining set:
dim(x_train)[1] 50000 32 32 3dim(y_train)[1] 50000 1Testing set:
dim(y_train)[1] 50000 1dim(y_test)[1] 10000 1For CNN, we keep the \(32 \times 32 \times 3\) tensor structure, instead of vectorizing into a long vector.
# Rescale
x_train = x_train / 255
x_test = x_test / 255
# Train
x_train.shape
# Test(50000, 32, 32, 3)x_test.shape(10000, 32, 32, 3)# rescale
x_train <- x_train / 255
x_test <- x_test / 255
dim(x_train)[1] 50000 32 32 3dim(x_test)[1] 10000 32 32 3Encode \(y\) as binary class matrix:
y_train = keras.utils.to_categorical(y_train, 100)
y_test = keras.utils.to_categorical(y_test, 100)
# Train
y_train.shape
# Test(50000, 100)y_test.shape
# First train instance(10000, 100)y_train[0]array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.],
dtype=float32)y_train <- to_categorical(y_train, 100)
y_test <- to_categorical(y_test, 100)
dim(y_train)[1] 50000 100dim(y_test)[1] 10000 100# head(y_train)Show a few images:
import matplotlib.pyplot as plt
# Feature: 32x32 color image
for i in range(25):
plt.figure()
plt.imshow(x_train[i]);
plt.show()
par(mar = c(0, 0, 0, 0), mfrow = c(5, 5))
index <- sample(seq(50000), 25)
for (i in index) plot(as.raster(x_train[i,,, ]))
Define a sequential model (a linear stack of layers) with 2 fully-connected hidden layers (256 and 128 neurons):
model = keras.Sequential(
[
keras.Input(shape = (32, 32, 3)),
layers.Conv2D(
filters = 32,
kernel_size = (3, 3),
padding = 'same',
activation = 'relu',
# input_shape = (32, 32, 3)
),
layers.MaxPooling2D(pool_size = (2, 2)),
layers.Conv2D(
filters = 64,
kernel_size = (3, 3),
padding = 'same',
activation = 'relu'
),
layers.MaxPooling2D(pool_size = (2, 2)),
layers.Conv2D(
filters = 128,
kernel_size = (3, 3),
padding = 'same',
activation = 'relu'
),
layers.MaxPooling2D(pool_size = (2, 2)),
layers.Conv2D(
filters = 256,
kernel_size = (3, 3),
padding = 'same',
activation = 'relu'
),
layers.MaxPooling2D(pool_size = (2, 2)),
layers.Flatten(),
layers.Dropout(rate = 0.5),
layers.Dense(units = 512, activation = 'relu'),
layers.Dense(units = 100, activation = 'softmax')
]
)
model.summary()Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 32, 32, 32) 896
max_pooling2d (MaxPooling2D (None, 16, 16, 32) 0
)
conv2d_1 (Conv2D) (None, 16, 16, 64) 18496
max_pooling2d_1 (MaxPooling (None, 8, 8, 64) 0
2D)
conv2d_2 (Conv2D) (None, 8, 8, 128) 73856
max_pooling2d_2 (MaxPooling (None, 4, 4, 128) 0
2D)
conv2d_3 (Conv2D) (None, 4, 4, 256) 295168
max_pooling2d_3 (MaxPooling (None, 2, 2, 256) 0
2D)
flatten (Flatten) (None, 1024) 0
dropout (Dropout) (None, 1024) 0
dense (Dense) (None, 512) 524800
dense_1 (Dense) (None, 100) 51300
=================================================================
Total params: 964,516
Trainable params: 964,516
Non-trainable params: 0
_________________________________________________________________Plot the model:
tf.keras.utils.plot_model(
model,
to_file = "model.png",
show_shapes = True,
show_dtype = False,
show_layer_names = True,
rankdir = "TB",
expand_nested = False,
dpi = 96,
layer_range = None,
show_layer_activations = False,
)
model <- keras_model_sequential() %>%
layer_conv_2d(
filters = 32,
kernel_size = c(3, 3),
padding = "same",
activation = "relu",
input_shape = c(32, 32, 3)
) %>%
layer_max_pooling_2d(pool_size = c(2, 2)) %>%
layer_conv_2d(
filters = 64,
kernel_size = c(3, 3),
padding = "same",
activation = "relu"
) %>%
layer_max_pooling_2d(pool_size = c(2, 2)) %>%
layer_conv_2d(
filters = 128,
kernel_size = c(3, 3),
padding = "same",
activation = "relu"
) %>%
layer_max_pooling_2d(pool_size = c(2, 2)) %>%
layer_conv_2d(
filters = 256,
kernel_size = c(3, 3),
padding = "same",
activation = "relu"
) %>%
layer_max_pooling_2d(pool_size = c(2, 2)) %>%
layer_flatten() %>%
layer_dropout(rate = 0.5) %>%
layer_dense(units = 512, activation = "relu") %>%
layer_dense(units = 100, activation = "softmax")
summary(model)Model: "sequential_1"
________________________________________________________________________________
Layer (type) Output Shape Param #
================================================================================
conv2d_7 (Conv2D) (None, 32, 32, 32) 896
max_pooling2d_7 (MaxPooling2D) (None, 16, 16, 32) 0
conv2d_6 (Conv2D) (None, 16, 16, 64) 18496
max_pooling2d_6 (MaxPooling2D) (None, 8, 8, 64) 0
conv2d_5 (Conv2D) (None, 8, 8, 128) 73856
max_pooling2d_5 (MaxPooling2D) (None, 4, 4, 128) 0
conv2d_4 (Conv2D) (None, 4, 4, 256) 295168
max_pooling2d_4 (MaxPooling2D) (None, 2, 2, 256) 0
flatten_1 (Flatten) (None, 1024) 0
dropout_1 (Dropout) (None, 1024) 0
dense_3 (Dense) (None, 512) 524800
dense_2 (Dense) (None, 100) 51300
================================================================================
Total params: 964,516
Trainable params: 964,516
Non-trainable params: 0
________________________________________________________________________________Compile the model with appropriate loss function, optimizer, and metrics:
model.compile(
loss = "categorical_crossentropy",
optimizer = "rmsprop",
metrics = ["accuracy"]
)model %>% compile(
loss = 'categorical_crossentropy',
optimizer = optimizer_rmsprop(),
metrics = c('accuracy')
)80%/20% split for the train/validation set. On my laptop, each epoch takes about half minute.
batch_size = 128
epochs = 30
history = model.fit(
x_train,
y_train,
batch_size = batch_size,
epochs = epochs,
validation_split = 0.2
)Plot training history:
hist = pd.DataFrame(history.history)
hist['epoch'] = np.arange(1, epochs + 1)
hist = hist.melt(
id_vars = ['epoch'],
value_vars = ['loss', 'accuracy', 'val_loss', 'val_accuracy'],
var_name = 'type',
value_name = 'value'
)
hist['split'] = np.where(['val' in s for s in hist['type']], 'validation', 'train')
hist['metric'] = np.where(['loss' in s for s in hist['type']], 'loss', 'accuracy')
# Accuracy trace plot
plt.figure()
sns.relplot(
data = hist[hist['metric'] == 'accuracy'],
kind = 'scatter',
x = 'epoch',
y = 'value',
hue = 'split'
).set(
xlabel = 'Epoch',
ylabel = 'Accuracy'
);
plt.show()
# Loss trace plot
plt.figure()
sns.relplot(
data = hist[hist['metric'] == 'loss'],
kind = 'scatter',
x = 'epoch',
y = 'value',
hue = 'split'
).set(
xlabel = 'Epoch',
ylabel = 'Loss'
);
plt.show()
system.time({
history <- model %>% fit(
x_train, y_train,
epochs = 30, batch_size = 128,
validation_split = 0.2
)
}) user system elapsed
6049.688 3113.319 930.251 plot(history)
Evaluate model performance on the test data:
score = model.evaluate(x_test, y_test, verbose = 0)
print("Test loss:", score[0])Test loss: 2.571018934249878print("Test accuracy:", score[1])Test accuracy: 0.4447999894618988model %>% evaluate(x_test, y_test) loss accuracy
2.532882 0.431200 Generate predictions on new data:
model %>% predict(x_test) %>% k_argmax()tf.Tensor([49 33 55 ... 51 42 92], shape=(10000), dtype=int64)