We’ll train our model for 5 epochs (5 cycles through all our data) using the
metrics getting displayed i.e
valid_loss? We use them to monitor model improvements over time.
Our best model was obtained at the 4th epoch.
fast.ai internally picks and saves only your best models when you run through your training and validation datasets.
Evaluating our Model
The competition submissions were evaluated based on the Area under the Receiver Operating Characteristic curve between the predicted probability and the observed target,
has_oilpalm. Learn more about AUC in this developer crash course, this video, or this Kaggle Learn Forum post. Fast.ai doesn’t come with this metric by default, so we’ll be using the scikit-learn library.
The beauty of using pre-trained models and fast.ai is that you get a really good prediction accuracy,
99.44% in our case without doing the most.
Let’s save our model and plot a confusion matrix about the predictions made.
View Results using confusion matrices
A confusion matrix is a graphical way of seeing the number of images your model accurately or inaccurately predicts.
From this plot, we can see that the model accurately predicts 2,863 images as not having oil palm plantations in them, 168 images with oil palm plantations were correctly classified. 10 images that contained oil palm plantation were classified as not having any and then 7images without oil palm plantations were classified as having them.
Not bad for a simple model ?.
Next, we find an ideal learning rate for this training iteration.
Fit the model using a maximum learning rate that is between
1e-4 for 7 epochs.
Observe the training metrics graphically to monitor the model’s performance after each training cycle.
Save the second stage of model training.
Print out the model’s accuracy, error rate and Area Under the Curve metrics.
As you’ll notice the model’s accuracy improved from
99.48%. The error rate reduced from
0.0052. The AUC also improved,
Upon comparison with the last confusion matrix we plotted, you’ll notice that the model makes better predictions.
Formerly 7 images that didn’t have oil palm plantations were misclassified, now we are down to 3. That’s progress.
You’ll notice we’ve been following a pattern during training and tweaking a few parameters in the course of this. What we’ve been doing is called tuning. Most deep learning experiments follow a similar iterative pattern.