Transfer Learning · Computer Vision

Transfer Learning for CIFAR-10: A Feature-Extraction Approach Using EfficientNetB0

Research Journal · Deep Learning Laboratory March 2026 15 min read

Abstract This paper describes a CNN pipeline for 10-class image classification on CIFAR-10 using a two-stage transfer-learning strategy. A pre-trained EfficientNetB0 backbone (ImageNet weights) is frozen and used as a fixed feature extractor, with a Lambda upsampling layer bridging the 32×32 to 224×224 resolution gap. The 1,280-dimensional feature vectors are cached and a lightweight classifier head is trained on them. The pipeline achieves ≥87% validation accuracy while remaining tractable on consumer hardware.

1. Introduction

Image classification is one of the foundational tasks in computer vision. The CIFAR-10 dataset consists of 60,000 colour images (50,000 train / 10,000 test), each 32×32 pixels, distributed evenly across ten categories: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck.

Training a deep CNN from random weights is feasible but slow — typically demanding hundreds of epochs, careful learning-rate scheduling, and aggressive regularisation. An alternative, widely used in industry, is transfer learning: borrowing feature representations learned by a large model on a bigger dataset and adapting them to the target task.

                Project goals: Use a Keras Applications backbone · Achieve ≥87% validation accuracy · Save the compiled model to cifar10.h5 · Expose a preprocess_data(X, Y) function · Guard training code with if __name__ == '__main__'
            

2. Materials and Methods

2.1 Dataset

CIFAR-10 was loaded via tensorflow.keras.datasets.cifar10, already partitioned into 50,000 training and 10,000 test images. No additional augmentation was applied in the baseline run — the large representational capacity of EfficientNetB0 features reduced the need for it during the head-training phase.

2.2 Pre-processing

The preprocess_data function performs two operations: it casts pixel arrays to float32 and applies EfficientNetB0's preprocess_input (scaling to approximately [−1, 1]), then one-hot-encodes labels using to_categorical with num_classes=10.

def preprocess_data(X, Y):
    X = tf.keras.applications.efficientnet.preprocess_input(
            X.astype("float32"))
    Y = tf.keras.utils.to_categorical(Y, num_classes=10)
    return X, Y

2.3 Model Architecture

The architecture is a three-block sequential pipeline. The frozen EfficientNetB0 backbone acts purely as a feature extractor, with a lightweight trainable head attached on top.

Block 1 — Upscaling

Lambda: tf.image.resize → (224, 224)

Output: (m, 224, 224, 3)

↓

Block 2 — Feature Extraction

EfficientNetB0 (frozen, ImageNet weights) + GlobalAveragePooling

Output: (m, 1280)

trainable = False · ~5.3M params

↓

Block 3a — Head Layer 1

Dense(512, ReLU) → BatchNorm → Dropout(0.4)

Output: (m, 512)

↓

Block 3b — Head Layer 2

Dense(256, ReLU) → BatchNorm → Dropout(0.3)

Output: (m, 256)

↓

Block 3c — Output

Dense(10, Softmax)

Output: (m, 10) — class probabilities

2.4 The Resolution Mismatch Problem

EfficientNetB0 was designed for 224×224 inputs; CIFAR-10 images are only 32×32. Feeding 32×32 images directly would pass through fewer spatial pooling operations, degrading feature quality. A Lambda layer bilinearly rescales every batch to 224×224 on the fly using tf.image.resize — adding negligible overhead and requiring no separate pre-processing pass.

2.5 Feature Pre-computation Strategy

Because all EfficientNetB0 layers are frozen, their output is deterministic for a given input. Instead of recomputing features every batch, a single forward pass was run over the entire dataset, caching 1,280-dimensional vectors in memory. The classifier head then trained exclusively on these cached vectors.

Benefits of Caching

Head training reduced from hours to minutes
Hyperparameter iteration becomes practical
No backbone recomputation per epoch

Trade-offs

~500 MB RAM to store two (50,000 × 1280) float32 arrays
Augmentation cannot be applied at this stage
Requires separate partial-model forward pass

2.6 Training Setup

The classifier head was compiled with Adam (lr = 1×10⁻³), categorical cross-entropy loss, and two callbacks:

callbacks = [
    EarlyStopping(monitor="val_accuracy", patience=8,
                  restore_best_weights=True),
    ReduceLROnPlateau(monitor="val_accuracy", patience=4,
                      factor=0.5, min_lr=1e-6)
]
# Max 60 epochs — early stopping typically halts sooner

If end-to-end validation accuracy fell below 87%, a fine-tuning fallback was designed to unfreeze the top 20 EfficientNetB0 layers and continue at lr = 5×10⁻⁵. In practice, this fallback was never triggered.

Features

Latent space · 10 CIFAR-10 classes

3D view unavailable — WebGL could not start. The measured accuracy below is unaffected.

Where the gradient goes

The cloud is illustrative — real EfficientNetB0 embeddings are 1280-dimensional and cannot be drawn, so this is a stand-in that runs from unstructured noise to well-separated clusters. The accuracy is measured. A nearest-centroid linear probe is fitted on 60% of the points and scored on the held-out 40%, recomputed every time you move the slider. Drag Features to the left and the probe collapses to chance: with no class structure in the representation, no linear head can recover it. Drag it right and the same probe becomes near-perfect without a single layer of the backbone changing. That is the whole argument for freezing — if the representation already separates the classes, the only thing left to fit is a head of 12,810 parameters, roughly 0.3% of the model.

3. Results

3.1 Training Dynamics

Metric	Head-only Training	Full Model (end-to-end)
Epochs to convergence	~18	N/A (backbone frozen)
Final training accuracy	~93%	—
Validation accuracy	~88%	~88%
Training time (V100 GPU)	< 3 min	< 1 min (eval only)

3.2 Final Validation Accuracy

The assembled end-to-end model consistently achieved: 87–89% validation accuracy satisfying the ≥87% requirement. The fine-tuning fallback was not triggered in any recorded run, confirming that frozen EfficientNetB0 features alone were sufficient.

3.3 Per-Class Observations

The model performed best on structurally distinctive classes — ships, airplanes, and automobiles — whose consistent shapes and colour statistics align well with low-level ImageNet features. Performance was lowest on visually similar pairs like cat vs. dog and deer vs. horse, consistent with known CIFAR-10 difficulty patterns in the literature.

4. Discussion

4.1 Why EfficientNetB0?

Multiple Keras Applications were evaluated (VGG16, ResNet50, MobileNetV2). EfficientNetB0 was selected for three reasons: compact parameter count (~5.3M), competitive ImageNet accuracy, and principled compound scaling — width, depth, and resolution are balanced together, which matters when the downstream task (32×32 CIFAR-10) differs significantly from pre-training (224×224 ImageNet).

4.2 Limitations

Resolution Upscaling

Bilinear interpolation from 32×32 to 224×224 introduces artificial blurring
Network operates outside its natural domain
Future work: patch-based methods or custom low-resolution backbones

No Data Augmentation

Cached features are fixed — augmentation can't be applied post-extraction
Augmentation before extraction requires multiple forward passes and extra memory
Trade-off deemed acceptable given accuracy achieved

4.3 Reproducibility

Random seeds were fixed (tf.random.set_seed(42), np.random.seed(42)) to maximise reproducibility. However, GPU non-determinism in TensorFlow means exact numerical results may vary slightly across runs and hardware.

4.4 Future Directions

Three avenues could push accuracy higher. First, selective fine-tuning of upper EfficientNetB0 blocks typically adds 1–3 percentage points. Second, replacing the Dense head with a residual MLP or attention-based aggregator may extract more information from the 1,280-d feature vectors. Third, mixed-precision training (float16) would allow larger batch sizes and more stable gradient estimates.

5. Key Hyperparameters

Hyperparameter	Value
Backbone	EfficientNetB0 (ImageNet weights)
Upscaling target	224 × 224 px
Feature dimension	1,280
Head Dense layers	512 → 256 → 10
Dropout rates	0.4, 0.3
Optimiser	Adam
Initial learning rate	1×10⁻³
Batch size (feature extraction)	256
Max epochs	60 (EarlyStopping patience = 8)
Min LR (ReduceLROnPlateau)	1×10⁻⁶
Fine-tune LR (if triggered)	5×10⁻⁵
Fine-tune layers unfrozen	Top 20 of EfficientNetB0 (non-BN)

Conclusion

This project demonstrates that transfer learning with a frozen EfficientNetB0 backbone is a highly effective and computationally efficient approach to CIFAR-10 classification. By pre-computing and caching feature vectors, head training is reduced from hours to minutes while achieving 87–89% validation accuracy.

The resolution mismatch between CIFAR-10's 32×32 images and EfficientNetB0's 224×224 native input is elegantly handled by a Lambda upsampling layer, requiring no offline pre-processing. The overall design prioritises speed and reproducibility without sacrificing accuracy.

Key takeaway: for small datasets with limited compute, feature extraction from a frozen pre-trained backbone combined with a lightweight classifier head is often the best starting point — beating training from scratch on both speed and generalisation.

References

Chollet, F. et al. (2015). Keras. https://keras.io
Krizhevsky, A. (2009). Learning Multiple Layers of Features from Tiny Images. Technical Report, University of Toronto.
Tan, M., & Le, Q. V. (2019). EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. ICML 2019, 97, 6105–6114.
Tan, M., & Le, Q. V. (2021). EfficientNetV2: Smaller Models and Faster Training. ICML 2021.
Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014). How transferable are features in deep neural networks? NeurIPS 2014, 27.
He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep Residual Learning for Image Recognition. CVPR 2016, 770–778.
Abadi, M. et al. (2016). TensorFlow: A System for Large-Scale Machine Learning. OSDI 2016.