<commit_before>---
layout: default
title: Caffe
---

Extracting Features Using Pre-trained Model
===========================================

CAFFE represents Convolution Architecture For Feature Extraction. Extracting features using pre-trained model is one of the strongest requirements users ask for.

Because of the record-breaking image classification accuracy and the flexible domain adaptability of [the network architecture proposed by Krizhevsky, Sutskever, and Hinton](http://books.nips.cc/papers/files/nips25/NIPS2012_0534.pdf), Caffe provides a pre-trained reference image model to save you from days of training. 

If you need detailed usage help information of the involved tools, please read the source code of them which provide everything you need to know about.

Get the Reference Model
-----------------------

Assume you are in the root directory of Caffe.

    cd models
    ./get_caffe_reference_imagenet_model.sh

After the downloading is finished, you will have models/caffe_reference_imagenet_model.

Preprocess the Data
-------------------

Generate a list of the files to process. 

    build/tools/generate_file_list.py /your/images/dir /your/images.txt

The network definition of the reference model only accepts 256*256 pixel images stored in the leveldb format. First, resize your images if they do not match the required size.

    build/tools/resize_and_crop_images.py --num_clients=8 --image_lib=opencv --output_side_length=256 --input=/your/images.txt --input_folder=/your/images/dir --output_folder=/your/resized/images/dir_256_256

Set the num_clients to be the number of CPU cores on your machine. Run "nproc" or "cat /proc/cpuinfo | grep processor | wc -l" to get the number on Linux.

    build/tools/generate_file_list.py /your/resized/images/dir_256_256 /your/resized/images_256_256.txt
    build/tools/convert_imageset /your/resized/images/dir_256_256 /your/resized/images_256_256.txt /your/resized/images_256_256_leveldb 1

In practice, subtracting the mean image from a dataset significantly improves classification accuracies.

    build/tools/compute_image_mean.bin /your/resized/images_256_256_leveldb /your/resized/images_256_256_mean.binaryproto<commit_msg>Change generate file list python script path in feature extraction doc<commit_after>---
layout: default
title: Caffe
---

Extracting Features Using Pre-trained Model
===========================================

CAFFE represents Convolution Architecture For Feature Extraction. Extracting features using pre-trained model is one of the strongest requirements users ask for.

Because of the record-breaking image classification accuracy and the flexible domain adaptability of [the network architecture proposed by Krizhevsky, Sutskever, and Hinton](http://books.nips.cc/papers/files/nips25/NIPS2012_0534.pdf), Caffe provides a pre-trained reference image model to save you from days of training. 

If you need detailed usage help information of the involved tools, please read the source code of them which provide everything you need to know about.

Get the Reference Model
-----------------------

Assume you are in the root directory of Caffe.

    cd models
    ./get_caffe_reference_imagenet_model.sh

After the downloading is finished, you will have models/caffe_reference_imagenet_model.

Preprocess the Data
-------------------

Generate a list of the files to process. 

    examples/feature_extraction/generate_file_list.py /your/images/dir /your/images.txt

The network definition of the reference model only accepts 256*256 pixel images stored in the leveldb format. First, resize your images if they do not match the required size.

    build/tools/resize_and_crop_images.py --num_clients=8 --image_lib=opencv --output_side_length=256 --input=/your/images.txt --input_folder=/your/images/dir --output_folder=/your/resized/images/dir_256_256

Set the num_clients to be the number of CPU cores on your machine. Run "nproc" or "cat /proc/cpuinfo | grep processor | wc -l" to get the number on Linux.

    build/tools/generate_file_list.py /your/resized/images/dir_256_256 /your/resized/images_256_256.txt
    build/tools/convert_imageset /your/resized/images/dir_256_256 /your/resized/images_256_256.txt /your/resized/images_256_256_leveldb 1

In practice, subtracting the mean image from a dataset significantly improves classification accuracies.

    build/tools/compute_image_mean.bin /your/resized/images_256_256_leveldb /your/resized/images_256_256_mean.binaryproto
{'fraction_non_alphanumeric': 0.09850679741475374, 'fraction_numerical': 0.030755515934923112, 'mean_word_length': 4.2737955346650995, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '1278488', 'n_tokens_mistral': 1351, 'n_tokens_neox': 1207, 'n_words': 449}