Applied

Computing

and

Informatics

Available

online

June

open

access

In

Press

Corrected

Proof

Ensemble

of

convolutional

neural

networks

for

bioimage

classification

Loris

Nanni

a

Stefano

Ghidoni

a

Sheryl

Brahnam

b

Get

rights

and

content

Under

a

Creative

Commons

license

Abstract

This

work

presents

a

system

based

on

an

ensemble

of

Convolutional

Neural

Networks

CNNs

and

descriptors

for

bioimage

classification

that

has

been

validated

on

different

datasets

of

color

images

The

proposed

system

represents

a

very

simple

yet

effective

way

of

boosting

the

performance

of

trained

CNNs

by

composing

multiple

CNNs

into

an

ensemble

and

combining

scores

by

sum

rule

Several

types

of

ensembles

are

considered

with

different

CNN

topologies

along

with

different

learning

parameter

sets

The

proposed

system

not

only

exhibits

strong

discriminative

power

but

also

generalizes

well

over

multiple

datasets

thanks

to

the

combination

of

multiple

descriptors

based

on

different

feature

types

both

learned

and

handcrafted

Separate

classifiers

are

trained

for

each

descriptor

and

the

entire

set

of

classifiers

is

combined

by

sum

rule

Results

show

that

the

proposed

system

obtains

stateoftheart

performance

across

four

different

bioimage

and

medical

datasets

The

MATLAB

code

of

the

descriptors

will

be

available

at

httpsgithubcomLorisNanni

Introduction

Despite

strong

advances

in

automatic

image

analysis

in

recent

years

in

the

field

of

medicine

expert

clinicians

remain

the

ones

who

typically

make

the

final

diagnostic

determination

of

medical

images

Automatic

and

semiautomatic

analysis

is

gaining

in

importance

however

due

to

the

massive

growth

in

medical

imaging

technologies

and

thanks

to

some

giant

strides

in

the

fields

of

image

processing

pattern

recognition

and

image

classification

all

of

which

have

made

automatic

analysis

of

medical

images

a

viable

alternative

In

general

bioimage

processing

often

relies

on

approaches

based

on

feature

extraction

from

images

that

contain

important

information

for

a

particular

diagnostic

task

Some

of

the

best

feature

extraction

methods

for

biological

tissue

analysis

consider

local

textured

patterns

A

large

variety

of

textural

features

have

been

employed

in

biomedical

imaging

classification

systems

with

some

of

these

features

combined

together

in

ensembles

under

the

assumption

that

different

textural

features

extract

different

types

of

information

from

the

same

image

Some

typical

methods

for

extracting

textural

features

include

Gabor

filters

and

Haralicks

cooccurrence

matrix

Other

feature

extraction

methods

commonly

used

today

are

the

ScaleInvariant

Feature

Transform

SIFT

and

Local

Binary

Patterns

LBP

along

with

its

many

variants

These

feature

extraction

methods

belong

to

what

is

often

referred

to

as

the

class

of

handcrafted

descriptors

so

named

because

the

algorithms

are

designed

by

researchers

to

detect

specific

characteristics

considered

important

in

the

analysis

of

images

Besides

handcrafted

features

some

machine

learning

techniques

have

been

developed

that

learn

features

automatically

This

class

of

socalled

learned

features

are

also

widely

used

in

bioimage

processing

but

they

tend

to

be

limited

in

power

because

they

rely

heavily

on

the

dataset

used

for

training

This

problem

can

be

overcome

by

training

on

a

very

large

dataset

or

an

ensemble

of

datasets

containing

a

broad

set

of

images

so

that

the

system

learns

a

wide

variety

of

different

patterns

In

this

way

the

learned

features

become

independent

of

any

specific

dataset

and

can

be

considered

as

general

feature

extractors

Like

the

handcrafted

features

mentioned

above

these

learned

features

can

be

used

alone

or

in

combination

with

other

sets

of

features

both

handcrafted

and

learned

to

analyze

new

problems

Some

examples

along

these

lines

include

where

learned

features

are

used

for

the

detection

of

ovarian

carcinomas

and

where

learned

features

are

combined

with

handcrafted

features

for

histopathology

image

representation

A

powerful

class

of

learned

descriptors

has

recently

been

proposed

that

are

based

on

the

deep

learning

paradigm

Deep

learning

has

proven

to

be

extremely

effective

in

several

image

classification

tasks

including

medical

image

analysis

Some

examples

include

the

detectioncounting

of

mitotic

events

the

segmentation

of

nuclei

and

many

cancerous

vs

noncancerous

tissue

evaluations

A

deep

learning

architecture

that

has

been

studied

extensively

is

the

Convolutional

Neural

Network

CNN

which

is

a

multilayered

image

classification

technique

that

incorporates

spatial

context

and

weight

sharing

between

pixels

A

CNN

learns

the

optimal

image

features

for

a

specific

image

classification

problem

by

adopting

an

effective

representation

of

the

original

image

Inspired

by

the

process

of

visual

perception

in

human

beings

it

requires

little

to

no

preprocessing

The

basic

components

of

a

CNN

are

stacks

of

different

types

of

specialized

layers

convolutional

activation

pooling

fullyconnected

softmax

etc

that

are

interconnected

and

whose

weights

are

trained

using

the

backpropagation

algorithm

The

deepest

layers

of

the

network

function

as

lowlevel

feature

extractors

The

training

phase

of

a

CNN

requires

huge

numbers

of

labelled

data

to

avoid

the

problem

of

overfitting

however

once

trained

CNNs

are

capable

of

producing

accurate

and

generalizable

models

that

achieve

stateoftheart

performance

in

general

pattern

recognition

tasks

Some

examples

include

LeNet

the

first

CNN

proposed

to

classify

handwritten

digits

AlexNet

a

deep

network

designed

for

image

classification

ZFNet

a

newer

model

that

outperforms

AlexNet

VGGNet

which

increases

depth

using

convolution

filters

GoogLeNet

which

includes

inception

modules

which

is

a

new

organizational

structure

and

ResNet

a

residual

network

that

is

much

easier

to

optimize

than

VGGNets

The

CNN

architecture

and

the

cited

examples

are

discussed

in

more

detail

in

Section

When

deep

neural

networks

are

trained

on

large

datasets

of

images

the

first

convolutional

filters

learned

by

the

network

often

resemble

either

Gabor

filters

or

color

blobs

that

are

easily

transferable

to

many

other

image

tasks

and

datasets

Pretrained

models

can

thus

be

used

to

extract

learned

features

from

novel

sets

of

images

and

these

features

can

then

be

fed

into

other

classifiers

similar

to

the

way

handcrafted

features

are

used

Conversely

features

computed

in

the

last

layer

of

a

pretrained

network

are

strongly

dependent

on

the

dataset

used

to

train

the

deep

learner

and

thus

on

the

specific

classification

problem

represented

by

a

given

dataset

Nonetheless

the

outputs

of

these

layers

can

be

used

for

other

tasks

if

CNN

finetuning

is

exploited

All

three

deep

learning

methods

described

above

are

used

in

medical

and

bioimage

classification

To

summarize

the

possibilities

mentioned

so

far

a

deep

learners

can

be

trained

on

images

from

scratch

as

in

b

pretrained

CNNs

can

function

as

additional

feature

extractors

that

can

be

combined

with

existing

handcrafted

image

features

as

in

and

c

the

outputs

of

pretrained

CNNs

can

be

finetuned

by

another

simpler

classifier

such

as

SVM

on

novel

target

images

as

in

Yet

another

class

of

approaches

combines

different

CNN

architectures

to

exploit

the

strengths

and

offset

the

weaknesses

of

a

given

architecture

In

this

work

we

investigate

methods

for

building

ensembles

of

CNNs

by

leveraging

pretrained

CNNs

We

consider

several

different

training

patterns

and

experiments

using

different

learning

rates

batch

sizes

and

topologies

What

is

interesting

is

that

this

simple

approach

produces

a

very

high

performing

system

one

that

strongly

outperforms

the

single

best

CNN

trained

specifically

on

a

given

dataset

Of

course

there

are

both

pros

and

cons

involved

in

combining

different

CNNs

Although

ensembles

of

CNNs

perform

exceptionally

well

training

such

models

requires

high

computational

power

in

this

work

we

used

three

TitanX

GPUs

Moreover

the

total

size

of

the

network

set

is

quite

large

requiring

considerable

computational

power

for

input

classification

Hence

this

approach

is

suitable

only

for

problems

where

computation

time

is

not

critical

Aside

from

exploring

different

ensembles

of

CNNs

we

also

consider

combining

heterogeneous

handcrafted

descriptors

for

bioimage

classification

The

best

system

proposed

in

this

work

combines

both

learned

and

handcrafted

descriptors

For

each

descriptor

a

different

classifier

is

trained

and

the

set

of

classifiers

along

with

the

classification

results

from

the

deep

learners

are

combined

by

sum

rule

The

handcrafted

descriptors

tested

in

this

paper

are

summarized

in

Section

and

the

power

of

this

approach

is

validated

on

four

different

biomedical

color

datasets

We

wish

to

stress

that

the

main

goal

of

the

proposed

system

is

to

produce

a

powerful

generalpurpose

image

classification

system

able

to

work

outofthebox

ie

requiring

little

to

no

parameter

tuning

on

any

bioimage

classification

problem

We

strive

to

produce

a

generalpurpose

system

that

performs

competitively

against

less

flexible

systems

that

have

been

optimized

for

very

specific

image

problems

and

datasets

Experimental

results

demonstrate

that

the

proposed

system

obtains

stateoftheart

performance

in

every

tested

bioimage

problem

Yet

the

same

set

of

descriptors

is

used

in

all

the

tested

datasets

demonstrating

the

generalizability

of

the

proposed

approach

Deep

learned

features

CNNs

are

a

class

of

deep

feedforward

neural

networks

Like

most

neural

networks

CNNs

are

composed

of

interconnected

neurons

that

have

inputs

with

learnable

weights

biases

and

activation

functions

CNN

layers

have

neurons

arranged

in

three

dimensions

width

height

and

depth

This

means

that

every

layer

in

a

CNN

transforms

a

input

volume

into

a

output

volume

of

neuron

activations

CNNs

are

built

with

five

classes

of

layers

conv

olutional

CONV

act

ivation

ACT

pool

ing

POOL

followed

by

a

last

stage

including

FullyConnected

FC

and

class

ification

CLASS

The

CONV

layer

is

the

core

building

block

of

a

CNN

and

is

also

what

makes

CNNs

so

computationally

expensive

These

layers

compute

the

outputs

of

neurons

that

are

connected

to

local

regions

by

applying

a

convolution

operation

to

the

input

The

spatial

extent

of

connectivity

of

these

local

regions

is

a

hyperparameter

called

the

receptive

field

and

a

parameter

sharing

scheme

is

used

in

CONV

Layers

to

control

the

number

of

parameters

This

means

that

the

parameters

of

CONV

layers

are

shared

sets

of

weights

also

called

kernels

or

filters

that

have

relatively

small

receptive

fields

POOL

layers

perform

nonlinear

downsampling

operations

Max

pooling

is

the

most

common

nonlinear

operation

it

partitions

the

input

into

a

set

of

nonoverlapping

rectangles

and

outputs

the

maximum

for

each

group

In

this

way

POOL

reduces

the

spatial

size

of

the

representation

while

simultaneously

reducing

the

number

of

parameters

the

possibility

of

overfitting

and

the

computational

complexity

of

the

network

It

is

common

practice

to

insert

a

POOL

layer

between

CONV

layers

ACT

layers

apply

some

activation

function

such

as

the

nonsaturating

ReLU

Rectified

Linear

Unit

function

or

the

saturating

hyperbolic

tangent

or

the

sigmoid

function

FC

layers

have

neurons

that

are

fully

connected

to

all

the

activations

in

the

previous

layer

and

are

applied

after

CONV

and

POOL

layers

In

this

work

we

test

and

combine

the

following

CNN

architectures

AlexNet

this

is

the

winner

of

the

ImageNet

ILSVRC

challenge

AlexNet

is

a

popular

CNN

that

is

composed

of

both

stacked

and

connected

layers

It

includes

five

CONV

layers

followed

by

three

FC

layers

with

some

maxPOOL

layers

inserted

in

the

middle

A

rectified

linear

unit

nonlinearity

is

applied

to

each

convolutional

along

with

a

fully

connected

layer

to

enable

faster

training

GoogleNet

this

is

the

winner

of

the

ImageNet

ILSVRC

challenge

The

main

novelty

of

this

CNN

is

the

introduction

of

an

inception

module

INC

ie

a

subnetwork

consisting

of

parallel

convolutional

filters

whose

outputs

are

concatenated

INC

greatly

reduces

the

number

of

parameters

required

much

lower

than

AlexNet

GoogleNet

is

composed

of

layers

that

require

training

layers

in

total

counting

the

POOL

layers

VGGNet

this

is

a

CNN

that

placed

second

in

ILSVRC

The

two

bestperforming

VGG

models

and

with

and

wt

layers

respectively

are

available

as

pretrained

models

Both

models

are

very

deep

and

include

CONVFC

layers

The

CONV

layers

are

extremely

homogeneous

and

use

very

small

convolution

filters

A

POOL

layer

is

inserted

after

two

or

three

CONV

layers

instead

after

each

CONV

layer

as

is

the

case

with

AlexNet

ResNet

this

is

the

winner

of

ILSVRC

This

network

is

approximately

twenty

times

deeper

than

AlexNet

and

eight

times

deeper

than

VGGNet

The

main

novelty

of

this

CNN

is

the

introduction

of

residual

RES

layers

making

it

a

networkinnetwork

architecture

ResNet

uses

special

skip

connections

and

batch

normalization

and

the

FC

layers

at

the

end

of

the

network

are

substituted

by

global

average

pooling

Instead

of

learning

unreferenced

functions

ResNet

explicitly

reformulates

layers

as

learning

residual

functions

with

reference

to

the

layer

inputs

As

a

result

ResNet

is

much

deeper

than

VGGNet

although

the

model

size

is

smaller

and

thus

Inception

is

a

variant

of

GoogleNet

based

on

the

factorization

of

convolutions

into

two

or

three

consecutive

layers

of

convolutions

is

an

Inception

style

networks

that

utilize

residual

connections

instead

of

filter

concatenation

As

noted

in

the

introduction

the

learning

effectiveness

of

a

CNN

depends

on

the

availability

of

large

training

data

Data

augmentation

is

one

effective

way

to

expand

training

data

when

necessary

and

to

reduce

overfitting

during

CNN

training

by

artificially

expanding

the

training

set

using

perturbations

of

individual

images

Data

augmentation

applies

transformations

and

deformations

to

the

labeled

data

thus

producing

new

samples

as

additional

training

data

A

key

attribute

of

the

data

augmentation

process

is

that

the

labels

remain

unchanged

after

applying

the

transformations

In

this

work

we

perform

random

data

augmentation

with

horizontal

and

vertical

flipping

rotation

in

a

range

of

translation

of

a

maximum

of

five

pixels

and

scaling

in

a

range

of

Finetuning

a

CNN

is

a

procedure

that

essentially

restarts

the

retraining

process

of

a

pretrained

network

so

that

it

learns

a

different

classification

problem

We

adopt

the

TwoRound

Tuning

for

finetuning

a

CNN

With

TwoRound

Tuning

the

first

round

of

tuning

is

performed

by

training

a

CNN

using

a

leaveoneout

dataset

strategy

eg

by

including

in

the

training

set

all

the

images

from

the

dataset

summarized

in

Table

except

for

the

target

dataset

The

final

number

of

classes

becomes

the

sum

of

all

the

classes

from

each

classification

problem

The

second

round

of

tuning

is

the

same

as

in

One

Round

Tuning

and

involves

only

the

training

set

of

the

target

problem

In

keeping

with

the

rationale

of

the

Data

Augmentation

step

we

use

the

following

datasets

in

the

first

round

of

tuning

PAP

the

PAP

SMEAR

dataset

which

contains

images

acquired

during

Pap

tests

to

identify

cervical

cancer

diagnosis

available

at

httplabsfmeaegeangrdecisiondownloads

LG

the

Liver

gender

dataset

which

includes

images

of

liver

tissue

sections

from

male

and

female

mice

on

a

caloric

restriction

diet

the

classes

are

the

genders

LA

the

Liver

aging

dataset

which

includes

images

of

liver

tissue

sections

from

female

mice

of

ages

on

an

adlibitum

diet

BR

the

BREAST

CANCER

dataset

which

contains

images

divided

into

the

control

malignant

cancer

and

benign

cancer

classes

HI

the

HISTOPATHOLOGY

dataset

which

contains

images

of

connective

epithelial

muscular

and

nervous

tissue

classes

RPE

a

dataset

composed

of

human

stem

cellderived

retinal

pigmented

epithelium

images

that

were

divided

into

subwindows

with

each

subwindow

divided

into

four

classes

by

two

trained

operators

available

at

We

finetune

the

weights

of

the

pretrained

CNNs

by

fixing

the

deep

CONV

layers

of

the

network

and

by

finetuning

only

the

higherlevel

FC

layers

since

these

layers

are

specific

to

the

details

of

the

classes

contained

in

the

target

dataset

The

last

FC

layer

is

designed

to

be

the

same

size

as

the

number

of

classes

in

the

new

dataset

All

the

FC

layers

are

initialized

with

random

values

and

trained

from

scratch

using

the

Stochastic

Gradient

Descent

SGD

algorithm

with

data

from

the

target

training

set

Handcrafted

features

In

Table

we

summarize

the

handcrafted

descriptors

used

in

our

tests

along

with

the

parameter

sets

used

to

extract

each

descriptor

Each

descriptor

is

trained

on

an

SVM

and

only

the

training

data

is

used

to

fix

its

parameters

Since

we

are

working

with

RGB

color

datasets

each

texture

descriptor

is

applied

separately

to

each

RGB

channel

with

the

final

score

given

by

the

sum

rule

of

the

three

classifiers

trained

with

the

three

set

of

features

Table

Summary

Handcrafted

Descriptors

Name

Parameters

Source

Section

LTP

Multiscale

Uniform

LTP

with

two

RP

configurations

and

threshold

MLPQ

Ensemble

of

LPQ

descriptors

obtained

by

varying

the

filter

sizes

the

scalar

frequency

and

the

correlation

coefficient

between

adjacent

pixel

values

CLBP

Completed

LBP

with

two

RP

configurations

and

RIC

Multiscale

Rotation

Invariant

Cooccurrence

of

Adjacent

LBP

with

R

FBSIF

Extension

of

the

BIF

by

varying

the

parameters

of

filter

size

size

and

the

threshold

for

binarizing

th

AHP

Adaptive

Hybrid

Pattern

with

quantization

level

and

the

RP

configurations

are

and

GOLD

Ensemble

of

Gaussians

of

LOcal

Descriptors

extracted

using

the

spatial

pyramid

decomposition

HOG

Histogram

of

Oriented

Gradients

with

cells

by

MOR

A

set

of

MORphological

features

CLM

CodebookLess

Model

We

use

the

ensemble

named

in

based

on

eSFT

PCA

for

dimensionality

reduction

and

onevsall

SVM

for

the

training

phase

LET

Same

parameters

used

in

the

source

code

of

Table

Descriptive

Summary

of

the

Datasets

Dataset

C

S

Size

URL

for

Download

BGR

LY

LAR

CO

As

it

can

be

observed

in

Table

many

of

the

handcrafted

texture

descriptors

are

based

on

Local

Binary

Patterns

LBP

a

descriptor

that

has

achieved

great

success

due

to

its

computational

efficiency

and

discriminative

power

The

traditional

LBP

is

expressed

as

where

is

the

difference

between

the

intensity

levels

of

a

central

pixel

q

c

and

a

set

of

neighbouring

pixels

q

p

A

neighbourhood

is

defined

by

a

circular

region

of

radius

R

and

P

neighbouring

points

The

function

s

x

in

Eq

is

defined

as

x

otherwise

LBP

descriptors

are

the

histograms

of

these

binary

numbers

The

Local

Ternary

Pattern

LTP

LTP

is

a

ternary

variant

of

LBP

and

is

designed

to

reduce

the

noise

in

the

feature

vector

when

uniform

regions

are

analyzed

LTP

proposes

a

threevalue

coding

scheme

that

includes

a

threshold

around

zero

for

the

evaluation

of

the

local

grayscale

difference

by

adding

to

Eq

the

threshold

τ

x

τ

x

τ

x

τ

Multithreshold

Local

Phase

Quantization

MLPQ

MLPQ

extends

the

multithreshold

approach

described

for

LBP

to

the

LPQ

feature

that

is

based

on

the

phase

of

the

ShortTerm

Fourier

Transform

STFT

evaluated

on

a

rectangular

neighborhood

of

size

R

The

MLPQ

features

used

in

our

experiments

are

computed

using

parameter

belonging

to

the

following

ranges

τ

R

a

and

ρ

Such

sets

were

proposed

in

Completed

LBP

CLBP

CLBP

proposed

in

encodes

a

texture

by

means

of

two

components

the

difference

sign

and

the

different

magnitude

that

is

computed

between

a

reference

pixel

and

all

the

pixels

belonging

to

a

given

neighborhood

CLBP

represents

a

local

region

by

its

centre

pixel

CLBPC

and

a

local

difference

signmagnitude

transform

LDSMT

This

is

what

produces

the

difference

signs

and

the

difference

magnitudes

Two

operators

CLBPSign

and

CLBPMagnitude

are

defined

for

the

difference

signs

and

the

difference

magnitudes

Since

all

three

descriptors

and

are

in

binary

format

they

can

be

combined

to

form

the

final

CLBP

histogram

Given

a

central

pixel

and

its

P

evenly

spaced

circular

neighbors

the

difference

between

and

can

be

calculated

as

and

be

decomposed

into

two

components

defining

the

LDSMT

transform

and

sign

where

is

the

sign

of

and

is

the

magnitude

of

Thus

the

LDSMT

transforms

the

vector

into

a

sign

vector

and

a

magnitude

vector

The

operator

is

the

traditional

LBP

operator

defined

in

Eq

The

is

defined

as

x

c

x

c

where

is

the

mean

value

of

The

center

pixels

represent

the

image

gray

level

and

thus

contains

discriminant

information

These

values

are

converted

into

a

binary

code

by

global

thresholding

which

makes

them

consistent

with

and

as

where

is

the

threshold

defined

in

Eq

and

is

the

average

gray

level

of

the

white

image

Combining

and

features

into

joint

or

hybrid

distributions

results

in

significant

improvement

for

rotation

invariant

texture

classification

The

and

histograms

are

concatenated

to

obtain

the

CLBP

descriptor

Multiscale

Rotation

Invariant

Cooccurrence

of

Adjacent

LBP

RIC

RIC

considers

the

cooccurrence

in

the

context

of

LBP

features

or

the

spatial

relations

among

pixels

This

feature

adds

rotational

invariance

for

angles

that

are

multiples

of

RIC

depends

on

two

parameters

namely

LBP

radius

and

the

displacement

among

the

LBPs

The

values

used

in

our

experiments

are

and

Full

BSIF

FBSIF

FBSIF

is

an

extension

of

the

Binarized

Statistical

Image

Feature

BSIF

that

assigns

each

pixel

of

the

input

image

a

nbit

label

obtained

by

means

of

a

set

of

n

linear

filters

Each

filter

operates

on

a

neighborhood

of

pixels

around

the

element

they

should

give

the

label

This

nbit

label

can

be

formalized

as

where

X

is

a

vector

of

length

l

obtained

from

the

neighborhood

while

W

is

a

n

l

matrix

including

the

filters

vector

notations

FBSIF

operates

by

evaluating

BSIF

using

several

values

of

the

filter

size

and

a

binarization

threshold

Values

considered

in

this

work

are

Each

combination

of

size

and

threshold

is

fed

to

a

separate

SVM

the

SVMs

are

then

combined

by

sum

rule

Adaptive

Hybrid

Pattern

AHP

AHP

descriptors

were

created

to

overcome

two

main

drawbacks

of

the

LBP

feature

its

noisy

behavior

in

quasiuniform

regions

and

its

reactivity

that

is

the

strong

variations

in

the

descriptor

that

are

possibly

induced

by

small

variation

in

the

input

image

which

is

caused

by

the

use

of

quantization

thresholds

AHP

overcomes

both

problems

by

using

a

Hybrid

Texture

Model

HTD

composed

of

local

primitive

features

and

global

spatial

structure

and

then

by

applying

an

adaptive

quantization

algorithm

AQA

to

improve

the

noise

robustness

of

the

angular

space

quantization

In

this

way

the

vector

quantization

thresholds

are

adaptive

to

the

content

of

the

local

patchAQA

extracts

the

discriminative

texture

information

provided

by

primitive

microfeatures

is

defined

as

where

represents

the

texture

the

global

texture

information

and

the

local

texture

information

is

the

joint

distribution

of

the

global

difference

between

gray

values

of

the

circular

symmetric

neighborhoods

and

the

mean

value

from

the

whole

texture

image

is

the

joint

distribution

of

the

local

differences

between

the

gray

value

of

the

center

pixel

and

the

gray

values

of

the

circularly

symmetric

neighborhoods

The

length

of

the

feature

histogram

of

the

whole

image

is

reduced

by

splitting

the

global

pattern

and

the

local

pattern

into

multiple

binary

patterns

using

the

threshold

calculations

in

Gaussian

of

Local

Descriptors

GOLD

GOLD

is

based

on

a

fourstep

algorithm

i

evaluation

of

SIFT

features

ii

spatial

pyramid

decomposition

iii

parametric

probability

density

estimation

iv

the

covariance

matrix

is

projected

onto

the

tangent

Euclidean

space

in

order

to

vectorize

the

feature

In

other

words

GOLD

descriptors

are

obtained

by

extracting

some

descriptors

from

an

image

to

obtain

where

D

i

by

collecting

and

weighting

them

in

a

spatial

pyramid

and

then

by

describing

each

subregion

by

the

estimated

parameters

of

a

multivariate

Gaussian

distribution

To

vectorize

the

descriptors

the

covariance

matrix

is

projected

onto

a

Euclidean

space

and

concatenated

to

the

mean

vector

to

obtain

the

final

descriptor

of

size

n

n

Finally

the

feature

vector

is

fed

into

an

SVM

with

a

histogram

kernel

Histogram

of

Oriented

Gradients

HOG

HOG

groups

pixels

into

small

windows

and

measures

intensity

gradients

in

each

of

them

It

is

possible

to

view

HOG

as

a

simplified

version

of

SIFT

HOG

calculates

intensity

gradients

pixel

by

pixel

and

the

selection

of

a

corresponding

histogram

bin

for

each

pixel

is

based

on

the

gradient

direction

A

histogram

is

then

evaluated

for

each

window

leading

to

the

final

descriptor

Windows

of

size

are

used

in

our

experiments

Color

descriptor

COL

COL

proposed

in

is

a

simple

and

compact

descriptor

acquired

combining

statistical

measures

extracted

from

each

color

channel

in

the

RGB

space

The

final

descriptor

is

obtained

as

the

concatenation

of

several

measures

the

mean

the

standard

deviation

the

and

moments

of

each

color

channel

and

the

marginal

Histograms

bins

per

channel

Morphological

descriptor

MOR

MOR

proposed

in

is

a

set

of

measures

extracted

from

a

segmented

version

of

the

image

including

the

aspect

ratio

number

of

objects

area

perimeter

eccentricity

and

other

measures

CodebookLess

Model

CLM

CLM

is

based

on

an

image

modeling

method

that

can

represent

an

image

by

means

of

a

single

Gaussian

This

is

obtained

by

first

evaluating

SIFT

features

on

a

regular

grid

placed

on

the

image

Thus

CLM

is

a

dense

sampling

features

model

and

fitting

them

using

a

Gaussian

model

The

main

difference

between

CLM

and

the

other

widely

used

dense

sampling

method

such

as

the

BoF

approach

is

the

absence

of

a

codebook

According

to

the

experiments

reported

in

we

select

for

CLM

the

ensemble

named

in

based

on

eSFT

PCA

for

dimensionality

reduction

and

onevsone

SVM

for

the

training

phase

LETRIST

descriptor

LET

LET

proposed

in

is

simple

but

effective

representation

that

encodes

the

joint

information

within

an

image

across

feature

and

scale

spaces

We

use

the

default

values

available

in

the

MATLAB

toolbox

Materials

Several

medical

datasets

were

used

to

test

our

system

and

demonstrate

the

generalizability

of

our

approach

Each

dataset

contains

different

types

of

medical

images

For

the

sake

of

easy

comparisons

the

datasets

used

in

our

experiments

were

selected

because

they

are

publicly

available

LY

the

LYMPHOMA

dataset

which

includes

images

of

malignant

lymphoma

subdivided

in

three

classes

CLL

chronic

lymphocytic

leukemia

FL

follicular

lymphoma

and

MCL

mantle

cell

lymphoma

BGR

the

BREAST

GRADING

CARCINOMA

which

is

a

medium

size

dataset

containing

images

Grade

Grade

and

Grade

images

of

resolution

corresponding

to

different

patients

with

invasive

ductal

carcinoma

of

the

breast

LAR

the

LARYNGEAL

dataset

which

contains

a

wellbalanced

set

of

patches

extracted

from

the

endoscopic

videos

of

patients

affected

by

laryngeal

squamous

cell

carcinoma

SCC

The

patches

are

relative

to

four

laryngeal

tissue

classes

LAR

contains

color

images

In

our

experiments

with

this

dataset

each

descriptor

is

separately

extracted

from

each

color

channel

CO

the

COLORECTAL

dataset

which

is

a

collection

of

textures

obtained

by

manual

annotation

and

tessellation

of

histological

images

of

human

colorectal

cancer

Table

summarizes

some

important

characteristics

of

each

dataset

including

the

number

of

classes

C

the

number

of

samples

S

ie

the

number

of

images

the

image

size

and

the

URL

for

downloading

the

dataset

The

testing

protocol

used

in

our

experiments

is

the

fivefold

crossvalidation

method

except

in

those

case

where

the

database

is

specifics

its

own

protocol

Experimental

results

The

experimental

evaluations

reported

in

this

section

are

intended

first

to

compare

the

performance

of

handcrafted

descriptors

to

deep

learned

descriptors

on

several

cancer

data

analysis

classification

tasks

and

second

to

evaluate

the

performance

of

several

ensembles

based

on

the

fusion

of

classifiers

Our

main

objective

is

to

design

a

method

that

is

both

robust

and

effective

on

different

classification

problems

To

assess

the

generalizability

and

robustness

of

our

system

our

best

performing

method

is

finally

compared

with

several

stateoftheart

results

published

by

different

researches

on

the

same

datasets

Note

Before

each

fusion

the

scores

of

the

classifiers

of

each

descriptor

are

normalized

to

mean

and

standard

deviation

Experiments

reported

below

were

statistically

validated

using

the

Wilcoxon

signed

rank

test

In

the

first

experiment

reported

in

Table

we

evaluate

the

performance

using

accuracy

as

the

performance

indicator

of

the

baseline

handcrafted

descriptors

described

in

Section

Moreover

the

performance

obtained

by

the

following

ensembles

of

handcrafted

methods

are

compared

FH

the

fusion

by

sum

rule

of

the

following

handcrafted

methods

LTP

CLBP

RIC

LET

MOR

AHP

COL

MLPQ

and

FullBSIF

We

have

not

use

GOLD

and

CLM

in

FH

since

they

are

computational

expensive

Note

the

scores

of

each

method

are

normalized

to

mean

zero

and

standard

deviation

so

that

the

importance

of

MLPQ

and

FullBSIF

that

are

methods

based

on

ensemble

combined

by

sum

rule

is

equal

to

the

other

approaches

FH

CLM

sum

rule

among

the

methods

belonging

to

FH

and

CLM

ie

the

sum

rule

among

the

nine

methods

of

FH

and

CLM

FH

CLMGOLD

sum

rule

among

the

methods

belonging

to

FH

GOLD

and

CLM

PREV

the

ensemble

of

handcrafted

features

proposed

in

ensemble

of

handcrafted

features

proposed

in

Table

Handcrafted

descriptors

LY

CO

BGR

LAR

LTP

MLPQ

CLBP

RIC

LET

MOR

AHP

COL

FBSIF

GOLD

CLM

FH

FHCLM

FHCLMGOLD

PREV

Bold

values

are

highest

performance

in

the

columns

Clearly

the

fusion

approaches

FH

and

FH

CLMGOLD

works

better

Wilcoxon

signed

rank

test

pvalue

of

than

the

standalone

methods

and

the

previous

handcrafted

ensembles

PREV

and

In

the

second

experiment

see

Table

Table

we

test

the

feasibility

of

building

an

ensemble

of

convolutional

neural

networks

as

follows

Training

CNNs

using

different

learning

rates

LR

ie

Training

CNNs

using

different

batch

sizes

BS

ie

and

Training

CNNs

using

different

topologies

Table

Standard

tuning

GoogleNet

Inception

SB

AB

Fus

SB

AB

Fus

SB

AB

Fus

SB

AB

Fus

LY

CO

BGR

LAR

AlexNet

FCNst

SB

AB

Fus

SB

AB

Fus

SB

AB

Fus

SB

AB

Fus

All

NoAlex

LY

CO

BGR

LAR

Bold

values

are

highest

performance

in

the

columns

Table

Tworound

tuning

AlexNet

GoogleNet

FCNTwo

SB

AB

Fus

SB

AB

Fus

SB

AB

Fus

SB

AB

Fus

All

NoAlex

LY

CO

BGR

LAR

Bold

values

are

highest

performance

in

the

columns

In

Table

we

report

experiments

using

standard

tuning

In

Table

we

report

the

performance

of

the

TwoRound

tuning

detailed

in

Section

The

following

methods

are

also

reported

in

Table

Table

SB

the

single

best

CNN

configuration

in

that

dataset

This

method

is

clearly

overfitted

since

we

report

the

best

result

on

the

testing

set

after

running

different

parameter

configurations

and

choosing

the

best

one

It

is

important

to

report

SB

as

baseline

performance

for

the

proposed

ensemble

AB

best

average

CNN

configuration

in

all

the

datasets

Fus

fusion

among

all

the

different

CNNs

trained

varying

the

parameter

configuration

If

the

CNN

does

not

converge

ie

it

produces

random

results

on

the

training

data

which

usually

happens

with

AlexNet

and

VggNet

with

LR

the

CNN

is

excluded

from

the

ensemble

Note

that

it

is

not

always

feasible

to

train

a

CNN

with

a

large

batch

size

In

other

words

if

for

a

given

BS

we

obtain

a

GPU

out

of

memory

error

message

we

discard

that

CNN

configuration

FCNst

fusion

among

the

methods

Fus

of

the

all

CNN

topologies

trained

using

standard

tuning

column

All

or

all

the

topologies

trained

using

standard

tuning

except

AlexNet

column

NoAlex

The

scores

of

each

given

CNN

topology

are

normalized

considering

how

many

CNNs

of

that

topology

are

effectively

used

in

the

fusion

Fus

ie

by

excluding

all

CNNs

that

produce

random

results

on

the

training

set

or

an

out

of

memory

error

message

FCNTwo

fusion

among

the

methods

Fus

of

the

all

CNN

topologies

trained

using

Tworound

tuning

column

All

or

all

the

topologies

trained

using

Tworound

tuning

except

AlexNet

column

NoAlex

The

scores

of

each

given

CNN

topology

are

normalized

considering

how

many

CNNs

of

that

topology

are

effectively

used

in

the

fusion

Fus

ie

by

excluding

all

CNNs

that

produce

random

results

on

the

training

set

or

an

out

of

memory

error

message

Notice

that

TwoRound

Tuning

is

applied

on

a

reduced

number

of

topologies

due

to

computational

issues

The

following

conclusions

can

be

drawn

from

the

results

reported

in

Table

Table

For

each

topology

Fus

outperforms

AB

Wilcoxon

signed

rank

test

pvalue

of

FCN

outperforms

each

Fus

Wilcoxon

signed

rank

test

pvalue

of

FCNtwo

obtains

performance

similar

to

FCNst

In

Table

the

ensemble

of

CNNs

is

combined

with

other

methods

The

ensembles

evaluated

in

Table

are

the

following

FCN

sum

rule

among

the

methods

that

belong

to

FCNst

and

FCNTwo

sum

rule

between

FCN

NoAlex

and

FH

before

fusion

the

scores

of

FCN

NoAlex

and

FH

are

normalized

to

mean

and

standard

deviation

sum

rule

between

FCN

NoAlex

and

FH

CLMGOLD

before

fusion

the

scores

of

FCN

NoAlex

and

FH

are

normalized

to

mean

and

standard

deviation

Table

Ensemble

proposed

here

FCN

All

NoAlex

LY

CO

BGR

LAR

Bold

values

are

highest

performance

in

the

columns

In

Table

we

compare

our

ensemble

with

the

literature

for

a

fair

comparison

we

have

reported

methods

based

on

the

same

testing

protocol

used

to

assess

the

performance

of

our

approaches

Table

Comparison

with

other

stateoftheart

approaches

accuracy

is

used

as

the

performance

indicator

Methods

LY

CO

BGR

LAR

Bold

values

are

highest

performance

in

the

columns

The

following

conclusions

can

be

drawn

from

the

results

reported

in

Table

Table

and

outperform

FCN

since

is

simpler

then

our

suggestion

is

to

use

obtains

stateoftheartperformance

eg

in

a

median

Fmeasure

of

is

obtained

in

the

LAR

dataset

while

our

ensemble

obtains

an

Fmeasure

of

Finally

in

Table

we

report

the

performance

obtained

by

some

ensemble

proposed

in

this

paper

using

the

Kappa

statistic

to

measure

the

agreement

between

true

and

predicted

class

labels

Table

Ensemble

tested

here

kstatistic

as

performance

indicator

FH

FCN

All

NoAlex

LY

CO

BGR

LAR

Bold

values

are

highest

performance

in

the

columns

The

conclusions

that

can

be

drawn

by

the

results

reported

in

Table

are

similar

to

those

that

can

be

drawn

by

the

performance

reported

in

Table

To

better

motivate

the

reason

of

the

good

performance

of

the

ensemble

of

CNNs

we

calculate

the

Yules

Qstatistic

among

the

methods

that

build

the

ensemble

The

Qstatistic

is

used

to

provide

information

about

the

correlation

among

the

output

of

different

classifiers

The

average

Qstatistic

among

the

different

CNNs

that

build

FCNst

is

hence

the

different

CNNs

brings

different

information

and

their

combination

permits

to

boost

the

performance

of

the

standalone

CNN

Conclusion

In

this

work

an

ensemble

of

CNNs

is

proposed

for

cancer

related

color

datasets

The

ensemble

is

built

in

a

very

simple

way

by

training

and

comparing

the

performance

of

CNNs

using

different

learning

rates

batch

sizes

and

topologies

The

set

of

CNNs

is

simply

combined

with

the

sum

rule

The

most

important

finding

of

this

work

is

that

this

simple

ensemble

outperforms

the

best

standalone

CNN

When

the

ensemble

of

CNNs

is

combined

with

other

features

based

on

handcrafted

features

the

final

ensemble

obtains

stateoftheart

performance

on

all

the

four

tested

datasets

For

each

handcrafted

features

a

different

support

vector

machine

is

trained

than

the

set

of

SVMs

is

combined

by

sum

rule

also

the

fusion

between

deep

learning

ensemble

and

handcrafted

features

ensemble

is

performed

by

sum

rule

Notice

that

before

the

fusion

the

set

of

scores

of

each

ensemble

is

normalized

to

mean

and

standard

deviation

In

the

future

we

plan

to

develop

and

test

different

approaches

for

representing

images

using

CNNs

Features

extracted

from

these

CNNs

will

then

be

used

to

train

SVM

classifiers

To

reproduce

our

experiments

MATLAB

source

code

will

be

available

at

httpsgithubcomLorisNanni

Acknowledgment

We

gratefully

acknowledge

the

support

of

NVIDIA

Corporation

for

the

NVIDIA

Hardware

Donation

Grant

of

a

Titan

X

used

in

this

research

References

J

Zhou

et

al

BIOCAT

a

pattern

recognition

platform

for

customizable

biological

image

classification

and

annotation

BMC

Bioinf

p

Google

Scholar

B

Misselwitz

et

alEnhanced

CellClassifier

a

multiclass

classification

tool

for

microscopy

images

BMC

Bioinform

Google

Scholar

G

Pau

et

al

EBImage

an

R

package

for

image

processing

with

applications

to

cellular

phenotypes

Bioinformatics

Google

Scholar

V

Uhlmann

S

Singh

AE

Carpenter

CPCHARM

segmentationfree

image

classification

made

accessible

BMC

Bioinf

p

Google

Scholar

A

Vailaya

et

al

Image

classification

for

contentbased

indexing

IEEE

Trans

Image

Process

pp

Google

Scholar

RC

Gonzalez

RE

Woods

Digital

Image

Processing

second

ed

AddisonWesley

Longman

Publishing

Co

Inc

Boston

Google

Scholar

L

Nanni

A

Lumini

S

Brahnam

Survey

on

LBP

based

texture

descriptors

for

image

classification

Expert

Syst

Appl

pp

Google

Scholar

L

Nanni

S

Brahnam

A

Lumini

Combining

different

Local

Binary

Pattern

variants

to

boost

performance

Expert

syst

Appl

pp

Google

Scholar

TH

Vu

et

al

Histopathological

image

classification

using

discriminative

featureoriented

dictionary

learning

IEEE

Trans

Med

Imaging

pp

Google

Scholar

S

Otalora

et

al

Combining

unsupervised

feature

learning

and

Riesz

wavelets

for

histopathology

image

representation

application

to

identifying

anaplastic

medulloblastoma

Munich

International

Conference

on

Medical

Image

Computing

and

Computer

Assisted

Intervention

pp

Google

Scholar

J

Schmidhuber

Deep

learning

in

neural

networks

an

overview

Neural

Networks

pp

Google

Scholar

H

Greenspan

B

van

Ginneken

RM

Summers

Deep

learning

in

medical

imaging

overview

and

future

promise

of

an

exciting

new

technique

IEEE

Trans

Med

Imaging

pp

Google

Scholar

A

Janowczyk

A

Madabhushi

Deep

learning

for

digital

pathology

image

analysis

a

comprehensive

tutorial

with

selected

use

cases

J

Pathol

Inform

Google

Scholar

J

Gua

et

al

Recent

advances

in

convolutional

neural

networks

Pattern

Recogn

pp

Google

Scholar

Y

LeCun

et

al

Gradientbased

learning

applied

to

document

recognition

Proc

IEEE

pp

Google

Scholar

A

Krizhevsky

I

Sutskever

GE

Hinton

ImageNet

Classification

with

Deep

Convolutional

Neural

Networks

F

Pereira

Ed

Advances

In

Neural

Information

Processing

Systems

Curran

Associates

Inc

Red

Hook

NY

pp

Google

Scholar

MD

Zeiler

R

Fergus

Visualizing

and

understanding

convolutional

networks

D

Fleet

Ed

Computer

Vision

ECCV

ECCV

Lecture

Notes

in

Computer

Science

Springer

Cham

Berlin

Google

Scholar

K

Simonyan

A

Zisserman

Very

Deep

Convolutional

Networks

for

LargeScale

Image

Recognition

Cornell

University

Google

Scholar

C

Szegedy

et

al

Going

deeper

with

convolutions

IEEE

Computer

Society

Conference

on

Computer

Vision

and

Pattern

Recognition

pp

Google

Scholar

K

He

et

al

Deep

residual

learning

for

image

recognition

IEEE

Conference

on

Computer

Vision

and

Pattern

Recognition

CVPR

IEEE

Las

Vegas

NV

pp

Google

Scholar

J

Yosinski

et

al

How

Transferable

are

Features

in

Deep

Neural

Networks

Cornell

University

Google

Scholar

HC

Shin

et

al

Deep

convolutional

neural

networks

for

computeraided

detection

Cnn

architectures

dataset

characteristics

and

transfer

learning

IEEE

Trans

Med

Imaging

pp

Google

Scholar

Y

Pan

et

al

Brain

tumor

grading

based

on

neural

networks

and

convolutional

neural

networks

IEEE

Engineering

in

Medicine

and

Biology

Society

EMBC

IEEE

pp

Google

Scholar

L

Nanni

et

al

Bioimage

Classification

with

Handcrafted

and

Learned

Features

IEEEACM

transactions

on

computational

biology

and

bioinformatics

In

Press

Google

Scholar

B

van

Ginneken

et

al

Offtheshelf

convolutional

neural

network

features

for

pulmonary

nodule

detection

in

computed

tomography

scans

IEEE

International

Symposium

on

Biomedical

Imaging

ISBI

IEEE

Google

Scholar

R

Li

et

al

Deep

learning

based

imaging

data

completion

for

improved

brain

disease

diagnosis

Medical

Image

Computing

and

ComputerAssisted

Intervention

pp

Google

Scholar

A

Kumar

et

al

An

ensemble

of

finetuned

convolutional

neural

networks

for

medical

image

classification

IEEE

J

Biomed

Health

Inf

pp

Google

Scholar

C

Szegedy

et

al

inceptionresnet

and

the

impact

of

residual

connections

on

learning

in

arxivorg

Cornell

University

pp

Google

Scholar

J

Jantzen

et

al

Papsmear

benchmark

data

for

pattern

classification

Albufeira

Portugal

Nature

Inspired

Smart

Information

Systems

NiSIS

pp

Google

Scholar

L

Shamir

et

al

IICBU

a

proposed

benchmark

suite

for

biological

image

analysis

Med

Biol

Eng

Compu

pp

Google

Scholar

GB

Junior

et

al

Classification

of

breast

tissues

using

Morans

index

and

Gearys

coefficient

as

texture

signatures

and

SVM

Comput

Biol

Med

pp

Google

Scholar

A

CruzRoa

JC

Caicedo

FA

González

Visual

pattern

mining

in

histology

image

collections

using

bag

of

features

Artif

Intell

Med

pp

Google

Scholar

X

Tan

B

Triggs

Enhanced

local

texture

feature

sets

for

face

recognition

under

difficult

lighting

conditions

LNCS

Analysis

and

Modelling

of

Faces

and

Gestures

pp

Google

Scholar

L

Nanni

S

Brahnam

A

Lumini

A

very

high

performing

system

to

discriminate

tissues

in

mammograms

as

benign

and

malignant

Expert

Syst

Appl

pp

Google

Scholar

Z

Guo

L

Zhang

D

Zhang

A

completed

modeling

of

local

binary

pattern

operator

for

texture

classification

IEEE

Trans

Image

Process

pp

Google

Scholar

R

Nosaka

K

Fukui

cell

classification

using

rotation

invariant

cooccurrence

among

local

binary

patterns

Pattern

Recogn

Bioinf

pp

Google

Scholar

L

Nanni

Review

on

texture

descriptors

for

image

classification

S

Alexander

Ed

Computer

Vision

and

Simulation

Methods

Applications

and

Technology

Nova

Publications

Hauppauge

NY

Google

Scholar

Z

Zhu

et

al

An

adaptive

hybrid

pattern

for

noiserobust

texture

analysis

Pattern

Recogn

pp

Google

Scholar

G

Serra

et

al

Gold

Gaussians

of

local

descriptors

for

image

representation

Comput

Vis

Image

Underst

May

pp

Google

Scholar

N

Dalal

B

Triggs

Histograms

of

oriented

gradients

for

human

detection

European

Conference

on

Computer

Vision

San

Diego

CA

Google

Scholar

P

Strandmark

J

Ulén

F

Kahl

Staining

Pattern

Classification

International

Conference

on

Pattern

Recognition

Google

Scholar

Q

Wang

et

al

Towards

effective

codebookless

model

for

image

classification

Pattern

Recogn

pp

Google

Scholar

T

Song

F

Meng

Letrist

locally

encoded

transform

feature

histogram

for

rotationinvariant

texture

classification

IEEE

Transactions

on

Circuits

and

Systems

for

Video

Technology

p

Google

Scholar

T

Ojala

M

Pietikainen

T

Maeenpaa

Multiresolution

grayscale

and

rotation

invariant

texture

classification

with

local

binary

patterns

IEEE

Trans

Pattern

Anal

Mach

Intell

pp

Google

Scholar

V

Ojansivu

J

Heikkila

Blur

insensitive

texture

classification

using

local

phase

quantization

ICISP

pp

Google

Scholar

C

Chan

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O