this script demonstrates the implementation of the sigmoid linear unit silu or swish function https en wikipedia orgwikirectifierneuralnetworks https en wikipedia orgwikiswishfunction the function takes a vector x of k real numbers as input and returns x sigmoidx swish is a smooth nonmonotonic function defined as fx x sigmoidx extensive experiments shows that swish consistently matches or outperforms relu on deep networks applied to a variety of challenging domains such as image classification and machine translation this script is inspired by a corresponding research paper https arxiv orgabs1710 05941 https blog paperspace comswishactivationfunction mathematical function sigmoid takes a vector x of k real numbers as input and returns 1 1 ex https en wikipedia orgwikisigmoidfunction sigmoidnp array1 0 1 0 2 0 array0 26894142 0 73105858 0 88079708 implements the sigmoid linear unit silu or swish function parameters vector np ndarray a numpy array consisting of real values returns swishvec np ndarray the input numpy array after applying swish examples sigmoidlinearunitnp array1 0 1 0 2 0 array0 26894142 0 73105858 1 76159416 sigmoidlinearunitnp array2 array0 23840584 parameters vector np ndarray a numpy array consisting of real values trainableparameter use to implement various swish activation functions returns swishvec np ndarray the input numpy array after applying swish examples swishnp array1 0 1 0 2 0 2 array0 11920292 0 88079708 1 96402758 swishnp array2 1 array0 23840584 mathematical function sigmoid takes a vector x of k real numbers as input and returns 1 1 e x https en wikipedia org wiki sigmoid_function sigmoid np array 1 0 1 0 2 0 array 0 26894142 0 73105858 0 88079708 implements the sigmoid linear unit silu or swish function parameters vector np ndarray a numpy array consisting of real values returns swish_vec np ndarray the input numpy array after applying swish examples sigmoid_linear_unit np array 1 0 1 0 2 0 array 0 26894142 0 73105858 1 76159416 sigmoid_linear_unit np array 2 array 0 23840584 parameters vector np ndarray a numpy array consisting of real values trainable_parameter use to implement various swish activation functions returns swish_vec np ndarray the input numpy array after applying swish examples swish np array 1 0 1 0 2 0 2 array 0 11920292 0 88079708 1 96402758 swish np array 2 1 array 0 23840584
import numpy as np


def sigmoid(vector: np.ndarray) -> np.ndarray:
    
    return 1 / (1 + np.exp(-vector))


def sigmoid_linear_unit(vector: np.ndarray) -> np.ndarray:
    
    return vector * sigmoid(vector)


def swish(vector: np.ndarray, trainable_parameter: int) -> np.ndarray:
    
    return vector * sigmoid(trainable_parameter * vector)


if __name__ == "__main__":
    import doctest

    doctest.testmod()