this script demonstrates an implementation of the gaussian error linear unit function https en wikipedia orgwikiactivationfunctioncomparisonofactivationfunctions the function takes a vector of k real numbers as input and returns x sigmoid1 702x gaussian error linear unit gelu is a highperforming neural network activation function this script is inspired by a corresponding research paper https arxiv orgabs1606 08415 mathematical function sigmoid takes a vector x of k real numbers as input and returns 1 1 ex https en wikipedia orgwikisigmoidfunction sigmoidnp array1 0 1 0 2 0 array0 26894142 0 73105858 0 88079708 implements the gaussian error linear unit gelu function parameters vector np ndarray a numpy array of shape 1 n consisting of real values returns geluvec np ndarray the input numpy array after applying gelu examples gaussianerrorlinearunitnp array1 0 1 0 2 0 array0 15420423 0 84579577 1 93565862 gaussianerrorlinearunitnp array3 array0 01807131 mathematical function sigmoid takes a vector x of k real numbers as input and returns 1 1 e x https en wikipedia org wiki sigmoid_function sigmoid np array 1 0 1 0 2 0 array 0 26894142 0 73105858 0 88079708 implements the gaussian error linear unit gelu function parameters vector np ndarray a numpy array of shape 1 n consisting of real values returns gelu_vec np ndarray the input numpy array after applying gelu examples gaussian_error_linear_unit np array 1 0 1 0 2 0 array 0 15420423 0 84579577 1 93565862 gaussian_error_linear_unit np array 3 array 0 01807131
import numpy as np


def sigmoid(vector: np.ndarray) -> np.ndarray:
    
    return 1 / (1 + np.exp(-vector))


def gaussian_error_linear_unit(vector: np.ndarray) -> np.ndarray:
    
    return vector * sigmoid(1.702 * vector)


if __name__ == "__main__":
    import doctest

    doctest.testmod()