Encrypted GELU

Definition

The Gaussian Error Linear Unit applied element by element to a secret vector. For a real vector $x = (x_0, \ldots, x_{N-1})$, the result $y$ has

where $\Phi$ is the cumulative distribution function of the standard normal distribution. Equivalently,

Each output element depends only on the corresponding input element. There is no interaction between positions and no reduction across the vector.

Mathematical Domain

GELU is a smooth activation: unlike the hard corner of ReLU it is differentiable everywhere, weighting each input by the probability that a standard normal variable falls below it. It is transcendental — defined through the error function — so it cannot be expressed as a polynomial. Arithmetic homomorphic encryption schemes evaluate only polynomials, so an implementation approximates GELU by a polynomial over a bounded input range.

The kernel fixes neither the input range, the vector length, nor the packing; these are defined by individual specifications.

Parameters Defined by Specifications

A specification narrows the general GELU operation into a concrete problem by pinning:

• the element domain and encryption scheme
• the input range on which the approximation must hold
• the vector length $N$
• the ciphertext packing and layout
• the correctness or accuracy rule and how it is scored
• which stages of the computation are timed

Applications

GELU is the standard activation of transformer models: it appears in the feed-forward block of each layer in architectures such as BERT and the GPT family. Private inference over these models needs an encrypted GELU wherever such a feed-forward block is evaluated.