Swish x·σ(βx) vs. GELU Activation Functions in Transformer Networks

Compares smooth non-monotonic neural network activations that eliminate ReLU dead-neuron gradients.

Expressions & Parameters

Primary Function f₁(x)

Comparison Curve f₂(x)

Plot Tangent Line at x₀ Shade Integral [a, b]

Live Parameter Sliders (a, b, c)

a1.00

b1.00

c0.00

Interactive Coordinate Plane (Drag to Pan · Wheel to Zoom)

(0.000, 0.000)

Pre-Computed Analytical Curve Reference

Primary Equationy = x / (1 + exp(-a*x))

Symbolic Derivative dy/dx(1 + exp(-a*x) - x*exp(-a*x)*-a)/(1 + exp(-a*x))^2

Canonical Viewportx ∈ [-5, 5], y ∈ [-1, 4.5]

Related Interactive Curves

Physics & Vibrations

Damped Harmonic Oscillator y = e^(−ax)·cos(bx) — Envelope & Phase Decay

Models underdamped mechanical springs and RLC circuits where exponential envelope e^(−ax) bounds rapid cosine oscillations.

y = exp(-a*x)*cos(b*x)

Probability & Statistics

Gaussian Normal Bell Curve f(x) = (1 / σ√2π)·e^(−(x−μ)²/2σ²)

Normalized probability density function with total area equal to 1, peak at x = μ, and inflection points at x = μ ± σ.

y = exp(-(x - c)^2 / (2*a^2)) / (a*sqrt(2*pi))

Machine Learning & Biology

Logistic Sigmoid Curve S(x) = L / (1 + e^(−k(x − x₀)))

Bounded S-shaped activation and population carrying-capacity curve with inflection symmetry at y = L/2.

y = a / (1 + exp(-b*(x - c)))

Signal Processing & Optics

Normalized Sinc Function sin(ax)/(ax) & Fraunhofer Diffraction Intensity

Ideal low-pass Nyquist reconstruction kernel in DSP; its square sinc²(x) governs single-slit optical diffraction fringes.

y = sinc(a*x)

Classical Algebraic Curves

Witch of Agnesi y = 8a³ / (x² + 4a²) — Cauchy-Lorentz Resonance Profile

Rational bell curve studied by Maria Gaetana Agnesi (1748), identical in form to spectral Lorentzian line shapes and Cauchy distributions.

y = 8*a^3 / (x^2 + 4*a^2)

Structural Mechanics

Catenary Arch y = a·cosh(x/a) vs. Parabolic Approximation

Exact equilibrium shape of a uniform flexible cable hanging under gravity, compared side-by-side against its osculating parabola.

y = a*cosh(x/a)

Mathematical Reference & Frequently Asked Questions

How can I modify the parameters of Swish x·σ(βx) vs. GELU Activation Functions in Transformer Networks?

Drag sliders a, b, and c in the expression sidebar or edit the formula directly to observe real-time shifts in amplitude, frequency, roots, and extrema.