Spectral Entropy for Market Research: Normalize the Spectrum

From frequency-bin power to bits or nats

For retained nonnegative powers q(k), define p(k) = q(k) / Σq(j). The total must be positive and finite. Then H = -Σp(k) log₂ p(k) is in bits; replacing log₂ with ln gives nats. Define a zero-probability contribution as zero. For K retained bins, Hnorm = H / log₂(K), provided K > 1. Natural logarithms give the same normalized result when the denominator is ln(K).

• K counts every retained bin, including bins with zero power. Counting only occupied bins would change the normalization when a signal becomes concentrated.
• The bounds are 0 ≤ H ≤ log₂(K) and 0 ≤ Hnorm ≤ 1. Equality at 1 requires equal retained powers. The flat row is a mathematical reference, not an observed white-noise sample.
• The middle row is 0.867563 nats because Hnats = Hbits × ln(2). A common multiplicative power scale cancels when forming probabilities.
• These formulas use bin powers on a uniform frequency grid. With a density on unequal-width bands, first form band masses using their widths or numerical integration. Normalizing density heights alone answers a different question.

Declare the DFT, DC and one-sided convention

This page uses a direct periodogram of N real, equally spaced observations. Subtract the sample mean first, multiply by a declared window w(t), and calculate X(k) = Σ[t=0..N-1] (x(t)-mean(x)) w(t) exp(-2πikt/N). There is no zero padding. Frequency k represents k/N cycles per observation, or k×fs/N when the sampling rate is fs.

• Drop k = 0, the DC bin, after the transform. Retain k = 1,...,floor(N/2), so K = floor(N/2). Demeaning removes DC before windowing; a nonconstant window can reintroduce a nonzero weighted mean, which this convention also excludes.
• Fold negative-frequency power onto its positive partner: q(k) = 2|X(k)|², except the Nyquist bin k = N/2 for even N, where q(k) = |X(k)|². For odd N there is no Nyquist bin and every retained bin is doubled.
• A two-sided spectrum counts conjugate bins separately. For the rectangular-window example, its two equal peaks give 1 bit across eight bins, while the folded non-DC spectrum gives 0 bits across four bins. Neither convention is interchangeable with the other.
• A constant series becomes all zeros after demeaning, so total power is zero and entropy is undefined. Do not label that case as maximally periodic or return a manufactured zero. With K = 1, raw entropy is zero for positive power but normalized entropy has a zero denominator.

Reproduce the eight-point periodic signal

Take x = [1, 0, -1, 0, 1, 0, -1, 0], two complete cycles in eight observations. Its mean is zero. A rectangular window w(t) = 1 gives the full DFT X = [0, 0, 4, 0, 0, 0, 4, 0]. Dropping DC and folding conjugate pairs produces q = [0, 32, 0, 0]. All retained power sits at 2/8 cycles per observation, so H = 0.

• Now apply the periodic Hann window w(t) = 0.5 - 0.5 cos(2πt/N). The windowed observations are [0, 0, -0.5, 0, 1, 0, -0.5, 0]. Its DFT is [0, -1, 2, -1, 0, -1, 2, -1].
• The folded powers become [2, 8, 2, 0]. Thus H = -2×(1/6)log₂(1/6) -(2/3)log₂(2/3) = 1.251629 bits and Hnorm = 1.251629/2 = 0.625815.
• No noise was added. The Hann taper spreads this bin-centered tone into adjacent bins. A symmetric Hann window using N-1 in its cosine denominator is a different window and will not reproduce these exact arrays.

Window sensitivity is not a one-way correction

A finite observation interval can spread a tone across bins when it contains a noninteger number of cycles. Tapering trades sidelobe suppression for a wider main lobe. The effect on entropy depends on the tone, sample length, frequency mask and preprocessing. A lower score after tapering is not evidence that the market became more predictable.

• Both rows subtract the unweighted sample mean before windowing, exclude DC and retain four folded bins. The 1.5-cycle row has mean 0.125; its rectangular powers are approximately [11.501142, 15.656854, 2.842004, 1].
• Changing N also changes resolution and the number of available bins. Zero padding samples the spectrum more densely without adding observations; it can change this discrete entropy and its normalization. Freeze both data length and FFT length for a comparison.
• Welch estimation averages periodograms from segments. Segment length, overlap and window become part of the statistic. Averaging spectra before taking entropy is generally different from averaging segment entropies, because entropy is nonlinear.

A finite white-noise sample need not have high entropy

An independent sequence taking +1 or -1 with equal probability is a zero-mean white-noise process. The particular vector [1, 1, -1, 1, -1, -1, 1, -1] has probability 1/256 under that model. Under the rectangular-window convention used here it has powers [9.372583, 0, 54.627417, 0], giving H = 0.600876 bits and Hnorm = 0.300438. This is a deliberately selected possible realization, not a representative noise benchmark.

• A flat population spectrum is an expectation statement about the process. The expected entropy is not the entropy of an expected spectrum. Finite periodograms have random peaks; even a long unsmoothed periodogram can remain variable bin by bin.
• With folding, interior bins combine two frequencies while an even-length Nyquist bin combines only one. Demeaning, tapering and endpoint weights also affect a finite-sample white-noise reference. Do not assume its expected retained bin powers are identical under every convention.
• For a reference distribution, simulate many samples from the explicitly chosen null model and apply exactly the same length, window, DC rule, spectrum estimator and frequency mask. Report its distribution, not a universal trading threshold such as 0.8.
• Amplitude rescaling cancels from p(k). The score cannot tell a quiet return series from an otherwise identical series with ten times its volatility. Power spectra also discard phase, so equal entropy cannot establish equal timing, independence or economic behavior.

Run the calculation without a signal-processing package

The Python example implements the exact convention above with a direct DFT. It is intended for short reproducibility examples, with quadratic running time. It rescales the input before centering to reduce overflow risk; the returned powers use that relative scale, while probabilities and entropy are unchanged by the scaling.

from cmath import exp
from math import cos, fsum, isfinite, log2, pi


def spectral_entropy(values, window="boxcar"):
    x = [float(value) for value in values]
    n = len(x)
    if n < 4 or not all(isfinite(value) for value in x):
        raise ValueError("Need at least four finite real observations")
    if window not in ("boxcar", "hann"):
        raise ValueError("Use boxcar or periodic hann")
    scale = max(abs(value) for value in x)
    if scale == 0:
        raise ValueError("Zero retained power: entropy is undefined")
    x = [value / scale for value in x]
    mean = fsum(x) / n
    y = [(value - mean) *
         (1 if window == "boxcar" else 0.5 - 0.5*cos(2*pi*t/n))
         for t, value in enumerate(x)]
    if max(abs(value) for value in y) == 0:
        raise ValueError("Zero retained power: entropy is undefined")
    power = []
    for k in range(1, n // 2 + 1):
        dft = sum(y[t] * exp(-2j*pi*k*t/n) for t in range(n))
        power.append(abs(dft)**2 * (1 if 2*k == n else 2))
    total = fsum(power)
    if not isfinite(total) or total <= 0:
        raise ValueError("Invalid retained power")
    probabilities = [value / total for value in power]
    bits = -fsum(p * log2(p) for p in probabilities if p > 0)
    return power, probabilities, bits, bits / log2(len(power))


examples = {
    "two cycles": [1, 0, -1, 0, 1, 0, -1, 0],
    "1.5 cycles": [cos(2*pi*1.5*t/8) for t in range(8)],
    "possible noise sample": [1, 1, -1, 1, -1, -1, 1, -1],
}
for name, values in examples.items():
    for window in ("boxcar", "hann"):
        power, probabilities, bits, normalized = spectral_entropy(values, window)
        print(name, window, round(bits, 6), round(normalized, 6))

Keep spectral, sample and permutation entropy distinct

These names describe different probability constructions. A numerical value from one method cannot be substituted for another, even when all three are used as time-series features.

Use a defined observation clock in market research

A DFT frequency needs a sampling interval. For bar-return research, retain timestamps and define the interval, session, missing-bar policy and price adjustments. Price levels and returns answer different questions: a large price offset can dominate DC, while drift can concentrate low-frequency power. Removing the mean alone does not make a price process stationary.

• A sequence of trade outcomes is sampled in trade order, not necessarily equal clock time. Its frequencies are cycles per trade. Do not relabel them cycles per day or treat a closed-trade equity curve as a fully marked account series.
• Use trailing observations available at the decision time. Freeze lookback, window and any entropy threshold on development data, then assess the rule on later dates with executable fill assumptions and costs. Overlapping trailing windows also produce dependent feature values.
• TradingView exports Strategy Report tabs separately: the List of Trades export describes trades, while Performance Summary exports metrics. Pineify Backtest Deep Report parses entry and exit times, prices, quantity and P&L, and calculates supported results such as net profit and profit factor.
• Obtain a regularly sampled price, return or marked-equity series separately for clock-time spectral analysis. The calculation on this page is independent; spectral entropy is not claimed as a built-in Backtest Deep Report field.
• Pine Script AI Coding Agent can help draft custom series or strategy logic from an explicit specification. Review the generated code and compare exported values with the independent calculation before evaluating any rule.