Systemic Docs

Name	Type
Filter Boolean	boolean
Filter Double	double
Filter Float	float
Filter Integer	integer
Filter String	string

A sweep gives you many candidates. Filter decides which of them deserve your capital.

Point a sweep at eight sector funds, or at five lookback lengths, and you get eight or five parallel branches — each a complete, independently computed candidate. Filter reads all of them at once, discards those that fail a test you define, ranks what is left by a number you define, and forwards the best to separate outputs: Rank 0, Rank 1, Rank 2. Everything downstream of Rank 0 receives the winner; everything downstream of Rank 1, the runner-up.

That is the whole job. The rest of this page is about doing it well — and about the trap that ruins more selection systems than any other.

The sweep itself is explained on the For Each page; merging many branches back into one lives on the Collapse page. This page assumes the candidates already exist.

The Selection Pipeline

flowchart TB
    subgraph IN["Candidates from the sweep"]
        C0["Candidate 0"]
        C1["Candidate 1"]
        CN["Candidate n-1"]
    end

    G["Condition gate<br/>keep only if the LAST<br/>condition value is true"]
    S["Rank survivors by the LAST<br/>Sort value - desc or asc"]
    T["Take<br/>keep at most Take survivors"]

    subgraph OUT["Outputs"]
        R0["Rank 0 - best"]
        R1["Rank 1"]
        R2["Rank 2 - empty if no survivor"]
        SI["Selection index"]
    end

    C0 --> G
    C1 --> G
    CN --> G
    G --> S --> T
    T --> R0
    T --> R1
    T --> R2
    T --> SI

    style IN fill:#ecf0f1,stroke:#bdc3c7
    style G fill:#fdebd0,stroke:#e67e22
    style S fill:#d6eaf8,stroke:#2980b9
    style T fill:#e8daef,stroke:#8e44ad
    style OUT fill:#d5f5e3,stroke:#27ae60
    style R2 fill:#f5b7b1,stroke:#c0392b

The Controls

Control	What you wire into it	Effect
Items	The candidate series itself	What gets forwarded
Condition (optional)	One true/false series per candidate	Survive only if the last value is true
Sort (optional)	One numeric series per candidate	Rank by the last value
Order	`desc` or `asc`	Highest-first or lowest-first. Absent, it behaves as `desc`.
Take	A whole number, at least 1	Maximum candidates forwarded

Two omissions behave differently. No Condition wired eliminates nothing — every candidate reaches the ranking stage. No Sort wired means no reordering happens at all: survivors keep the order the sweep produced them in, and Rank 0 is simply the first survivor.

Only the Last Value Counts

Condition and Sort are full time series, but Filter reads one number from each: the final one. For candidate $i$ with condition series $c_i$ and sort series $s_i$ of length $T_i$ , the survival set is

\mathcal{S} = \{\, i : c_i[T_i - 1] = \text{true} \,\}

and survivors are ordered by the key $\kappa_i = s_i[T_i - 1]$ .

Selection is a decision made now: a candidate whose Sharpe ratio was magnificent two years ago and dreadful last month is judged on last month. This also makes Filter indifferent to padding — when branches produce series of different lengths, shorter ones are extended by repeating their final data point, which leaves the final value unchanged.

Worked Example 1: Sector Rotation With a Trend Gate

The classic use of Filter is cross-sectional: sweep a basket of assets, rank them against each other, hold the leaders.

Setup. Eight Select Sector SPDR funds — XLK (technology), XLE (energy), XLF (financials), XLV (health care), XLU (utilities), XLY (consumer discretionary), XLP (consumer staples), XLI (industrials) — on monthly dividend-adjusted closes.

Decision date: 31 October 2018 — the month of a sharp, broad equity drawdown, chosen because it shows the gate doing real work.

Sort — trailing 12-month total return, from the 31 October 2017 close to the 31 October 2018 close. Order desc.
Condition — price above its 10-month simple moving average (the January–October 2018 closes), the trend filter popularised in Meb Faber's tactical allocation work.
Take — 4.

The Candidate Table

Input order	Sector	Price (31 Oct 2018)	Price (31 Oct 2017)	Sort: 12m return	10-month average	Condition: above?
0	XLK	32.2629	28.9060	+11.6132%	32.3338	❌ false
1	XLE	23.8372	23.4526	+1.6399%	25.6199	❌ false
2	XLF	22.6715	22.5615	+0.4876%	23.7876	❌ false
3	XLV	77.6620	69.9577	+11.0128%	75.5209	✅ true
4	XLU	21.1190	20.9425	+0.8428%	20.1053	✅ true
5	XLY	48.9960	42.2628	+15.9317%	50.1302	❌ false
6	XLP	44.6904	41.9583	+6.5115%	42.7509	✅ true
7	XLI	61.4393	61.7574	−0.5151%	65.6120	❌ false

Check a row by hand: XLV's return is $\frac{77.6620}{69.9577} - 1 = 0.110128$ , i.e. $+11.0128\%$ ; its condition is $77.6620 > 75.5209$ , true.

What Filter Forwards

Five of eight candidates fail the gate. Three survive, and they are ranked by the Sort key:

Output	Sector	Sort key	Selection index
Rank 0	XLV	+11.0128%	3
Rank 1	XLP	+6.5115%	6
Rank 2	XLU	+0.8428%	4
Rank 3	(empty — branch dead)	—	—

Take was 4 but only three candidates survived, so Rank 3 is empty and everything downstream of it is dead. This is normal and must be planned for. If you size positions as a fixed fraction per rank, an empty rank leaves that fraction in cash — usually the right behaviour, but it should be a choice, not a surprise.

Why the Gate Earned Its Place

Ranking all eight by 12-month return alone, ignoring the Condition, gives XLY (+15.93%), XLK (+11.61%), XLV (+11.01%). The two highest-momentum sectors on the board — consumer discretionary and technology — had both already broken below their own 10-month averages. The gate removed them and left the three defensives.

That is the point of separating eligibility from ranking. Momentum tells you what has been working; the trend gate tells you whether it is still working. Sort answers the first question, Condition the second.

Ties, and Why Input Order Matters

Continuous statistics rarely tie. Discrete ones tie constantly — and counts, ratings and rounded scores are exactly what traders often rank on.

Suppose the Sort input is instead the count of months in the trailing 12 that closed higher than the previous month, still as of 31 October 2018:

Rank	Sector	Sort key	Input order	Why here
0	XLE	9	1	Tied at 9; lower input order wins
1	XLY	9	5	Tied at 9
2	XLV	8	3	Tied at 8; lower input order wins
3	XLP	8	6	Tied at 8
4	XLK	7	0	Three-way tie at 7
5	XLU	7	4
6	XLI	7	7
7	XLF	5	2

The sort is stable: equal keys never change their relative order, so XLE precedes XLY because the sweep produced it first, and XLK precedes XLU precedes XLI for the same reason.

The consequence: when ties are likely, the order you list your sweep values in becomes a silent tiebreaker. Put your preferred default first — or choose a Sort statistic with enough resolution that exact ties are rare.

Worked Example 2: Choosing a Parameter, With a Risk Gate

The second archetype: one asset, many settings, pick the setting.

Setup. Weekly dividend-adjusted closes for QQQ, and a long-or-flat trend rule: hold QQQ for the coming week if last week's close finished above its own $W$ -week simple moving average, otherwise hold cash and earn nothing. The sweep runs over $W \in \{5, 10, 20, 30, 40\}$ weeks.

In-sample window: 1 January 2016 – 31 December 2025 (521 weekly returns per candidate).

Condition — maximum drawdown shallower than 20%.
Sort — annualised Sharpe ratio of the rule's weekly returns, at a zero cash rate: $\text{Sharpe} = \frac{\bar{r}}{\sigma_r}\sqrt{52}$ . Order desc.
Take — 2.

The Candidate Table

Input order	$W$	CAGR	Sort: Sharpe	Condition: max drawdown	Survives?
0	5 weeks	12.7077%	0.9635	24.2478%	❌ deeper than 20%
1	10 weeks	13.2497%	0.9794	19.1371%	✅
2	20 weeks	15.0850%	1.0355	19.9863%	✅
3	30 weeks	15.0721%	0.9935	23.8374%	❌ deeper than 20%
4	40 weeks	17.6548%	1.1275	18.0276%	✅
—	buy and hold	19.5926%	0.9849	35.0585%	(not a candidate)

Two candidates fail the drawdown gate, and the calls are close: 19.9863% survives, 23.8374% does not. A gate is a cliff, not a slope.

What Filter Forwards

Output	Candidate	Sort key	Selection index
Rank 0	40 weeks	1.1275	4
Rank 1	20 weeks	1.0355	2

Take was 2, so the 10-week candidate — a legitimate survivor, ranked third — is discarded. Survivors beyond Take appear on no output at all.

Now the Uncomfortable Part

Every number in that table was measured on the same decade the winner was chosen from. Here is what the five candidates did over the following period, 1 January 2026 to 4 September 2026 (36 weekly returns), nothing else changed:

xychart-beta
    title "QQQ trend rule: Sharpe by lookback window"
    x-axis "Moving-average window (weeks)" [5, 10, 20, 30, 40]
    y-axis "Annualised Sharpe" 0.5 --> 1.3
    line [0.9635, 0.9794, 1.0355, 0.9935, 1.1275]
    line [0.7268, 0.5601, 1.2561, 1.1060, 0.8486]

Pale line: the 2016–2025 Sharpe the selection was made on. Dark line: the 2026 Sharpe that followed.

$W$	In-sample Sharpe	Rank	Out-of-sample Sharpe	Rank	Out-of-sample return
5	0.9635	5	0.7268	4	+7.6805%
10	0.9794	4	0.5601	5	+5.4632%
20	1.0355	2	1.2561	1	+15.0972%
30	0.9935	3	1.1060	2	+13.2282%
40	1.1275	1	0.8486	3	+9.8884%

The in-sample champion came third of five. Rank 1 — the candidate Filter deemed merely second-best — actually led. The ranking did not survive contact with new data.

The Selection Bias You Have Just Committed

Filter makes it trivially easy to sweep twenty variants, rank them on a backtested statistic, and route capital to the winner. That procedure has a name — data snooping — and a consequence: overfitting.

The mechanism is arithmetic. If you evaluate $n$ candidates whose true edge is identical, the measured best is not the best — it is the luckiest, and the expected in-sample outperformance of the maximum grows with $n$ even when every candidate is worthless. Bailey, Borwein, López de Prado and Zhu quantify this in Pseudo-Mathematics and Financial Charlatanism: with enough trials, a strategy with zero true expected return can be made to show almost any desired backtested Sharpe. Harvey, Liu and Zhu reach the same conclusion for published asset-pricing factors in "... and the Cross-Section of Expected Returns".

The sweep is capped at 20 combinations, which stops you searching a million-point grid — but 20 is ample to manufacture a flattering winner.

Defences That Actually Work

Defence	What it looks like in practice
Rank on something economic, not something fitted	Momentum, volatility, yield and valuation are properties of the asset. A backtested Sharpe is a property of your search. Example 1 is far safer than Example 2 for this reason.
Keep the grid small and pre-committed	Decide the sweep values before you look at results.
Demand a plateau, not a peak	If $W = 40$ is excellent while its neighbours are poor, you found noise. A parameter worth trusting has good neighbours.
Use Condition as a floor, not a scoreboard	"Drawdown under 20%" expresses risk tolerance. It is far more robust than a gate tuned to maximise a backtested number.
Reserve data you never look at	Choose on one period, verify on another. The out-of-sample table above is that test, and it failed.

Momentum ranking carries the strongest independent evidence of any Filter-shaped strategy — Jegadeesh and Titman (1993) documented it in US equities and it has since been found across countries and asset classes (overview). Even so, it has suffered multi-year droughts and violent crashes.

Missing Data

Filter never invents a candidate and never invents a ranking key.

Situation	Behaviour
A candidate's branch died upstream	It arrives empty. It holds its place in the input ordering, but is never eligible for selection.
Condition series is missing or empty	That candidate is eliminated. Absence of evidence is not treated as a pass.
Sort value cannot be read as a number	The candidate ranks last — in both `desc` and `asc`. Unrankable is unrankable regardless of direction.
Two unrankable candidates	They tie, separated by their original input order.

The "last in both directions" rule matters. Under asc you might expect an unreadable value to sort as if it were negative infinity and win. It does not: a candidate you cannot evaluate is never promoted by the evaluation failing, so a data outage makes your system ignore that candidate rather than bet on it.

Edge Cases

Case	Result
Every candidate eliminated	All ranks empty, all downstream branches dead. The system holds nothing.
Fewer survivors than Take	Early ranks fill; the rest are empty and dead. See Example 1.
More survivors than Take	Ranks beyond Take appear on no output. See Example 2.
Take = 1	A pure "pick the best" node — the most common setting.
Take larger than the sweep	Harmless; the extra ranks are always empty.
Sweep of one	A pass-through gate: forwarded to `Rank 0` if Condition passes, otherwise nothing.
All Sort values equal	Stable ordering returns exactly the sweep's input order.
Maximum breadth	The sweep caps at 20 combinations, so Filter never sees more than 20 candidates.

Reading the Selection Index

Alongside the rank outputs, Filter reports a Selection index: for each rank, which input candidate was forwarded — $3, 6, 4$ in Example 1. Plot it and you get an honest picture of turnover: an index that changes every period is a system paying transaction costs to chase noise; one that changes twice a year is a system with conviction. Indices that are out of range, repeated, or beyond Take are discarded, so the reported selection is always a clean one-to-one account of what was forwarded.

Filter

Variants

The Selection Pipeline

The Controls

Only the Last Value Counts

Worked Example 1: Sector Rotation With a Trend Gate

The Candidate Table

What Filter Forwards

Why the Gate Earned Its Place

Ties, and Why Input Order Matters

Worked Example 2: Choosing a Parameter, With a Risk Gate

The Candidate Table

What Filter Forwards

Now the Uncomfortable Part

The Selection Bias You Have Just Committed

Defences That Actually Work

Missing Data

Edge Cases

Reading the Selection Index

Further Reading