Replication Study: Optimal Stopping via Randomized Neural Networks

Introduction

This document presents a replication study of experiments from the paper "Optimal Stopping via Randomized Neural Networks". The goal was to replicate selected results using the authors' provided Python code, with necessary modernizations.

Code Modernization Challenges

1. Legacy Code Issues

The original code is outdated, with several packages no longer available in the versions used by the authors. This made exact replication impossible.

Solution: Modernize the code by updating all packages and Python to the latest versions, and adapting incompatible functions.

Example: The function df2[name].fillna(method="ffill", inplace=True) was replaced with df2[name].ffill()

2. Code Structure and Usability

The code was designed specifically to replicate exact results from the paper's tables, making it unintuitive for running custom experiments with different parameters.

3. Computational Limitations

RAM Constraints:

d ≤ 50: Possible, sometimes with issues
d = 100: Always results in some problems
d = 500: Severe memory problems
d > 500: Computer aborts execution

Important Note: Models like RLSM and RFQI require only seconds to minutes even for d > 500. The issue is RAM for storing high-dimensional matrices, not computational time. For LSM and FQI, dimensions > 50 exceed computational capacity.

Experimental Setup

Two simplified comparison tables were created, replicating selected results from the original paper with dimension constraints.

Standard Parameters (used in both tables):

Simulations: 10 runs per model
Statistics: Mean and standard deviation of prices, median computation time
Payoff: MaxCall
Drift: 0.0
Volatility: 0.2
Paths: 20,000
Dates: 10
Strike: 100
Dividend: 0.0
Maturity: 1
Hidden size: 20
Epochs: 30
Hurst: 0.75
Ridge coefficient: 1.0
Train ITM only: True

Table 1: Black-Scholes Model (Max Call, varying spot prices and dimensions)

Model: Black-Scholes | Dimensions tested: d = 5, 10, 50 | Spot prices (x₀): 80, 100, 120

Article Table 1 (Original Results)

		price							duration
d	x₀	LSM	DOS	NLSM	RLSM	FQI	RFQI	EOP	LSM	DOS	NLSM	RLSM	FQI	RFQI	EOP
5	80	5.23 (0.07)	5.12 (0.12)	5.19 (0.09)	5.28 (0.12)	5.26 (0.10)	5.20 (0.06)	5.31 (0.05)	11s	9s	0s	0s	2s	0s	0s
	100	24.95 (0.14)	24.64 (0.21)	24.72 (0.15)	24.91 (0.16)	24.96 (0.17)	25.00 (0.19)	24.97 (0.15)	11s	8s	2s	0s	2s	0s	0s
	120	49.73 (0.21)	49.45 (0.18)	49.47 (0.22)	49.62 (0.25)	49.68 (0.22)	49.75 (0.17)	49.77 (0.15)	11s	7s	2s	0s	2s	0s	0s
10	80	9.20 (0.07)	9.19 (0.14)	8.82 (0.15)	9.24 (0.11)	9.25 (0.12)	9.25 (0.10)	9.27 (0.09)	28s	7s	1s	0s	6s	0s	0s
	100	34.33 (0.15)	34.03 (0.17)	33.69 (0.20)	34.28 (0.11)	34.25 (0.19)	34.17 (0.11)	34.26 (0.09)	29s	7s	2s	0s	7s	0s	0s
	120	60.94 (0.24)	60.90 (0.20)	60.33 (0.25)	61.08 (0.23)	61.10 (0.19)	61.07 (0.21)	61.20 (0.13)	29s	7s	2s	0s	6s	0s	0s
50	80	22.45 (0.11)	23.17 (0.10)	21.78 (0.34)	22.03 (0.16)	23.51 (0.13)	23.42 (0.11)	23.52 (0.09)	8m39s	8s	2s	0s	6m28s	1s	0s
	100	53.49 (0.10)	53.93 (0.12)	52.15 (0.60)	52.44 (0.21)	54.24 (0.09)	54.23 (0.08)	54.37 (0.09)	8m42s	8s	3s	0s	6m57s	1s	0s
	120	84.31 (0.12)	84.72 (0.12)	82.48 (0.79)	82.98 (0.16)	85.03 (0.18)	85.00 (0.20)	85.28 (0.07)	8m46s	9s	3s	0s	7m4s	1s	0s

My Replication Results - Table 1

		price							duration
d	x₀	LSM	DOS	NLSM	RLSM	FQI	RFQI	EOP	LSM	DOS	NLSM	RLSM	FQI	RFQI	EOP
5	80	5.21 (0.05)	5.11 (0.08)	5.11 (0.07)	5.23 (0.07)	5.27 (0.08)	4.67 (0.85)	5.31 (0.06)	0s	10s	3s	0s	20s	9s	0s
	100	24.91 (0.11)	24.67 (0.12)	24.68 (0.17)	24.87 (0.11)	24.99 (0.14)	18.59 (7.71)	24.99 (0.11)	1s	12s	6s	0s	20s	9s	0s
	120	49.70 (0.15)	49.40 (0.17)	49.35 (0.21)	49.69 (0.18)	49.76 (0.13)	45.69 (6.88)	49.77 (0.11)	1s	10s	7s	0s	20s	9s	0s
10	80	9.18 (0.12)	9.13 (0.12)	9.00 (0.11)	9.26 (0.12)	9.31 (0.07)	8.84 (0.94)	9.29 (0.07)	3s	10s	3s	0s	49s	10s	0s
	100	34.25 (0.07)	34.07 (0.14)	33.74 (0.29)	34.19 (0.14)	34.32 (0.12)	23.64 (11.58)	34.29 (0.09)	7s	10s	8s	0s	50s	10s	0s
	120	61.08 (0.21)	60.78 (0.19)	60.48 (0.29)	61.00 (0.17)	61.06 (0.11)	46.56 (15.25)	61.14 (0.10)	7s	8s	7s	0s	49s	11s	0s
50	80	22.51 (0.08)	23.25 (0.14)	21.79 (0.34)	22.05 (0.06)	23.37 (0.13)	23.37 (0.10)	23.50 (0.10)	3m41s	14s	9s	1s	48m27s	10s	0s
	100	53.51 (0.08)	53.94 (0.18)	52.42 (0.41)	52.51 (0.15)	-	54.17 (0.19)	54.37 (0.04)	4m22s	15s	12s	1s	-	12s	0s
	120	84.13 (0.13)	84.67 (0.16)	82.51 (0.50)	82.97 (0.32)	-	85.09 (0.15)	85.25 (0.11)	4m18s	13s	13s	1s	-	11s	0s

Table 2: Heston Model with Variance (Max Call, varying dimensions)

Model: Heston with variance | Dimensions tested: d = 5, 10, 50, 100, 500

Article Table 2 (Original Results)

	price							duration
d	LSM	DOS	NLSM	RLSM	FQI	RFQI	EOP	LSM	DOS	NLSM	RLSM	FQI	RFQI	EOP
5	8.34 (0.08)	8.36 (0.07)	8.22 (0.09)	8.37 (0.07)	8.25 (0.03)	8.33 (0.07)	8.23 (0.04)	31s	6s	3s	0s	8s	0s	0s
10	11.81 (0.06)	11.83 (0.07)	11.51 (0.12)	11.83 (0.02)	11.79 (0.06)	11.83 (0.05)	11.79 (0.07)	1m30s	6s	3s	0s	28s	0s	0s
50	16.85 (0.07)	20.01 (0.06)	18.60 (0.32)	19.31 (0.05)	20.05 (0.06)	20.09 (0.05)	20.04 (0.04)	39m37s	8s	4s	0s	1h22m45s	1s	0s
100	-	23.49 (0.06)	21.75 (0.41)	22.90 (0.02)	-	23.69 (0.06)	23.66 (0.04)	-	14s	6s	0s	-	1s	0s
500	-	31.31 (0.06)	29.93 (0.32)	31.35 (0.06)	-	32.14 (0.06)	32.13 (0.07)	-	1m19s	24s	3s	-	2s	0s

My Replication Results - Table 2

	Price (std)							Duration
d	LSM	DOS	NLSM	RLSM	FQI	RFQI	EOP	LSM	DOS	NLSM	RLSM	FQI	RFQI	EOP
5	8.35 (0.06)	8.34 (0.05)	8.21 (0.07)	8.34 (0.04)	8.26 (0.04)	7.86 (0.92)	8.24 (0.06)	0s	15s	8s	0s	48s	3s	0s
10	11.77 (0.07)	11.80 (0.06)	11.53 (0.06)	11.80 (0.05)	11.82 (0.07)	9.07 (3.03)	11.78 (0.05)	5s	17s	6s	0s	2m53s	2s	0s
50	16.86 (0.07)	19.92 (0.05)	18.46 (0.19)	19.36 (0.08)	20.09 (-)	20.13 (0.10)	20.09 (0.07)	1h47m14s	36s	14s	0s	2h33m48s	2s	0s
100	-	23.40 (0.06)	21.78 (0.45)	22.85 (0.06)	-	23.68 (0.05)	23.68 (0.03)	-	36s	31s	1s	-	4s	0s
500	-	31.14 (0.05)	29.86 (0.35)	31.31 (0.02)	-	32.08 (0.04)	32.14 (0.04)	-	5m44s	6m58s	9m50s	-	1m48s	0s

Comparison Tables: How far are my estimates from the article's

How the comparison tables are produced:

Price errors (%): Calculated as (My estimate - Article estimate) / Article estimate × 100%.
Green = my estimate higher (positive error), Red = my estimate lower (negative error)
Duration differences: Absolute time difference (My time - Article time).
Green = faster (better), Red = slower (worse)

Comparison Table 1 (Errors vs. Article)

Parameters		Price (Relative % Error)							Duration (Absolute Difference)
d	x₀	LSM	DOS	NLSM	RLSM	FQI	RFQI	EOP	LSM	DOS	NLSM	RLSM	FQI	RFQI	EOP
5	80	-0.38%	-0.20%	-1.54%	-0.95%	+0.19%	-10.19%	0.00%	-11s	+1s	+3s	0s	+18s	+9s	0s
	100	-0.16%	+0.12%	-0.16%	-0.16%	+0.12%	-25.64%	+0.08%	-10s	+4s	+4s	0s	+18s	+9s	0s
	120	-0.06%	-0.10%	-0.24%	+0.14%	+0.16%	-8.16%	0.00%	-10s	+3s	+5s	0s	+18s	+9s	0s
10	80	-0.22%	-0.65%	+2.04%	+0.22%	+0.65%	-4.43%	+0.22%	-25s	+3s	+2s	0s	+43s	+10s	0s
	100	-0.23%	+0.12%	+0.15%	-0.26%	+0.20%	-30.82%	+0.09%	-22s	+3s	+6s	0s	+43s	+10s	0s
	120	+0.23%	-0.20%	+0.25%	-0.13%	-0.07%	-23.77%	-0.10%	-22s	+1s	+5s	0s	+43s	+11s	0s
50	80	+0.27%	+0.35%	+0.05%	+0.09%	-0.60%	-0.21%	-0.09%	-4m58s	+6s	+7s	+1s	+41m59s	+9s	0s
	100	+0.04%	+0.02%	+0.52%	+0.13%	-	-0.11%	0.00%	-4m20s	+7s	+9s	+1s	-	+11s	0s
	120	-0.21%	-0.06%	+0.04%	-0.01%	-	+0.11%	-0.04%	-4m28s	+4s	+10s	+1s	-	+10s	0s

Comparison Table 2 (Errors vs. Article)

d	Price (Relative % Error)							Duration (Absolute Difference)
	LSM	DOS	NLSM	RLSM	FQI	RFQI	EOP	LSM	DOS	NLSM	RLSM	FQI	RFQI	EOP
5	+0.12%	-0.24%	-0.12%	-0.36%	+0.12%	-5.64%	+0.12%	-31s	+9s	+5s	0s	+40s	+3s	0s
10	-0.34%	-0.25%	+0.17%	-0.25%	+0.25%	-23.33%	-0.08%	-1m25s	+11s	+3s	0s	+2m25s	+2s	0s
50	+0.06%	-0.45%	-0.75%	+0.26%	+0.20%	+0.20%	+0.25%	+1h7m37s	+28s	+10s	0s	+1h11m3s	+1s	0s
100	-	-0.38%	+0.14%	-0.22%	-	-0.04%	+0.08%	-	+22s	+25s	+1s	-	+3s	0s
500	-	-0.54%	-0.23%	-0.13%	-	-0.19%	+0.03%	-	+4m25s	+6m34s	+9m47s	-	+1m46s	0s

Analysis of Results (Tables 1 & 2)

Duration Performance:

LSM: Faster for small dimensions (saving 11 seconds to 1.5 minutes), but dramatically slower for d=50 (loses 5 minutes to over 1 hour). Computation time grows much more rapidly in my implementation, becoming unsustainable beyond d=50
FQI: Similar to LSM - becomes computationally prohibitive beyond d=50
RLSM: Consistently stays under 10 seconds in both implementations, with one exception at d=500 (~10 minutes due to high-dimensional matrix memory issues)
RFQI: Maintains sub-10 second performance across all dimensions in both implementations
DOS & NLSM: Moderately slower in my implementation across all dimensions, significantly slower for d=500
EOP: Consistently 0 seconds across all applications

Price Accuracy:

EOP (benchmark): Maximum absolute error 0.25%, typically 0-0.10% - excellent agreement validates the overall implementation
Most models (LSM, DOS, RLSM, FQI): Majority show absolute errors < 0.50%, which is good considering the stochastic nature and reported standard deviations
NLSM: Errors up to 2% for d < 10, but improves significantly for higher dimensions
RFQI: Shows 23-30% errors for d < 10, but dramatically improves to < 0.2% for d ≥ 50

Key Insight: Results confirm the authors' conclusion: RLSM should be used for lower dimensions (d < 10), while RFQI is optimal for higher dimensions (d ≥ 50) due to both accuracy and computational efficiency.

Statistical Context and Confidence Intervals

The authors' own standard deviations sometimes exceed 2% of the estimate for dimensions d < 10, becoming residual only for very large dimensions.

95% Confidence Interval Calculation:

With n=10 simulations per model, the 95% confidence interval is:

±1.96 × σ / √n = ±1.96 × 2% / √10 ≈ ±1.24%

Conclusion: In general, all model estimates (except RFQI for low dimensions) fall within the 95% confidence interval, indicating statistically acceptable replication.

Overall Conclusions

Key Findings:

Benchmark Accuracy: EOP results show excellent agreement (0-0.25% error), validating the overall implementation
Dimension-Dependent Performance:
- For d < 10: RLSM recommended (RFQI shows large errors)
- For d ≥ 50: RFQI recommended (excellent accuracy, fast computation)
Computational Challenges:
- LSM and FQI become computationally prohibitive beyond d=50
- RLSM and RFQI maintain reasonable computation times even for high dimensions
- My implementation is generally slower, particularly for LSM and FQI at high dimensions
Statistical Validity: Most replications fall within expected confidence intervals, considering the stochastic nature of the algorithms and standard deviations reported by the authors
Confirms Authors' Recommendations: The results strongly support the paper's conclusion that RLSM should be used for lower dimensions and RFQI for higher dimensions