Production Scanner — Going Live | Build a Stock Scanner — Part 6/6

The Production Scanner Architecture

You've built the scanner logic (filters, scoring, regime detection) and validated it with backtesting. Now it's time to wrap it in a production-grade pipeline that runs reliably every day without manual intervention. Here's the complete architecture.

Pipeline Stages

23:00

Daily Run Time

~15 min

Execution Time

99.5%

Target Uptime

The 5-Stage Pipeline

1 Data Ingestion

Pull market data from your data provider(s). This includes end-of-day prices, volume, fundamentals, insider transactions, options flow, and sentiment data. The ingestion layer handles API rate limits, retries, and data validation.

Price data: EOD quotes for entire universe (~3,000 symbols)
Fundamentals: Quarterly financials, earnings estimates
Sentiment: Short interest, analyst ratings, news sentiment
Market regime: VIX, breadth, yield curve, credit spreads

2 Regime Detection

Before running any scanner logic, determine the current market regime. This sets the parameters for everything that follows — which strategies to deploy, position sizing multipliers, and exposure limits.

Output: Regime label (Risk-On, Transition, Risk-Off, Crisis) + confidence level (High/Medium/Low).

3 Filter & Score

Apply the DSL filters (Part 2) to define the candidate universe, then score all candidates (Part 3) using the composite scoring engine. Apply boosts and penalties. This is the core scanner logic.

Output: Ranked list of all scored candidates with grades.

4 Selection & Validation

Select the top N candidates based on grade threshold and diversification rules. Validate each pick: check spot price vs calculated entry (must be within 10%), verify volume is still sufficient, and cross-check for any breaking news that might invalidate the thesis.

Output: Final shortlist of 5-10 A+ picks with entry, stop, and target levels.

5 Output & Alert

Generate the scan report (HTML or PDF), send alerts to configured channels (Discord, email, Telegram), and log all results for retrospective analysis. Archive the full scoring data for audit trail.

Output: Published scan report + notifications + historical log entry.

Fail-Safe Design

Production systems fail. APIs go down, data feeds have gaps, your server reboots. Design your pipeline with fail-safe defaults:

If data ingestion fails: Don't run the scanner. Better no picks than picks based on stale data.
If regime detection fails: Default to Transition (cautious) with low confidence (0.5x exposure).
If scoring produces < 3 A+ picks: Report as "no actionable setups" rather than lowering the bar.
If alerting fails: Log to local file and retry. Don't trade without reviewing the output.

The golden rule: when in doubt, do nothing. No trade is always better than a bad trade based on faulty data.

Daily Scheduling

Your scanner needs to run at the same time every day, reliably, without manual intervention. The timing matters — you want results ready for the next trading day's open.

Optimal Run Schedule

Time (ET)	Task	Duration	Notes
20:00	EOD data finalized	—	Most providers finalize by 8 PM ET
22:00	Data ingestion begins	5-10 min	Pull all required data sources
22:10	Regime detection + scanning	3-5 min	Full pipeline execution
22:15	Validation + report generation	2-3 min	Spot price check, HTML report build
22:20	Alerts sent	1 min	Discord/email/Telegram notifications
06:30	Pre-market review	15 min	Manual review of scanner output + overnight news
09:30	Market open — execute	—	Place orders based on reviewed picks

Why Run at Night?

Running at night (after market close) has three advantages: (1) EOD data is complete and finalized — no partial candles or delayed feeds. (2) You have the entire overnight period to review results and prepare orders before market open. (3) Intraday scanning generates excessive noise — daily timeframe signals are more reliable for swing trading setups.

Some scanners also run a pre-market refresh at 7 AM ET to check for overnight news that might invalidate setups (earnings miss, geopolitical event, pre-market gap). This is a quick validation pass, not a full re-scan.

Automation Options

Method	Complexity	Reliability	Cost	Best For
Cron job (Linux/Mac)	Low	High (local)	Free	Personal scanner on your machine
Task Scheduler (Windows)	Low	Medium	Free	Windows users
Cloud function (AWS Lambda)	Medium	Very high	$1-5/month	Reliable, serverless deployment
Docker + cron on VPS	Medium	High	$5-20/month	Full control, portable environment
Bot (Discord/Telegram)	High	Very high	$5-10/month	Team use, interactive commands
GitHub Actions	Low	High	Free (2000 min/mo)	Open-source scanners, CI/CD native

For most individual traders, a cron job on a local machine or a cloud function is sufficient. If you need team access or want interactive queries ("run scanner with custom filters"), a Discord/Telegram bot adds significant value. This is the approach DailyTickers uses — a Discord bot that triggers the scanner via scheduled commands, with the flexibility to run ad-hoc scans on demand.

Building a Reliable Data Pipeline

Your scanner is only as good as its data. A missing data point, a stale quote, or a bad fundamental value can produce a pick that looks great on paper but is based on fiction. The data pipeline is the foundation of production reliability.

Data Sources by Type

Data Type	Free Options	Paid Options	Update Frequency
EOD Prices	Yahoo Finance, Alpha Vantage	Polygon, IEX Cloud, Tiingo	Daily (after 8 PM ET)
Intraday Prices	Alpha Vantage (limited)	Polygon, Alpaca	1-min to 15-min bars
Fundamentals	SEC EDGAR, Financial Modeling Prep	Sharadar, Bloomberg	Quarterly (within days of filing)
Insider Transactions	SEC EDGAR (Form 4)	OpenInsider, Quiver Quant	Within 2 business days of trade
Options Flow	Limited free options	Unusual Whales, FlowAlgo	Real-time during market hours
Sentiment	StockTwits API	Sentifi, MarketPsych	Real-time
Market Regime	FRED (VIX, yield curve)	Most providers include this	Daily

Data Quality Checks

Every data ingestion run should pass through quality checks before the scanner processes it. These are your data circuit breakers:

Completeness: Are all symbols in the universe present? Missing symbols = missed opportunities or stale positions.
Freshness: Is the latest bar from today? Stale data produces yesterday's picks with today's label.
Range validation: RSI between 0-100? Price > 0? Volume > 0? Catches data feed corruption.
Outlier detection: A stock that moved +500% in a day is either wrong data or an extraordinary event. Flag for manual review.
Cross-validation: Compare price from two independent sources. If they diverge by more than 1%, something is wrong.

Data Quality Rule

If quality_score < 95% → Abort scan and alert operator

The Backup Data Source

Always have a secondary data source configured. If your primary provider goes down (it happens more often than you'd expect), the pipeline should automatically fall back to the secondary source rather than skipping the scan entirely. Log the fallback event for investigation the next day. The key metrics to verify on fallback: price accuracy (<0.5% divergence from primary) and universe coverage (>95% of symbols available).

Multi-Channel Alert System

Your scanner produces results. Now they need to reach you — and possibly your team or subscribers — through the right channels at the right time. The key principle is tiered urgency: not every alert deserves to wake you up at midnight.

Channel Comparison

Channel	Latency	Rich Content	Team Support	Setup Effort
Discord	< 1s	Embeds, charts, reactions	Excellent (channels, roles)	Medium (bot + webhook)
Telegram	< 1s	Good (Markdown, images)	Good (groups, channels)	Low (bot token + API)
Email	1-30s	Excellent (full HTML)	Limited	Low (SMTP or API)
SMS	< 5s	None (text only)	None	Medium (Twilio)
Web Dashboard	Real-time	Full (interactive charts)	Excellent	High (web app)

Alert Tiers

Tier	Channels	Examples	Frequency
Critical	SMS + Discord	Scanner failure, data quality issue, regime → Crisis	~2/month
High	Discord + Email	New scan results, A+ picks, regime change	Daily (scan nights)
Medium	Discord only	B+ picks, price approaching entry/stop levels	2-3x/week
Low	Dashboard only	Watchlist updates, performance stats, market commentary	Continuous

Discord Integration Architecture

Discord is the most popular choice for trading communities and automated scanners because it supports rich embeds, file attachments (for scan reports), role-based access, and real-time interaction. Here's what a well-structured Discord scanner setup looks like:

Channel	Purpose	Update Frequency
#scanner-daily	Daily scan results — top 10 picks with entry/stop/target	Once/day at 10:20 PM ET
#regime-alerts	Regime changes and confidence level updates	On regime change only
#trade-alerts	Entry/stop triggers for active picks during market hours	Real-time during market
#retrospective	Weekly performance review and scoring adjustments	Once/week (Friday)
#scanner-logs	System health, errors, debugging, data quality reports	Continuous

Keeping Your Scanner Healthy

A production scanner is a living system. It can degrade silently — data quality drops, a signal stops working, market structure changes. Without monitoring, you won't notice until your P&L tells you, and by then the damage is done.

Key Health Metrics

Metric	Normal Range	Warning	Critical
Data completeness	> 99%	95-99%	< 95% (abort scan)
Execution time	< 10 min	10-20 min	> 20 min (likely hanging)
Candidates scored	2,000-3,000	< 1,500 (data gap?)	< 500 (feed down)
A+ picks generated	2-6 per scan	0-1 or 10+ (calibration issue)	Always 0 or always 10+
Rolling 30-day hit rate	55-70%	45-55% (strategy degrading)	< 45% (broken — kill switch)
Avg score drift	Stable (mean ~62)	Mean shifting 5+ points	Scores clustering at extremes

When to Pull the Kill Switch

Stop the scanner and investigate if any of these occur:

Rolling 14-day hit rate drops below 40%
Three consecutive scans produce 0 A+ picks (unless Crisis regime)
Data completeness below 90% for 2+ consecutive days
Max drawdown exceeds backtest worst-case by 50%
Scanner picks correlate > 0.8 with simple SPY buy-and-hold (no alpha being generated)

Pulling the kill switch is not failure — it's prudent risk management. Better to pause for a week of debugging than to bleed capital into a broken system.

Debugging Common Issues

Symptom	Likely Cause	Diagnostic Step	Fix
All picks hitting stops	Regime mismatch	Check regime classifier accuracy	Recalibrate regime thresholds
Same 5 stocks every scan	Filters too narrow	Log candidate count at each filter	Widen filters or add signal types
Score inflation (many A+)	Correlated signals	Check signal independence matrix	Remove correlated signals
No picks found	Overly restrictive or data gap	Run each filter independently	Check data, relax weakest filter
Good picks, poor execution	Entry timing or slippage	Compare entry price vs signal price	Add limit orders or intraday logic
Decreasing hit rate over time	Market structure shift	Compare recent vs historical regime	Re-run walk-forward backtest

The Weekly Retrospective

The retrospective is the most important process in running a production scanner. It closes the feedback loop: the scanner generates picks, the market delivers outcomes, and you analyze the delta. Without retrospectives, your scanner is static — it will inevitably degrade as markets evolve.

Retrospective Structure (45 minutes weekly)

Section	Contents	Time
1. Results Dashboard	Hit rate, avg P&L, best/worst picks, profit factor for the week	5 min
2. Pick-by-Pick Review	For each pick: entry, current price, TP1 hit?, stop hit?, still open	15 min
3. Strategy Analysis	Hit rate by strategy type (momentum, value, breakout, short)	5 min
4. Regime Accuracy	Was the regime classifier correct? Did strategy allocation match?	5 min
5. Lessons & Adjustments	What worked, what didn't, parameter changes for next week	10 min
6. Forward Look	Regime outlook, earnings calendar, macro events, adjusted strategy	5 min

The 5 Rules of Retrospectives

Never skip it. Even when you're winning — especially when you're winning. Winning streaks breed complacency, and the retro catches brewing problems before they explode.
Be brutally honest. Don't rationalize losers ("it would have worked if not for that tweet"). The market is always right. Your job is to understand why, not to argue.
Track cumulative patterns. A single losing trade is noise. The same strategy losing 4 weeks in a row is a signal that demands action.
Don't over-adjust. If last week's retro says "tighten stops," don't widen them this week because one tight stop was premature. Wait for 3+ data points before making parameter changes.
Document everything. Every retro goes into a permanent log. In 6 months, you'll look back and understand exactly when and why your scanner evolved.

Cumulative Performance Tracking

Beyond weekly retros, track cumulative performance metrics that reveal the scanner's health over longer horizons:

Rolling 30-day hit rate — Should stay in the 55-65% corridor. Drift below 50% = investigate immediately.
Rolling 30-day Sharpe — Should stay above 1.0. Below 0.5 = the scanner is not generating alpha.
Hit rate by grade bracket — A+ should always outperform A, which outperforms B+. If this ordering breaks, scoring calibration is off.
Hit rate by regime — Monitors regime detection accuracy. If Risk-On hit rate matches Risk-Off hit rate, regime detection isn't working.
Unique tickers ratio — Are you scanning the whole market or recycling the same 20 names? Target: > 60% unique tickers month-over-month.

The Improvement Flywheel

A production scanner is never "done." Markets evolve, correlations shift, new data sources become available, and your own trading style matures. The improvement flywheel ensures your scanner keeps pace with these changes.

The Flywheel

Scan → Trade → Retrospective → Adjust → Backtest → Deploy → Repeat

A/B Testing Scanner Variants

When you want to test a change (new signal, different weight, adjusted threshold), don't just apply it globally. Run two variants in parallel and compare results over 30+ trading days.

Element	Variant A (Control)	Variant B (Test)
Description	Current production scanner	Production + new insider buying signal
Duration	30 trading days minimum	30 trading days minimum
Capital allocation	80% of normal	20% of normal (or paper trade)
Tracking	Hit rate, PF, Sharpe, MDD	Same metrics, tracked separately
Decision rule	Baseline performance	Promote if improvement > 10%

The 10% Improvement Threshold

Don't deploy a variant just because it's marginally better. Require a meaningful improvement (at least 10% better on key metrics) before promoting it to production. Small improvements may be noise — statistical fluctuations that don't represent genuine edge. Additionally, the variant must perform well across all walk-forward windows in a re-backtest, not just the recent 30-day test period.

Be especially cautious about adding complexity. A variant with 2 extra signals that improves hit rate by 3% but makes the scanner harder to maintain and debug is probably not worth deploying. Simplicity has compounding value in production systems — every additional dependency is a potential point of failure.

Scaling Considerations

As your scanner matures and proves its edge, you may want to scale along several dimensions. The critical rule: scale one dimension at a time with a 30-day validation period between changes.

Dimension	Starter	Intermediate	Advanced
Universe	US large-cap (500)	US all-cap + Europe (3,000)	Global + crypto (10,000+)
Strategies	1 (momentum only)	3 (momentum + value + breakout)	5+ with regime-based rotation
Run frequency	Daily (EOD)	Daily + pre-market refresh	Intraday (every 4 hours)
Team	Solo trader	2-3 person group	Full trading desk
Output	Email alert	Discord bot + dashboard	Full web app + API + published reports
Retrospective	Monthly (manual)	Weekly (semi-automated)	Daily (fully automated with exception alerts)

If you scale multiple dimensions simultaneously and performance degrades, you won't know which change caused the problem. One dimension at a time, with validation between each step, gives you a clear audit trail and easy rollback path.

Test Your Understanding

Quiz 1: Your data feed fails at 10 PM. What should the scanner do?

Answer: Abort the scan and alert the operator. Never run the scanner on stale or missing data. The fail-safe principle is "when in doubt, do nothing." Send a critical alert (SMS + Discord) so you can investigate. Try the backup data source if configured. If backup also fails, skip tonight's scan entirely. No scan is always better than a scan based on yesterday's data labeled as today's.

Quiz 2: Why run the scanner at night instead of during market hours?

Answer: Three reasons: (1) Data completeness — EOD bars are finalized after market close, eliminating partial candles and delayed feeds. (2) Review time — Running at night gives you the overnight period to review results, check overnight news, and prepare orders for market open. (3) Signal quality — Daily timeframe signals (RSI, MACD, volume) are far more reliable than intraday signals, which generate excessive noise and false positives for swing trading.

Quiz 3: Your scanner's rolling 30-day hit rate dropped from 62% to 48%. What do you do?

Answer: This crosses the warning-to-critical threshold. Systematic response: (1) Check the regime classifier — did the market change regimes without detection? (2) Audit data quality — are there feed issues degrading signal accuracy? (3) Review recent losers — look for patterns (same sector, same strategy type, same signal failing). (4) Pull the kill switch if you can't identify the root cause within 1-2 days. (5) Re-run walk-forward backtest with the most recent 6 months to check if the scanner's edge has structurally decayed. Never keep trading a broken system while you investigate.

Quiz 4: What are the 5 rules of a good retrospective?

Answer: (1) Never skip it — even when winning, especially when winning. (2) Be brutally honest — don't rationalize losers or attribute losses to bad luck. (3) Track cumulative patterns — one bad trade is noise; four weeks of the same strategy failing is a signal. (4) Don't over-adjust — wait for 3+ data points before changing parameters. Knee-jerk reactions introduce instability. (5) Document everything — every retro goes into a permanent log for long-term pattern analysis and institutional memory.

Quiz 5: You have a new insider buying signal that improved backtest results by 8%. Should you deploy it?

Answer: Not yet. An 8% improvement is below the 10% threshold for promoting a variant to production. Steps: (1) Run it as Variant B in a 30-day A/B test with 20% capital allocation (or paper trade). (2) Verify the improvement holds across all walk-forward windows, not just the recent period. (3) Assess operational complexity — does the new signal add a data dependency that could break the pipeline? Only promote if the improvement is sustained, statistically significant, and operationally manageable. A simple scanner that works reliably beats a complex scanner that breaks unpredictably.

Quiz 6: What's the danger of scaling multiple dimensions simultaneously?

Answer: Each scaling step (larger universe, more strategies, higher frequency, more team members) introduces new failure modes. If you expand the universe AND add new strategies AND increase frequency simultaneously, and performance degrades, you won't know which change caused the problem. This makes debugging nearly impossible and rollback ambiguous. Scale one dimension at a time with a 30-day validation period between changes. If something breaks, you know exactly what caused it and can surgically roll back that specific change.

Part 6 — Key Takeaways

5-stage pipeline — Data ingestion, regime detection, filter & score, selection & validation, output & alert. Each stage has quality checks and fail-safes.
Run at night, review in the morning — EOD data is clean, overnight gives review time, daily signals are more reliable than intraday.
Data quality is non-negotiable — If quality drops below 95%, abort the scan. Bad data produces bad picks. Always have a backup data source.
Multi-channel alerting with tiered urgency — Critical alerts via SMS, daily results via Discord, performance via dashboard. Tiered urgency prevents alert fatigue.
Monitor health continuously — Track data completeness, execution time, candidate count, rolling hit rate, and score distribution. Pull the kill switch when metrics breach critical thresholds.
Weekly retrospectives are mandatory — The feedback loop is what makes a scanner improve over time. Never skip it. Never rationalize losers.
A/B test before deploying changes — 30-day parallel test, 10% improvement threshold, verified across all walk-forward windows before promotion.
Scale one dimension at a time — Universe, strategies, frequency, team. Validate each expansion independently with 30-day checkpoints.

Series Complete — You Built a Scanner

Part 1 — Philosophy: The 3 pillars (universe, signals, ranking), the funnel model, and 7 common mistakes to avoid
Part 2 — DSL Filters: Expression-based filtering, boolean logic, reusable filter templates for momentum, value, and breakout strategies
Part 3 — Scoring: Weighted composite scores, normalization methods, grade thresholds, boosts & penalties, sector-relative scoring
Part 4 — Regime Detection: 4 market regimes, 6 indicator voting system, dynamic strategy rotation, asymmetric transition rules
Part 5 — Backtesting: Walk-forward optimization, 5 bias elimination techniques, 8 key metrics, the go/no-go checklist
Part 6 — Production: 5-stage pipeline, scheduling, data quality, multi-channel alerting, monitoring, retrospectives, continuous improvement

You now have the complete blueprint for a professional-grade stock scanner. The rest is execution. Start simple (Parts 1-2), add scoring (Part 3), integrate regime detection (Part 4), validate with backtesting (Part 5), and deploy to production (Part 6). Iterate weekly with retrospectives. Good trading.

Back to the Beginning

Part 1: Scanner Philosophy — What Makes a Great Scanner