sami7777/biggfish
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Scalping strategy overhaul: bidirectional trading, oil/JPY focus

Browse Source 
- Switch to oil stocks (USO, XLE, OXY, CVX, XOM, SLB, HAL, DVN, MPC, VLO)
- Add JPY/USD forex pairs for Japan targeting
- 7-action RL space: long, short, close (was 5 long-only actions)
- Bollinger Band mean-reversion scalp entries both directions
- 5-minute candles with 60-second cycles for scalping
- 35 features (added VWAP, fast RSI, fast ROC for scalping)
- Short position support in backtest, executor, and RL environment
- GA tuned for scalping: tighter SL/TP, shorter hold times

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
sami7777
2026-03-12 16:45:19 -07:00
commit 82c68fa8b0
51 changed files with 7971 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.claude/settings.local.json
+19
View File
@@ -0,0 +1,19 @@
{
"permissions": {
"allow": [
"Bash(python -c:*)",
"Bash(ssh:*)",
"Bash(scp:*)",
"WebFetch(domain:docs.alpaca.markets)",
"Bash(curl:*)",
"Bash(python -m json.tool:*)",
"Bash(python3:*)",
"WebFetch(domain:sag.sh)",
"WebFetch(domain:summarize.sh)",
"Bash(git init:*)",
"Bash(git remote add:*)",
"Bash(git remote set-url:*)",
"Bash(git add:*)"
]
}
}
.gitignore
+51
View File
@@ -0,0 +1,51 @@
# Configuration
config/config.json
# Python
__pycache__/
*.py[cod]
*$py.class
*.so
.Python
env/
venv/
ENV/
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
wheels/
*.egg-info/
.installed.cfg
*.egg
# Logs
logs/
*.log
# Data
data/
*.db
*.sqlite
# IDE
.vscode/
.idea/
*.swp
*.swo
*~
# OS
.DS_Store
Thumbs.db
# Environment
.env
.env.local
BUILD_SUMMARY.md
+288
View File
@@ -0,0 +1,288 @@
# 🐟 BIGGFISH - Build Complete!
## What We Built
A complete autonomous stock trading system designed to turn $100 into $1,000 in 2 months through intelligent paper trading.
## Project Stats
```
Total Files Created: 25+
Lines of Code: 2,500+
Python Modules: 7
Documentation Pages: 5
Development Time: 1 session
Status: ✅ READY TO USE
```
## Core Components
### 1. Trading Engine ✅
**File**: `src/trading/broker.py` (210 lines)
- Alpaca API integration
- Paper trading (hardcoded safety)
- Order execution (market & limit)
- Portfolio tracking
- Historical data fetching
### 2. Research Module ✅
**File**: `src/research/screener.py` (240 lines)
- 50+ stock universe (small/mid/large caps)
- Multi-factor scoring system
- Volume surge detection
- Momentum analysis
- Volatility filtering
### 3. Strategy Engine ✅
**File**: `src/strategies/manager.py` (180 lines)
- 3 strategy types (momentum, mean reversion, swing)
- Automatic position sizing
- Risk/reward calculation
- Stop loss & target setting
- Approval workflow
### 4. Reporting System ✅
**File**: `src/reporting/reporter.py` (150 lines)
- Daily performance reports
- Strategy proposals
- Progress tracking
- Goal visualization
- File-based logging
### 5. Main Orchestrator ✅
**File**: `src/main.py` (180 lines)
- System initialization
- Scheduled tasks
- Trading cycles
- Market hours awareness
- Error handling
### 6. CLI Tools ✅
**File**: `src/cli.py` (150 lines)
- Portfolio status command
- Market scanning
- Strategy generation
- Market hours check
- Manual control
### 7. Configuration System ✅
**File**: `config/config.example.json`
- Alpaca API settings
- Risk parameters
- Portfolio progression rules
- Research intervals
- Reporting preferences
## Documentation
1. **README.md** - Project overview and architecture
2. **SETUP.md** - Detailed setup instructions
3. **QUICKSTART.md** - 5-minute getting started guide
4. **PROJECT_OVERVIEW.md** - Comprehensive system documentation
5. **BUILD_SUMMARY.md** - This file
## Features Implemented
### Trading Features
- ✅ Paper trading on Alpaca
- ✅ Market & limit orders
- ✅ Position tracking
- ✅ Portfolio management
- ✅ Automatic stop losses
- ✅ Risk-based position sizing
### Research Features
- ✅ Multi-timeframe scanning
- ✅ Volume analysis
- ✅ Momentum detection
- ✅ Volatility filtering
- ✅ Scoring system (0-100)
- ✅ Multi-cap universe (small/mid/large)
### Strategy Features
- ✅ Momentum breakout strategy
- ✅ Mean reversion strategy
- ✅ Swing trading strategy
- ✅ Automatic entry/exit calculation
- ✅ Risk/reward optimization
- ✅ Approval gate
### Safety Features
- ✅ Paper trading lock
- ✅ Position limits (20% max)
- ✅ Daily loss limit (5%)
- ✅ Total loss limit (15%)
- ✅ Approval required for trades
- ✅ Stop losses on all positions
- ✅ Market hours enforcement
### Automation Features
- ✅ Scheduled scanning (every 6h)
- ✅ News monitoring (every 2h)
- ✅ Daily reports (4:30 PM)
- ✅ Continuous operation
- ✅ Error recovery
### Reporting Features
- ✅ Daily performance reports
- ✅ Strategy proposals
- ✅ Progress tracking
- ✅ Position summaries
- ✅ JSON data export
- ✅ Goal visualization
## Technology Stack
**Core**:
- Python 3.9+
- Alpaca API (paper trading)
- yfinance (market data)
- pandas (data analysis)
**Trading**:
- alpaca-py 0.8.2
- pandas 2.1.4
- numpy 1.26.2
**Analysis**:
- yfinance 0.2.35
- ta 0.11.0
- pandas-ta 0.3.14b0
**Utilities**:
- schedule 1.2.0
- loguru 0.7.2
- python-dotenv 1.0.0
## Project Structure
```
biggfish/
├── src/
│ ├── main.py # Main system orchestrator
│ ├── cli.py # Command-line interface
│ ├── trading/
│ │ └── broker.py # Alpaca integration
│ ├── research/
│ │ └── screener.py # Stock screening
│ ├── strategies/
│ │ └── manager.py # Strategy generation
│ └── reporting/
│ └── reporter.py # Reports & notifications
├── config/
│ └── config.example.json # Configuration template
├── data/
│ ├── stocks/ # Stock data cache
│ ├── reports/ # Daily reports
│ └── trades/ # Trade history
├── logs/ # System logs
├── tests/ # Unit tests (TODO)
├── README.md
├── SETUP.md
├── QUICKSTART.md
├── PROJECT_OVERVIEW.md
├── requirements.txt
└── verify.sh
```
## What's Next?
### Immediate (You Need to Do)
1. Get Alpaca paper trading API keys (free)
2. Run `./verify.sh` to check installation
3. Copy config and add your keys
4. Install dependencies
5. Run first scan
### Phase 1 (Weeks 1-2)
- [ ] Execute first trades
- [ ] Monitor daily performance
- [ ] Refine screening parameters
- [ ] Track win rate
### Phase 2 (Weeks 3-4)
- [ ] Add Telegram integration
- [ ] Implement news sentiment
- [ ] Enhance strategy engine
- [ ] Add backtesting
### Phase 3 (Weeks 5-8)
- [ ] ML pattern recognition
- [ ] Multi-timeframe analysis
- [ ] Sector rotation
- [ ] Options strategies (if successful)
## How to Use
```bash
# 1. Verify installation
cd /workspace/extra/repos/biggfish
./verify.sh
# 2. Setup config
cp config/config.example.json config/config.json
nano config/config.json # Add your Alpaca keys
# 3. Install dependencies
pip install -r requirements.txt
# 4. Test connection
python src/cli.py market
# 5. Run first scan
python src/cli.py scan
# 6. Check portfolio
python src/cli.py status
# 7. Start the system
python src/main.py
```
## Success Criteria
**Technical**:
- ✅ System builds without errors
- ✅ Alpaca connection works
- ✅ Stock screening functions
- ✅ Strategies generate correctly
- ✅ Reports save properly
**Trading**:
- 🎯 $100 → $1,000 in 8 weeks
- 🎯 >60% win rate
- 🎯 Average gain >10% per trade
- 🎯 Max drawdown <15%
- 🎯 Consistent daily activity
## Important Notes
⚠️ **PAPER TRADING ONLY** - This is an experimental system. Do not use with real money without extensive testing.
⚠️ **Not Financial Advice** - This is a learning project. You are responsible for any trading decisions.
⚠️ **High Risk** - Small cap stocks are volatile. Even in paper trading, expect significant swings.
✅ **Safe to Experiment** - Paper trading means zero real-world risk. Perfect for learning!
## Support
- 📖 **Documentation**: See SETUP.md and QUICKSTART.md
- 🔍 **Debugging**: Check `logs/biggfish_*.log`
- 💬 **Questions**: Review PROJECT_OVERVIEW.md
## Credits
**Built By**: Nanoclaw (Claude AI)
**Built For**: Sami
**Purpose**: Autonomous stock trading experiment
**Goal**: $100 → $1,000 in 2 months
**Method**: Research-driven, risk-managed paper trading
---
## Let's Go! 🐟🚀
Everything is ready. Time to catch that BIGGFISH!
**Next Step**: `./verify.sh` then `python src/main.py`
PROJECT_OVERVIEW.md
+278
View File
@@ -0,0 +1,278 @@
# 🐟 BIGGFISH Project Overview
## Mission
Turn $100 into $1,000 in 2 months through intelligent, autonomous stock trading.
## Core Philosophy
- **Start Small, Scale Smart**: Begin with small cap momentum plays, gradually transition to blue chips
- **Research-Driven**: Continuous market analysis and learning
- **Risk-Managed**: Strict position limits and stop losses
- **Transparent**: Daily reports and strategy proposals
- **Paper Trading**: Zero real-world risk during development
## System Architecture
### 1. Trading Engine (`src/trading/broker.py`)
**Purpose**: Interface with Alpaca Markets for paper trading
**Capabilities**:
- Connect to Alpaca paper trading API
- Execute market and limit orders
- Track portfolio value and positions
- Monitor account status and buying power
- Get historical price data
**Safety Features**:
- Always uses paper trading (hardcoded)
- Position size limits enforced
- Market hours checking
### 2. Research Module (`src/research/screener.py`)
**Purpose**: Scan markets and identify trading opportunities
**Stock Universe**:
- **Small Caps** (~30 tickers): High growth potential, higher volatility
- **Mid Caps** (~10 tickers): Balanced growth and stability
- **Large Caps** (~10 tickers): Blue chips for stability
**Screening Criteria**:
- Volume surge detection (>1.5x average = bullish)
- Price momentum (weekly/monthly trends)
- Volatility analysis (sweet spot: 2-5%)
- Minimum price filter ($2+, avoid penny stocks)
- Minimum volume filter (500K+ daily)
**Scoring System** (0-100):
- Base: 50 points
- Momentum bonus: +10 to +15 points
- Volume surge: +10 to +20 points
- Volatility: +10 if optimal, -10 if excessive
- Penalties for low price/volume
### 3. Strategy Engine (`src/strategies/manager.py`)
**Purpose**: Generate trading strategies from opportunities
**Strategy Types**:
1. **Momentum Breakout**
- Trigger: Volume surge >1.5x + price momentum >3%
- Target: 15% gain
- Stop Loss: 7%
- Best for: Strong trending stocks
2. **Mean Reversion**
- Trigger: Recent pullback + low volatility
- Target: 10% gain
- Stop Loss: 5%
- Best for: Oversold quality stocks
3. **Swing Trade**
- Trigger: General opportunity
- Target: 12% gain
- Stop Loss: 6%
- Best for: Mixed signals
**Risk Management**:
- Max position size: 20% of portfolio
- Max cash per trade: 30%
- Risk/Reward calculated for each trade
- Stop losses automatically set
### 4. Reporting System (`src/reporting/reporter.py`)
**Purpose**: Track performance and communicate insights
**Daily Reports Include**:
- Portfolio value and cash position
- Day P/L (profit/loss)
- Goal progress ($100 → $1,000)
- Open positions with P/L
- Performance metrics
**Strategy Proposals Include**:
- Entry, target, and stop prices
- Position size and risk
- Detailed rationale
- Technical signals
## Trading Progression
### Phase 1: Small Cap Focus ($100 → $200)
- **Timeframe**: Weeks 1-2
- **Universe**: 100% small caps
- **Strategy**: Aggressive momentum plays
- **Goal**: Double initial capital through high-volatility winners
### Phase 2: Mid Cap Mixed ($200 → $400)
- **Timeframe**: Weeks 3-4
- **Universe**: 80% small, 20% mid caps
- **Strategy**: Balance momentum with stability
- **Goal**: Consistent gains with reduced risk
### Phase 3: Diversified ($400 → $700)
- **Timeframe**: Weeks 5-6
- **Universe**: 60% small, 30% mid, 10% large caps
- **Strategy**: Portfolio diversification
- **Goal**: Protect gains while growing
### Phase 4: Balanced ($700 → $1,000)
- **Timeframe**: Weeks 7-8
- **Universe**: 40% small, 30% mid, 30% large caps
- **Strategy**: Capital preservation with selective opportunities
- **Goal**: Cross $1,000 finish line safely
## Configuration System
All parameters in `config/config.json`:
```json
{
"trading": {
"initial_capital": 100,
"target_capital": 1000,
"max_position_size_pct": 20, // Max 20% per position
"max_daily_loss_pct": 5, // Stop trading if -5% in a day
"max_total_loss_pct": 15, // Emergency brake at -15%
"require_approval": true // Human approval required
},
"research": {
"screening_interval_hours": 6, // Scan every 6 hours
"news_check_interval_hours": 2, // News every 2 hours
"max_watchlist_size": 50,
"min_volume": 500000,
"min_price": 2.0
}
}
```
## Automation Features
**Scheduled Tasks**:
- ✅ Market scanning every 6 hours
- ✅ News monitoring every 2 hours
- ✅ Daily reports at 4:30 PM ET
- ✅ Automatic strategy generation
**Manual Control**:
- Strategy approval/rejection
- Emergency stop
- Parameter adjustments
- Manual trade execution
## Safety Mechanisms
1. **Paper Trading Lock**: Hardcoded to use paper API
2. **Position Limits**: Max 20% of portfolio per position
3. **Daily Circuit Breaker**: Stop if -5% in one day
4. **Total Loss Limit**: Emergency stop at -15% total loss
5. **Approval Gate**: All strategies require human approval
6. **Stop Losses**: Automatic stops on every position
7. **Market Hours**: Only trade during market hours
## Data Storage
```
data/
├── stocks/ # Stock data cache
├── reports/ # Daily performance reports (JSON)
└── trades/ # Trade history and logs
```
## CLI Tools
```bash
# Real-time portfolio status
python src/cli.py status
# Scan for opportunities
python src/cli.py scan [--focus small_cap|mid_cap_mixed|balanced]
# Generate strategies
python src/cli.py strategies
# Check market hours
python src/cli.py market
```
## Workflow Example
**Morning** (9:00 AM):
1. System wakes up, checks if market is open
2. Runs initial scan of small cap universe
3. Generates 3-5 strategy proposals
4. Sends proposals to you via Telegram/logs
**You Review** (9:30 AM):
- Review proposals: "SOUN momentum breakout looks good ✅"
- Approve or reject each strategy
- System executes approved trades
**Midday** (12:00 PM):
- System checks news for holdings
- Monitors positions against stop losses
- No new scans (next scan at 3 PM)
**Afternoon** (3:00 PM):
- Second scan of the day
- May generate new proposals for next day
**Market Close** (4:30 PM):
- Daily report generated
- Shows: portfolio value, P/L, goal progress
- Highlights: best/worst performers
- Tomorrow: strategy preview
## Success Metrics
**Week 1-2**: $100 → $200 (100% gain)
- Minimum 3 profitable trades
- Max 2 losses
- Average gain per winner: 15%+
**Week 3-4**: $200 → $400 (100% gain)
- Consistent 10%+ weekly gains
- Diversification into mid caps
- Reduced volatility
**Week 5-6**: $400 → $700 (75% gain)
- Blue chips added for stability
- Portfolio beta reduction
- Risk-adjusted returns optimized
**Week 7-8**: $700 → $1,000 (43% gain)
- Capital preservation mode
- Selective high-confidence plays
- Goal achievement
## Future Enhancements
**Phase 2 Features** (after reaching $1,000):
- [ ] ML-based pattern recognition
- [ ] Sentiment analysis from news/social
- [ ] Options trading strategies
- [ ] Backtesting engine
- [ ] Multi-timeframe analysis
- [ ] Sector rotation strategies
- [ ] Earnings play automation
**Integration Ideas**:
- Telegram bot for mobile approval
- Discord/Slack notifications
- Web dashboard for monitoring
- Real-time alerts for big moves
## Risk Disclaimer
This is an **experimental system** operating in **paper trading mode**.
- ⚠️ Not financial advice
- ⚠️ Past performance ≠ future results
- ⚠️ High risk strategies used
- ⚠️ Do not use with real money without extensive testing
## Getting Started
See `SETUP.md` for detailed setup instructions.
---
**Let's go catch that BIGGFISH! 🐟🚀**
QUICKSTART.md
+227
View File
@@ -0,0 +1,227 @@
# 🐟 BIGGFISH Quick Start
Get up and running in 5 minutes!
## TL;DR
```bash
cd /workspace/extra/repos/biggfish
# 1. Verify installation
./verify.sh
# 2. Copy config
cp config/config.example.json config/config.json
# 3. Add your Alpaca API keys to config.json
# Get free paper trading keys at: https://alpaca.markets/
# 4. Install dependencies
pip install -r requirements.txt
# 5. Test it works
python src/cli.py market
# 6. Run your first scan
python src/cli.py scan
# 7. Start the system
python src/main.py
```
## What Happens Next?
### When You Run `python src/main.py`:
1. **System Initializes** 🚀
- Connects to Alpaca (paper trading)
- Loads your $100 starting balance
- Begins monitoring
2. **First Scan** (immediate)
- Scans 30+ small cap stocks
- Scores each based on momentum, volume, volatility
- Identifies top 5-10 opportunities
3. **Strategy Generation** 🧠
- Creates 3-5 trading strategies
- Calculates entry, target, and stop prices
- Determines position sizes (max 20% per position)
- Generates detailed rationale
4. **Awaiting Your Approval** ⏳
- Strategies are logged and saved
- System waits for you to approve/reject
- No trades executed without approval
5. **Ongoing Monitoring** 👀
- Scans market every 6 hours
- Checks news every 2 hours
- Daily report at 4:30 PM ET
- Continuous learning and adaptation
## Your First Trade
1. **Review Strategies**
```bash
python src/cli.py strategies
```
2. **You'll see something like**:
```
Strategy #1: Momentum Breakout - SOUN
BUY 15 shares @ $5.50
Target: $6.33 (+15.0%)
Stop: $5.12 (-7.0%)
Position Size: $82.50
Score: 85/100
Rationale:
📊 Signals:
• Score: 85/100
• 1W Change: +8.2%
• Volume Surge: 2.3x
• Volatility: 3.4%
• Sector: Technology
🚀 Strong volume surge with positive momentum suggests breakout potential.
```
3. **Approve It** (in future version with Telegram bot)
- For now, strategies are logged for your review
- Manual execution via CLI coming soon
## CLI Commands
```bash
# Check portfolio value and positions
python src/cli.py status
# Scan for opportunities (default: small caps)
python src/cli.py scan
# Scan with different focus
python src/cli.py scan --focus mid_cap_mixed
# Generate fresh strategies
python src/cli.py strategies
# Check if market is open
python src/cli.py market
```
## Understanding the Output
### Portfolio Status
```
💰 Portfolio Value: $100.00
💵 Cash: $100.00
📊 Positions Value: $0.00
📈 Day P/L: $0.00 (+0.00%)
🎯 Goal Progress: $100.00 / $1000.00 (10.0%)
[██░░░░░░░░░░░░░░░░░░] 10.0%
```
### Opportunity Scan
```
1. SOUN - Score: 85/100
Price: $5.50 | 1W: +8.2%
Volume Surge: 2.3x | Volatility: 3.4%
Sector: Technology
```
**Score Meaning**:
- 90-100: Exceptional setup
- 80-89: Strong opportunity
- 70-79: Good opportunity
- 60-69: Acceptable
- <60: Filtered out
## Progression Path
**Week 1**: Learn the system, make first trades
- Goal: $100 → $150 (50% gain)
- Focus: Understanding signals
- Trades: 3-5 small positions
**Week 2**: Gain confidence
- Goal: $150 → $225 (50% gain)
- Focus: Pattern recognition
- Trades: Start increasing position sizes
**Week 3-4**: Scale up
- Goal: $225 → $400 (78% gain)
- Focus: Consistency
- Trades: Add mid caps to mix
**Week 5-6**: Diversify
- Goal: $400 → $700 (75% gain)
- Focus: Risk management
- Trades: Add blue chips
**Week 7-8**: Final push
- Goal: $700 → $1,000 (43% gain)
- Focus: Capital preservation
- Trades: Selective, high-confidence only
## Key Files to Know
- `config/config.json` - All settings
- `logs/biggfish_*.log` - System logs
- `data/reports/report_*.json` - Daily reports
- `src/cli.py` - Command line interface
- `src/main.py` - Main system
## Common Questions
**Q: Is this safe?**
A: Yes! It's 100% paper trading. No real money involved.
**Q: Do I need to approve every trade?**
A: Yes, `require_approval: true` in config. Change to `false` for full automation (not recommended at first).
**Q: What if I lose money?**
A: System has safety limits:
- Max 5% loss per day (circuit breaker)
- Max 15% total loss (emergency stop)
- Stop losses on every position
**Q: Can I run this 24/7?**
A: Stock market is only open Mon-Fri 9:30 AM - 4:00 PM ET. System will wait when market is closed.
**Q: How do I stop it?**
A: Press `Ctrl+C` in the terminal
**Q: Where are the reports?**
A: Check `data/reports/` directory
## Troubleshooting
**"Invalid API credentials"**
→ Check config.json has correct Alpaca keys
**"Market is closed"**
→ Normal! Wait for market hours
**No opportunities found**
→ Try different focus: `--focus mid_cap_mixed`
**Module not found**
→ Run `pip install -r requirements.txt`
## Ready to Go?
```bash
# One command to verify everything
./verify.sh
# Then start trading
python src/main.py
```
**Let's catch that BIGGFISH! 🐟🚀**
---
Need help? Check the logs: `tail -f logs/biggfish_*.log`
README.md
+65
View File
@@ -0,0 +1,65 @@
# 🐟 BIGGFISH - Autonomous Stock Trading System
**Goal:** Turn $100 → $1,000 in 2 months through intelligent paper trading
## Overview
BIGGFISH is an AI-powered autonomous trading system that:
- Starts with small cap stocks, gradually transitions to blue chips
- Continuously researches and learns market patterns
- Proposes strategies for approval before execution
- Provides daily performance reports and insights
- Operates on Alpaca paper trading (zero risk)
## Architecture
### Core Components
1. **Trading Engine** (`src/trading/`)
- Alpaca API integration
- Order execution
- Position management
- Portfolio tracking
2. **Research Module** (`src/research/`)
- Market data analysis
- Small cap screening
- News sentiment analysis
- Pattern recognition
- Continuous learning
3. **Strategy Engine** (`src/strategies/`)
- Strategy development
- Backtesting
- Risk assessment
- Performance optimization
4. **Reporting System** (`src/reporting/`)
- Daily performance reports
- Trade logs
- Strategy proposals
- Learning insights
## Trading Rules
- ✅ Paper trading only (Alpaca)
- ✅ Start with small caps, move to blue chips as portfolio grows
- ✅ All strategies require approval before execution
- ✅ Daily reports and transparency
- ✅ Risk limits enforced programmatically
- ✅ Continuous research and adaptation
## Setup
```bash
cd /workspace/extra/repos/biggfish
pip install -r requirements.txt
cp config/config.example.json config/config.json
# Add your Alpaca paper trading API keys to config.json
python src/main.py
```
## Goal Timeline
- **Week 1-2:** Small cap momentum plays ($100 → $200)
- **Week 3-4:** Diversify into mid caps ($200 → $400)
- **Week 5-6:** Add blue chip positions ($400 → $700)
- **Week 7-8:** Balanced portfolio ($700 → $1,000)
## Status
🚧 **In Development** - Building initial components
SETUP.md
+126
View File
@@ -0,0 +1,126 @@
# 🐟 BIGGFISH Setup Guide
## Prerequisites
- Python 3.9+
- Alpaca account (free paper trading)
## Step 1: Get Alpaca API Keys
1. Go to [Alpaca](https://alpaca.markets/) and create a free account
2. Navigate to your dashboard
3. Generate **Paper Trading** API keys (NOT live trading!)
4. Save your API Key and Secret Key
## Step 2: Install Dependencies
```bash
cd /workspace/extra/repos/biggfish
pip install -r requirements.txt
```
## Step 3: Configure BIGGFISH
```bash
# Copy example config
cp config/config.example.json config/config.json
# Edit config.json and add your Alpaca keys
nano config/config.json
```
Update these fields:
```json
{
"alpaca": {
"api_key": "YOUR_ALPACA_PAPER_API_KEY",
"secret_key": "YOUR_ALPACA_PAPER_SECRET_KEY",
"base_url": "https://paper-api.alpaca.markets"
}
}
```
## Step 4: Test the Connection
```bash
python src/cli.py market
```
You should see market status. If you get an error, check your API keys.
## Step 5: Run Your First Scan
```bash
python src/cli.py scan
```
This will scan for small cap opportunities and show you the top candidates.
## Step 6: Check Portfolio Status
```bash
python src/cli.py status
```
## Step 7: Start the Trading System
```bash
python src/main.py
```
The system will:
- ✅ Scan for opportunities every 6 hours
- ✅ Check news every 2 hours
- ✅ Generate daily reports at 4:30 PM
- ✅ Propose strategies for your approval
- ✅ Execute approved trades
## CLI Commands
```bash
# Show portfolio status
python src/cli.py status
# Scan for opportunities
python src/cli.py scan
python src/cli.py scan --focus mid_cap_mixed
# Generate strategies
python src/cli.py strategies
# Check market hours
python src/cli.py market
```
## Next Steps
1. **Review Daily Reports** - Check `data/reports/` for performance tracking
2. **Approve Strategies** - When strategies are proposed, review and approve them
3. **Monitor Performance** - Track progress toward the $100 → $1,000 goal
4. **Adjust Configuration** - Fine-tune risk parameters in `config/config.json`
## Safety Features
- ✅ **Paper Trading Only** - No real money at risk
- ✅ **Position Limits** - Max 20% per position
- ✅ **Daily Loss Limits** - Max 5% daily loss
- ✅ **Total Loss Limits** - Max 15% total loss
- ✅ **Approval Required** - All strategies need approval before execution
## Troubleshooting
### "Invalid API credentials"
- Double-check your API keys in config.json
- Make sure you're using PAPER trading keys, not live keys
### "Market is closed"
- Stock market is only open Mon-Fri, 9:30 AM - 4:00 PM ET
- System will wait for market to open
### No opportunities found
- This is normal - the screener is selective
- Try different focus modes (small_cap, mid_cap_mixed, balanced)
- Market conditions may not be favorable
## Support
Check the logs in `logs/` for detailed error messages.
config/auto_config.json
+87
View File
@@ -0,0 +1,87 @@
{
"alpaca": {
"api_key": "PKIJPFNMNZ3YKYP765XD6ZPPJY",
"secret_key": "42PuPEYG2nGbeMJiogiFKKLyPtkEHKtwwXJamRKpf4tL",
"base_url": "https://paper-api.alpaca.markets"
},
"oanda": {
"api_token": "860db6509bc2f430b0cbfe197012a628-310ea40d575131302e6a30c958260837",
"account_id": "101-001-38661051-001",
"practice": true
},
"trading": {
"symbols": [
"USO", "XLE", "OXY", "CVX", "XOM",
"SLB", "HAL", "DVN", "MPC", "VLO"
],
"forex_symbols": [
"USD_JPY", "EUR_JPY", "GBP_JPY", "CAD_JPY",
"AUD_JPY", "EUR_USD", "GBP_USD", "USD_CAD"
],
"cycle_interval_seconds": 60,
"initial_capital": 100,
"target_capital": 1000,
"commission_rate": 0.0,
"min_trade_value": 3,
"require_approval": false,
"max_position_pct": 12,
"max_concurrent_positions": 12
},
"safety": {
"max_position_pct": 12,
"max_concurrent_positions": 12,
"max_daily_trades": 50,
"max_daily_loss_pct": 4,
"max_total_loss_pct": 15,
"min_trade_value": 3,
"initial_capital": 100
},
"rl": {
"gamma": 0.97,
"epsilon_start": 1.0,
"epsilon_min": 0.08,
"epsilon_decay": 0.9990,
"learning_rate": 0.0005,
"batch_size": 128,
"memory_size": 100000,
"target_update_freq": 50,
"hidden_dim": 128,
"live_epsilon": 0.12,
"train_interval_hours": 1,
"checkpoint_interval_hours": 1
},
"ga": {
"population_size": 60,
"elite_count": 8,
"mutation_rate": 0.20,
"mutation_strength": 0.25,
"crossover_rate": 0.7,
"tournament_size": 5,
"evolution_interval_hours": 3,
"generations_per_cycle": 15
},
"backtest": {
"interval_seconds": 900,
"lookback_days": 14,
"initial_capital": 100
},
"cache": {
"warmup_lookback_days": 60,
"update_interval_seconds": 60,
"timeframes": ["5m", "1h"]
},
"database": {
"path": "data/biggfish.db"
},
"reporting": {
"dashboard_interval_seconds": 60,
"save_to_file": true
},
"telegram": {
"enabled": true,
"bot_token": "8499352620:AAGVxZ2krfHS219xGRb-1-yGthMs45GjNg4",
"chat_id": "637130179",
"daily_report_hour": 21,
"trade_alerts": true
}
}
config/config.example.json
+37
View File
@@ -0,0 +1,37 @@
{
"alpaca": {
"api_key": "YOUR_ALPACA_PAPER_API_KEY",
"secret_key": "YOUR_ALPACA_PAPER_SECRET_KEY",
"base_url": "https://paper-api.alpaca.markets"
},
"trading": {
"initial_capital": 100,
"target_capital": 1000,
"max_position_size_pct": 20,
"max_daily_loss_pct": 5,
"max_total_loss_pct": 15,
"require_approval": true
},
"portfolio": {
"small_cap_threshold": 2000000000,
"mid_cap_threshold": 10000000000,
"initial_focus": "small_cap",
"transition_rules": {
"at_200": "80% small, 20% mid",
"at_400": "60% small, 30% mid, 10% large",
"at_700": "40% small, 30% mid, 30% large"
}
},
"research": {
"screening_interval_hours": 6,
"news_check_interval_hours": 2,
"max_watchlist_size": 50,
"min_volume": 500000,
"min_price": 2.0
},
"reporting": {
"daily_report_time": "16:30",
"telegram_enabled": true,
"save_to_file": true
}
}
config/strategies.json
+56
View File
@@ -0,0 +1,56 @@
{
"strategies": [
{
"name": "example-crypto-rebalance",
"description": "Rebalance crypto portfolio every 6 hours",
"type": "rebalance",
"assetType": "crypto",
"exchange": "binance",
"schedule": "0 */6 * * *",
"allocations": {
"BTC/USDT": 40,
"ETH/USDT": 30,
"SOL/USDT": 20,
"USDT": 10
},
"threshold": 5,
"enabled": false
},
{
"name": "example-stock-dca",
"description": "Weekly DCA into index funds",
"type": "dca",
"assetType": "stock",
"broker": "alpaca",
"schedule": "0 10 * * 1",
"investments": [
{
"symbol": "SPY",
"amount": 100
},
{
"symbol": "QQQ",
"amount": 50
}
],
"enabled": false
},
{
"name": "example-conditional-buy",
"description": "Buy BTC when price drops 10%",
"type": "conditional",
"assetType": "crypto",
"exchange": "binance",
"conditions": {
"symbol": "BTC/USDT",
"priceDropPercent": 10,
"timeframe": "24h"
},
"action": {
"type": "buy",
"amount": 100
},
"enabled": false
}
]
}
requirements.txt
+32
View File
@@ -0,0 +1,32 @@
# Core Trading
alpaca-py==0.8.2
pandas==2.1.4
numpy==1.26.2
# Market Data & Analysis
yfinance==0.2.35
ta==0.11.0
pandas-ta==0.3.14b0
# News & Sentiment
feedparser==6.0.10
requests==2.31.0
beautifulsoup4==4.12.2
# ML & Research
scikit-learn==1.3.2
scipy==1.11.4
# Deep Learning (RL Agent)
torch>=2.0.0
# Scheduling
apscheduler>=3.10.0
# Utilities
python-dotenv==1.0.0
schedule==1.2.0
pytz==2023.3
# Logging & Monitoring
loguru==0.7.2
scripts/krystie/read-events.sh
+2
View File
@@ -0,0 +1,2 @@
#!/bin/bash
python3 /root/.openclaw/agents/main/workspace/skills/biggfish/scripts/read_events.py
scripts/krystie/read-status.sh
+2
View File
@@ -0,0 +1,2 @@
#!/bin/bash
python3 /root/.openclaw/agents/main/workspace/skills/biggfish/scripts/read_status.py
scripts/krystie/read-trades.sh
+2
View File
@@ -0,0 +1,2 @@
#!/bin/bash
python3 /root/.openclaw/agents/main/workspace/skills/biggfish/scripts/read_trades.py "$@"
scripts/krystie/read_events.py
+48
View File
@@ -0,0 +1,48 @@
#!/usr/bin/env python3
"""Read BIGGFISH recent events for Krystie."""
import json, sys
EVENTS_FILE = "/opt/biggfish/src/data/krystie-events.json"
try:
with open(EVENTS_FILE) as f:
data = json.load(f)
except FileNotFoundError:
print("No events file found. BIGGFISH may not have generated any events yet.")
sys.exit(1)
events = data.get("events", [])
if not events:
print("No events recorded yet.")
sys.exit(0)
print("=== BIGGFISH EVENTS (last {}) ===".format(len(events)))
print()
for e in reversed(events[-20:]):
t = e.get("time", "?")
etype = e.get("type", "?")
d = e.get("data", {})
if etype == "trade_open":
print("[{}] TRADE OPENED: {} {} @ ${:.4f} (amount: ${:.2f})".format(
t, d.get("side", "?").upper(), d.get("symbol", "?"),
d.get("entry_price", 0), d.get("amount", 0)))
elif etype == "trade_close":
pnl = d.get("pnl", 0)
print("[{}] TRADE CLOSED: {} ${:.4f} -> ${:.4f} P&L: ${:+.2f} ({:+.1f}%) [{}]".format(
t, d.get("symbol", "?"), d.get("entry_price", 0),
d.get("exit_price", 0), pnl, d.get("pnl_pct", 0),
d.get("exit_reason", "?")))
elif etype == "ga_milestone":
print("[{}] GA MILESTONE: Gen {} | Fitness: {:.4f}".format(
t, d.get("generation", 0), d.get("fitness", 0)))
elif etype == "daily_report":
print("[{}] DAILY REPORT: Equity ${:,.2f} | Day P&L: ${:+.2f} | Trades: {}".format(
t, d.get("equity", 0), d.get("day_pnl", 0), d.get("trades_count", 0)))
elif etype == "bot_started":
print("[{}] BOT STARTED".format(t))
elif etype == "bot_stopped":
print("[{}] BOT STOPPED".format(t))
else:
print("[{}] {}: {}".format(t, etype, json.dumps(d)))
scripts/krystie/read_status.py
+70
View File
@@ -0,0 +1,70 @@
#!/usr/bin/env python3
"""Read BIGGFISH live status for Krystie."""
import json, sys
STATUS_FILE = "/opt/biggfish/src/data/krystie-status.json"
try:
with open(STATUS_FILE) as f:
s = json.load(f)
except FileNotFoundError:
print("BIGGFISH status file not found. The bot may not be running.")
print("Check: systemctl status biggfish.service")
sys.exit(1)
updated = s.get("updated_at", "unknown")
uptime = s.get("uptime_hours", 0)
markets = s.get("markets", {})
portfolio = s.get("portfolio", {})
positions = s.get("positions", [])
learning = s.get("learning", {})
today = s.get("today_summary", {})
config = s.get("config", {})
print("=== BIGGFISH STATUS ===")
print("Last updated:", updated)
print("Uptime: {:.1f} hours".format(uptime))
print()
market_parts = ["{}: {}".format(k, v) for k, v in markets.items()]
print("Markets:", " | ".join(market_parts))
print()
equity = portfolio.get("equity", 0)
initial = config.get("initial_capital", 100000)
target = config.get("target_capital", 1000000)
total_pnl = equity - initial if equity else 0
pnl_pct = (total_pnl / initial * 100) if initial else 0
progress = (equity / target * 100) if target else 0
print("Portfolio: ${:,.2f}".format(equity))
print("Total P&L: ${:+,.2f} ({:+.1f}%)".format(total_pnl, pnl_pct))
print("Goal: ${:,.0f} / ${:,.0f} ({:.1f}%)".format(equity, target, progress))
print()
if positions:
print("Open Positions ({}):".format(len(positions)))
for p in positions:
pnl = p.get("unrealized_pnl", 0)
print(" {:8s} {:>6} @ ${:.4f} P&L: ${:+.2f}".format(
p.get("symbol", "?"), str(p.get("qty", 0)),
p.get("current_price", 0), pnl))
else:
print("No open positions")
print()
print("Today: {} trades | {} wins | {} losses | P&L: ${:+.2f}".format(
today.get("trades_count", 0), today.get("wins", 0),
today.get("losses", 0), today.get("total_pnl", 0)))
print()
print("Learning:")
print(" GA: Gen {} | Fitness: {:.4f}".format(
learning.get("ga_generation", 0), learning.get("ga_best_fitness", 0)))
print(" RL: Epsilon: {:.4f} | Experiences: {:,} | Loss: {:.6f}".format(
learning.get("rl_epsilon", 0), learning.get("rl_experiences", 0),
learning.get("rl_loss", 0)))
print()
print("Stocks:", ", ".join(config.get("stock_symbols", [])))
print("Forex:", ", ".join(config.get("forex_symbols", [])))
scripts/krystie/read_trades.py
+51
View File
@@ -0,0 +1,51 @@
#!/usr/bin/env python3
"""Read BIGGFISH trade history from SQLite database for Krystie."""
import sqlite3, sys
DB = "/opt/biggfish/src/data/biggfish.db"
LIMIT = int(sys.argv[1]) if len(sys.argv) > 1 else 20
try:
db = sqlite3.connect(DB)
db.row_factory = sqlite3.Row
except Exception as e:
print("BIGGFISH database not found at", DB)
sys.exit(1)
rows = db.execute("""
SELECT symbol, side, amount, entry_price, exit_price,
entry_time, exit_time, pnl, pnl_pct, status, strategy_id
FROM trades
ORDER BY entry_time DESC
LIMIT ?
""", (LIMIT,)).fetchall()
if not rows:
print("No trades recorded yet.")
sys.exit(0)
print("=== BIGGFISH TRADE HISTORY (last {}) ===".format(len(rows)))
print()
print("{:<10} {:<6} {:>10} {:>10} {:>10} {:>8} {:<10} {:<20}".format(
"Symbol", "Side", "Entry", "Exit", "P&L", "P&L%", "Status", "Time"))
print("-" * 90)
for r in rows:
entry = "${:.4f}".format(r["entry_price"]) if r["entry_price"] else "-"
exit_p = "${:.4f}".format(r["exit_price"]) if r["exit_price"] else "-"
pnl = "${:+.2f}".format(r["pnl"]) if r["pnl"] is not None else "-"
pnl_pct = "{:+.1f}%".format(r["pnl_pct"]) if r["pnl_pct"] is not None else "-"
print("{:<10} {:<6} {:>10} {:>10} {:>10} {:>8} {:<10} {:<20}".format(
r["symbol"], r["side"], entry, exit_p, pnl, pnl_pct,
r["status"], str(r["entry_time"])[:19]))
closed = [r for r in rows if r["status"] == "closed" and r["pnl"] is not None]
if closed:
total_pnl = sum(r["pnl"] for r in closed)
wins = sum(1 for r in closed if r["pnl"] > 0)
losses = sum(1 for r in closed if r["pnl"] <= 0)
print()
print("Summary: {} closed trades | {} wins | {} losses | Total P&L: ${:+.2f}".format(
len(closed), wins, losses, total_pnl))
db.close()
src/__init__.py
View File
src/adapters/__init__.py
View File
src/adapters/base_adapter.py
+138
View File
@@ -0,0 +1,138 @@
"""
Base Adapter Interface for BiggFish
Defines the common interface for all exchange/broker adapters
"""
from abc import ABC, abstractmethod
from typing import Dict, List, Optional
from decimal import Decimal
class BaseAdapter(ABC):
"""Base class for all trading adapters (exchanges and brokers)"""
def __init__(self, config: Dict):
"""
Initialize the adapter with configuration
Args:
config: Dictionary containing API credentials and settings
"""
self.config = config
self.api_key = config.get('apiKey')
self.api_secret = config.get('apiSecret')
self.enabled = config.get('enabled', False)
@abstractmethod
def connect(self) -> bool:
"""
Establish connection to the exchange/broker
Returns:
True if connection successful, False otherwise
"""
pass
@abstractmethod
def get_balance(self) -> Dict[str, Decimal]:
"""
Get current account balances
Returns:
Dictionary mapping symbols to balances
Example: {"BTC": Decimal("1.5"), "USDT": Decimal("10000")}
"""
pass
@abstractmethod
def get_price(self, symbol: str) -> Decimal:
"""
Get current market price for a symbol
Args:
symbol: Trading pair or stock symbol (e.g., "BTC/USDT" or "AAPL")
Returns:
Current market price
"""
pass
@abstractmethod
def get_portfolio_value(self) -> Decimal:
"""
Get total portfolio value in base currency
Returns:
Total portfolio value
"""
pass
@abstractmethod
def create_market_order(self, symbol: str, side: str, amount: Decimal) -> Dict:
"""
Create a market order
Args:
symbol: Trading pair or stock symbol
side: "buy" or "sell"
amount: Amount to trade
Returns:
Order result dictionary
"""
pass
@abstractmethod
def create_limit_order(self, symbol: str, side: str, amount: Decimal, price: Decimal) -> Dict:
"""
Create a limit order
Args:
symbol: Trading pair or stock symbol
side: "buy" or "sell"
amount: Amount to trade
price: Limit price
Returns:
Order result dictionary
"""
pass
@abstractmethod
def get_order_status(self, order_id: str) -> Dict:
"""
Get status of an order
Args:
order_id: Order identifier
Returns:
Order status dictionary
"""
pass
@abstractmethod
def cancel_order(self, order_id: str) -> bool:
"""
Cancel an open order
Args:
order_id: Order identifier
Returns:
True if successfully cancelled
"""
pass
def validate_connection(self) -> bool:
"""
Validate that the adapter can connect and authenticate
Returns:
True if valid connection
"""
try:
return self.connect()
except Exception as e:
print(f"Connection validation failed: {e}")
return False
src/adapters/crypto_adapter.py
+197
View File
@@ -0,0 +1,197 @@
"""
Crypto Exchange Adapter using CCXT
Supports 100+ cryptocurrency exchanges
"""
import ccxt
from typing import Dict, List, Optional
from decimal import Decimal
from .base_adapter import BaseAdapter
class CryptoAdapter(BaseAdapter):
"""Adapter for cryptocurrency exchanges using CCXT library"""
SUPPORTED_EXCHANGES = {
'binance': ccxt.binance,
'coinbase': ccxt.coinbase,
'kraken': ccxt.kraken,
'kucoin': ccxt.kucoin,
'bybit': ccxt.bybit,
'okx': ccxt.okx,
# Add more as needed
}
def __init__(self, exchange_name: str, config: Dict):
"""
Initialize crypto exchange adapter
Args:
exchange_name: Name of the exchange (e.g., 'binance')
config: Configuration dictionary with API credentials
"""
super().__init__(config)
self.exchange_name = exchange_name.lower()
self.exchange = None
self.testnet = config.get('testnet', False)
def connect(self) -> bool:
"""Establish connection to the exchange"""
try:
if self.exchange_name not in self.SUPPORTED_EXCHANGES:
raise ValueError(f"Exchange {self.exchange_name} not supported")
exchange_class = self.SUPPORTED_EXCHANGES[self.exchange_name]
self.exchange = exchange_class({
'apiKey': self.api_key,
'secret': self.api_secret,
'enableRateLimit': True,
})
if self.testnet:
self.exchange.set_sandbox_mode(True)
# Test connection
self.exchange.load_markets()
return True
except Exception as e:
print(f"Failed to connect to {self.exchange_name}: {e}")
return False
def get_balance(self) -> Dict[str, Decimal]:
"""Get current account balances"""
try:
balance = self.exchange.fetch_balance()
return {
symbol: Decimal(str(amount))
for symbol, amount in balance['total'].items()
if amount > 0
}
except Exception as e:
print(f"Error fetching balance: {e}")
return {}
def get_price(self, symbol: str) -> Decimal:
"""Get current market price for a trading pair"""
try:
ticker = self.exchange.fetch_ticker(symbol)
return Decimal(str(ticker['last']))
except Exception as e:
print(f"Error fetching price for {symbol}: {e}")
return Decimal(0)
def get_portfolio_value(self, base_currency: str = 'USDT') -> Decimal:
"""
Calculate total portfolio value in base currency
Args:
base_currency: Currency to value portfolio in (default: USDT)
Returns:
Total portfolio value
"""
try:
balances = self.get_balance()
total_value = Decimal(0)
for symbol, amount in balances.items():
if symbol == base_currency:
total_value += amount
else:
# Try to get price in base currency
pair = f"{symbol}/{base_currency}"
try:
price = self.get_price(pair)
total_value += amount * price
except:
# If pair doesn't exist, skip or try alternative
pass
return total_value
except Exception as e:
print(f"Error calculating portfolio value: {e}")
return Decimal(0)
def create_market_order(self, symbol: str, side: str, amount: Decimal) -> Dict:
"""Create a market order"""
try:
order = self.exchange.create_market_order(
symbol=symbol,
side=side,
amount=float(amount)
)
return order
except Exception as e:
print(f"Error creating market order: {e}")
return {'error': str(e)}
def create_limit_order(self, symbol: str, side: str, amount: Decimal, price: Decimal) -> Dict:
"""Create a limit order"""
try:
order = self.exchange.create_limit_order(
symbol=symbol,
side=side,
amount=float(amount),
price=float(price)
)
return order
except Exception as e:
print(f"Error creating limit order: {e}")
return {'error': str(e)}
def get_order_status(self, order_id: str) -> Dict:
"""Get order status"""
try:
order = self.exchange.fetch_order(order_id)
return order
except Exception as e:
print(f"Error fetching order status: {e}")
return {'error': str(e)}
def cancel_order(self, order_id: str) -> bool:
"""Cancel an order"""
try:
self.exchange.cancel_order(order_id)
return True
except Exception as e:
print(f"Error cancelling order: {e}")
return False
def get_current_allocation(self, base_currency: str = 'USDT') -> Dict[str, float]:
"""
Get current portfolio allocation as percentages
Args:
base_currency: Base currency for valuation
Returns:
Dictionary mapping symbols to percentage allocations
"""
try:
balances = self.get_balance()
total_value = self.get_portfolio_value(base_currency)
if total_value == 0:
return {}
allocations = {}
for symbol, amount in balances.items():
if symbol == base_currency:
value = amount
else:
pair = f"{symbol}/{base_currency}"
try:
price = self.get_price(pair)
value = amount * price
except:
continue
percentage = float((value / total_value) * 100)
if percentage > 0.01: # Filter out dust
allocations[symbol] = round(percentage, 2)
return allocations
except Exception as e:
print(f"Error calculating allocation: {e}")
return {}
src/adapters/stock_adapter.py
+211
View File
@@ -0,0 +1,211 @@
"""
Stock Broker Adapter
Supports Alpaca and other stock brokers
"""
from typing import Dict, List, Optional
from decimal import Decimal
from .base_adapter import BaseAdapter
try:
from alpaca_trade_api import REST
ALPACA_AVAILABLE = True
except ImportError:
ALPACA_AVAILABLE = False
class StockAdapter(BaseAdapter):
"""Adapter for stock brokers (Alpaca, Interactive Brokers, etc.)"""
SUPPORTED_BROKERS = ['alpaca', 'interactiveBrokers']
def __init__(self, broker_name: str, config: Dict):
"""
Initialize stock broker adapter
Args:
broker_name: Name of the broker (e.g., 'alpaca')
config: Configuration dictionary with API credentials
"""
super().__init__(config)
self.broker_name = broker_name.lower()
self.client = None
self.base_url = config.get('baseUrl', 'https://paper-api.alpaca.markets')
def connect(self) -> bool:
"""Establish connection to the broker"""
try:
if self.broker_name == 'alpaca':
return self._connect_alpaca()
elif self.broker_name == 'interactivebrokers':
return self._connect_ib()
else:
raise ValueError(f"Broker {self.broker_name} not supported")
except Exception as e:
print(f"Failed to connect to {self.broker_name}: {e}")
return False
def _connect_alpaca(self) -> bool:
"""Connect to Alpaca"""
if not ALPACA_AVAILABLE:
raise ImportError("alpaca-trade-api not installed. Run: pip install alpaca-trade-api")
self.client = REST(
key_id=self.api_key,
secret_key=self.api_secret,
base_url=self.base_url
)
# Test connection
account = self.client.get_account()
return account.status == 'ACTIVE'
def _connect_ib(self) -> bool:
"""Connect to Interactive Brokers"""
# Placeholder for IB implementation
raise NotImplementedError("Interactive Brokers support coming soon")
def get_balance(self) -> Dict[str, Decimal]:
"""Get current account balances"""
try:
if self.broker_name == 'alpaca':
account = self.client.get_account()
positions = self.client.list_positions()
balances = {
'USD': Decimal(str(account.cash))
}
for position in positions:
balances[position.symbol] = Decimal(str(position.qty))
return balances
except Exception as e:
print(f"Error fetching balance: {e}")
return {}
def get_price(self, symbol: str) -> Decimal:
"""Get current market price for a stock"""
try:
if self.broker_name == 'alpaca':
# Get latest trade
trade = self.client.get_latest_trade(symbol)
return Decimal(str(trade.price))
except Exception as e:
print(f"Error fetching price for {symbol}: {e}")
return Decimal(0)
def get_portfolio_value(self) -> Decimal:
"""Calculate total portfolio value"""
try:
if self.broker_name == 'alpaca':
account = self.client.get_account()
return Decimal(str(account.portfolio_value))
except Exception as e:
print(f"Error calculating portfolio value: {e}")
return Decimal(0)
def create_market_order(self, symbol: str, side: str, amount: Decimal) -> Dict:
"""Create a market order"""
try:
if self.broker_name == 'alpaca':
order = self.client.submit_order(
symbol=symbol,
qty=float(amount),
side=side,
type='market',
time_in_force='day'
)
return {
'id': order.id,
'symbol': order.symbol,
'side': order.side,
'qty': order.qty,
'status': order.status
}
except Exception as e:
print(f"Error creating market order: {e}")
return {'error': str(e)}
def create_limit_order(self, symbol: str, side: str, amount: Decimal, price: Decimal) -> Dict:
"""Create a limit order"""
try:
if self.broker_name == 'alpaca':
order = self.client.submit_order(
symbol=symbol,
qty=float(amount),
side=side,
type='limit',
limit_price=float(price),
time_in_force='day'
)
return {
'id': order.id,
'symbol': order.symbol,
'side': order.side,
'qty': order.qty,
'status': order.status
}
except Exception as e:
print(f"Error creating limit order: {e}")
return {'error': str(e)}
def get_order_status(self, order_id: str) -> Dict:
"""Get order status"""
try:
if self.broker_name == 'alpaca':
order = self.client.get_order(order_id)
return {
'id': order.id,
'status': order.status,
'filled_qty': order.filled_qty
}
except Exception as e:
print(f"Error fetching order status: {e}")
return {'error': str(e)}
def cancel_order(self, order_id: str) -> bool:
"""Cancel an order"""
try:
if self.broker_name == 'alpaca':
self.client.cancel_order(order_id)
return True
except Exception as e:
print(f"Error cancelling order: {e}")
return False
def get_current_allocation(self) -> Dict[str, float]:
"""
Get current portfolio allocation as percentages
Returns:
Dictionary mapping symbols to percentage allocations
"""
try:
if self.broker_name == 'alpaca':
account = self.client.get_account()
total_value = Decimal(str(account.portfolio_value))
if total_value == 0:
return {}
positions = self.client.list_positions()
allocations = {}
# Cash allocation
cash = Decimal(str(account.cash))
cash_pct = float((cash / total_value) * 100)
if cash_pct > 0.01:
allocations['USD'] = round(cash_pct, 2)
# Position allocations
for position in positions:
value = Decimal(str(position.market_value))
percentage = float((value / total_value) * 100)
if percentage > 0.01:
allocations[position.symbol] = round(percentage, 2)
return allocations
except Exception as e:
print(f"Error calculating allocation: {e}")
return {}
src/backtest/__init__.py
View File
src/backtest/engine.py
+517
View File
@@ -0,0 +1,517 @@
"""
Backtesting Engine
Simulates trading strategies against historical OHLCV data.
Includes both callback-based and vectorized fast paths.
"""
import pandas as pd
import numpy as np
from dataclasses import dataclass, field
from datetime import datetime
from typing import Callable, Dict, List, Optional, Tuple
from loguru import logger
from backtest.metrics import compute_metrics
@dataclass
class BacktestResult:
"""Container for backtest results"""
trades: List[Dict] = field(default_factory=list)
equity_curve: pd.Series = field(default_factory=lambda: pd.Series(dtype=float))
metrics: Dict = field(default_factory=dict)
params: Dict = field(default_factory=dict)
symbol: str = ""
timeframe: str = ""
class BacktestEngine:
"""Simulates trading against historical OHLCV data"""
def __init__(self, initial_capital: float = 100.0,
commission_rate: float = 0.001):
self.initial_capital = initial_capital
self.commission_rate = commission_rate
def run(self, strategy_fn: Callable, candles_df: pd.DataFrame,
params: Dict = None, symbol: str = "") -> BacktestResult:
"""
Run a backtest using the callback-based strategy function.
Args:
strategy_fn: Callable with signature:
(state: Dict, candle_idx: int, df: pd.DataFrame, params: Dict) -> Dict
candles_df: DataFrame with columns: open, high, low, close, volume
params: Strategy parameters dict
symbol: Symbol name for labeling
Returns:
BacktestResult
"""
if candles_df is None or len(candles_df) < 50:
return BacktestResult(symbol=symbol)
params = params or {}
trades, equity_curve = self._simulate(candles_df, strategy_fn, params)
metrics = compute_metrics(equity_curve, trades)
return BacktestResult(
trades=trades,
equity_curve=equity_curve,
metrics=metrics,
params=params,
symbol=symbol,
timeframe='1h',
)
def run_fast(self, signals: Dict, candles_df: pd.DataFrame,
symbol: str = "") -> BacktestResult:
"""
Fast backtest using pre-computed signal arrays from genome_to_signals().
Avoids per-candle indicator computation entirely.
Args:
signals: dict from genome_to_signals() with 'entry', 'exit',
'stop_loss', 'take_profit', 'amount_pct', 'max_hold_candles'
candles_df: DataFrame with columns: open, high, low, close, volume
symbol: Symbol name for labeling
Returns:
BacktestResult
"""
if candles_df is None or len(candles_df) < 50:
return BacktestResult(symbol=symbol)
trades, equity_values = self._simulate_fast(candles_df, signals)
equity_times = list(range(len(equity_values)))
if hasattr(candles_df.index, '__getitem__'):
equity_times = list(candles_df.index[:len(equity_values)])
equity_curve = pd.Series(equity_values, index=equity_times)
metrics = compute_metrics(equity_curve, trades)
return BacktestResult(
trades=trades,
equity_curve=equity_curve,
metrics=metrics,
params={},
symbol=symbol,
timeframe='1h',
)
def _simulate_fast(self, df: pd.DataFrame, signals: Dict) -> Tuple[List[Dict], List[float]]:
"""
Core fast simulation loop using pre-computed signal arrays.
Supports both long and short positions for scalping.
~10-50x faster than callback-based _simulate for GA evaluation.
"""
close = df['close'].values.astype(np.float64)
high = df['high'].values.astype(np.float64)
low = df['low'].values.astype(np.float64)
n = len(close)
entry_signals = signals['entry']
exit_signals = signals['exit']
short_entry_signals = signals.get('short_entry', np.zeros(n, dtype=bool))
short_exit_signals = signals.get('short_exit', np.zeros(n, dtype=bool))
sl_levels = signals['stop_loss']
tp_levels = signals['take_profit']
short_sl_levels = signals.get('short_stop_loss', np.zeros(n))
short_tp_levels = signals.get('short_take_profit', np.zeros(n))
amount_pct = signals['amount_pct']
max_hold = signals['max_hold_candles']
commission = self.commission_rate
capital = self.initial_capital
position_shares = 0.0 # positive = long, negative = short
position_entry_price = 0.0
position_sl = 0.0
position_tp = 0.0
position_entry_idx = 0
position_side = '' # 'long' or 'short'
trades = []
equity_values = []
for i in range(n):
current_price = close[i]
# Check exit conditions for LONG position
if position_shares > 0:
closed = False
exit_price = 0.0
exit_reason = ''
if position_sl > 0 and low[i] <= position_sl:
exit_price = position_sl
closed = True
exit_reason = 'stop_loss'
elif position_tp > 0 and high[i] >= position_tp:
exit_price = position_tp
closed = True
exit_reason = 'take_profit'
elif exit_signals[i]:
exit_price = current_price
closed = True
exit_reason = 'signal'
elif (i - position_entry_idx) >= max_hold:
exit_price = current_price
closed = True
exit_reason = 'max_hold'
if closed:
pnl = (exit_price - position_entry_price) * position_shares
fees = abs(exit_price * position_shares * commission)
pnl -= fees
pnl_pct = (exit_price - position_entry_price) / position_entry_price * 100
capital += position_shares * exit_price - fees
trades.append({
'entry_price': position_entry_price,
'exit_price': exit_price,
'shares': position_shares,
'pnl': round(pnl, 4),
'pnl_pct': round(pnl_pct, 4),
'fees': round(fees, 4),
'entry_idx': position_entry_idx,
'exit_idx': i,
'exit_reason': exit_reason,
'side': 'long',
})
position_shares = 0.0
position_entry_price = 0.0
position_side = ''
# Check exit conditions for SHORT position
elif position_shares < 0:
abs_shares = abs(position_shares)
closed = False
exit_price = 0.0
exit_reason = ''
# Short SL: price goes UP above stop
if position_sl > 0 and high[i] >= position_sl:
exit_price = position_sl
closed = True
exit_reason = 'stop_loss'
# Short TP: price goes DOWN below target
elif position_tp > 0 and low[i] <= position_tp:
exit_price = position_tp
closed = True
exit_reason = 'take_profit'
elif short_exit_signals[i]:
exit_price = current_price
closed = True
exit_reason = 'signal'
elif (i - position_entry_idx) >= max_hold:
exit_price = current_price
closed = True
exit_reason = 'max_hold'
if closed:
pnl = (position_entry_price - exit_price) * abs_shares
fees = abs(exit_price * abs_shares * commission)
pnl -= fees
pnl_pct = (position_entry_price - exit_price) / position_entry_price * 100
# Return collateral + profit (or - loss)
capital += (position_entry_price * abs_shares) + pnl - fees
trades.append({
'entry_price': position_entry_price,
'exit_price': exit_price,
'shares': abs_shares,
'pnl': round(pnl, 4),
'pnl_pct': round(pnl_pct, 4),
'fees': round(fees, 4),
'entry_idx': position_entry_idx,
'exit_idx': i,
'exit_reason': exit_reason,
'side': 'short',
})
position_shares = 0.0
position_entry_price = 0.0
position_side = ''
# Check LONG entry (only if flat)
if position_shares == 0 and entry_signals[i] and capital > 5:
invest = capital * min(amount_pct, 0.5)
if invest > 1:
fees = invest * commission
shares = (invest - fees) / current_price
if capital >= invest:
capital -= invest
position_shares = shares
position_entry_price = current_price
position_sl = sl_levels[i]
position_tp = tp_levels[i]
position_entry_idx = i
position_side = 'long'
# Check SHORT entry (only if flat and no long entry this bar)
elif position_shares == 0 and short_entry_signals[i] and capital > 5:
invest = capital * min(amount_pct, 0.5)
if invest > 1:
fees = invest * commission
shares = (invest - fees) / current_price
if capital >= invest:
# Short: set aside collateral, owe shares
capital -= invest # collateral
position_shares = -shares
position_entry_price = current_price
position_sl = short_sl_levels[i]
position_tp = short_tp_levels[i]
position_entry_idx = i
position_side = 'short'
# Track equity
if position_shares > 0:
equity = capital + position_shares * current_price
elif position_shares < 0:
abs_shares = abs(position_shares)
short_pnl = (position_entry_price - current_price) * abs_shares
equity = capital + (position_entry_price * abs_shares) + short_pnl
else:
equity = capital
equity_values.append(equity)
# Close remaining position at end
if position_shares > 0:
final_price = close[-1]
pnl = (final_price - position_entry_price) * position_shares
fees = abs(final_price * position_shares * commission)
pnl -= fees
capital += position_shares * final_price - fees
trades.append({
'entry_price': position_entry_price,
'exit_price': final_price,
'shares': position_shares,
'pnl': round(pnl, 4),
'pnl_pct': round((final_price - position_entry_price) / position_entry_price * 100, 4),
'fees': round(fees, 4),
'entry_idx': position_entry_idx,
'exit_idx': n - 1,
'exit_reason': 'end_of_data',
'side': 'long',
})
elif position_shares < 0:
final_price = close[-1]
abs_shares = abs(position_shares)
pnl = (position_entry_price - final_price) * abs_shares
fees = abs(final_price * abs_shares * commission)
pnl -= fees
capital += (position_entry_price * abs_shares) + pnl - fees
trades.append({
'entry_price': position_entry_price,
'exit_price': final_price,
'shares': abs_shares,
'pnl': round(pnl, 4),
'pnl_pct': round((position_entry_price - final_price) / position_entry_price * 100, 4),
'fees': round(fees, 4),
'entry_idx': position_entry_idx,
'exit_idx': n - 1,
'exit_reason': 'end_of_data',
'side': 'short',
})
return trades, equity_values
def _simulate(self, df: pd.DataFrame, strategy_fn: Callable,
params: Dict) -> Tuple[List[Dict], pd.Series]:
"""
Core simulation loop (callback-based, used for non-GA backtests).
Supports long, short, sell, and cover actions.
"""
capital = self.initial_capital
position = None
trades = []
equity_values = []
equity_times = []
for i in range(len(df)):
candle = df.iloc[i]
current_price = float(candle['close'])
high_price = float(candle['high'])
low_price = float(candle['low'])
# Check exit conditions for open position
if position is not None:
closed = False
pos_dir = position.get('direction', 'long')
if pos_dir == 'long':
if position['stop_loss'] and low_price <= position['stop_loss']:
exit_price = position['stop_loss']
closed = True
exit_reason = 'stop_loss'
elif position['take_profit'] and high_price >= position['take_profit']:
exit_price = position['take_profit']
closed = True
exit_reason = 'take_profit'
elif pos_dir == 'short':
# Short: SL above, TP below
if position['stop_loss'] and high_price >= position['stop_loss']:
exit_price = position['stop_loss']
closed = True
exit_reason = 'stop_loss'
elif position['take_profit'] and low_price <= position['take_profit']:
exit_price = position['take_profit']
closed = True
exit_reason = 'take_profit'
if closed:
if pos_dir == 'long':
pnl = (exit_price - position['entry_price']) * position['shares']
else:
pnl = (position['entry_price'] - exit_price) * position['shares']
fees = abs(exit_price * position['shares'] * self.commission_rate)
pnl -= fees
if pos_dir == 'long':
capital += position['shares'] * exit_price - fees
else:
capital += (position['entry_price'] * position['shares']) + pnl - fees
pnl_pct = pnl / (position['entry_price'] * position['shares']) * 100
trades.append({
'entry_price': position['entry_price'],
'exit_price': exit_price,
'shares': position['shares'],
'pnl': round(pnl, 4),
'pnl_pct': round(pnl_pct, 4),
'fees': round(fees, 4),
'entry_idx': position['entry_idx'],
'exit_idx': i,
'exit_reason': exit_reason,
'side': pos_dir,
})
position = None
# Get strategy signal
pos_for_state = None
if position is not None:
pos_for_state = {**position}
state = {
'capital': capital,
'position': pos_for_state,
'num_trades': len(trades),
'equity': capital + (position['shares'] * current_price if position and position.get('direction') == 'long' else 0),
}
try:
signal = strategy_fn(state, i, df, params)
except Exception:
signal = {'action': 'hold'}
action = signal.get('action', 'hold')
# Execute signal
if action == 'buy' and position is None and capital > 5:
amount_pct = min(signal.get('amount_pct', 0.1), 0.5)
invest = capital * amount_pct
shares = invest / current_price
fees = invest * self.commission_rate
if invest > 1 and capital >= invest + fees:
capital -= invest + fees
position = {
'entry_price': current_price,
'shares': shares,
'stop_loss': signal.get('stop_loss'),
'take_profit': signal.get('take_profit'),
'entry_idx': i,
'amount': invest,
'direction': 'long',
}
elif action == 'short' and position is None and capital > 5:
amount_pct = min(signal.get('amount_pct', 0.1), 0.5)
invest = capital * amount_pct
shares = invest / current_price
fees = invest * self.commission_rate
if invest > 1 and capital >= invest + fees:
capital -= invest + fees # collateral
position = {
'entry_price': current_price,
'shares': shares,
'stop_loss': signal.get('short_stop_loss') or signal.get('stop_loss'),
'take_profit': signal.get('short_take_profit') or signal.get('take_profit'),
'entry_idx': i,
'amount': invest,
'direction': 'short',
}
elif action in ('sell', 'cover') and position is not None:
exit_price = current_price
pos_dir = position.get('direction', 'long')
if pos_dir == 'long':
pnl = (exit_price - position['entry_price']) * position['shares']
else:
pnl = (position['entry_price'] - exit_price) * position['shares']
fees = abs(exit_price * position['shares'] * self.commission_rate)
pnl -= fees
pnl_pct = pnl / (position['entry_price'] * position['shares']) * 100
if pos_dir == 'long':
capital += position['shares'] * exit_price - fees
else:
capital += (position['entry_price'] * position['shares']) + pnl - fees
trades.append({
'entry_price': position['entry_price'],
'exit_price': exit_price,
'shares': position['shares'],
'pnl': round(pnl, 4),
'pnl_pct': round(pnl_pct, 4),
'fees': round(fees, 4),
'entry_idx': position['entry_idx'],
'exit_idx': i,
'exit_reason': 'signal',
'side': pos_dir,
})
position = None
# Track equity
if position and position.get('direction') == 'long':
equity = capital + position['shares'] * current_price
elif position and position.get('direction') == 'short':
short_pnl = (position['entry_price'] - current_price) * position['shares']
equity = capital + (position['entry_price'] * position['shares']) + short_pnl
else:
equity = capital
equity_values.append(equity)
equity_times.append(df.index[i] if hasattr(df.index, '__getitem__') else i)
# Close any remaining position at the end
if position is not None:
final_price = float(df.iloc[-1]['close'])
pos_dir = position.get('direction', 'long')
if pos_dir == 'long':
pnl = (final_price - position['entry_price']) * position['shares']
else:
pnl = (position['entry_price'] - final_price) * position['shares']
fees = abs(final_price * position['shares'] * self.commission_rate)
pnl -= fees
pnl_pct = pnl / (position['entry_price'] * position['shares']) * 100
trades.append({
'entry_price': position['entry_price'],
'exit_price': final_price,
'shares': position['shares'],
'pnl': round(pnl, 4),
'pnl_pct': round(pnl_pct, 4),
'fees': round(fees, 4),
'entry_idx': position['entry_idx'],
'exit_idx': len(df) - 1,
'exit_reason': 'end_of_data',
'side': pos_dir,
})
equity_curve = pd.Series(equity_values, index=equity_times)
return trades, equity_curve
src/backtest/metrics.py
+134
View File
@@ -0,0 +1,134 @@
"""
Performance Metrics for Backtesting
Computes Sharpe, Sortino, max drawdown, win rate, profit factor, etc.
"""
import numpy as np
import pandas as pd
from typing import Dict, List, Tuple
def compute_metrics(equity_curve: pd.Series, trades: List[Dict],
periods_per_year: int = 252 * 7) -> Dict:
"""
Compute comprehensive performance metrics.
Args:
equity_curve: Series of portfolio values at each step
trades: List of trade dicts with 'pnl' and 'pnl_pct' fields
periods_per_year: Annualization factor (252*7 for hourly on trading days)
Returns:
Dict with all performance metrics
"""
if len(equity_curve) < 2:
return _empty_metrics()
returns = equity_curve.pct_change().dropna()
if len(returns) < 2:
return _empty_metrics()
# Filter to closed trades with P&L
closed = [t for t in trades if t.get('pnl') is not None]
wins = [t for t in closed if t['pnl'] > 0]
losses = [t for t in closed if t['pnl'] <= 0]
total_trades = len(closed)
win_rate = len(wins) / total_trades * 100 if total_trades > 0 else 0
gross_profit = sum(t['pnl'] for t in wins) if wins else 0
gross_loss = abs(sum(t['pnl'] for t in losses)) if losses else 0
profit_factor = gross_profit / gross_loss if gross_loss > 0 else float('inf') if gross_profit > 0 else 0
avg_win = np.mean([t['pnl_pct'] for t in wins]) if wins else 0
avg_loss = np.mean([abs(t['pnl_pct']) for t in losses]) if losses else 0
total_return = (equity_curve.iloc[-1] / equity_curve.iloc[0] - 1) * 100
max_dd, max_dd_duration = compute_max_drawdown(equity_curve)
sharpe = compute_sharpe(returns, periods_per_year=periods_per_year)
sortino = compute_sortino(returns, periods_per_year=periods_per_year)
expectancy = (win_rate / 100 * avg_win) - ((1 - win_rate / 100) * avg_loss)
avg_trade_pnl = np.mean([t['pnl'] for t in closed]) if closed else 0
return {
'sharpe_ratio': round(sharpe, 4),
'sortino_ratio': round(sortino, 4),
'max_drawdown': round(max_dd, 4),
'max_drawdown_duration': max_dd_duration,
'total_return': round(total_return, 4),
'win_rate': round(win_rate, 4),
'loss_rate': round(100 - win_rate, 4),
'profit_factor': round(profit_factor, 4) if profit_factor != float('inf') else 999.0,
'avg_win': round(avg_win, 4),
'avg_loss': round(avg_loss, 4),
'expectancy': round(expectancy, 4),
'total_trades': total_trades,
'winning_trades': len(wins),
'losing_trades': len(losses),
'avg_trade_pnl': round(avg_trade_pnl, 4),
'gross_profit': round(gross_profit, 4),
'gross_loss': round(gross_loss, 4),
'final_equity': round(float(equity_curve.iloc[-1]), 2),
}
def compute_sharpe(returns: pd.Series, risk_free_rate: float = 0.0,
periods_per_year: int = 252 * 7) -> float:
"""Annualized Sharpe ratio"""
if len(returns) < 2 or returns.std() == 0:
return 0.0
excess = returns - risk_free_rate / periods_per_year
return float(excess.mean() / excess.std() * np.sqrt(periods_per_year))
def compute_sortino(returns: pd.Series, risk_free_rate: float = 0.0,
periods_per_year: int = 252 * 7) -> float:
"""Annualized Sortino ratio (penalizes downside deviation only)"""
if len(returns) < 2:
return 0.0
excess = returns - risk_free_rate / periods_per_year
downside = returns[returns < 0]
if len(downside) < 2 or downside.std() == 0:
return compute_sharpe(returns, risk_free_rate, periods_per_year)
return float(excess.mean() / downside.std() * np.sqrt(periods_per_year))
def compute_max_drawdown(equity_curve: pd.Series) -> Tuple[float, int]:
"""
Returns (max_drawdown_pct, duration_in_steps).
Max drawdown is peak-to-trough decline as a percentage.
"""
if len(equity_curve) < 2:
return 0.0, 0
peak = equity_curve.expanding().max()
drawdown = (equity_curve - peak) / peak * 100
max_dd = abs(float(drawdown.min()))
# Duration: longest stretch below previous peak
is_underwater = drawdown < 0
if not is_underwater.any():
return 0.0, 0
groups = (~is_underwater).cumsum()
underwater_periods = is_underwater.groupby(groups).sum()
max_duration = int(underwater_periods.max()) if len(underwater_periods) > 0 else 0
return max_dd, max_duration
def _empty_metrics() -> Dict:
"""Return empty metrics dict"""
return {
'sharpe_ratio': 0, 'sortino_ratio': 0, 'max_drawdown': 0,
'max_drawdown_duration': 0, 'total_return': 0, 'win_rate': 0,
'loss_rate': 0, 'profit_factor': 0, 'avg_win': 0, 'avg_loss': 0,
'expectancy': 0, 'total_trades': 0, 'winning_trades': 0,
'losing_trades': 0, 'avg_trade_pnl': 0, 'gross_profit': 0,
'gross_loss': 0, 'final_equity': 0,
}
src/cli.py
+133
View File
@@ -0,0 +1,133 @@
#!/usr/bin/env python3
"""
BIGGFISH CLI - Command line interface for manual control
"""
import json
import sys
from pathlib import Path
from loguru import logger
import argparse
# Add parent directory to path
sys.path.insert(0, str(Path(__file__).parent))
from trading.broker import AlpacaBroker
from research.screener import StockScreener
from strategies.manager import StrategyManager
from reporting.reporter import Reporter
def load_config():
"""Load configuration"""
config_path = Path(__file__).parent.parent / "config" / "config.json"
with open(config_path) as f:
return json.load(f)
def cmd_status(args):
"""Show current portfolio status"""
config = load_config()
broker = AlpacaBroker(config["alpaca"])
reporter = Reporter(config["reporting"])
portfolio = broker.get_portfolio()
positions = broker.get_positions()
report = reporter.generate_daily_report(portfolio, positions)
print(reporter.format_daily_report(report))
def cmd_scan(args):
"""Run stock screening"""
config = load_config()
screener = StockScreener(config["research"])
focus = args.focus or "small_cap"
opportunities = screener.scan(focus)
print(f"\n🔍 Top Opportunities ({focus})\n")
print("=" * 70)
for i, opp in enumerate(opportunities[:10], 1):
print(f"\n{i}. {opp['symbol']} - Score: {opp['score']}/100")
print(f" Price: ${opp['current_price']:.2f} | 1W: {opp['price_change_1w']:+.1f}%")
print(f" Volume Surge: {opp['volume_surge']:.1f}x | Volatility: {opp['volatility']:.1f}%")
print(f" Sector: {opp.get('sector', 'Unknown')}")
def cmd_strategies(args):
"""Show pending strategies"""
config = load_config()
broker = AlpacaBroker(config["alpaca"])
screener = StockScreener(config["research"])
strategy_mgr = StrategyManager(config)
# Generate fresh strategies
portfolio = broker.get_portfolio()
opportunities = screener.scan("small_cap")
strategies = strategy_mgr.generate_strategies(opportunities, portfolio)
if not strategies:
print("\n❌ No strategies generated")
return
print(f"\n🧠 Generated {len(strategies)} Strategies\n")
print("=" * 70)
for i, s in enumerate(strategies, 1):
print(f"\n{i}. {s['type'].replace('_', ' ').title()} - {s['symbol']}")
print(f" {s['action'].upper()} {s['shares']} shares @ ${s['entry_price']:.2f}")
print(f" Target: ${s['target_price']:.2f} (+{s['reward_pct']:.1f}%)")
print(f" Stop: ${s['stop_loss']:.2f} (-{s['risk_pct']:.1f}%)")
print(f" Risk: ${s['position_value']:.2f}")
def cmd_market(args):
"""Show market status"""
config = load_config()
broker = AlpacaBroker(config["alpaca"])
hours = broker.get_market_hours()
print("\n📊 Market Status\n")
print("=" * 40)
print(f"Open: {'✅ YES' if hours['is_open'] else '❌ NO'}")
print(f"Next Open: {hours['next_open']}")
print(f"Next Close: {hours['next_close']}")
def main():
parser = argparse.ArgumentParser(description="BIGGFISH CLI")
subparsers = parser.add_subparsers(dest='command', help='Commands')
# Status command
subparsers.add_parser('status', help='Show portfolio status')
# Scan command
scan_parser = subparsers.add_parser('scan', help='Scan for opportunities')
scan_parser.add_argument('--focus', choices=['small_cap', 'mid_cap_mixed', 'balanced'],
help='Market cap focus')
# Strategies command
subparsers.add_parser('strategies', help='Generate and show strategies')
# Market command
subparsers.add_parser('market', help='Show market status')
args = parser.parse_args()
if not args.command:
parser.print_help()
return
commands = {
'status': cmd_status,
'scan': cmd_scan,
'strategies': cmd_strategies,
'market': cmd_market
}
try:
commands[args.command](args)
except Exception as e:
logger.error(f"Error: {e}", exc_info=True)
sys.exit(1)
if __name__ == "__main__":
main()
src/core/__init__.py
View File
src/core/portfolio.py
+235
View File
@@ -0,0 +1,235 @@
"""
Portfolio Management and Calculations
Handles portfolio analysis, allocation calculations, and rebalancing math
"""
from typing import Dict, List, Tuple
from decimal import Decimal, ROUND_DOWN
class Portfolio:
"""Portfolio management and calculation utilities"""
def __init__(self, adapter):
"""
Initialize portfolio manager
Args:
adapter: Exchange or broker adapter instance
"""
self.adapter = adapter
def get_current_allocation(self) -> Dict[str, float]:
"""
Get current portfolio allocation percentages
Returns:
Dictionary mapping symbols to percentages
"""
return self.adapter.get_current_allocation()
def calculate_rebalance_trades(
self,
target_allocation: Dict[str, float],
threshold: float = 0,
min_trade_value: Decimal = Decimal("10")
) -> List[Dict]:
"""
Calculate trades needed to rebalance portfolio to target allocation
Args:
target_allocation: Target allocation percentages (e.g., {"BTC": 40, "ETH": 30, "USDT": 30})
threshold: Minimum drift percentage before rebalancing (default: 0)
min_trade_value: Minimum trade value to execute
Returns:
List of trade dictionaries with symbol, action, and amount
"""
current_allocation = self.get_current_allocation()
total_value = self.adapter.get_portfolio_value()
if total_value == 0:
print("Portfolio value is zero, cannot rebalance")
return []
# Normalize target allocation to 100%
total_target = sum(target_allocation.values())
if total_target == 0:
print("Target allocation sums to zero")
return []
normalized_target = {
symbol: (pct / total_target) * 100
for symbol, pct in target_allocation.items()
}
# Calculate drifts
drifts = {}
for symbol in set(list(current_allocation.keys()) + list(normalized_target.keys())):
current = current_allocation.get(symbol, 0)
target = normalized_target.get(symbol, 0)
drift = target - current
drifts[symbol] = drift
# Check if rebalancing is needed
max_drift = max(abs(d) for d in drifts.values())
if max_drift < threshold:
print(f"Maximum drift {max_drift:.2f}% is below threshold {threshold}%")
return []
# Calculate trade amounts
trades = []
for symbol, drift in drifts.items():
if abs(drift) < 0.1: # Ignore tiny drifts
continue
# Calculate trade value
trade_value = (Decimal(str(drift)) / 100) * total_value
if abs(trade_value) < min_trade_value:
continue
# Determine action
if drift > 0:
action = "buy"
else:
action = "sell"
trade_value = abs(trade_value)
trades.append({
'symbol': symbol,
'action': action,
'value': trade_value,
'drift': drift
})
return trades
def calculate_trade_amounts(
self,
trades: List[Dict],
base_currency: str = 'USDT'
) -> List[Dict]:
"""
Convert trade values to actual amounts based on current prices
Args:
trades: List of trades from calculate_rebalance_trades
base_currency: Base currency for valuation
Returns:
Updated trade list with amounts
"""
updated_trades = []
for trade in trades:
symbol = trade['symbol']
value = trade['value']
# If trading the base currency, amount = value
if symbol == base_currency:
trade['amount'] = value
updated_trades.append(trade)
continue
# Get current price
pair = f"{symbol}/{base_currency}"
try:
price = self.adapter.get_price(pair)
if price > 0:
amount = (value / price).quantize(Decimal('0.00000001'), rounding=ROUND_DOWN)
trade['amount'] = amount
trade['price'] = price
updated_trades.append(trade)
except Exception as e:
print(f"Error calculating amount for {symbol}: {e}")
continue
return updated_trades
def validate_trade(self, trade: Dict, safety_config: Dict) -> Tuple[bool, str]:
"""
Validate a trade against safety parameters
Args:
trade: Trade dictionary
safety_config: Safety configuration
Returns:
Tuple of (is_valid, reason)
"""
min_trade_value = Decimal(str(safety_config.get('minTradeValue', 10)))
# Check minimum trade value
if trade.get('value', 0) < min_trade_value:
return False, f"Trade value {trade['value']} below minimum {min_trade_value}"
# Add more validations as needed
return True, "Valid"
def execute_rebalance(
self,
target_allocation: Dict[str, float],
threshold: float = 5,
dry_run: bool = True,
safety_config: Dict = None
) -> Dict:
"""
Execute full rebalancing operation
Args:
target_allocation: Target allocation percentages
threshold: Drift threshold for rebalancing
dry_run: If True, don't execute trades
safety_config: Safety parameters
Returns:
Results dictionary with trades and status
"""
if safety_config is None:
safety_config = {'minTradeValue': 10}
# Calculate trades
trades = self.calculate_rebalance_trades(target_allocation, threshold)
if not trades:
return {'status': 'no_rebalance_needed', 'trades': []}
# Calculate amounts
trades_with_amounts = self.calculate_trade_amounts(trades)
# Validate and execute
results = {
'status': 'completed' if not dry_run else 'dry_run',
'trades': [],
'errors': []
}
for trade in trades_with_amounts:
# Validate
is_valid, reason = self.validate_trade(trade, safety_config)
if not is_valid:
results['errors'].append({
'trade': trade,
'reason': reason
})
continue
# Execute if not dry run
if not dry_run:
try:
order = self.adapter.create_market_order(
symbol=trade['symbol'],
side=trade['action'],
amount=trade['amount']
)
trade['order'] = order
results['trades'].append(trade)
except Exception as e:
results['errors'].append({
'trade': trade,
'error': str(e)
})
else:
results['trades'].append(trade)
return results
src/core/rebalancer.py
+130
View File
@@ -0,0 +1,130 @@
"""
Rebalancing Strategy Implementation
Handles portfolio rebalancing logic and execution
"""
from typing import Dict, List
from decimal import Decimal
from .portfolio import Portfolio
class Rebalancer:
"""Portfolio rebalancing strategy executor"""
def __init__(self, adapter, config: Dict):
"""
Initialize rebalancer
Args:
adapter: Exchange or broker adapter
config: Strategy configuration
"""
self.adapter = adapter
self.config = config
self.portfolio = Portfolio(adapter)
def execute(self, dry_run: bool = False) -> Dict:
"""
Execute rebalancing strategy
Args:
dry_run: If True, calculate but don't execute trades
Returns:
Results dictionary
"""
target_allocation = self.config.get('allocations', {})
threshold = self.config.get('threshold', 5)
safety_config = self.config.get('safety', {})
print(f"\n{'='*50}")
print(f"Rebalancing Strategy: {self.config.get('name', 'Unnamed')}")
print(f"{'='*50}")
# Get current state
current_allocation = self.portfolio.get_current_allocation()
total_value = self.adapter.get_portfolio_value()
print(f"\nCurrent Portfolio Value: ${total_value}")
print(f"\nCurrent Allocation:")
for symbol, pct in sorted(current_allocation.items(), key=lambda x: x[1], reverse=True):
print(f" {symbol}: {pct:.2f}%")
print(f"\nTarget Allocation:")
for symbol, pct in sorted(target_allocation.items(), key=lambda x: x[1], reverse=True):
print(f" {symbol}: {pct:.2f}%")
# Execute rebalance
results = self.portfolio.execute_rebalance(
target_allocation=target_allocation,
threshold=threshold,
dry_run=dry_run,
safety_config=safety_config
)
# Print results
print(f"\nRebalance Status: {results['status']}")
if results['trades']:
print(f"\nTrades {'(DRY RUN)' if dry_run else '(EXECUTED)'}:")
for trade in results['trades']:
action = trade['action'].upper()
symbol = trade['symbol']
amount = trade.get('amount', 0)
value = trade.get('value', 0)
drift = trade.get('drift', 0)
print(f" {action} {amount} {symbol} (${value:.2f}) - Drift: {drift:+.2f}%")
else:
print("\nNo trades needed")
if results.get('errors'):
print(f"\nErrors:")
for error in results['errors']:
print(f" {error}")
print(f"{'='*50}\n")
return results
def check_drift(self) -> Dict[str, float]:
"""
Check current drift from target allocation
Returns:
Dictionary mapping symbols to drift percentages
"""
current = self.portfolio.get_current_allocation()
target = self.config.get('allocations', {})
# Normalize target
total_target = sum(target.values())
if total_target == 0:
return {}
normalized_target = {
symbol: (pct / total_target) * 100
for symbol, pct in target.items()
}
# Calculate drifts
drifts = {}
for symbol in set(list(current.keys()) + list(normalized_target.keys())):
current_pct = current.get(symbol, 0)
target_pct = normalized_target.get(symbol, 0)
drift = target_pct - current_pct
drifts[symbol] = drift
return drifts
def should_rebalance(self) -> bool:
"""
Check if rebalancing is needed based on threshold
Returns:
True if rebalancing is needed
"""
drifts = self.check_drift()
threshold = self.config.get('threshold', 5)
max_drift = max(abs(d) for d in drifts.values()) if drifts else 0
return max_drift >= threshold
src/core/safety.py
+155
View File
@@ -0,0 +1,155 @@
"""
Safety Manager
Circuit breakers, drawdown limits, and position size enforcement.
Last line of defense before any trade executes.
"""
from datetime import datetime, timedelta
from typing import Dict, Tuple
from loguru import logger
class SafetyManager:
"""Circuit breakers, drawdown limits, position size enforcement"""
def __init__(self, config: Dict, store=None):
self.config = config
self.store = store
# Limits from config
self.max_position_pct = config.get('max_position_pct', 20) / 100
self.max_concurrent_positions = config.get('max_concurrent_positions', 5)
self.max_daily_trades = config.get('max_daily_trades', 20)
self.max_daily_loss_pct = config.get('max_daily_loss_pct', 5) / 100
self.max_total_loss_pct = config.get('max_total_loss_pct', 15) / 100
self.min_trade_value = config.get('min_trade_value', 5.0)
self.initial_capital = config.get('initial_capital', 100.0)
# State tracking
self.trading_halted = False
self.halt_reason = ""
self.daily_pnl = 0.0
self.daily_trades = 0
self.daily_reset_time = datetime.utcnow().replace(hour=0, minute=0, second=0)
self.peak_equity = self.initial_capital
def is_trading_allowed(self) -> bool:
"""Check if trading is currently allowed"""
self._check_daily_reset()
return not self.trading_halted
def validate_trade(self, symbol: str, side: str, amount: float,
price: float, portfolio_value: float,
open_positions: int = 0) -> Tuple[bool, str]:
"""
Validate a proposed trade against safety constraints.
Returns: (allowed: bool, reason: str)
"""
self._check_daily_reset()
# Circuit breaker
if self.trading_halted:
return False, f"Trading halted: {self.halt_reason}"
trade_value = amount * price
# Min trade value
if trade_value < self.min_trade_value:
return False, f"Trade value ${trade_value:.2f} below minimum ${self.min_trade_value}"
# Max position size
if portfolio_value > 0:
position_pct = trade_value / portfolio_value
if position_pct > self.max_position_pct:
return False, (f"Position {position_pct:.1%} exceeds max "
f"{self.max_position_pct:.1%}")
# Max concurrent positions (for buys only)
if side == 'buy' and open_positions >= self.max_concurrent_positions:
return False, f"Max {self.max_concurrent_positions} concurrent positions reached"
# Max daily trades
if self.daily_trades >= self.max_daily_trades:
return False, f"Max {self.max_daily_trades} daily trades reached"
# Daily drawdown check
if portfolio_value > 0:
daily_loss = abs(self.daily_pnl) if self.daily_pnl < 0 else 0
if daily_loss / portfolio_value > self.max_daily_loss_pct:
self.trigger_circuit_breaker(
f"Daily loss {daily_loss/portfolio_value:.1%} exceeds limit"
)
return False, self.halt_reason
# Total drawdown check
if portfolio_value > 0 and self.peak_equity > 0:
total_dd = (self.peak_equity - portfolio_value) / self.peak_equity
if total_dd > self.max_total_loss_pct:
self.trigger_circuit_breaker(
f"Total drawdown {total_dd:.1%} exceeds {self.max_total_loss_pct:.1%}"
)
return False, self.halt_reason
return True, "Valid"
def record_trade_result(self, pnl: float):
"""Update daily P&L tracking after a trade closes"""
self.daily_pnl += pnl
self.daily_trades += 1
def update_peak_equity(self, equity: float):
"""Update peak equity for drawdown tracking"""
if equity > self.peak_equity:
self.peak_equity = equity
def check_daily_drawdown(self, portfolio_value: float):
"""Check daily P&L against limit"""
self._check_daily_reset()
if portfolio_value <= 0:
return
if self.daily_pnl < 0:
daily_loss_pct = abs(self.daily_pnl) / portfolio_value
if daily_loss_pct > self.max_daily_loss_pct:
self.trigger_circuit_breaker(
f"Daily loss {daily_loss_pct:.1%} exceeds {self.max_daily_loss_pct:.1%}"
)
def trigger_circuit_breaker(self, reason: str):
"""Halt trading"""
self.trading_halted = True
self.halt_reason = reason
logger.warning(f"CIRCUIT BREAKER: {reason}")
def reset_circuit_breaker(self):
"""Resume trading"""
self.trading_halted = False
self.halt_reason = ""
logger.info("Circuit breaker reset - trading resumed")
def _check_daily_reset(self):
"""Reset daily counters at midnight UTC"""
now = datetime.utcnow()
if now.date() > self.daily_reset_time.date():
self.daily_pnl = 0.0
self.daily_trades = 0
self.daily_reset_time = now.replace(hour=0, minute=0, second=0)
# Auto-reset circuit breaker on new day (if triggered by daily limit)
if self.trading_halted and 'Daily' in self.halt_reason:
self.reset_circuit_breaker()
logger.info("Daily circuit breaker auto-reset on new trading day")
def get_status(self) -> Dict:
"""Get safety manager status"""
return {
'trading_allowed': not self.trading_halted,
'halt_reason': self.halt_reason,
'daily_pnl': round(self.daily_pnl, 2),
'daily_trades': self.daily_trades,
'peak_equity': round(self.peak_equity, 2),
'max_position_pct': self.max_position_pct,
'max_daily_loss_pct': self.max_daily_loss_pct,
'max_total_loss_pct': self.max_total_loss_pct,
}
src/core/strategy_engine.py
+256
View File
@@ -0,0 +1,256 @@
"""
Strategy Engine
Manages and executes trading strategies
"""
import json
from typing import Dict, List, Optional
from datetime import datetime
from apscheduler.schedulers.background import BackgroundScheduler
from apscheduler.triggers.cron import CronTrigger
from ..adapters.crypto_adapter import CryptoAdapter
from ..adapters.stock_adapter import StockAdapter
from .rebalancer import Rebalancer
class StrategyEngine:
"""Main engine for managing and executing trading strategies"""
def __init__(self, config_path: str = 'config/config.json', strategies_path: str = 'config/strategies.json'):
"""
Initialize strategy engine
Args:
config_path: Path to main configuration file
strategies_path: Path to strategies configuration file
"""
self.config_path = config_path
self.strategies_path = strategies_path
self.config = self._load_config(config_path)
self.strategies = self._load_config(strategies_path)
self.adapters = {}
self.scheduler = BackgroundScheduler()
self.running_strategies = {}
def _load_config(self, path: str) -> Dict:
"""Load configuration from JSON file"""
try:
with open(path, 'r') as f:
return json.load(f)
except FileNotFoundError:
print(f"Config file not found: {path}")
return {}
except json.JSONDecodeError as e:
print(f"Error parsing {path}: {e}")
return {}
def _save_strategies(self):
"""Save strategies back to file"""
try:
with open(self.strategies_path, 'w') as f:
json.dump(self.strategies, f, indent=2)
except Exception as e:
print(f"Error saving strategies: {e}")
def initialize_adapters(self):
"""Initialize all enabled exchange and broker adapters"""
# Initialize crypto exchanges
for exchange_name, exchange_config in self.config.get('exchanges', {}).items():
if exchange_config.get('enabled', False):
try:
adapter = CryptoAdapter(exchange_name, exchange_config)
if adapter.connect():
self.adapters[exchange_name] = adapter
print(f"✓ Connected to {exchange_name}")
else:
print(f"✗ Failed to connect to {exchange_name}")
except Exception as e:
print(f"✗ Error initializing {exchange_name}: {e}")
# Initialize stock brokers
for broker_name, broker_config in self.config.get('brokers', {}).items():
if broker_config.get('enabled', False):
try:
adapter = StockAdapter(broker_name, broker_config)
if adapter.connect():
self.adapters[broker_name] = adapter
print(f"✓ Connected to {broker_name}")
else:
print(f"✗ Failed to connect to {broker_name}")
except Exception as e:
print(f"✗ Error initializing {broker_name}: {e}")
def get_adapter(self, name: str):
"""Get adapter by name"""
return self.adapters.get(name)
def load_strategies(self):
"""Load and schedule all enabled strategies"""
for strategy in self.strategies.get('strategies', []):
if strategy.get('enabled', False):
self.schedule_strategy(strategy)
def schedule_strategy(self, strategy: Dict):
"""
Schedule a strategy for execution
Args:
strategy: Strategy configuration dictionary
"""
strategy_name = strategy.get('name')
strategy_type = strategy.get('type')
schedule = strategy.get('schedule')
if not all([strategy_name, strategy_type, schedule]):
print(f"Invalid strategy configuration: {strategy_name}")
return
# Get appropriate adapter
adapter_name = strategy.get('exchange') or strategy.get('broker')
adapter = self.get_adapter(adapter_name)
if not adapter:
print(f"Adapter {adapter_name} not available for strategy {strategy_name}")
return
# Create strategy executor
if strategy_type == 'rebalance':
executor = Rebalancer(adapter, strategy)
else:
print(f"Strategy type {strategy_type} not yet implemented")
return
# Schedule execution
try:
dry_run = self.config.get('general', {}).get('dryRun', True)
def execute_strategy():
print(f"\n[{datetime.now().strftime('%Y-%m-%d %H:%M:%S')}] Executing strategy: {strategy_name}")
executor.execute(dry_run=dry_run)
# Parse cron schedule
trigger = CronTrigger.from_crontab(schedule)
job = self.scheduler.add_job(
execute_strategy,
trigger=trigger,
id=strategy_name,
name=strategy_name,
replace_existing=True
)
self.running_strategies[strategy_name] = {
'strategy': strategy,
'executor': executor,
'job': job
}
print(f"✓ Scheduled strategy: {strategy_name} ({schedule})")
except Exception as e:
print(f"✗ Error scheduling strategy {strategy_name}: {e}")
def execute_strategy_now(self, strategy_name: str, dry_run: Optional[bool] = None) -> Dict:
"""
Execute a strategy immediately
Args:
strategy_name: Name of the strategy to execute
dry_run: Override dry run setting
Returns:
Execution results
"""
if strategy_name in self.running_strategies:
executor = self.running_strategies[strategy_name]['executor']
if dry_run is None:
dry_run = self.config.get('general', {}).get('dryRun', True)
print(f"\n[{datetime.now().strftime('%Y-%m-%d %H:%M:%S')}] Manually executing: {strategy_name}")
return executor.execute(dry_run=dry_run)
else:
# Try to find strategy in config
for strategy in self.strategies.get('strategies', []):
if strategy.get('name') == strategy_name:
# Get adapter
adapter_name = strategy.get('exchange') or strategy.get('broker')
adapter = self.get_adapter(adapter_name)
if not adapter:
return {'error': f'Adapter {adapter_name} not available'}
# Execute
if strategy.get('type') == 'rebalance':
executor = Rebalancer(adapter, strategy)
if dry_run is None:
dry_run = self.config.get('general', {}).get('dryRun', True)
return executor.execute(dry_run=dry_run)
return {'error': f'Strategy {strategy_name} not found'}
def enable_strategy(self, strategy_name: str):
"""Enable a strategy"""
for strategy in self.strategies.get('strategies', []):
if strategy.get('name') == strategy_name:
strategy['enabled'] = True
self.schedule_strategy(strategy)
self._save_strategies()
return True
return False
def disable_strategy(self, strategy_name: str):
"""Disable a strategy"""
for strategy in self.strategies.get('strategies', []):
if strategy.get('name') == strategy_name:
strategy['enabled'] = False
if strategy_name in self.running_strategies:
self.scheduler.remove_job(strategy_name)
del self.running_strategies[strategy_name]
self._save_strategies()
return True
return False
def get_status(self) -> Dict:
"""Get engine status"""
return {
'adapters': list(self.adapters.keys()),
'strategies': {
name: {
'enabled': info['strategy'].get('enabled', False),
'schedule': info['strategy'].get('schedule'),
'next_run': info['job'].next_run_time.isoformat() if info['job'].next_run_time else None
}
for name, info in self.running_strategies.items()
},
'dry_run': self.config.get('general', {}).get('dryRun', True)
}
def start(self):
"""Start the strategy engine"""
print("\n" + "="*50)
print("BiggFish Strategy Engine Starting...")
print("="*50 + "\n")
# Initialize adapters
self.initialize_adapters()
if not self.adapters:
print("No adapters initialized. Please check configuration.")
return False
# Load strategies
self.load_strategies()
# Start scheduler
if self.running_strategies:
self.scheduler.start()
print(f"\n✓ Engine started with {len(self.running_strategies)} active strategies")
return True
else:
print("\nNo enabled strategies found")
return False
def stop(self):
"""Stop the strategy engine"""
self.scheduler.shutdown()
print("\nBiggFish Strategy Engine stopped")
src/main.py
+174
View File
@@ -0,0 +1,174 @@
"""
BIGGFISH - Autonomous Stock Trading System
Main entry point
"""
import json
import sys
from pathlib import Path
from loguru import logger
import schedule
import time
from datetime import datetime
# Setup logging
log_path = Path(__file__).parent.parent / "logs"
log_path.mkdir(exist_ok=True)
logger.add(
log_path / "biggfish_{time}.log",
rotation="1 day",
retention="30 days",
level="INFO"
)
from trading.broker import AlpacaBroker
from research.screener import StockScreener
from strategies.manager import StrategyManager
from reporting.reporter import Reporter
class BIGGFISH:
"""Main trading system orchestrator"""
def __init__(self, config_path="config/config.json"):
logger.info("🐟 Initializing BIGGFISH...")
# Load configuration
with open(config_path) as f:
self.config = json.load(f)
# Initialize components
self.broker = AlpacaBroker(self.config["alpaca"])
self.screener = StockScreener(self.config["research"])
self.strategy_manager = StrategyManager(self.config)
self.reporter = Reporter(self.config["reporting"])
self.running = False
logger.info("✅ BIGGFISH initialized successfully")
def start(self):
"""Start the trading system"""
logger.info("🚀 Starting BIGGFISH trading system")
self.running = True
# Initial market check
if self.broker.is_market_open():
logger.info("📈 Market is open - running initial scan")
self.run_cycle()
else:
logger.info("🔒 Market is closed")
# Schedule tasks
schedule.every(6).hours.do(self.run_research)
schedule.every(2).hours.do(self.check_news)
schedule.every().day.at("16:30").do(self.generate_daily_report)
# Main loop
logger.info("♾️ Entering main loop...")
while self.running:
try:
schedule.run_pending()
time.sleep(60) # Check every minute
except KeyboardInterrupt:
logger.info("⏹️ Shutting down BIGGFISH...")
self.running = False
except Exception as e:
logger.error(f"❌ Error in main loop: {e}", exc_info=True)
def run_cycle(self):
"""Run a complete trading cycle"""
logger.info("🔄 Running trading cycle...")
try:
# Get current portfolio status
portfolio = self.broker.get_portfolio()
logger.info(f"💰 Current portfolio value: ${portfolio['equity']:.2f}")
# Run research and screening
opportunities = self.run_research()
# Generate strategies
if opportunities:
strategies = self.strategy_manager.generate_strategies(
opportunities,
portfolio
)
# Check if strategies need approval
if self.config["trading"]["require_approval"] and strategies:
self.reporter.send_strategy_proposal(strategies)
logger.info("📋 Strategy proposals sent for approval")
except Exception as e:
logger.error(f"❌ Error in trading cycle: {e}", exc_info=True)
def run_research(self):
"""Run market research and screening"""
logger.info("🔬 Running market research...")
try:
# Get current portfolio value to determine focus
portfolio = self.broker.get_portfolio()
equity = float(portfolio["equity"])
# Determine market cap focus based on portfolio size
focus = self._determine_focus(equity)
logger.info(f"🎯 Current focus: {focus}")
# Screen for opportunities
opportunities = self.screener.scan(focus)
logger.info(f"📊 Found {len(opportunities)} opportunities")
return opportunities
except Exception as e:
logger.error(f"❌ Error in research: {e}", exc_info=True)
return []
def check_news(self):
"""Check for market-moving news"""
logger.info("📰 Checking news...")
# TODO: Implement news checking
pass
def generate_daily_report(self):
"""Generate and send daily performance report"""
logger.info("📊 Generating daily report...")
try:
portfolio = self.broker.get_portfolio()
positions = self.broker.get_positions()
report = self.reporter.generate_daily_report(portfolio, positions)
self.reporter.send_report(report)
logger.info("✅ Daily report sent")
except Exception as e:
logger.error(f"❌ Error generating report: {e}", exc_info=True)
def _determine_focus(self, equity):
"""Determine market cap focus based on portfolio size"""
rules = self.config["portfolio"]["transition_rules"]
if equity >= 700:
return "balanced"
elif equity >= 400:
return "mid_cap_heavy"
elif equity >= 200:
return "mid_cap_mixed"
else:
return "small_cap"
def main():
"""Main entry point"""
config_path = Path(__file__).parent.parent / "config" / "config.json"
if not config_path.exists():
logger.error("❌ Config file not found. Copy config.example.json to config.json")
sys.exit(1)
fish = BIGGFISH(config_path)
fish.start()
if __name__ == "__main__":
main()
src/main_auto.py
+804
View File
@@ -0,0 +1,804 @@
"""
BIGGFISH Autonomous Self-Learning Trading Bot
24/7 entry point - runs forever, learns continuously, trades autonomously.
Usage:
python -m src.main_auto
python -m src.main_auto --config config/auto_config.json
"""
import json
import sys
import time
import threading
import signal as signal_module
from pathlib import Path
from datetime import datetime, timedelta
from loguru import logger
import numpy as np
# Setup logging
log_path = Path(__file__).parent.parent / "logs"
log_path.mkdir(exist_ok=True)
logger.add(
log_path / "biggfish_auto_{time}.log",
rotation="1 day",
retention="30 days",
level="INFO"
)
from trading.broker import AlpacaBroker
from data.store import DataStore
from data.candle_cache import CandleCache
try:
from trading.oanda_broker import OandaBroker
except ImportError:
OandaBroker = None
from data.features import FeatureEngine
from backtest.engine import BacktestEngine
from backtest.metrics import compute_metrics
from ml.rl_agent import RLAgent
from ml.rl_environment import TradingEnvironment
from ml.genetic import GeneticEvolver, StrategyGenome
from strategies.auto_strategy import genome_to_strategy, evaluate_genome_signal
from trading.executor import TradingExecutor
from core.safety import SafetyManager
from reporting.telegram_reporter import TelegramReporter
from reporting.krystie_bridge import KrystieBridge
class BiggFishAuto:
"""
24/7 Autonomous self-learning trading bot.
Coordinates trading, backtesting, RL training, and GA evolution.
"""
def __init__(self, config_path: str = "config/auto_config.json"):
logger.info("Initializing BIGGFISH Autonomous Trader...")
# Load config
with open(config_path) as f:
self.config = json.load(f)
# Initialize components
self.store = DataStore(self.config['database']['path'])
self.store.initialize()
self.broker = AlpacaBroker(self.config['alpaca'])
self.feature_engine = FeatureEngine()
# OANDA broker for forex (optional)
self.oanda_broker = None
oanda_config = self.config.get('oanda')
if oanda_config and oanda_config.get('api_token') and OandaBroker:
try:
self.oanda_broker = OandaBroker(oanda_config)
logger.info("OANDA forex broker connected")
except Exception as e:
logger.warning(f"OANDA connection failed (forex disabled): {e}")
self.candle_cache = CandleCache(self.store, oanda_broker=self.oanda_broker)
self.backtest_engine = BacktestEngine(
initial_capital=self.config['backtest']['initial_capital'],
commission_rate=self.config['trading'].get('commission_rate', 0.001)
)
self.safety = SafetyManager(self.config['safety'], self.store)
# Executors per broker
self.executor = TradingExecutor(
self.broker, self.store, self.safety, self.config['trading']
)
self.forex_executor = None
if self.oanda_broker:
self.forex_executor = TradingExecutor(
self.oanda_broker, self.store, self.safety, self.config['trading']
)
# RL Agent
env_temp = TradingEnvironment(self.feature_engine,
initial_capital=self.config['trading']['initial_capital'])
self.rl_agent = RLAgent(
state_dim=env_temp.state_dim,
action_dim=TradingEnvironment.NUM_ACTIONS,
config=self.config['rl']
)
# GA Evolver
self.ga_evolver = GeneticEvolver(
self.config['ga'], self.backtest_engine, self.store
)
# Telegram daily reporter
tg_config = self.config.get('telegram', {})
if tg_config.get('enabled') and tg_config.get('bot_token') and tg_config.get('chat_id'):
self.telegram = TelegramReporter(tg_config['bot_token'], tg_config['chat_id'])
logger.info("Telegram reporting enabled")
else:
self.telegram = None
logger.info("Telegram reporting disabled (no config)")
# Krystie bridge (writes status/events to JSON for Krystie to read)
self.krystie = KrystieBridge(self.config['database']['path'].rsplit('/', 1)[0] or 'data')
# State
self.running = False
self.start_time = None
self.cycle_count = 0
self.last_backtest = datetime.min
self.last_rl_train = datetime.min
self.last_ga_evolve = datetime.min
self.last_dashboard = datetime.min
self.last_daily_report = datetime.min
self.recent_trades = [] # Last 10 trades for dashboard
logger.info("BIGGFISH Autonomous Trader initialized")
def start(self):
"""Start the autonomous trading system"""
self.running = True
self.start_time = datetime.utcnow()
# Register shutdown handler
signal_module.signal(signal_module.SIGINT, self._signal_handler)
logger.info("=" * 60)
logger.info(" BIGGFISH AUTONOMOUS TRADER STARTING")
logger.info("=" * 60)
# 1. Load saved state
self._load_state()
self.krystie.log_startup()
# 2. Warm candle cache
self._warm_cache()
# 3. Initial GA population
self.ga_evolver.initialize_population()
if not self.ga_evolver.population:
logger.info("Running initial GA evolution...")
self._run_ga_evolution()
# 4. Initial RL training if no checkpoint
if self.rl_agent.steps == 0:
logger.info("Running initial RL training...")
self._run_rl_training()
# 5. Main loop
logger.info("Entering main loop...")
self._print_dashboard()
while self.running:
try:
cycle_start = datetime.utcnow()
# Trading cycle (run if any market is open)
any_market_open = self.broker.is_market_open()
if self.oanda_broker:
any_market_open = any_market_open or self.oanda_broker.is_market_open()
if any_market_open:
self._trading_cycle()
else:
logger.debug("All markets closed - running learning tasks")
# Periodic tasks (run regardless of market hours)
now = datetime.utcnow()
# Cache update (every 5 min)
cache_interval = self.config['cache']['update_interval_seconds']
if (now - self.last_backtest).total_seconds() > cache_interval:
self._update_cache()
# Backtest (every 30 min)
bt_interval = self.config['backtest']['interval_seconds']
if (now - self.last_backtest).total_seconds() > bt_interval:
self._run_backtest()
self.last_backtest = now
# RL training (every 2 hours)
rl_interval = self.config['rl']['train_interval_hours'] * 3600
if (now - self.last_rl_train).total_seconds() > rl_interval:
self._run_rl_training()
self.last_rl_train = now
# GA evolution (every 6 hours)
ga_interval = self.config['ga']['evolution_interval_hours'] * 3600
if (now - self.last_ga_evolve).total_seconds() > ga_interval:
self._run_ga_evolution()
self.last_ga_evolve = now
# Daily Telegram report (once per day, around market close ~21:00 UTC)
if self.telegram and self._should_send_daily_report(now):
self._send_daily_telegram_report()
self.last_daily_report = now
# Dashboard (every minute)
dash_interval = self.config['reporting']['dashboard_interval_seconds']
if (now - self.last_dashboard).total_seconds() > dash_interval:
self._print_dashboard()
self.last_dashboard = now
# Sleep until next cycle
elapsed = (datetime.utcnow() - cycle_start).total_seconds()
sleep_time = max(1, self.config['trading']['cycle_interval_seconds'] - elapsed)
time.sleep(sleep_time)
except KeyboardInterrupt:
self._shutdown()
break
except Exception as e:
logger.error(f"Main loop error: {e}", exc_info=True)
time.sleep(30)
def _is_forex(self, symbol: str) -> bool:
"""Check if symbol is a forex pair (OANDA format: XXX_YYY)"""
return '_' in symbol and len(symbol) == 7
def _get_broker(self, symbol: str):
"""Get the appropriate broker for a symbol"""
if self._is_forex(symbol) and self.oanda_broker:
return self.oanda_broker
return self.broker
def _get_executor(self, symbol: str):
"""Get the appropriate executor for a symbol"""
if self._is_forex(symbol) and self.forex_executor:
return self.forex_executor
return self.executor
def _trading_cycle(self):
"""Run one trading cycle across all markets"""
self.cycle_count += 1
all_symbols = self._get_tradeable_symbols()
# Check exits first
try:
current_prices = {}
for symbol in all_symbols:
broker = self._get_broker(symbol)
price = broker.get_latest_price(symbol)
if price:
current_prices[symbol] = price
# Check exits per executor
stock_prices = {s: p for s, p in current_prices.items() if not self._is_forex(s)}
forex_prices = {s: p for s, p in current_prices.items() if self._is_forex(s)}
if stock_prices:
closed = self.executor.check_exits(stock_prices)
for trade in closed:
logger.info(f"CLOSED {trade['symbol']}: ${trade['pnl']:+.2f} "
f"({trade['pnl_pct']:+.1f}%) [{trade['exit_reason']}]")
self.recent_trades.append(trade)
self.krystie.log_trade(trade)
if forex_prices and self.forex_executor:
closed = self.forex_executor.check_exits(forex_prices)
for trade in closed:
logger.info(f"CLOSED {trade['symbol']}: ${trade['pnl']:+.2f} "
f"({trade['pnl_pct']:+.1f}%) [{trade['exit_reason']}]")
self.recent_trades.append(trade)
self.krystie.log_trade(trade)
except Exception as e:
logger.error(f"Exit check error: {e}")
# Check safety
if not self.safety.is_trading_allowed():
logger.warning(f"Trading halted: {self.safety.halt_reason}")
return
# Evaluate each symbol
for symbol in all_symbols:
try:
self._trade_symbol(symbol)
except Exception as e:
logger.debug(f"Error trading {symbol}: {e}")
# Keep only last 20 trades
self.recent_trades = self.recent_trades[-20:]
def _get_tradeable_symbols(self) -> list:
"""Get symbols that can be traded right now"""
stock_symbols = self.config['trading'].get('symbols', [])
forex_symbols = self.config['trading'].get('forex_symbols', [])
tradeable = []
# Stocks: only if Alpaca market is open
if self.broker.is_market_open():
tradeable.extend(stock_symbols)
# Forex: if OANDA is connected and forex market is open (24/5)
if self.oanda_broker and self.oanda_broker.is_market_open():
tradeable.extend(forex_symbols)
return tradeable
def _trade_symbol(self, symbol: str):
"""Evaluate and potentially trade a single symbol (scalping mode)"""
broker = self._get_broker(symbol)
executor = self._get_executor(symbol)
# Use 5m candles for scalping, fall back to 1h
df = self.candle_cache.get_cached(
symbol, '5m',
start=datetime.utcnow() - timedelta(days=14)
)
if df is None or len(df) < 60:
df = self.candle_cache.get_cached(
symbol, '1h',
start=datetime.utcnow() - timedelta(days=14)
)
if df is None or len(df) < 60:
return
# Compute features
features_df = self.feature_engine.compute_and_normalize(df)
if features_df is None or len(features_df) < 10:
return
# Get market state
state = self.feature_engine.get_state_vector(features_df, -1)
# Get GA signal
best_genome = self.ga_evolver.get_best_genome()
ga_signal = {'signal': 'hold', 'confidence': 0.0}
strategy_params = {}
if best_genome:
raw_df = self.feature_engine.compute(df)
if raw_df is not None and len(raw_df) > 0:
ga_signal = evaluate_genome_signal(best_genome, raw_df, len(raw_df) - 1)
strategy_params = {
'stop_loss': ga_signal.get('stop_loss'),
'take_profit': ga_signal.get('take_profit'),
'short_stop_loss': ga_signal.get('short_stop_loss'),
'short_take_profit': ga_signal.get('short_take_profit'),
'position_pct': ga_signal.get('position_pct', 0.1),
'strategy_id': f"ga_gen{best_genome.generation}",
}
# Augment state with GA signal + portfolio
portfolio_state = executor.get_portfolio_state()
signal_dir = 1.0 if ga_signal['signal'] == 'buy' else (
-1.0 if ga_signal['signal'] == 'sell' else 0.0)
augmented_state = np.concatenate([
state,
[portfolio_state.get('position_ratio', 0),
portfolio_state.get('unrealized_pnl', 0),
portfolio_state.get('time_in_position', 0)],
[signal_dir, ga_signal.get('confidence', 0.0)],
])
# RL agent decides (now with 7 actions including shorts)
action = self.rl_agent.select_action(augmented_state, live_mode=True)
if action == 0: # Hold
return
# Get current price
current_price = broker.get_latest_price(symbol)
if not current_price:
return
# Execute
trade = executor.execute_signal(
symbol, action, current_price, strategy_params
)
if trade:
action_name = TradingEnvironment.ACTION_NAMES[action]
logger.info(f"TRADE: {action_name} {symbol} @ ${current_price:.4f}")
self.recent_trades.append(trade)
self.krystie.log_trade(trade)
def _all_symbols(self) -> list:
"""Get all configured symbols (stocks + forex)"""
symbols = list(self.config['trading'].get('symbols', []))
symbols.extend(self.config['trading'].get('forex_symbols', []))
return symbols
def _warm_cache(self):
"""Warm the candle cache"""
logger.info("Warming candle cache...")
symbols = self._all_symbols()
timeframes = self.config['cache'].get('timeframes', ['1h'])
lookback = self.config['cache'].get('warmup_lookback_days', 90)
self.candle_cache.warm_cache(symbols, timeframes, lookback)
logger.info("Cache warmup complete")
def _update_cache(self):
"""Update candle cache incrementally"""
symbols = self._all_symbols()
timeframes = self.config['cache'].get('timeframes', ['1h'])
self.candle_cache.update_cache(symbols, timeframes)
def _run_backtest(self):
"""Run background backtest of current strategy (scalping timeframe)"""
best_genome = self.ga_evolver.get_best_genome()
if not best_genome:
return
strategy_fn = genome_to_strategy(best_genome)
symbols = self._all_symbols()[:5] # Top 5 for breadth
lookback = self.config['backtest'].get('lookback_days', 14)
for symbol in symbols:
# Prefer 5m data for scalping backtest
df = self.candle_cache.get_cached(
symbol, '5m',
start=datetime.utcnow() - timedelta(days=lookback)
)
if df is None or len(df) < 60:
df = self.candle_cache.get_cached(
symbol, '1h',
start=datetime.utcnow() - timedelta(days=lookback)
)
if df is None or len(df) < 60:
continue
featured_df = self.feature_engine.compute(df)
if featured_df is None or len(featured_df) < 50:
continue
result = self.backtest_engine.run(
strategy_fn, featured_df,
params=best_genome.to_dict(), symbol=symbol
)
if result.metrics.get('total_trades', 0) > 0:
self.store.record_strategy_result(
strategy_id=f"ga_gen{best_genome.generation}",
params=best_genome.to_dict(),
metrics=result.metrics
)
logger.info(f"Backtest {symbol}: Sharpe={result.metrics['sharpe_ratio']:.2f} "
f"WR={result.metrics['win_rate']:.0f}% "
f"Return={result.metrics['total_return']:.1f}%")
def _run_rl_training(self):
"""Run RL batch training"""
logger.info("Starting RL training...")
env = TradingEnvironment(
self.feature_engine,
initial_capital=self.config['trading']['initial_capital']
)
total_metrics = {'avg_loss': 0, 'episodes': 0}
for symbol in self._all_symbols()[:5]:
# Prefer 5m data for RL training (more scalping episodes)
df = self.candle_cache.get_cached(
symbol, '5m',
start=datetime.utcnow() - timedelta(days=14)
)
if df is None or len(df) < 100:
df = self.candle_cache.get_cached(
symbol, '1h',
start=datetime.utcnow() - timedelta(days=14)
)
if df is None or len(df) < 100:
continue
# Create GA signal function for this data
best_genome = self.ga_evolver.get_best_genome()
ga_fn = None
if best_genome:
featured = self.feature_engine.compute(df)
if featured is not None and len(featured) > 0:
def make_ga_fn(genome, feat_df):
def fn(step):
if step < len(feat_df):
return evaluate_genome_signal(genome, feat_df, step)
return {'signal': 'hold', 'confidence': 0.0}
return fn
ga_fn = make_ga_fn(best_genome, featured)
metrics = self.rl_agent.train_on_episode(env, df, ga_signal_fn=ga_fn)
total_metrics['avg_loss'] += metrics.get('avg_loss', 0)
total_metrics['episodes'] += 1
# Save checkpoint
self.rl_agent.save(self.store)
if total_metrics['episodes'] > 0:
avg = total_metrics['avg_loss'] / total_metrics['episodes']
logger.info(f"RL training complete: {total_metrics['episodes']} episodes, "
f"avg_loss={avg:.6f}, epsilon={self.rl_agent.epsilon:.4f}")
def _run_ga_evolution(self):
"""Run GA evolution cycle"""
logger.info("Starting GA evolution...")
# Prepare candle data for fitness evaluation
candles_data = {}
symbols = self._all_symbols()[:5]
for symbol in symbols:
df = self.candle_cache.get_cached(
symbol, '5m',
start=datetime.utcnow() - timedelta(days=14)
)
if df is None or len(df) < 60:
df = self.candle_cache.get_cached(
symbol, '1h',
start=datetime.utcnow() - timedelta(days=14)
)
if df is not None and len(df) > 60:
featured = self.feature_engine.compute(df)
if featured is not None and len(featured) > 50:
candles_data[symbol] = featured
if not candles_data:
logger.warning("No data available for GA evolution")
return
# Strategy function factory
def strategy_fn_factory(genome):
return genome_to_strategy(genome)
# Run evolution
num_gens = self.config['ga'].get('generations_per_cycle', 10)
best = self.ga_evolver.run_evolution_cycle(
strategy_fn_factory, candles_data, num_generations=num_gens
)
if best:
logger.info(f"GA evolution complete: Gen {self.ga_evolver.generation}, "
f"best fitness={best.fitness:.4f}")
self.krystie.log_ga_milestone(self.ga_evolver.generation, best.fitness)
def _should_send_daily_report(self, now: datetime) -> bool:
"""Check if it's time to send the daily report (once per day after 21:00 UTC)"""
report_hour = self.config.get('telegram', {}).get('daily_report_hour', 21)
if now.hour >= report_hour and (now - self.last_daily_report).total_seconds() > 20 * 3600:
return True
return False
def _send_daily_telegram_report(self):
"""Gather data and send the daily Telegram report"""
try:
portfolio = self.broker.get_portfolio()
positions = self.broker.get_positions()
# Add OANDA positions if available
if self.oanda_broker:
try:
oanda_positions = self.oanda_broker.get_positions()
positions.extend(oanda_positions)
except Exception:
pass
# Get today's trades from DB
today_start = datetime.utcnow().replace(hour=0, minute=0, second=0, microsecond=0)
today_trades = self.store.get_trades(start=today_start, limit=50)
# Learning stats
rl_stats = self.rl_agent.get_stats()
best_genome = self.ga_evolver.get_best_genome()
learning_stats = {
'rl': rl_stats,
'ga': {
'generation': self.ga_evolver.generation,
'best_fitness': best_genome.fitness if best_genome else 0,
},
}
self.telegram.send_daily_report(
portfolio, positions, today_trades, learning_stats,
self.config['trading']
)
self.krystie.log_daily_report(
equity=portfolio.get('equity', 0),
day_pnl=portfolio.get('day_pnl', 0),
trades_count=len(today_trades),
)
logger.info("Daily Telegram report sent")
except Exception as e:
logger.error(f"Failed to send daily Telegram report: {e}")
def _print_dashboard(self):
"""Print live console dashboard"""
try:
portfolio = self.broker.get_portfolio()
positions = self.broker.get_positions()
except Exception as e:
logger.error(f"Dashboard error: {e}")
return
equity = portfolio['equity']
target = self.config['trading']['target_capital']
initial = self.config['trading']['initial_capital']
progress = (equity / target) * 100
total_pnl = equity - initial
total_pnl_pct = (total_pnl / initial) * 100
uptime = datetime.utcnow() - self.start_time if self.start_time else timedelta()
hours = int(uptime.total_seconds() // 3600)
minutes = int((uptime.total_seconds() % 3600) // 60)
best_genome = self.ga_evolver.get_best_genome()
gen = self.ga_evolver.generation
best_fit = best_genome.fitness if best_genome else 0
rl_stats = self.rl_agent.get_stats()
safety_status = self.safety.get_status()
stock_status = "OPEN" if self.broker.is_market_open() else "CLOSED"
forex_status = ""
if self.oanda_broker:
forex_status = " | FX: " + ("OPEN" if self.oanda_broker.is_market_open() else "CLOSED")
# Build dashboard
lines = []
lines.append("")
lines.append("=" * 64)
lines.append(f" BIGGFISH AUTONOMOUS TRADER | {hours}h {minutes}m | "
f"Gen {gen} | Stocks: {stock_status}{forex_status}")
lines.append("=" * 64)
# Progress bar
bar_width = 30
filled = int(bar_width * min(progress, 100) / 100)
bar = "#" * filled + "-" * (bar_width - filled)
lines.append(f" Portfolio: ${equity:.2f} / ${target} [{bar}] {progress:.1f}%")
lines.append(f" Day P&L: ${portfolio['day_pnl']:+.2f} ({portfolio['day_pnl_pct']:+.1f}%)")
lines.append(f" Total P&L: ${total_pnl:+.2f} ({total_pnl_pct:+.1f}%)")
# Positions
lines.append("-" * 64)
if positions:
lines.append(f" Active Positions ({len(positions)}):")
for p in positions[:5]:
pl_str = f"${p['unrealized_pl']:+.2f} ({p['unrealized_plpc']:+.1f}%)"
lines.append(f" {p['symbol']:6s} {p['qty']:.0f} @ ${p['avg_entry_price']:.2f}"
f" -> ${p['current_price']:.2f} {pl_str}")
else:
lines.append(" No active positions")
# Learning status
lines.append("-" * 64)
lines.append(" Learning Status:")
lines.append(f" RL Agent: epsilon={rl_stats['epsilon']:.3f} | "
f"loss={rl_stats['avg_loss']:.6f} | "
f"{rl_stats['memory_size']:,} experiences")
lines.append(f" GA: gen {gen} | best fitness={best_fit:.4f}")
# Recent trades
if self.recent_trades:
lines.append("-" * 64)
lines.append(" Recent Trades:")
for t in self.recent_trades[-5:]:
side = t.get('side', '?').upper()
symbol = t.get('symbol', '?')
pnl = t.get('pnl')
if pnl is not None:
pnl_str = f" P&L: ${pnl:+.2f}"
else:
pnl_str = ""
price = t.get('entry_price') or t.get('exit_price', 0)
lines.append(f" {side:4s} {symbol:6s} "
f"{t.get('amount', 0):.1f} @ ${price:.2f}{pnl_str}")
# Safety
if not safety_status['trading_allowed']:
lines.append("-" * 64)
lines.append(f" !! TRADING HALTED: {safety_status['halt_reason']}")
# Next events
lines.append("-" * 64)
now = datetime.utcnow()
bt_next = max(0, self.config['backtest']['interval_seconds'] -
(now - self.last_backtest).total_seconds())
rl_next = max(0, self.config['rl']['train_interval_hours'] * 3600 -
(now - self.last_rl_train).total_seconds())
ga_next = max(0, self.config['ga']['evolution_interval_hours'] * 3600 -
(now - self.last_ga_evolve).total_seconds())
lines.append(f" Next: backtest {bt_next/60:.0f}m | "
f"RL train {rl_next/60:.0f}m | "
f"GA evolve {ga_next/3600:.1f}h")
lines.append("=" * 64)
print("\n".join(lines))
# Update Krystie status file
try:
markets = {"stocks": stock_status}
if self.oanda_broker:
markets["forex"] = "OPEN" if self.oanda_broker.is_market_open() else "CLOSED"
learning_stats = {
'rl': rl_stats,
'ga': {'generation': gen, 'best_fitness': best_fit},
}
today_start = datetime.utcnow().replace(hour=0, minute=0, second=0, microsecond=0)
today_trades = self.store.get_trades(start=today_start, limit=50)
self.krystie.update_status(
portfolio=portfolio,
positions=positions,
learning_stats=learning_stats,
markets=markets,
config=self.config['trading'],
uptime_seconds=uptime.total_seconds(),
today_trades=today_trades,
)
except Exception as e:
logger.debug(f"Krystie status update error: {e}")
def _load_state(self):
"""Load saved state from database"""
# Load RL model
if self.rl_agent.load(self.store):
logger.info("Loaded RL model from checkpoint")
# Load timing state
last_bt = self.store.load_state('last_backtest')
if last_bt:
self.last_backtest = datetime.fromisoformat(last_bt)
last_rl = self.store.load_state('last_rl_train')
if last_rl:
self.last_rl_train = datetime.fromisoformat(last_rl)
last_ga = self.store.load_state('last_ga_evolve')
if last_ga:
self.last_ga_evolve = datetime.fromisoformat(last_ga)
last_report = self.store.load_state('last_daily_report')
if last_report:
self.last_daily_report = datetime.fromisoformat(last_report)
logger.info("State loaded from database")
def _save_state(self):
"""Save state to database for recovery"""
self.rl_agent.save(self.store)
self.store.save_state('last_backtest', self.last_backtest.isoformat())
self.store.save_state('last_rl_train', self.last_rl_train.isoformat())
self.store.save_state('last_ga_evolve', self.last_ga_evolve.isoformat())
self.store.save_state('last_daily_report', self.last_daily_report.isoformat())
self.store.save_state('last_shutdown', datetime.utcnow().isoformat())
logger.info("State saved to database")
def _signal_handler(self, signum, frame):
"""Handle SIGINT for graceful shutdown"""
self._shutdown()
def _shutdown(self):
"""Graceful shutdown"""
logger.info("Shutting down BIGGFISH...")
self.running = False
self.krystie.log_shutdown()
self._save_state()
self.store.close()
logger.info("Shutdown complete. State saved. Resume anytime.")
def main():
import argparse
parser = argparse.ArgumentParser(description="BIGGFISH Autonomous Trader")
parser.add_argument('--config', default='config/auto_config.json',
help='Path to configuration file')
args = parser.parse_args()
config_path = Path(args.config)
if not config_path.exists():
print(f"Config file not found: {config_path}")
print("Copy config/auto_config.json and fill in your Alpaca API keys")
sys.exit(1)
bot = BiggFishAuto(str(config_path))
bot.start()
if __name__ == '__main__':
main()
src/ml/__init__.py
View File
src/ml/genetic.py
+483
View File
@@ -0,0 +1,483 @@
"""
Genetic Algorithm for Strategy Parameter Optimization
Evolves a population of StrategyGenome instances to find profitable trading parameters.
Optimized: uses vectorized backtesting and parallel genome evaluation.
"""
import random
import math
import json
from concurrent.futures import ProcessPoolExecutor, as_completed
from dataclasses import dataclass, field, asdict
from typing import Dict, List, Optional, Tuple
from loguru import logger
# Gene ranges for scalping: (min, max, is_int)
# Shorter periods, tighter SL/TP, faster hold times
GENE_RANGES = {
'fast_ma_period': (3, 20, True),
'slow_ma_period': (10, 60, True),
'rsi_period': (5, 21, True),
'rsi_overbought': (60, 80, False),
'rsi_oversold': (20, 40, False),
'bb_period': (8, 25, True),
'bb_std': (1.5, 2.5, False),
'atr_period': (5, 14, True),
'macd_fast': (5, 12, True),
'macd_slow': (12, 26, True),
'macd_signal': (5, 9, True),
'volume_surge_threshold': (1.1, 2.5, False),
'stop_loss_atr_mult': (0.5, 2.5, False),
'take_profit_atr_mult': (0.8, 3.0, False),
'max_position_pct': (0.05, 0.15, False),
'min_hold_candles': (1, 6, True),
'max_hold_candles': (3, 36, True),
}
@dataclass
class StrategyGenome:
"""A genome encoding all tunable strategy parameters"""
# Indicator periods (scalping-tuned defaults)
fast_ma_period: int = 8
slow_ma_period: int = 21
rsi_period: int = 9
rsi_overbought: float = 70.0
rsi_oversold: float = 30.0
bb_period: int = 15
bb_std: float = 2.0
atr_period: int = 10
macd_fast: int = 8
macd_slow: int = 17
macd_signal: int = 7
# Entry thresholds
volume_surge_threshold: float = 1.3
# Risk management (tighter for scalping)
stop_loss_atr_mult: float = 1.2
take_profit_atr_mult: float = 1.8
max_position_pct: float = 0.10
# Timing (short holds for scalping)
min_hold_candles: int = 1
max_hold_candles: int = 12
# Fitness (set after evaluation)
fitness: float = 0.0
generation: int = 0
def to_dict(self) -> Dict:
return asdict(self)
@classmethod
def from_dict(cls, d: Dict) -> 'StrategyGenome':
valid_fields = {f.name for f in cls.__dataclass_fields__.values()}
filtered = {k: v for k, v in d.items() if k in valid_fields}
return cls(**filtered)
@classmethod
def random(cls, generation: int = 0) -> 'StrategyGenome':
"""Create a random genome within valid parameter ranges"""
genes = {}
for gene_name, (lo, hi, is_int) in GENE_RANGES.items():
if is_int:
genes[gene_name] = random.randint(int(lo), int(hi))
else:
genes[gene_name] = round(random.uniform(lo, hi), 4)
# Constraint: slow_ma > fast_ma
if genes['slow_ma_period'] <= genes['fast_ma_period']:
genes['slow_ma_period'] = genes['fast_ma_period'] + random.randint(10, 50)
# Constraint: macd_slow > macd_fast
if genes['macd_slow'] <= genes['macd_fast']:
genes['macd_slow'] = genes['macd_fast'] + random.randint(8, 16)
# Constraint: take_profit > stop_loss
if genes['take_profit_atr_mult'] <= genes['stop_loss_atr_mult']:
genes['take_profit_atr_mult'] = genes['stop_loss_atr_mult'] + random.uniform(0.5, 2.0)
# Constraint: max_hold > min_hold
if genes['max_hold_candles'] <= genes['min_hold_candles']:
genes['max_hold_candles'] = genes['min_hold_candles'] + random.randint(10, 50)
genes['generation'] = generation
return cls(**genes)
def _evaluate_genome_worker(genome_dict: Dict, candles_dict: Dict[str, Dict],
initial_capital: float, commission_rate: float) -> float:
"""
Worker function for parallel genome evaluation.
Runs in a separate process, so must be a top-level function.
Uses vectorized signal generation + fast backtest.
"""
from strategies.auto_strategy import genome_to_signals
from backtest.engine import BacktestEngine
import pandas as pd
genome = StrategyGenome.from_dict(genome_dict)
engine = BacktestEngine(initial_capital=initial_capital, commission_rate=commission_rate)
fitness_scores = []
for symbol, candle_data in candles_dict.items():
df = pd.DataFrame(candle_data)
if len(df) < 60:
continue
try:
signals = genome_to_signals(genome, df)
result = engine.run_fast(signals, df, symbol=symbol)
except Exception:
continue
m = result.metrics
sharpe = max(m.get('sharpe_ratio', 0), 0)
max_dd = m.get('max_drawdown', 0)
total_trades = m.get('total_trades', 0)
win_rate = m.get('win_rate', 0) / 100.0 # 0-1
dd_penalty = max(1 - max_dd / 100, 0)
# Scalping: reward higher trade frequency more aggressively
trade_bonus = math.sqrt(max(total_trades, 0))
# Bonus for win rate > 50%
wr_bonus = 1.0 + max(0, win_rate - 0.5) * 0.5
if total_trades < 5:
trade_bonus *= 0.3 # Scalping needs more trades
score = sharpe * dd_penalty * trade_bonus * wr_bonus
fitness_scores.append(score)
if not fitness_scores:
return 0.0
return sum(fitness_scores) / len(fitness_scores)
class GeneticEvolver:
"""
Genetic algorithm for optimizing strategy parameters.
Evolves a population of StrategyGenome instances.
"""
def __init__(self, config: Dict, backtest_engine, store=None):
self.population_size = config.get('population_size', 50)
self.elite_count = config.get('elite_count', 5)
self.mutation_rate = config.get('mutation_rate', 0.15)
self.mutation_strength = config.get('mutation_strength', 0.2)
self.crossover_rate = config.get('crossover_rate', 0.7)
self.tournament_size = config.get('tournament_size', 5)
self.parallel_workers = config.get('parallel_workers', 4)
self.backtest_engine = backtest_engine
self.store = store
self.population: List[StrategyGenome] = []
self.generation = 0
self.best_ever: Optional[StrategyGenome] = None
def initialize_population(self):
"""
Create initial population.
If evolution history exists in DB, load the latest generation.
Otherwise, create random genomes.
"""
if self.store:
latest = self.store.get_latest_generation()
if latest:
self.generation = latest['generation']
self.population = [
StrategyGenome.from_dict(g) for g in latest['population']
]
if self.population:
self.best_ever = max(self.population, key=lambda g: g.fitness)
logger.info(f"Loaded GA population from generation {self.generation} "
f"({len(self.population)} genomes)")
return
# Create random population
self.population = [
StrategyGenome.random(generation=0)
for _ in range(self.population_size)
]
self.generation = 0
logger.info(f"Created random population of {self.population_size} genomes")
def evaluate_fitness(self, genome: StrategyGenome, strategy_fn_factory,
candles_data: Dict[str, 'pd.DataFrame']) -> float:
"""
Evaluate a genome using vectorized fast backtest.
Falls back to callback-based if signals not available.
"""
from strategies.auto_strategy import genome_to_signals
fitness_scores = []
for symbol, df in candles_data.items():
if df is None or len(df) < 60:
continue
try:
signals = genome_to_signals(genome, df)
result = self.backtest_engine.run_fast(signals, df, symbol=symbol)
except Exception:
# Fallback to callback-based
strategy_fn = strategy_fn_factory(genome)
result = self.backtest_engine.run(
strategy_fn, df, params=genome.to_dict(), symbol=symbol
)
m = result.metrics
sharpe = max(m.get('sharpe_ratio', 0), 0)
max_dd = m.get('max_drawdown', 0)
total_trades = m.get('total_trades', 0)
win_rate = m.get('win_rate', 0) / 100.0
dd_penalty = max(1 - max_dd / 100, 0)
trade_bonus = math.sqrt(max(total_trades, 0))
wr_bonus = 1.0 + max(0, win_rate - 0.5) * 0.5
if total_trades < 5:
trade_bonus *= 0.3
score = sharpe * dd_penalty * trade_bonus * wr_bonus
fitness_scores.append(score)
if not fitness_scores:
return 0.0
return sum(fitness_scores) / len(fitness_scores)
def evaluate_population(self, strategy_fn_factory,
candles_data: Dict[str, 'pd.DataFrame']):
"""Evaluate all genomes - uses parallel workers if available."""
unevaluated = [(i, g) for i, g in enumerate(self.population) if g.fitness == 0.0]
if not unevaluated:
return
# Prepare serializable candle data for parallel workers
candles_dict = {}
for symbol, df in candles_data.items():
if df is not None and len(df) >= 60:
candles_dict[symbol] = {
'open': df['open'].values.tolist(),
'high': df['high'].values.tolist(),
'low': df['low'].values.tolist(),
'close': df['close'].values.tolist(),
'volume': df['volume'].values.tolist(),
}
if not candles_dict:
return
initial_capital = self.backtest_engine.initial_capital
commission_rate = self.backtest_engine.commission_rate
# Try parallel evaluation
if self.parallel_workers > 1 and len(unevaluated) > 4:
try:
self._evaluate_parallel(
unevaluated, candles_dict, initial_capital,
commission_rate, strategy_fn_factory, candles_data
)
return
except Exception as e:
logger.debug(f"Parallel evaluation failed, falling back to sequential: {e}")
# Sequential fallback (still uses vectorized fast path)
for idx, (i, genome) in enumerate(unevaluated):
genome.fitness = self.evaluate_fitness(
genome, strategy_fn_factory, candles_data
)
if (idx + 1) % 10 == 0:
logger.debug(f"Evaluated {idx+1}/{len(unevaluated)} genomes")
def _evaluate_parallel(self, unevaluated, candles_dict, initial_capital,
commission_rate, strategy_fn_factory, candles_data):
"""Evaluate genomes in parallel using ProcessPoolExecutor."""
genome_dicts = [(i, g.to_dict()) for i, g in unevaluated]
with ProcessPoolExecutor(max_workers=self.parallel_workers) as executor:
futures = {}
for i, gd in genome_dicts:
fut = executor.submit(
_evaluate_genome_worker, gd, candles_dict,
initial_capital, commission_rate
)
futures[fut] = i
done_count = 0
for future in as_completed(futures):
pop_idx = futures[future]
try:
fitness = future.result(timeout=30)
self.population[pop_idx].fitness = fitness
except Exception:
# Fallback for this genome
self.population[pop_idx].fitness = self.evaluate_fitness(
self.population[pop_idx], strategy_fn_factory, candles_data
)
done_count += 1
if done_count % 10 == 0:
logger.debug(f"Evaluated {done_count}/{len(unevaluated)} genomes (parallel)")
def select_parent(self) -> StrategyGenome:
"""Tournament selection"""
tournament = random.sample(
self.population,
min(self.tournament_size, len(self.population))
)
return max(tournament, key=lambda g: g.fitness)
def crossover(self, parent1: StrategyGenome,
parent2: StrategyGenome) -> StrategyGenome:
"""Uniform crossover: for each gene, randomly pick from parent1 or parent2"""
child_genes = {}
p1 = parent1.to_dict()
p2 = parent2.to_dict()
for gene_name in GENE_RANGES:
child_genes[gene_name] = p1[gene_name] if random.random() < 0.5 else p2[gene_name]
# Repair constraints
if child_genes['slow_ma_period'] <= child_genes['fast_ma_period']:
child_genes['slow_ma_period'] = child_genes['fast_ma_period'] + 10
if child_genes['macd_slow'] <= child_genes['macd_fast']:
child_genes['macd_slow'] = child_genes['macd_fast'] + 8
if child_genes['take_profit_atr_mult'] <= child_genes['stop_loss_atr_mult']:
child_genes['take_profit_atr_mult'] = child_genes['stop_loss_atr_mult'] + 0.5
if child_genes['max_hold_candles'] <= child_genes['min_hold_candles']:
child_genes['max_hold_candles'] = child_genes['min_hold_candles'] + 10
child_genes['fitness'] = 0.0
child_genes['generation'] = self.generation + 1
return StrategyGenome(**child_genes)
def mutate(self, genome: StrategyGenome) -> StrategyGenome:
"""Gaussian mutation on each gene with probability mutation_rate"""
genes = genome.to_dict()
for gene_name, (lo, hi, is_int) in GENE_RANGES.items():
if random.random() < self.mutation_rate:
gene_range = hi - lo
delta = random.gauss(0, gene_range * self.mutation_strength)
new_val = genes[gene_name] + delta
new_val = max(lo, min(hi, new_val))
if is_int:
new_val = round(new_val)
else:
new_val = round(new_val, 4)
genes[gene_name] = new_val
# Repair constraints after mutation
if genes['slow_ma_period'] <= genes['fast_ma_period']:
genes['slow_ma_period'] = genes['fast_ma_period'] + 10
if genes['macd_slow'] <= genes['macd_fast']:
genes['macd_slow'] = genes['macd_fast'] + 8
if genes['take_profit_atr_mult'] <= genes['stop_loss_atr_mult']:
genes['take_profit_atr_mult'] = genes['stop_loss_atr_mult'] + 0.5
if genes['max_hold_candles'] <= genes['min_hold_candles']:
genes['max_hold_candles'] = genes['min_hold_candles'] + 10
genes['fitness'] = 0.0
genes['generation'] = self.generation + 1
return StrategyGenome.from_dict(genes)
def evolve_generation(self, strategy_fn_factory,
candles_data: Dict[str, 'pd.DataFrame']) -> Dict:
"""
Run one generation of evolution:
1. Evaluate fitness of all genomes
2. Sort by fitness
3. Keep elite_count best unchanged
4. Fill remaining via tournament selection + crossover + mutation
5. Save to database
"""
# Evaluate
self.evaluate_population(strategy_fn_factory, candles_data)
# Sort by fitness
self.population.sort(key=lambda g: g.fitness, reverse=True)
best = self.population[0]
avg_fitness = sum(g.fitness for g in self.population) / len(self.population)
if self.best_ever is None or best.fitness > self.best_ever.fitness:
self.best_ever = StrategyGenome.from_dict(best.to_dict())
self.best_ever.fitness = best.fitness
# Elitism: keep top N unchanged
new_population = [
StrategyGenome.from_dict(g.to_dict())
for g in self.population[:self.elite_count]
]
# Preserve their fitness
for i in range(min(self.elite_count, len(self.population))):
new_population[i].fitness = self.population[i].fitness
# Fill the rest
while len(new_population) < self.population_size:
parent1 = self.select_parent()
parent2 = self.select_parent()
if random.random() < self.crossover_rate:
child = self.crossover(parent1, parent2)
else:
child = StrategyGenome.from_dict(parent1.to_dict())
child.fitness = 0.0
child = self.mutate(child)
new_population.append(child)
self.population = new_population
self.generation += 1
# Save to database
if self.store:
self.store.record_generation(
self.generation,
[g.to_dict() for g in self.population],
best.fitness,
avg_fitness
)
stats = {
'generation': self.generation,
'best_fitness': round(best.fitness, 4),
'avg_fitness': round(avg_fitness, 4),
'best_genome': best.to_dict(),
}
logger.info(f"GA Gen {self.generation}: best={best.fitness:.4f} avg={avg_fitness:.4f}")
return stats
def run_evolution_cycle(self, strategy_fn_factory,
candles_data: Dict[str, 'pd.DataFrame'],
num_generations: int = 10) -> Optional[StrategyGenome]:
"""
Run multiple generations.
Returns the best genome found.
"""
for _ in range(num_generations):
self.evolve_generation(strategy_fn_factory, candles_data)
return self.get_best_genome()
def get_best_genome(self) -> Optional[StrategyGenome]:
"""Return the highest-fitness genome"""
if self.best_ever:
return self.best_ever
if self.population:
return max(self.population, key=lambda g: g.fitness)
return None
src/ml/rl_agent.py
+297
View File
@@ -0,0 +1,297 @@
"""
Reinforcement Learning Agent for Trading
DQN with experience replay and target network, implemented in PyTorch.
"""
import io
import random
import numpy as np
from collections import deque
from typing import Dict, List, Optional, Tuple
from loguru import logger
try:
import torch
import torch.nn as nn
import torch.optim as optim
TORCH_AVAILABLE = True
except ImportError:
TORCH_AVAILABLE = False
logger.warning("PyTorch not installed. RL agent will use random actions. "
"Install with: pip install torch")
class TradingNetwork:
"""Neural network for the RL agent (PyTorch or fallback)"""
pass
if TORCH_AVAILABLE:
class TradingNetwork(nn.Module):
"""MLP with 2 hidden layers for Q-value prediction"""
def __init__(self, state_dim: int, action_dim: int, hidden_dim: int = 128):
super().__init__()
self.net = nn.Sequential(
nn.Linear(state_dim, hidden_dim),
nn.ReLU(),
nn.Dropout(0.1),
nn.Linear(hidden_dim, hidden_dim),
nn.ReLU(),
nn.Dropout(0.1),
nn.Linear(hidden_dim, action_dim)
)
def forward(self, x):
return self.net(x)
class RLAgent:
"""
DQN-based RL agent for trading decisions.
Falls back to random actions if PyTorch is not available.
"""
def __init__(self, state_dim: int, action_dim: int = 7, config: Dict = None):
config = config or {}
self.state_dim = state_dim
self.action_dim = action_dim
# Hyperparameters
self.gamma = config.get('gamma', 0.99)
self.epsilon = config.get('epsilon_start', 1.0)
self.epsilon_min = config.get('epsilon_min', 0.05)
self.epsilon_decay = config.get('epsilon_decay', 0.9995)
self.learning_rate = config.get('learning_rate', 0.0003)
self.batch_size = config.get('batch_size', 64)
self.memory_size = config.get('memory_size', 50000)
self.target_update_freq = config.get('target_update_freq', 100)
self.live_epsilon = config.get('live_epsilon', 0.1)
# Experience replay buffer
self.memory = deque(maxlen=self.memory_size)
self.steps = 0
self.training_losses = []
# PyTorch setup
self.use_torch = TORCH_AVAILABLE
if self.use_torch:
self.device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
hidden_dim = config.get('hidden_dim', 128)
self.policy_net = TradingNetwork(state_dim, action_dim, hidden_dim).to(self.device)
self.target_net = TradingNetwork(state_dim, action_dim, hidden_dim).to(self.device)
self.target_net.load_state_dict(self.policy_net.state_dict())
self.target_net.eval()
self.optimizer = optim.Adam(self.policy_net.parameters(), lr=self.learning_rate)
logger.info(f"RL Agent initialized (PyTorch, device={self.device}, "
f"state_dim={state_dim}, action_dim={action_dim})")
else:
self.device = None
self.policy_net = None
self.target_net = None
self.optimizer = None
logger.info("RL Agent initialized (random mode - no PyTorch)")
def select_action(self, state: np.ndarray, live_mode: bool = False) -> int:
"""
Epsilon-greedy action selection.
In live mode, uses live_epsilon instead of training epsilon.
"""
eps = self.live_epsilon if live_mode else self.epsilon
if random.random() < eps:
return random.randint(0, self.action_dim - 1)
if not self.use_torch:
return random.randint(0, self.action_dim - 1)
with torch.no_grad():
state_tensor = torch.FloatTensor(state).unsqueeze(0).to(self.device)
q_values = self.policy_net(state_tensor)
return int(q_values.argmax(dim=1).item())
def store_experience(self, state, action, reward, next_state, done):
"""Store transition in replay buffer"""
self.memory.append((state, action, reward, next_state, done))
def train_step(self) -> Optional[float]:
"""
Sample mini-batch from replay buffer, compute DQN loss, update.
Returns loss value or None if not enough samples.
"""
if not self.use_torch:
return None
if len(self.memory) < self.batch_size:
return None
batch = random.sample(self.memory, self.batch_size)
states, actions, rewards, next_states, dones = zip(*batch)
states = torch.FloatTensor(np.array(states)).to(self.device)
actions = torch.LongTensor(actions).to(self.device)
rewards = torch.FloatTensor(rewards).to(self.device)
next_states = torch.FloatTensor(np.array(next_states)).to(self.device)
dones = torch.BoolTensor(dones).to(self.device)
# Current Q values
current_q = self.policy_net(states).gather(1, actions.unsqueeze(1)).squeeze(1)
# Target Q values
with torch.no_grad():
next_q = self.target_net(next_states).max(1)[0]
next_q[dones] = 0.0
target_q = rewards + self.gamma * next_q
# Loss and backprop
loss = nn.functional.smooth_l1_loss(current_q, target_q)
self.optimizer.zero_grad()
loss.backward()
torch.nn.utils.clip_grad_norm_(self.policy_net.parameters(), 1.0)
self.optimizer.step()
self.steps += 1
# Update target network
if self.steps % self.target_update_freq == 0:
self.target_net.load_state_dict(self.policy_net.state_dict())
# Decay epsilon
self.epsilon = max(self.epsilon_min, self.epsilon * self.epsilon_decay)
loss_val = loss.item()
self.training_losses.append(loss_val)
return loss_val
def train_on_episode(self, env, candles_df, ga_signal_fn=None) -> Dict:
"""
Train on a full episode (backtest run through candles).
Returns training metrics.
"""
state = env.reset(candles_df)
if state is None:
return {'avg_loss': 0, 'total_reward': 0, 'steps': 0}
total_reward = 0
total_loss = 0
loss_count = 0
step = 0
done = False
while not done:
# Get GA signal if available
ga_signal = None
if ga_signal_fn and step < len(candles_df):
ga_signal = ga_signal_fn(step)
action = self.select_action(state)
next_state, reward, done, info = env.step(action, ga_signal=ga_signal)
self.store_experience(state, action, reward, next_state, done)
loss = self.train_step()
if loss is not None:
total_loss += loss
loss_count += 1
total_reward += reward
state = next_state
step += 1
avg_loss = total_loss / max(loss_count, 1)
return {
'avg_loss': round(avg_loss, 6),
'total_reward': round(total_reward, 4),
'steps': step,
'epsilon': round(self.epsilon, 4),
'total_trades': env.total_trades,
'total_pnl': round(env.total_pnl, 4),
'final_equity': round(env.equity_history[-1] if env.equity_history else 0, 2),
'memory_size': len(self.memory),
}
def update_from_live_trade(self, state, action, reward, next_state):
"""
Online learning: called after each live trade result.
Stores experience and does one training step.
"""
self.store_experience(state, action, reward, next_state, False)
self.train_step()
def save(self, store, epoch: int = None):
"""Save model checkpoint to DataStore"""
if not self.use_torch:
return
if epoch is None:
epoch = self.steps
state_bytes = self._get_state_dict_bytes()
metrics = {
'epsilon': self.epsilon,
'steps': self.steps,
'memory_size': len(self.memory),
'avg_loss': round(np.mean(self.training_losses[-100:]), 6)
if self.training_losses else 0,
}
store.save_model_checkpoint('rl_agent', epoch, state_bytes, metrics)
logger.info(f"RL model saved (epoch {epoch}, epsilon={self.epsilon:.4f})")
def load(self, store) -> bool:
"""Load latest checkpoint from DataStore"""
if not self.use_torch:
return False
checkpoint = store.load_latest_checkpoint('rl_agent')
if checkpoint is None:
logger.info("No RL checkpoint found, starting fresh")
return False
try:
buffer = io.BytesIO(checkpoint['state_dict'])
state_dict = torch.load(buffer, map_location=self.device, weights_only=True)
# Check dimension compatibility before loading
first_layer_key = 'net.0.weight'
if first_layer_key in state_dict:
saved_input_dim = state_dict[first_layer_key].shape[1]
if saved_input_dim != self.state_dim:
logger.warning(f"RL checkpoint dimension mismatch (saved={saved_input_dim}, "
f"current={self.state_dim}). Starting fresh with new architecture.")
return False
self.policy_net.load_state_dict(state_dict)
self.target_net.load_state_dict(state_dict)
import json
metrics = json.loads(checkpoint.get('metrics', '{}'))
self.epsilon = metrics.get('epsilon', self.epsilon)
self.steps = metrics.get('steps', self.steps)
logger.info(f"RL model loaded (epoch {checkpoint['epoch']}, "
f"epsilon={self.epsilon:.4f})")
return True
except Exception as e:
logger.warning(f"Could not load RL checkpoint (likely dimension change): {e}. Starting fresh.")
return False
def _get_state_dict_bytes(self) -> bytes:
"""Serialize model state dict to bytes"""
buffer = io.BytesIO()
torch.save(self.policy_net.state_dict(), buffer)
return buffer.getvalue()
def get_stats(self) -> Dict:
"""Get current agent statistics"""
return {
'epsilon': round(self.epsilon, 4),
'steps': self.steps,
'memory_size': len(self.memory),
'avg_loss': round(np.mean(self.training_losses[-100:]), 6)
if self.training_losses else 0,
'device': str(self.device) if self.device else 'random',
'use_torch': self.use_torch,
}
src/ml/rl_environment.py
+402
View File
@@ -0,0 +1,402 @@
"""
Trading Environment for Reinforcement Learning
Defines state space, action space, and reward function.
"""
import numpy as np
import pandas as pd
from typing import Dict, Tuple, Optional
from loguru import logger
from data.features import FeatureEngine
class TradingEnvironment:
"""
Trading environment for RL agent.
Modes:
- 'backtest': steps through historical candles
- 'live': receives state updates from the trading loop
"""
# Actions
HOLD = 0
BUY_SMALL = 1 # Buy with 25% of available capital (go long)
BUY_LARGE = 2 # Buy with 50% of available capital (go long)
CLOSE_HALF = 3 # Close 50% of position (long or short)
CLOSE_ALL = 4 # Close 100% of position (long or short)
SHORT_SMALL = 5 # Short with 25% of available capital
SHORT_LARGE = 6 # Short with 50% of available capital
ACTION_NAMES = ['hold', 'buy_25%', 'buy_50%', 'close_50%', 'close_all',
'short_25%', 'short_50%']
NUM_ACTIONS = 7
def __init__(self, feature_engine: FeatureEngine,
initial_capital: float = 100.0,
commission_rate: float = 0.001):
self.feature_engine = feature_engine
self.initial_capital = initial_capital
self.commission_rate = commission_rate
# Portfolio features appended to market features
# [position_ratio, unrealized_pnl_pct, time_in_position_normalized]
self.portfolio_features = 3
# GA signal features: [signal_direction, confidence]
self.ga_features = 2
self.state_dim = FeatureEngine.NUM_FEATURES + self.portfolio_features + self.ga_features
# State tracking
self.reset()
def reset(self, candles_df: pd.DataFrame = None) -> Optional[np.ndarray]:
"""Reset environment for a new episode"""
self.capital = self.initial_capital
self.position_shares = 0.0
self.position_price = 0.0
self.step_count = 0
self.entry_step = 0
self.equity_history = [self.initial_capital]
self.peak_equity = self.initial_capital
self.last_action = self.HOLD
self.last_action_step = -10
self.total_trades = 0
self.total_pnl = 0.0
self.wins = 0
self.losses = 0
self.candles_df = candles_df
self.features_df = None
if candles_df is not None and len(candles_df) > 50:
self.features_df = self.feature_engine.compute_and_normalize(candles_df)
if len(self.features_df) > 0:
return self._get_state(0)
return np.zeros(self.state_dim, dtype=np.float32)
def step(self, action: int, ga_signal: Dict = None,
current_candle: pd.Series = None) -> Tuple[np.ndarray, float, bool, Dict]:
"""
Execute action, advance one timestep.
Args:
action: integer action (0-4)
ga_signal: optional GA strategy signal {signal, confidence}
current_candle: optional candle for live mode
Returns: (next_state, reward, done, info)
"""
prev_equity = self._get_equity()
# Get current price
if self.features_df is not None and self.step_count < len(self.features_df):
idx = self.step_count
current_price = float(self.candles_df['close'].iloc[
self.candles_df.index.get_indexer(
[self.features_df.index[idx]], method='nearest'
)[0]
]) if self.candles_df is not None else 0
# Simpler: just use the close from the original df aligned by position
try:
orig_idx = self.features_df.index[idx]
if orig_idx in self.candles_df.index:
current_price = float(self.candles_df.loc[orig_idx, 'close'])
else:
current_price = float(self.candles_df['close'].iloc[-1])
except (IndexError, KeyError):
current_price = float(self.candles_df['close'].iloc[-1])
elif current_candle is not None:
current_price = float(current_candle.get('close', current_candle.get('Close', 0)))
else:
return np.zeros(self.state_dim, dtype=np.float32), 0.0, True, {}
# Execute action
trade_info = self._execute_action(action, current_price)
self.step_count += 1
curr_equity = self._get_equity(current_price)
self.equity_history.append(curr_equity)
if curr_equity > self.peak_equity:
self.peak_equity = curr_equity
# Compute reward
reward = self._compute_reward(action, prev_equity, curr_equity, current_price)
# Check if done
done = False
if self.features_df is not None:
done = self.step_count >= len(self.features_df) - 1
if curr_equity < self.initial_capital * 0.5: # 50% loss = episode over
done = True
# Get next state
next_state = self._get_state(self.step_count, ga_signal)
info = {
'equity': curr_equity,
'position_value': abs(self.position_shares) * current_price if self.position_shares != 0 else 0,
'position_side': 'long' if self.position_shares > 0 else ('short' if self.position_shares < 0 else 'flat'),
'capital': self.capital,
'total_trades': self.total_trades,
'total_pnl': self.total_pnl,
**trade_info,
}
return next_state, reward, done, info
def _execute_action(self, action: int, current_price: float) -> Dict:
"""Execute a trading action, return trade info.
position_shares > 0 means long, < 0 means short."""
info = {'trade': None}
if current_price <= 0:
return info
# GO LONG (only if flat)
if action == self.BUY_SMALL and self.position_shares == 0:
invest = self.capital * 0.25
if invest > 1:
fees = invest * self.commission_rate
shares = (invest - fees) / current_price
self.capital -= invest
self.position_shares = shares
self.position_price = current_price
self.entry_step = self.step_count
self.last_action = action
self.last_action_step = self.step_count
info['trade'] = 'buy_25%'
elif action == self.BUY_LARGE and self.position_shares == 0:
invest = self.capital * 0.50
if invest > 1:
fees = invest * self.commission_rate
shares = (invest - fees) / current_price
self.capital -= invest
self.position_shares = shares
self.position_price = current_price
self.entry_step = self.step_count
self.last_action = action
self.last_action_step = self.step_count
info['trade'] = 'buy_50%'
# CLOSE POSITION (long or short)
elif action == self.CLOSE_HALF and self.position_shares != 0:
close_shares = abs(self.position_shares) * 0.5
if self.position_shares > 0:
# Close half of long
proceeds = close_shares * current_price
fees = proceeds * self.commission_rate
pnl = (current_price - self.position_price) * close_shares - fees
self.capital += proceeds - fees
self.position_shares -= close_shares
else:
# Cover half of short
cost = close_shares * current_price
fees = cost * self.commission_rate
pnl = (self.position_price - current_price) * close_shares - fees
self.capital += (self.position_price * close_shares) - cost - fees
self.position_shares += close_shares
self.total_trades += 1
self.total_pnl += pnl
if pnl > 0:
self.wins += 1
else:
self.losses += 1
self.last_action = action
self.last_action_step = self.step_count
info['trade'] = 'close_50%'
info['pnl'] = pnl
elif action == self.CLOSE_ALL and self.position_shares != 0:
abs_shares = abs(self.position_shares)
if self.position_shares > 0:
# Close all long
proceeds = abs_shares * current_price
fees = proceeds * self.commission_rate
pnl = (current_price - self.position_price) * abs_shares - fees
self.capital += proceeds - fees
else:
# Cover all short
cost = abs_shares * current_price
fees = cost * self.commission_rate
pnl = (self.position_price - current_price) * abs_shares - fees
self.capital += (self.position_price * abs_shares) - cost - fees
self.position_shares = 0
self.position_price = 0
self.total_trades += 1
self.total_pnl += pnl
if pnl > 0:
self.wins += 1
else:
self.losses += 1
self.last_action = action
self.last_action_step = self.step_count
info['trade'] = 'close_all'
info['pnl'] = pnl
# GO SHORT (only if flat)
elif action == self.SHORT_SMALL and self.position_shares == 0:
invest = self.capital * 0.25
if invest > 1:
fees = invest * self.commission_rate
shares = (invest - fees) / current_price
# Short: we receive proceeds upfront, owe shares later
self.capital += invest - fees # margin collateral stays, proceeds added
self.position_shares = -shares
self.position_price = current_price
self.entry_step = self.step_count
self.last_action = action
self.last_action_step = self.step_count
info['trade'] = 'short_25%'
elif action == self.SHORT_LARGE and self.position_shares == 0:
invest = self.capital * 0.50
if invest > 1:
fees = invest * self.commission_rate
shares = (invest - fees) / current_price
self.capital += invest - fees
self.position_shares = -shares
self.position_price = current_price
self.entry_step = self.step_count
self.last_action = action
self.last_action_step = self.step_count
info['trade'] = 'short_50%'
return info
def _compute_reward(self, action: int, prev_equity: float,
curr_equity: float, current_price: float) -> float:
"""
Reward function for scalping:
- Base: portfolio return (amplified for scalping sensitivity)
- Penalty: drawdown, overtrading
- Bonus: profitable close (long or short)
- Scalp bonus: quick profitable round trips
"""
if prev_equity <= 0:
return 0.0
# Base reward: portfolio return (amplified 2x for scalping sensitivity)
base_reward = (curr_equity - prev_equity) / prev_equity * 2.0
# Drawdown penalty
drawdown = (self.peak_equity - curr_equity) / self.peak_equity if self.peak_equity > 0 else 0
dd_penalty = -0.5 * max(0, drawdown - 0.03)
# Overtrading penalty (reduced for scalping - allow faster re-entry)
overtrade_penalty = 0.0
if action != self.HOLD and (self.step_count - self.last_action_step) < 2:
overtrade_penalty = -0.0005
# Holding penalty (stronger for scalping - don't sit idle)
hold_penalty = 0.0
if action == self.HOLD and self.position_shares == 0:
hold_penalty = -0.0002
# Profitable close bonus (works for both long and short)
close_bonus = 0.0
if action in (self.CLOSE_HALF, self.CLOSE_ALL) and self.position_price > 0:
if self.position_shares > 0 and current_price > self.position_price:
# Profitable long close
pnl_pct = (current_price - self.position_price) / self.position_price
close_bonus = 0.02 * pnl_pct
elif self.position_shares < 0 and current_price < self.position_price:
# Profitable short close
pnl_pct = (self.position_price - current_price) / self.position_price
close_bonus = 0.02 * pnl_pct
# Quick scalp bonus: reward fast profitable round trips
scalp_bonus = 0.0
if action in (self.CLOSE_HALF, self.CLOSE_ALL) and self.position_shares != 0:
hold_time = self.step_count - self.entry_step
if hold_time < 12 and close_bonus > 0: # Quick + profitable
scalp_bonus = 0.005
reward = base_reward + dd_penalty + overtrade_penalty + hold_penalty + close_bonus + scalp_bonus
# Clip to [-1, 1]
return max(-1.0, min(1.0, reward))
def _get_equity(self, current_price: float = None) -> float:
"""Calculate current total equity (handles long and short positions)"""
if current_price is None:
if self.candles_df is not None and self.step_count < len(self.candles_df):
current_price = float(self.candles_df['close'].iloc[self.step_count])
else:
current_price = self.position_price if self.position_price > 0 else 0
if self.position_shares > 0:
# Long: capital + shares * price
return self.capital + self.position_shares * current_price
elif self.position_shares < 0:
# Short: capital + unrealized P&L from short
abs_shares = abs(self.position_shares)
short_pnl = (self.position_price - current_price) * abs_shares
return self.capital + short_pnl
return self.capital
def _get_state(self, step: int, ga_signal: Dict = None) -> np.ndarray:
"""Build full state vector"""
# Market features
if self.features_df is not None and step < len(self.features_df):
market_features = self.feature_engine.get_state_vector(self.features_df, step)
else:
market_features = np.zeros(FeatureEngine.NUM_FEATURES, dtype=np.float32)
# Portfolio features (handles long and short)
equity = self._get_equity()
position_value = abs(self.position_shares) * self.position_price
# Positive ratio = long, negative ratio = short
position_ratio = (self.position_shares * self.position_price) / equity if equity > 0 else 0.0
unrealized_pnl = 0.0
if self.position_shares != 0 and self.position_price > 0:
if self.candles_df is not None and step < len(self.candles_df):
current = float(self.candles_df['close'].iloc[min(step, len(self.candles_df) - 1)])
if self.position_shares > 0:
unrealized_pnl = (current - self.position_price) / self.position_price
else:
unrealized_pnl = (self.position_price - current) / self.position_price
time_in_position = 0.0
if self.position_shares != 0:
time_in_position = min((self.step_count - self.entry_step) / 48.0, 1.0)
portfolio_features = np.array([
position_ratio, unrealized_pnl, time_in_position
], dtype=np.float32)
# GA signal features
if ga_signal:
signal_dir = 1.0 if ga_signal.get('signal') == 'buy' else (
-1.0 if ga_signal.get('signal') == 'sell' else 0.0)
confidence = float(ga_signal.get('confidence', 0.0))
else:
signal_dir = 0.0
confidence = 0.0
ga_features = np.array([signal_dir, confidence], dtype=np.float32)
return np.concatenate([market_features, portfolio_features, ga_features])
def get_portfolio_state(self, current_price: float = 0) -> Dict:
"""Get portfolio state dict for external use (handles long and short)"""
equity = self._get_equity(current_price)
position_value = self.position_shares * current_price if current_price > 0 else 0
unrealized = 0.0
if self.position_price > 0 and current_price > 0 and self.position_shares != 0:
if self.position_shares > 0:
unrealized = (current_price - self.position_price) / self.position_price
else:
unrealized = (self.position_price - current_price) / self.position_price
return {
'position_ratio': position_value / equity if equity > 0 else 0,
'unrealized_pnl': unrealized,
'time_in_position': min((self.step_count - self.entry_step) / 48.0, 1.0)
if self.position_shares != 0 else 0,
}
src/reporting/__init__.py
View File
src/reporting/krystie_bridge.py
+167
View File
@@ -0,0 +1,167 @@
"""
Krystie Bridge - BIGGFISH ↔ Krystie Communication Layer
Writes live status and event logs to JSON files that Krystie (OpenClaw agent)
can read from her workspace. Both run on the same VPS, so file-based IPC works.
Status file: data/krystie-status.json (overwritten every cycle)
Events file: data/krystie-events.json (rolling log, last 50 events)
"""
import json
import os
from datetime import datetime, timezone
from typing import Dict, List, Optional
from loguru import logger
class KrystieBridge:
"""Writes BIGGFISH state to JSON files for Krystie to read."""
MAX_EVENTS = 50
def __init__(self, data_dir: str = "data"):
self.data_dir = data_dir
self.status_path = os.path.join(data_dir, "krystie-status.json")
self.events_path = os.path.join(data_dir, "krystie-events.json")
os.makedirs(data_dir, exist_ok=True)
# Load existing events
self._events = self._load_events()
logger.info("Krystie bridge initialized")
def _now_iso(self) -> str:
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
def _write_json(self, path: str, data: dict):
"""Atomic write: write to tmp then rename"""
tmp = path + ".tmp"
try:
with open(tmp, 'w') as f:
json.dump(data, f, indent=2, default=str)
os.replace(tmp, path)
except Exception as e:
logger.error(f"Krystie bridge write error ({path}): {e}")
try:
os.unlink(tmp)
except OSError:
pass
def _load_events(self) -> list:
try:
with open(self.events_path) as f:
data = json.load(f)
return data.get("events", [])
except (FileNotFoundError, json.JSONDecodeError):
return []
def update_status(self, portfolio: Dict, positions: List[Dict],
learning_stats: Dict, markets: Dict,
config: Dict, uptime_seconds: float = 0,
today_trades: Optional[List[Dict]] = None):
"""Write the live status snapshot."""
today_trades = today_trades or []
wins = [t for t in today_trades if (t.get('pnl') or 0) > 0]
losses = [t for t in today_trades if (t.get('pnl') or 0) < 0]
total_pnl = sum(t.get('pnl', 0) for t in today_trades if t.get('pnl') is not None)
rl = learning_stats.get('rl', {})
ga = learning_stats.get('ga', {})
status = {
"updated_at": self._now_iso(),
"uptime_hours": round(uptime_seconds / 3600, 1),
"markets": markets,
"portfolio": portfolio,
"positions": [
{
"symbol": p.get("symbol"),
"qty": p.get("qty"),
"entry_price": p.get("avg_entry_price"),
"current_price": p.get("current_price"),
"unrealized_pnl": p.get("unrealized_pl", 0),
}
for p in positions
],
"learning": {
"ga_generation": ga.get("generation", 0),
"ga_best_fitness": round(ga.get("best_fitness", 0), 4),
"rl_epsilon": round(rl.get("epsilon", 0), 4),
"rl_experiences": rl.get("memory_size", 0),
"rl_loss": round(rl.get("avg_loss", 0), 6),
},
"today_summary": {
"trades_count": len(today_trades),
"wins": len(wins),
"losses": len(losses),
"total_pnl": round(total_pnl, 2),
},
"config": {
"stock_symbols": config.get("symbols", []),
"forex_symbols": config.get("forex_symbols", []),
"initial_capital": config.get("initial_capital", 0),
"target_capital": config.get("target_capital", 0),
},
}
self._write_json(self.status_path, status)
def log_event(self, event_type: str, data: Dict):
"""Append an event to the rolling log."""
event = {
"time": self._now_iso(),
"type": event_type,
"data": data,
}
self._events.append(event)
self._events = self._events[-self.MAX_EVENTS:]
self._write_json(self.events_path, {"events": self._events})
def log_trade(self, trade: Dict):
"""Log a trade event."""
pnl = trade.get('pnl')
if pnl is not None:
self.log_event("trade_close", {
"symbol": trade.get("symbol"),
"side": trade.get("side"),
"entry_price": trade.get("entry_price"),
"exit_price": trade.get("exit_price"),
"pnl": round(pnl, 2),
"pnl_pct": round(trade.get("pnl_pct", 0), 2),
"exit_reason": trade.get("exit_reason", "signal"),
})
else:
self.log_event("trade_open", {
"symbol": trade.get("symbol"),
"side": trade.get("side"),
"amount": trade.get("amount"),
"entry_price": trade.get("entry_price"),
})
def log_ga_milestone(self, generation: int, fitness: float):
"""Log a GA evolution milestone."""
self.log_event("ga_milestone", {
"generation": generation,
"fitness": round(fitness, 4),
})
def log_daily_report(self, equity: float, day_pnl: float, trades_count: int):
"""Log that a daily report was sent."""
self.log_event("daily_report", {
"equity": round(equity, 2),
"day_pnl": round(day_pnl, 2),
"trades_count": trades_count,
})
def log_startup(self):
"""Log bot startup."""
self.log_event("bot_started", {
"message": "BIGGFISH autonomous trader started",
})
def log_shutdown(self):
"""Log bot shutdown."""
self.log_event("bot_stopped", {
"message": "BIGGFISH autonomous trader stopped",
})
src/reporting/reporter.py
+144
View File
@@ -0,0 +1,144 @@
"""
Reporting and communication module
"""
from loguru import logger
from datetime import datetime
import json
from pathlib import Path
class Reporter:
"""Generate reports and send notifications"""
def __init__(self, config):
self.config = config
self.report_dir = Path(__file__).parent.parent.parent / "data" / "reports"
self.report_dir.mkdir(parents=True, exist_ok=True)
def generate_daily_report(self, portfolio, positions):
"""Generate comprehensive daily performance report"""
report = {
"date": datetime.now().strftime("%Y-%m-%d"),
"timestamp": datetime.now().isoformat(),
"portfolio": portfolio,
"positions": positions,
"performance": self._calculate_performance(portfolio, positions)
}
# Save to file
if self.config.get("save_to_file", True):
report_file = self.report_dir / f"report_{datetime.now().strftime('%Y%m%d')}.json"
with open(report_file, 'w') as f:
json.dump(report, f, indent=2)
return report
def _calculate_performance(self, portfolio, positions):
"""Calculate performance metrics"""
total_pl = sum(p["unrealized_pl"] for p in positions)
return {
"total_value": portfolio["equity"],
"cash": portfolio["cash"],
"positions_value": portfolio["long_market_value"],
"day_pnl": portfolio["day_pnl"],
"day_pnl_pct": portfolio["day_pnl_pct"],
"total_unrealized_pl": total_pl,
"num_positions": len(positions),
"positions_summary": [
{
"symbol": p["symbol"],
"value": p["market_value"],
"pl": p["unrealized_pl"],
"pl_pct": p["unrealized_plpc"]
}
for p in positions
]
}
def format_daily_report(self, report):
"""Format daily report as readable text"""
text = "🐟 BIGGFISH Daily Report\n"
text += f"📅 {report['date']}\n"
text += "=" * 40 + "\n\n"
# Portfolio Summary
p = report["portfolio"]
text += f"💰 Portfolio Value: ${p['equity']:.2f}\n"
text += f"💵 Cash: ${p['cash']:.2f}\n"
text += f"📊 Positions Value: ${p['long_market_value']:.2f}\n"
text += f"📈 Day P/L: ${p['day_pnl']:.2f} ({p['day_pnl_pct']:+.2f}%)\n\n"
# Goal Progress
target = 1000
current = p['equity']
progress = (current / target) * 100
text += f"🎯 Goal Progress: ${current:.2f} / ${target:.2f} ({progress:.1f}%)\n"
text += self._draw_progress_bar(progress) + "\n\n"
# Positions
if report["positions"]:
text += f"📋 Open Positions ({len(report['positions'])})\n"
text += "-" * 40 + "\n"
for pos in report["positions"]:
text += f"{pos['symbol']}: "
text += f"{pos['qty']} @ ${pos['current_price']:.2f} "
text += f"| P/L: ${pos['unrealized_pl']:.2f} ({pos['unrealized_plpc']:+.2f}%)\n"
else:
text += "📋 No open positions\n"
text += "\n" + "=" * 40 + "\n"
return text
def _draw_progress_bar(self, percentage, width=20):
"""Draw a simple progress bar"""
filled = int(width * min(percentage, 100) / 100)
bar = "█" * filled + "░" * (width - filled)
return f"[{bar}] {percentage:.1f}%"
def send_report(self, report):
"""Send report via configured channels"""
text = self.format_daily_report(report)
logger.info("\n" + text)
# TODO: Send to Telegram if enabled
if self.config.get("telegram_enabled"):
self._send_to_telegram(text)
def send_strategy_proposal(self, strategies):
"""Send strategy proposals for approval"""
text = "🧠 BIGGFISH Strategy Proposals\n"
text += f"⏰ {datetime.now().strftime('%Y-%m-%d %H:%M')}\n"
text += "=" * 40 + "\n\n"
for i, strategy in enumerate(strategies, 1):
text += f"Strategy #{i}: {strategy['type'].replace('_', ' ').title()}\n"
text += f"Symbol: {strategy['symbol']}\n"
text += f"Action: {strategy['action'].upper()} {strategy['shares']} shares\n"
text += f"Entry: ${strategy['entry_price']:.2f}\n"
text += f"Target: ${strategy['target_price']:.2f} (+{strategy['reward_pct']:.1f}%)\n"
text += f"Stop: ${strategy['stop_loss']:.2f} (-{strategy['risk_pct']:.1f}%)\n"
text += f"Position Size: ${strategy['position_value']:.2f}\n"
text += f"Score: {strategy['score']}/100\n\n"
text += f"Rationale:\n{strategy['rationale']}\n"
text += "-" * 40 + "\n\n"
text += "⚠️ Awaiting approval to execute\n"
logger.info("\n" + text)
if self.config.get("telegram_enabled"):
self._send_to_telegram(text)
def _send_to_telegram(self, text):
"""Send message to Telegram"""
# TODO: Implement Telegram integration
logger.debug("Telegram notification would be sent here")
pass
src/reporting/telegram_reporter.py
+158
View File
@@ -0,0 +1,158 @@
"""
Telegram Daily Reporter for BIGGFISH
Sends daily trade summaries via Krystie's Telegram bot.
"""
import requests
from datetime import datetime, timedelta
from typing import Dict, List, Optional
from loguru import logger
class TelegramReporter:
"""Sends daily reports to Telegram via Krystie bot"""
API_URL = "https://api.telegram.org/bot{token}/{method}"
def __init__(self, bot_token: str, chat_id: str):
self.bot_token = bot_token
self.chat_id = chat_id
def _send_message(self, text: str, parse_mode: str = "HTML") -> bool:
"""Send a message via Telegram Bot API"""
url = self.API_URL.format(token=self.bot_token, method="sendMessage")
try:
resp = requests.post(url, json={
"chat_id": self.chat_id,
"text": text,
"parse_mode": parse_mode,
}, timeout=15)
if resp.status_code == 200 and resp.json().get("ok"):
logger.info("Telegram message sent successfully")
return True
else:
logger.error(f"Telegram send failed: {resp.status_code} {resp.text}")
return False
except Exception as e:
logger.error(f"Telegram send error: {e}")
return False
def send_daily_report(self, portfolio: Dict, positions: List[Dict],
today_trades: List[Dict], learning_stats: Dict,
config: Dict) -> bool:
"""
Build and send end-of-day summary.
Args:
portfolio: broker portfolio dict (equity, cash, day_pnl, etc.)
positions: list of open position dicts
today_trades: trades executed/closed today
learning_stats: RL + GA metrics
config: trading config (initial_capital, target_capital)
"""
text = self._format_daily_report(
portfolio, positions, today_trades, learning_stats, config
)
return self._send_message(text)
def send_trade_alert(self, trade: Dict) -> bool:
"""Send an immediate trade notification"""
side = trade.get('side', '?').upper()
symbol = trade.get('symbol', '?')
price = trade.get('entry_price') or trade.get('exit_price', 0)
pnl = trade.get('pnl')
if pnl is not None:
emoji = "\u2705" if pnl >= 0 else "\u274c"
text = (f"{emoji} <b>BIGGFISH Trade Closed</b>\n"
f"{symbol} | Exit @ ${price:.2f}\n"
f"P&L: <b>${pnl:+.2f}</b> ({trade.get('pnl_pct', 0):+.1f}%)")
else:
text = (f"\U0001f41f <b>BIGGFISH Trade Opened</b>\n"
f"{side} {symbol} @ ${price:.2f}\n"
f"Amount: ${trade.get('amount', 0):.2f}")
return self._send_message(text)
def _format_daily_report(self, portfolio: Dict, positions: List[Dict],
today_trades: List[Dict], learning_stats: Dict,
config: Dict) -> str:
"""Format the daily report as HTML for Telegram"""
equity = portfolio.get('equity', 0)
initial = config.get('initial_capital', 100000)
target = config.get('target_capital', 1000000)
day_pnl = portfolio.get('day_pnl', 0)
day_pnl_pct = portfolio.get('day_pnl_pct', 0)
total_pnl = equity - initial
total_pnl_pct = (total_pnl / initial * 100) if initial > 0 else 0
progress = (equity / target * 100) if target > 0 else 0
date_str = datetime.utcnow().strftime("%Y-%m-%d")
lines = []
lines.append(f"\U0001f41f <b>BIGGFISH Daily Report</b>")
lines.append(f"\U0001f4c5 {date_str}")
lines.append("")
# Portfolio
lines.append(f"\U0001f4b0 <b>Portfolio:</b> ${equity:,.2f}")
lines.append(f"\U0001f4c8 Day P&L: <b>${day_pnl:+,.2f}</b> ({day_pnl_pct:+.1f}%)")
lines.append(f"\U0001f4ca Total P&L: ${total_pnl:+,.2f} ({total_pnl_pct:+.1f}%)")
lines.append(f"\U0001f3af Goal: ${equity:,.0f} / ${target:,.0f} ({progress:.1f}%)")
lines.append(self._progress_bar(progress))
lines.append("")
# Today's trades
wins = [t for t in today_trades if (t.get('pnl') or 0) > 0]
losses = [t for t in today_trades if (t.get('pnl') or 0) < 0]
total_trade_pnl = sum(t.get('pnl', 0) for t in today_trades if t.get('pnl') is not None)
lines.append(f"\U0001f4cb <b>Today's Trades:</b> {len(today_trades)}")
if today_trades:
lines.append(f" \u2705 Wins: {len(wins)} | \u274c Losses: {len(losses)}")
lines.append(f" \U0001f4b5 Trade P&L: ${total_trade_pnl:+,.2f}")
lines.append("")
for t in today_trades:
side = t.get('side', '?').upper()
symbol = t.get('symbol', '?')
pnl = t.get('pnl')
entry = t.get('entry_price', 0)
exit_p = t.get('exit_price')
if exit_p and pnl is not None:
emoji = "\u2705" if pnl >= 0 else "\u274c"
lines.append(f" {emoji} {symbol}: ${entry:.2f} \u2192 ${exit_p:.2f} "
f"(<b>${pnl:+.2f}</b>)")
else:
lines.append(f" \U0001f41f {side} {symbol} @ ${entry:.2f}")
else:
lines.append(" No trades today")
lines.append("")
# Open positions
if positions:
lines.append(f"\U0001f4ca <b>Open Positions:</b> {len(positions)}")
for p in positions:
pl = p.get('unrealized_pl', 0)
emoji = "\U0001f7e2" if pl >= 0 else "\U0001f534"
lines.append(f" {emoji} {p['symbol']}: {p.get('qty', 0):.0f} shares "
f"@ ${p.get('current_price', 0):.2f} "
f"(${pl:+.2f})")
lines.append("")
# Learning stats
rl = learning_stats.get('rl', {})
ga = learning_stats.get('ga', {})
lines.append(f"\U0001f9e0 <b>Learning:</b>")
lines.append(f" RL: \u03b5={rl.get('epsilon', 0):.3f} | "
f"{rl.get('memory_size', 0):,} experiences")
lines.append(f" GA: Gen {ga.get('generation', 0)} | "
f"Best fitness: {ga.get('best_fitness', 0):.2f}")
return "\n".join(lines)
def _progress_bar(self, pct: float, width: int = 15) -> str:
filled = int(width * min(pct, 100) / 100)
bar = "\u2588" * filled + "\u2591" * (width - filled)
return f"[{bar}] {pct:.1f}%"
src/research/__init__.py
View File
src/research/screener.py
+187
View File
@@ -0,0 +1,187 @@
"""
Stock screening and research module
"""
import yfinance as yf
import pandas as pd
from loguru import logger
from datetime import datetime, timedelta
import time
class StockScreener:
"""Screen stocks based on various criteria"""
def __init__(self, config):
self.config = config
self.watchlist = set()
# Small cap stock universe (example tickers - will expand)
self.small_cap_universe = [
# Technology
"SOUN", "MARA", "RIOT", "BBAI", "SWI", "MNDY",
# Healthcare/Biotech
"NVCR", "RXRX", "SDGR", "VERA", "RDNT",
# Industrial/Energy
"PTEN", "NINE", "PUMP", "BORR", "IMPP",
# Consumer
"BYND", "GOGO", "XPOF", "REAL", "CBRL",
# Financial
"VRTS", "CVLT", "OCSL", "TPVG",
# Other sectors
"GOCO", "PRPL", "ROOT", "SEAT", "HYZN"
]
# Mid cap expanding universe
self.mid_cap_universe = [
"PLTR", "RBLX", "RIVN", "DKNG", "HOOD",
"SOFI", "LCID", "COIN", "UPST", "CELH"
]
# Blue chip universe
self.large_cap_universe = [
"AAPL", "MSFT", "GOOGL", "AMZN", "NVDA",
"META", "TSLA", "V", "MA", "JPM"
]
def scan(self, focus="small_cap"):
"""Scan for trading opportunities based on focus"""
logger.info(f"🔍 Scanning {focus} stocks...")
opportunities = []
try:
# Determine which universe to scan
if focus == "small_cap":
universe = self.small_cap_universe
weight = 1.0
elif focus == "mid_cap_mixed":
universe = self.small_cap_universe + self.mid_cap_universe[:5]
weight = 0.8
elif focus == "mid_cap_heavy":
universe = self.mid_cap_universe + self.small_cap_universe[:10]
weight = 0.6
elif focus == "balanced":
universe = (self.small_cap_universe[:10] +
self.mid_cap_universe +
self.large_cap_universe[:5])
weight = 0.4
else:
universe = self.small_cap_universe
weight = 1.0
# Scan each ticker
for ticker in universe:
try:
analysis = self.analyze_ticker(ticker)
if analysis and analysis["score"] > 60:
analysis["weight"] = weight
opportunities.append(analysis)
logger.info(f"✅ {ticker}: Score {analysis['score']}")
time.sleep(0.1) # Rate limiting
except Exception as e:
logger.debug(f"⚠️ Error analyzing {ticker}: {e}")
continue
# Sort by score
opportunities.sort(key=lambda x: x["score"], reverse=True)
logger.info(f"📊 Found {len(opportunities)} opportunities")
return opportunities[:10] # Return top 10
except Exception as e:
logger.error(f"❌ Error in scan: {e}", exc_info=True)
return []
def analyze_ticker(self, ticker):
"""Analyze a single ticker"""
try:
stock = yf.Ticker(ticker)
# Get recent data
hist = stock.history(period="1mo")
if hist.empty or len(hist) < 10:
return None
info = stock.info
current_price = hist['Close'][-1]
# Calculate metrics
volume_avg = hist['Volume'].mean()
price_change_1w = ((hist['Close'][-1] - hist['Close'][-5]) / hist['Close'][-5] * 100) if len(hist) >= 5 else 0
price_change_1m = ((hist['Close'][-1] - hist['Close'][0]) / hist['Close'][0] * 100)
volatility = hist['Close'].pct_change().std() * 100
# Volume surge detection
recent_volume = hist['Volume'][-1]
volume_surge = (recent_volume / volume_avg) if volume_avg > 0 else 1
# Scoring system
score = 50 # Base score
# Momentum
if price_change_1w > 5:
score += 15
elif price_change_1w > 2:
score += 10
elif price_change_1w < -5:
score -= 10
# Volume
if volume_surge > 1.5:
score += 20
elif volume_surge > 1.2:
score += 10
# Volatility (want some, but not too much)
if 2 < volatility < 5:
score += 10
elif volatility > 8:
score -= 10
# Price range (avoid penny stocks)
if current_price < self.config.get("min_price", 2.0):
score -= 30
# Volume requirement
if volume_avg < self.config.get("min_volume", 500000):
score -= 20
return {
"symbol": ticker,
"score": score,
"current_price": round(current_price, 2),
"price_change_1w": round(price_change_1w, 2),
"price_change_1m": round(price_change_1m, 2),
"volume_avg": int(volume_avg),
"volume_surge": round(volume_surge, 2),
"volatility": round(volatility, 2),
"market_cap": info.get("marketCap", 0),
"sector": info.get("sector", "Unknown")
}
except Exception as e:
logger.debug(f"Error analyzing {ticker}: {e}")
return None
def get_quote(self, ticker):
"""Get real-time quote for a ticker"""
try:
stock = yf.Ticker(ticker)
info = stock.info
return {
"symbol": ticker,
"price": info.get("currentPrice", info.get("regularMarketPrice")),
"change": info.get("regularMarketChange"),
"change_pct": info.get("regularMarketChangePercent"),
"volume": info.get("volume"),
"market_cap": info.get("marketCap")
}
except Exception as e:
logger.error(f"Error getting quote for {ticker}: {e}")
return None
src/strategies/__init__.py
View File
src/strategies/auto_strategy.py
+402
View File
@@ -0,0 +1,402 @@
"""
Auto Strategy
Converts a StrategyGenome into a callable trading strategy for backtesting and live trading.
Optimized: indicators are pre-computed once as arrays, not per-candle.
"""
import numpy as np
import pandas as pd
from typing import Dict, Callable
from loguru import logger
def genome_to_strategy(genome) -> Callable:
"""
Convert a StrategyGenome into a scalping strategy with long AND short signals.
Returns a callable with signature:
(state: Dict, candle_idx: int, df: pd.DataFrame, params: Dict) -> Dict
Indicators are pre-computed on first call and cached via closure.
"""
g = genome
_cache = {}
def strategy_fn(state: Dict, idx: int, df: pd.DataFrame, params: Dict) -> Dict:
"""Evaluate scalping strategy at candle index"""
if idx < max(g.slow_ma_period, 30):
return {'action': 'hold'}
# Pre-compute all indicators once, cache by DataFrame id
df_id = id(df)
if df_id not in _cache:
_cache.clear()
_cache[df_id] = _precompute_indicators(df, g)
ind = _cache[df_id]
if idx >= len(ind['fast_ma']):
return {'action': 'hold'}
current_price = ind['close'][idx]
fast_ma = ind['fast_ma'][idx]
slow_ma = ind['slow_ma'][idx]
rsi = ind['rsi'][idx]
macd_hist = ind['macd_hist'][idx]
atr = ind['atr'][idx]
vol_ratio = ind['vol_ratio'][idx]
# BB for mean-reversion scalps
bb_upper = ind.get('bb_upper')
bb_lower = ind.get('bb_lower')
position = state.get('position')
# --- EXIT CONDITIONS ---
if position is not None:
held_too_long = False
if 'entry_idx' in position:
candles_held = idx - position['entry_idx']
held_too_long = candles_held >= g.max_hold_candles
pos_dir = position.get('direction', 'long')
if pos_dir == 'long':
if rsi > g.rsi_overbought or held_too_long or macd_hist < 0:
return {'action': 'sell'}
elif pos_dir == 'short':
if rsi < g.rsi_oversold or held_too_long or macd_hist > 0:
return {'action': 'cover'}
return {'action': 'hold'}
# --- LONG ENTRY (scalp) ---
bullish_trend = current_price > fast_ma
rsi_buy_zone = g.rsi_oversold < rsi < (g.rsi_overbought - 5)
macd_bullish = macd_hist > 0
volume_active = vol_ratio > g.volume_surge_threshold
# Bollinger bounce: price near lower band = mean reversion long
bb_bounce_long = False
if bb_lower is not None and idx < len(bb_lower):
bb_bounce_long = current_price <= bb_lower[idx] * 1.005
long_signal = (bullish_trend and rsi_buy_zone and macd_bullish and volume_active) or \
(bb_bounce_long and rsi < 35 and volume_active)
if long_signal:
stop_loss = current_price - (atr * g.stop_loss_atr_mult)
take_profit = current_price + (atr * g.take_profit_atr_mult)
confidence = min(1.0, (vol_ratio - 1) * 0.4 + 0.3)
if bb_bounce_long:
confidence = min(1.0, confidence + 0.15)
return {
'action': 'buy',
'amount_pct': g.max_position_pct,
'stop_loss': stop_loss,
'take_profit': take_profit,
'confidence': confidence,
}
# --- SHORT ENTRY (scalp) ---
bearish_trend = current_price < fast_ma
rsi_sell_zone = (g.rsi_oversold + 5) < rsi < g.rsi_overbought
macd_bearish = macd_hist < 0
# Bollinger rejection: price near upper band = mean reversion short
bb_bounce_short = False
if bb_upper is not None and idx < len(bb_upper):
bb_bounce_short = current_price >= bb_upper[idx] * 0.995
short_signal = (bearish_trend and rsi_sell_zone and macd_bearish and volume_active) or \
(bb_bounce_short and rsi > 65 and volume_active)
if short_signal:
# For shorts: stop is ABOVE, take profit is BELOW
short_stop = current_price + (atr * g.stop_loss_atr_mult)
short_tp = current_price - (atr * g.take_profit_atr_mult)
confidence = min(1.0, (vol_ratio - 1) * 0.4 + 0.3)
if bb_bounce_short:
confidence = min(1.0, confidence + 0.15)
return {
'action': 'short',
'amount_pct': g.max_position_pct,
'stop_loss': short_stop,
'take_profit': short_tp,
'short_stop_loss': short_stop,
'short_take_profit': short_tp,
'confidence': confidence,
}
return {'action': 'hold'}
return strategy_fn
def genome_to_signals(genome, df: pd.DataFrame) -> Dict[str, np.ndarray]:
"""
Vectorized signal generation for fast backtesting (long + short scalping).
Pre-computes all indicators and generates entry/exit signal arrays for both sides.
Returns dict with:
'entry': boolean array (True = long entry signal)
'exit': boolean array (True = long exit signal)
'short_entry': boolean array (True = short entry signal)
'short_exit': boolean array (True = short exit/cover signal)
'stop_loss': float array (long SL price at each bar)
'take_profit': float array (long TP price at each bar)
'short_stop_loss': float array (short SL price, ABOVE entry)
'short_take_profit': float array (short TP price, BELOW entry)
'amount_pct': float (position size)
'indicators': dict of pre-computed indicator arrays
"""
g = genome
ind = _precompute_indicators(df, g)
min_idx = max(g.slow_ma_period, 30)
# --- LONG ENTRY ---
bullish_trend = ind['close'] > ind['fast_ma']
rsi_buy_zone = (ind['rsi'] > g.rsi_oversold) & (ind['rsi'] < (g.rsi_overbought - 5))
macd_bullish = ind['macd_hist'] > 0
volume_active = ind['vol_ratio'] > g.volume_surge_threshold
# BB bounce long
bb_bounce_long = np.zeros(len(ind['close']), dtype=bool)
if 'bb_lower' in ind:
bb_bounce_long = ind['close'] <= ind['bb_lower'] * 1.005
bb_long_entry = bb_bounce_long & (ind['rsi'] < 35) & volume_active
entry = (bullish_trend & rsi_buy_zone & macd_bullish & volume_active) | bb_long_entry
entry[:min_idx] = False
# Long exit: RSI overbought or MACD turns bearish
exit_signal = (ind['rsi'] > g.rsi_overbought) | (ind['macd_hist'] < 0)
exit_signal[:min_idx] = False
# --- SHORT ENTRY ---
bearish_trend = ind['close'] < ind['fast_ma']
rsi_sell_zone = (ind['rsi'] > (g.rsi_oversold + 5)) & (ind['rsi'] < g.rsi_overbought)
macd_bearish = ind['macd_hist'] < 0
# BB bounce short
bb_bounce_short = np.zeros(len(ind['close']), dtype=bool)
if 'bb_upper' in ind:
bb_bounce_short = ind['close'] >= ind['bb_upper'] * 0.995
bb_short_entry = bb_bounce_short & (ind['rsi'] > 65) & volume_active
short_entry = (bearish_trend & rsi_sell_zone & macd_bearish & volume_active) | bb_short_entry
short_entry[:min_idx] = False
# Short exit: RSI oversold or MACD turns bullish
short_exit = (ind['rsi'] < g.rsi_oversold) | (ind['macd_hist'] > 0)
short_exit[:min_idx] = False
# SL/TP levels (long)
stop_loss = ind['close'] - (ind['atr'] * g.stop_loss_atr_mult)
take_profit = ind['close'] + (ind['atr'] * g.take_profit_atr_mult)
# SL/TP levels (short - inverted)
short_stop_loss = ind['close'] + (ind['atr'] * g.stop_loss_atr_mult)
short_take_profit = ind['close'] - (ind['atr'] * g.take_profit_atr_mult)
return {
'entry': entry,
'exit': exit_signal,
'short_entry': short_entry,
'short_exit': short_exit,
'stop_loss': stop_loss,
'take_profit': take_profit,
'short_stop_loss': short_stop_loss,
'short_take_profit': short_take_profit,
'amount_pct': g.max_position_pct,
'max_hold_candles': g.max_hold_candles,
'indicators': ind,
}
def evaluate_genome_signal(genome, df: pd.DataFrame, idx: int) -> Dict:
"""
Evaluate a genome's strategy at a specific index.
Returns signal dict with action, confidence, stop_loss, take_profit.
Used by the RL agent to get the GA signal component.
Supports long, short, and hold signals.
"""
if idx < max(genome.slow_ma_period, 30) or idx >= len(df):
return {'signal': 'hold', 'confidence': 0.0}
strategy_fn = genome_to_strategy(genome)
state = {'position': None, 'capital': 1000, 'num_trades': 0}
result = strategy_fn(state, idx, df, genome.to_dict())
action = result.get('action', 'hold')
# Normalize: 'short' action -> 'sell' signal for RL
signal = action
if action == 'short':
signal = 'sell'
elif action == 'cover':
signal = 'buy'
return {
'signal': signal,
'confidence': result.get('confidence', 0.0),
'stop_loss': result.get('stop_loss'),
'take_profit': result.get('take_profit'),
'short_stop_loss': result.get('short_stop_loss'),
'short_take_profit': result.get('short_take_profit'),
'position_pct': result.get('amount_pct', 0.1),
}
# --- Pre-computation helpers ---
def _precompute_indicators(df: pd.DataFrame, genome) -> Dict[str, np.ndarray]:
"""Pre-compute all indicators as numpy arrays in one pass."""
close = df['close'].values.astype(np.float64)
high = df['high'].values.astype(np.float64)
low = df['low'].values.astype(np.float64)
volume = df['volume'].values.astype(np.float64)
# Moving averages (simple cumsum trick)
fast_ma = _rolling_mean(close, genome.fast_ma_period)
slow_ma = _rolling_mean(close, genome.slow_ma_period)
# RSI
rsi = _compute_rsi_array(close, genome.rsi_period)
# MACD
ema_fast = _ema_array(close, genome.macd_fast)
ema_slow = _ema_array(close, genome.macd_slow)
macd_line = ema_fast - ema_slow
signal_line = _ema_array(macd_line, genome.macd_signal)
macd_hist = macd_line - signal_line
# ATR
atr = _compute_atr_array(high, low, close, genome.atr_period)
# Volume ratio (current volume / 20-period SMA of volume)
vol_sma = _rolling_mean(volume, 20)
vol_ratio = np.where(vol_sma > 0, volume / vol_sma, 1.0)
# Bollinger Bands for mean-reversion scalping
bb_mid = _rolling_mean(close, genome.bb_period)
bb_std = np.full(len(close), 0.0)
for i in range(genome.bb_period - 1, len(close)):
bb_std[i] = np.std(close[max(0, i - genome.bb_period + 1):i + 1])
for i in range(genome.bb_period - 1):
bb_std[i] = np.std(close[:i + 1]) if i > 0 else 0.0
bb_upper = bb_mid + genome.bb_std * bb_std
bb_lower = bb_mid - genome.bb_std * bb_std
return {
'close': close,
'high': high,
'low': low,
'volume': volume,
'fast_ma': fast_ma,
'slow_ma': slow_ma,
'rsi': rsi,
'macd_line': macd_line,
'macd_signal': signal_line,
'macd_hist': macd_hist,
'atr': atr,
'vol_ratio': vol_ratio,
'bb_upper': bb_upper,
'bb_mid': bb_mid,
'bb_lower': bb_lower,
}
def _rolling_mean(arr: np.ndarray, period: int) -> np.ndarray:
"""Fast rolling mean using cumsum."""
n = len(arr)
result = np.full(n, np.nan)
if n < period:
for i in range(n):
result[i] = np.mean(arr[:i + 1])
return result
cs = np.cumsum(arr)
result[period - 1:] = (cs[period - 1:] - np.concatenate([[0], cs[:n - period]])) / period
# Fill initial values with expanding mean
for i in range(period - 1):
result[i] = np.mean(arr[:i + 1])
return result
def _ema_array(arr: np.ndarray, period: int) -> np.ndarray:
"""Compute EMA for entire array in one pass."""
n = len(arr)
result = np.empty(n)
multiplier = 2.0 / (period + 1)
result[0] = arr[0]
for i in range(1, n):
result[i] = (arr[i] - result[i - 1]) * multiplier + result[i - 1]
return result
def _compute_rsi_array(close: np.ndarray, period: int) -> np.ndarray:
"""Compute RSI for entire array using Wilder's smoothing."""
n = len(close)
rsi = np.full(n, 50.0)
if n < period + 1:
return rsi
deltas = np.diff(close)
gains = np.where(deltas > 0, deltas, 0.0)
losses = np.where(deltas < 0, -deltas, 0.0)
# Initial averages (SMA)
avg_gain = np.mean(gains[:period])
avg_loss = np.mean(losses[:period])
if avg_loss > 0:
rs = avg_gain / avg_loss
rsi[period] = 100.0 - (100.0 / (1.0 + rs))
else:
rsi[period] = 100.0
# Wilder's smoothing for remaining
for i in range(period, len(deltas)):
avg_gain = (avg_gain * (period - 1) + gains[i]) / period
avg_loss = (avg_loss * (period - 1) + losses[i]) / period
if avg_loss > 0:
rs = avg_gain / avg_loss
rsi[i + 1] = 100.0 - (100.0 / (1.0 + rs))
else:
rsi[i + 1] = 100.0
return rsi
def _compute_atr_array(high: np.ndarray, low: np.ndarray,
close: np.ndarray, period: int) -> np.ndarray:
"""Compute ATR for entire array using Wilder's smoothing."""
n = len(close)
atr = np.full(n, 0.0)
if n < 2:
return atr
# True range
tr = np.empty(n)
tr[0] = high[0] - low[0]
for i in range(1, n):
tr[i] = max(high[i] - low[i],
abs(high[i] - close[i - 1]),
abs(low[i] - close[i - 1]))
# Initial ATR = SMA of first `period` TRs
if n >= period:
atr[period - 1] = np.mean(tr[:period])
# Wilder's smoothing
for i in range(period, n):
atr[i] = (atr[i - 1] * (period - 1) + tr[i]) / period
# Fill early values with expanding mean
for i in range(period - 1):
atr[i] = np.mean(tr[:i + 1])
else:
for i in range(n):
atr[i] = np.mean(tr[:i + 1])
return atr
src/strategies/manager.py
+138
View File
@@ -0,0 +1,138 @@
"""
Strategy generation and management
"""
from loguru import logger
import json
from datetime import datetime
class StrategyManager:
"""Generate and manage trading strategies"""
def __init__(self, config):
self.config = config
self.active_strategies = []
self.approved_strategies = []
def generate_strategies(self, opportunities, portfolio):
"""Generate trading strategies from opportunities"""
logger.info(f"🧠 Generating strategies from {len(opportunities)} opportunities...")
strategies = []
try:
equity = portfolio["equity"]
cash = portfolio["cash"]
# Get risk limits
max_position_pct = self.config["trading"]["max_position_size_pct"]
max_position_value = equity * (max_position_pct / 100)
# Generate strategies for top opportunities
for opp in opportunities[:5]: # Top 5 opportunities
# Skip if score too low
if opp["score"] < 65:
continue
# Calculate position size
position_value = min(max_position_value, cash * 0.3) # Max 30% of cash per trade
shares = int(position_value / opp["current_price"])
if shares < 1:
continue
# Determine strategy type based on signals
if opp["volume_surge"] > 1.5 and opp["price_change_1w"] > 3:
strategy_type = "momentum_breakout"
target_gain = 15 # 15% target
stop_loss = 7 # 7% stop
elif opp["price_change_1w"] < -3 and opp["volatility"] < 5:
strategy_type = "mean_reversion"
target_gain = 10
stop_loss = 5
else:
strategy_type = "swing_trade"
target_gain = 12
stop_loss = 6
# Calculate prices
entry_price = opp["current_price"]
target_price = entry_price * (1 + target_gain / 100)
stop_price = entry_price * (1 - stop_loss / 100)
strategy = {
"id": f"strat_{datetime.now().strftime('%Y%m%d_%H%M%S')}_{opp['symbol']}",
"type": strategy_type,
"symbol": opp["symbol"],
"action": "buy",
"shares": shares,
"entry_price": round(entry_price, 2),
"target_price": round(target_price, 2),
"stop_loss": round(stop_price, 2),
"position_value": round(shares * entry_price, 2),
"risk_pct": round((entry_price - stop_price) / entry_price * 100, 2),
"reward_pct": round((target_price - entry_price) / entry_price * 100, 2),
"score": opp["score"],
"rationale": self._generate_rationale(opp, strategy_type),
"created_at": datetime.now().isoformat(),
"status": "pending_approval"
}
strategies.append(strategy)
logger.info(f"📋 Generated {strategy_type} strategy for {opp['symbol']}")
self.active_strategies.extend(strategies)
return strategies
except Exception as e:
logger.error(f"❌ Error generating strategies: {e}", exc_info=True)
return []
def _generate_rationale(self, opp, strategy_type):
"""Generate human-readable rationale for a strategy"""
rationale = f"{opp['symbol']} - {strategy_type.replace('_', ' ').title()}\n\n"
rationale += f"📊 Signals:\n"
rationale += f"• Score: {opp['score']}/100\n"
rationale += f"• 1W Change: {opp['price_change_1w']:+.1f}%\n"
rationale += f"• Volume Surge: {opp['volume_surge']:.1f}x\n"
rationale += f"• Volatility: {opp['volatility']:.1f}%\n"
rationale += f"• Sector: {opp.get('sector', 'Unknown')}\n\n"
if strategy_type == "momentum_breakout":
rationale += "🚀 Strong volume surge with positive momentum suggests breakout potential."
elif strategy_type == "mean_reversion":
rationale += "📉 Recent pullback in low-volatility stock suggests mean reversion opportunity."
else:
rationale += "📈 Swing trade setup based on technical indicators."
return rationale
def approve_strategy(self, strategy_id):
"""Approve a strategy for execution"""
for strategy in self.active_strategies:
if strategy["id"] == strategy_id:
strategy["status"] = "approved"
self.approved_strategies.append(strategy)
logger.info(f"✅ Strategy {strategy_id} approved")
return True
return False
def reject_strategy(self, strategy_id, reason=""):
"""Reject a strategy"""
for strategy in self.active_strategies:
if strategy["id"] == strategy_id:
strategy["status"] = "rejected"
strategy["rejection_reason"] = reason
logger.info(f"❌ Strategy {strategy_id} rejected: {reason}")
return True
return False
def get_pending_strategies(self):
"""Get all strategies pending approval"""
return [s for s in self.active_strategies if s["status"] == "pending_approval"]
def get_approved_strategies(self):
"""Get all approved strategies ready for execution"""
return [s for s in self.active_strategies if s["status"] == "approved"]
src/trading/__init__.py
View File
src/trading/broker.py
+254
View File
@@ -0,0 +1,254 @@
"""
Alpaca broker interface for paper trading
"""
from alpaca.trading.client import TradingClient
from alpaca.trading.requests import MarketOrderRequest, LimitOrderRequest
from alpaca.trading.enums import OrderSide, TimeInForce
from alpaca.data.historical import StockHistoricalDataClient
from alpaca.data.requests import StockBarsRequest
from alpaca.data.timeframe import TimeFrame
from loguru import logger
from datetime import datetime, timedelta
class AlpacaBroker:
"""Alpaca paper trading broker interface"""
def __init__(self, config):
self.config = config
# Initialize trading client
self.trading_client = TradingClient(
api_key=config["api_key"],
secret_key=config["secret_key"],
paper=True # Always use paper trading
)
# Initialize data client
self.data_client = StockHistoricalDataClient(
api_key=config["api_key"],
secret_key=config["secret_key"]
)
logger.info("✅ Alpaca broker connected (PAPER TRADING)")
def get_account(self):
"""Get account information"""
return self.trading_client.get_account()
def get_portfolio(self):
"""Get current portfolio status"""
account = self.get_account()
return {
"equity": float(account.equity),
"cash": float(account.cash),
"buying_power": float(account.buying_power),
"portfolio_value": float(account.portfolio_value),
"long_market_value": float(account.long_market_value),
"day_pnl": float(account.equity) - float(account.last_equity),
"day_pnl_pct": ((float(account.equity) - float(account.last_equity)) / float(account.last_equity) * 100) if float(account.last_equity) > 0 else 0
}
def get_positions(self):
"""Get all open positions"""
positions = self.trading_client.get_all_positions()
return [
{
"symbol": p.symbol,
"qty": float(p.qty),
"market_value": float(p.market_value),
"cost_basis": float(p.cost_basis),
"unrealized_pl": float(p.unrealized_pl),
"unrealized_plpc": float(p.unrealized_plpc) * 100,
"current_price": float(p.current_price),
"avg_entry_price": float(p.avg_entry_price)
}
for p in positions
]
def is_market_open(self):
"""Check if market is currently open"""
clock = self.trading_client.get_clock()
return clock.is_open
def get_market_hours(self):
"""Get today's market hours"""
clock = self.trading_client.get_clock()
return {
"is_open": clock.is_open,
"next_open": clock.next_open,
"next_close": clock.next_close
}
def place_market_order(self, symbol, qty, side):
"""Place a market order"""
try:
order_data = MarketOrderRequest(
symbol=symbol,
qty=qty,
side=OrderSide.BUY if side == "buy" else OrderSide.SELL,
time_in_force=TimeInForce.DAY
)
order = self.trading_client.submit_order(order_data)
logger.info(f"✅ Market order placed: {side.upper()} {qty} {symbol}")
return {
"id": order.id,
"symbol": order.symbol,
"qty": float(order.qty),
"side": order.side,
"type": order.type,
"status": order.status
}
except Exception as e:
logger.error(f"❌ Error placing order: {e}")
raise
def place_limit_order(self, symbol, qty, side, limit_price):
"""Place a limit order"""
try:
order_data = LimitOrderRequest(
symbol=symbol,
qty=qty,
side=OrderSide.BUY if side == "buy" else OrderSide.SELL,
time_in_force=TimeInForce.DAY,
limit_price=limit_price
)
order = self.trading_client.submit_order(order_data)
logger.info(f"✅ Limit order placed: {side.upper()} {qty} {symbol} @ ${limit_price}")
return {
"id": order.id,
"symbol": order.symbol,
"qty": float(order.qty),
"side": order.side,
"type": order.type,
"limit_price": float(order.limit_price),
"status": order.status
}
except Exception as e:
logger.error(f"❌ Error placing limit order: {e}")
raise
def cancel_order(self, order_id):
"""Cancel an open order"""
try:
self.trading_client.cancel_order_by_id(order_id)
logger.info(f"✅ Order {order_id} cancelled")
return True
except Exception as e:
logger.error(f"❌ Error cancelling order: {e}")
return False
def get_orders(self, status="open"):
"""Get orders by status"""
from alpaca.trading.enums import QueryOrderStatus
status_map = {
"open": QueryOrderStatus.OPEN,
"closed": QueryOrderStatus.CLOSED,
"all": QueryOrderStatus.ALL
}
orders = self.trading_client.get_orders(filter=status_map.get(status))
return [
{
"id": o.id,
"symbol": o.symbol,
"qty": float(o.qty),
"side": o.side,
"type": o.type,
"status": o.status,
"created_at": o.created_at
}
for o in orders
]
def get_bars(self, symbol, timeframe="1Day", limit=100):
"""Get historical price bars"""
try:
request_params = StockBarsRequest(
symbol_or_symbols=symbol,
timeframe=TimeFrame.Day if timeframe == "1Day" else TimeFrame.Hour,
limit=limit
)
bars = self.data_client.get_stock_bars(request_params)
if symbol in bars:
return [
{
"timestamp": bar.timestamp,
"open": float(bar.open),
"high": float(bar.high),
"low": float(bar.low),
"close": float(bar.close),
"volume": int(bar.volume)
}
for bar in bars[symbol]
]
return []
except Exception as e:
logger.error(f"❌ Error fetching bars for {symbol}: {e}")
return []
def fetch_bars_range(self, symbol, timeframe="1Hour", start=None, end=None):
"""Fetch historical bars for a date range (for backtesting cache)"""
try:
tf = TimeFrame.Day if timeframe == "1Day" else TimeFrame.Hour
kwargs = {
"symbol_or_symbols": symbol,
"timeframe": tf,
}
if start:
kwargs["start"] = start
if end:
kwargs["end"] = end
request_params = StockBarsRequest(**kwargs)
bars = self.data_client.get_stock_bars(request_params)
if symbol in bars:
return [
{
"timestamp": bar.timestamp,
"open": float(bar.open),
"high": float(bar.high),
"low": float(bar.low),
"close": float(bar.close),
"volume": int(bar.volume)
}
for bar in bars[symbol]
]
return []
except Exception as e:
logger.error(f"Error fetching bars range for {symbol}: {e}")
return []
def get_latest_price(self, symbol):
"""Get latest price for a symbol"""
try:
bars = self.get_bars(symbol, timeframe="1Day", limit=1)
if bars:
return bars[-1]['close']
return None
except Exception as e:
logger.error(f"Error getting latest price for {symbol}: {e}")
return None
def close_position(self, symbol):
"""Close an entire position for a symbol"""
try:
self.trading_client.close_position(symbol)
logger.info(f"Closed position: {symbol}")
return True
except Exception as e:
logger.error(f"Error closing position {symbol}: {e}")
return False
src/trading/executor.py
+379
View File
@@ -0,0 +1,379 @@
"""
Trade Executor
Executes live trades via Alpaca broker with position management.
"""
from datetime import datetime
from typing import Dict, List, Optional
from loguru import logger
class TradingExecutor:
"""
Executes live trades based on RL agent decisions.
Manages positions, stop losses, and take profits.
"""
def __init__(self, broker, store, safety, config: Dict):
"""
Args:
broker: AlpacaBroker instance
store: DataStore instance
safety: SafetyManager instance
config: Trading configuration dict
"""
self.broker = broker
self.store = store
self.safety = safety
self.config = config
self.commission_rate = config.get('commission_rate', 0.001)
def execute_signal(self, symbol: str, action: int, current_price: float,
strategy_params: Dict = None) -> Optional[Dict]:
"""
Execute a trading action from the RL agent.
Actions:
0: Hold
1: Buy 25% of available capital (go long)
2: Buy 50% of available capital (go long)
3: Close 50% of position (long or short)
4: Close 100% of position (long or short)
5: Short 25% of available capital
6: Short 50% of available capital
Returns: trade record dict, or None if no action taken
"""
if action == 0: # Hold
return None
if current_price <= 0:
return None
strategy_params = strategy_params or {}
# Get portfolio state
try:
portfolio = self.broker.get_portfolio()
equity = portfolio['equity']
cash = portfolio['cash']
except Exception as e:
logger.error(f"Error getting portfolio: {e}")
return None
# Update safety peak
self.safety.update_peak_equity(equity)
# Get current positions
open_positions = self.store.get_open_positions()
open_for_symbol = [p for p in open_positions if p['symbol'] == symbol]
num_positions = len(set(p['symbol'] for p in open_positions))
# BUY LONG actions
if action in (1, 2):
if open_for_symbol:
logger.debug(f"Already have position in {symbol}, skipping buy")
return None
pct = 0.25 if action == 1 else 0.50
invest = cash * pct
max_position_pct = self.config.get('max_position_pct', 12) / 100
max_invest = equity * max_position_pct
if invest > max_invest:
invest = max_invest
if invest < self.config.get('min_trade_value', 3):
return None
shares = int(invest / current_price)
if shares < 1:
shares = round(invest / current_price, 4)
if shares * current_price < self.config.get('min_trade_value', 3):
return None
allowed, reason = self.safety.validate_trade(
symbol, 'buy', shares, current_price, equity, num_positions
)
if not allowed:
logger.debug(f"Trade blocked: {reason}")
return None
try:
order = self.broker.place_market_order(symbol, shares, 'buy')
logger.info(f"BUY {shares} {symbol} @ ~${current_price:.4f}")
except Exception as e:
logger.error(f"Error placing buy order: {e}")
return None
stop_loss = strategy_params.get('stop_loss')
take_profit = strategy_params.get('take_profit')
trade = {
'symbol': symbol,
'side': 'buy',
'amount': shares,
'entry_price': current_price,
'entry_time': datetime.utcnow().isoformat(),
'strategy_id': strategy_params.get('strategy_id', 'auto'),
'stop_loss': stop_loss,
'take_profit': take_profit,
'order_id': str(order.get('id', '')),
'status': 'open',
'metadata': {
'action': action,
'invest_pct': pct,
'equity_at_entry': equity,
'direction': 'long',
},
}
trade_id = self.store.record_trade(trade)
trade['id'] = trade_id
return trade
# CLOSE POSITION actions (long or short)
elif action in (3, 4):
if not open_for_symbol:
return None
position = open_for_symbol[0]
total_shares = position['amount']
is_short = position.get('metadata', {}).get('direction') == 'short'
if action == 3:
close_shares = abs(total_shares) * 0.5
else:
close_shares = abs(total_shares)
close_shares = round(close_shares, 4)
if close_shares * current_price < self.config.get('min_trade_value', 1):
close_shares = abs(total_shares)
# Determine order side (opposite of position direction)
close_side = 'buy' if is_short else 'sell'
try:
order = self.broker.place_market_order(symbol, close_shares, close_side)
logger.info(f"CLOSE({close_side.upper()}) {close_shares} {symbol} @ ~${current_price:.4f}")
except Exception as e:
logger.error(f"Error placing close order: {e}")
return None
# Calculate P&L
if is_short:
pnl = (position['entry_price'] - current_price) * close_shares
else:
pnl = (current_price - position['entry_price']) * close_shares
if close_shares >= abs(total_shares) * 0.99:
self.store.close_position(
position['id'], current_price, datetime.utcnow(),
fees=close_shares * current_price * self.commission_rate
)
self.safety.record_trade_result(pnl)
return {
'symbol': symbol,
'side': close_side,
'amount': close_shares,
'exit_price': current_price,
'pnl': round(pnl, 2),
'pnl_pct': round(pnl / (position['entry_price'] * close_shares) * 100, 2),
'action': action,
'direction': 'short' if is_short else 'long',
}
else:
self.safety.record_trade_result(pnl)
self.store.close_position(
position['id'], current_price, datetime.utcnow(),
fees=close_shares * current_price * self.commission_rate
)
remaining = abs(total_shares) - close_shares
if remaining > 0:
self.store.record_trade({
'symbol': symbol,
'side': position['side'],
'amount': remaining,
'entry_price': position['entry_price'],
'entry_time': position['entry_time'],
'strategy_id': position.get('strategy_id', 'auto'),
'stop_loss': position.get('stop_loss'),
'take_profit': position.get('take_profit'),
'status': 'open',
'metadata': position.get('metadata', {}),
})
return {
'symbol': symbol,
'side': close_side,
'amount': close_shares,
'exit_price': current_price,
'pnl': round(pnl, 2),
'action': action,
'direction': 'short' if is_short else 'long',
}
# SHORT actions
elif action in (5, 6):
if open_for_symbol:
logger.debug(f"Already have position in {symbol}, skipping short")
return None
pct = 0.25 if action == 5 else 0.50
invest = cash * pct
max_position_pct = self.config.get('max_position_pct', 12) / 100
max_invest = equity * max_position_pct
if invest > max_invest:
invest = max_invest
if invest < self.config.get('min_trade_value', 3):
return None
shares = int(invest / current_price)
if shares < 1:
shares = round(invest / current_price, 4)
if shares * current_price < self.config.get('min_trade_value', 3):
return None
allowed, reason = self.safety.validate_trade(
symbol, 'sell', shares, current_price, equity, num_positions
)
if not allowed:
logger.debug(f"Short blocked: {reason}")
return None
try:
order = self.broker.place_market_order(symbol, shares, 'sell')
logger.info(f"SHORT {shares} {symbol} @ ~${current_price:.4f}")
except Exception as e:
logger.error(f"Error placing short order: {e}")
return None
# For shorts, stop_loss is ABOVE entry, take_profit is BELOW
stop_loss = strategy_params.get('short_stop_loss') or strategy_params.get('stop_loss')
take_profit = strategy_params.get('short_take_profit') or strategy_params.get('take_profit')
trade = {
'symbol': symbol,
'side': 'sell',
'amount': shares,
'entry_price': current_price,
'entry_time': datetime.utcnow().isoformat(),
'strategy_id': strategy_params.get('strategy_id', 'auto'),
'stop_loss': stop_loss,
'take_profit': take_profit,
'order_id': str(order.get('id', '')),
'status': 'open',
'metadata': {
'action': action,
'invest_pct': pct,
'equity_at_entry': equity,
'direction': 'short',
},
}
trade_id = self.store.record_trade(trade)
trade['id'] = trade_id
return trade
return None
def check_exits(self, current_prices: Dict[str, float]) -> List[Dict]:
"""
Check all open positions for stop loss / take profit exits.
Handles both long and short positions.
Returns list of closed trades.
"""
closed = []
open_positions = self.store.get_open_positions()
for position in open_positions:
symbol = position['symbol']
price = current_prices.get(symbol)
if price is None:
continue
should_exit = False
exit_reason = ''
is_short = position.get('metadata', {}).get('direction') == 'short'
if is_short:
# Short position: stop_loss is ABOVE entry, take_profit is BELOW
if position.get('stop_loss') and price >= position['stop_loss']:
should_exit = True
exit_reason = 'stop_loss'
elif position.get('take_profit') and price <= position['take_profit']:
should_exit = True
exit_reason = 'take_profit'
else:
# Long position: stop_loss is BELOW entry, take_profit is ABOVE
if position.get('stop_loss') and price <= position['stop_loss']:
should_exit = True
exit_reason = 'stop_loss'
elif position.get('take_profit') and price >= position['take_profit']:
should_exit = True
exit_reason = 'take_profit'
if should_exit:
# Close side is opposite of position direction
close_side = 'buy' if is_short else 'sell'
try:
self.broker.place_market_order(
symbol, position['amount'], close_side
)
logger.info(f"EXIT ({exit_reason}) {symbol} @ ${price:.4f} [{'SHORT' if is_short else 'LONG'}]")
except Exception as e:
logger.error(f"Error executing exit for {symbol}: {e}")
continue
if is_short:
pnl = (position['entry_price'] - price) * position['amount']
else:
pnl = (price - position['entry_price']) * position['amount']
self.store.close_position(
position['id'], price, datetime.utcnow(),
fees=position['amount'] * price * self.commission_rate
)
self.safety.record_trade_result(pnl)
closed.append({
'symbol': symbol,
'side': close_side,
'amount': position['amount'],
'entry_price': position['entry_price'],
'exit_price': price,
'pnl': round(pnl, 2),
'pnl_pct': round(pnl / (position['entry_price'] * position['amount']) * 100, 2),
'exit_reason': exit_reason,
'direction': 'short' if is_short else 'long',
})
return closed
def get_portfolio_state(self) -> Dict:
"""Get current portfolio state for RL agent"""
try:
portfolio = self.broker.get_portfolio()
positions = self.broker.get_positions()
open_db = self.store.get_open_positions()
total_position_value = sum(p.get('market_value', 0) for p in positions)
equity = portfolio['equity']
return {
'equity': equity,
'cash': portfolio['cash'],
'position_ratio': total_position_value / equity if equity > 0 else 0,
'unrealized_pnl': sum(p.get('unrealized_pl', 0) for p in positions) / equity
if equity > 0 else 0,
'time_in_position': 0, # Simplified
'num_positions': len(positions),
}
except Exception as e:
logger.error(f"Error getting portfolio state: {e}")
return {
'equity': 0, 'cash': 0, 'position_ratio': 0,
'unrealized_pnl': 0, 'time_in_position': 0, 'num_positions': 0,
}
src/trading/oanda_broker.py
+321
View File
@@ -0,0 +1,321 @@
"""
OANDA broker interface for forex paper trading.
Uses OANDA v20 REST API - no extra dependency, just requests.
"""
import requests
from datetime import datetime, timedelta
from typing import Dict, List, Optional
from loguru import logger
class OandaBroker:
"""OANDA practice (paper) trading broker interface"""
PRACTICE_URL = "https://api-fxpractice.oanda.com"
LIVE_URL = "https://api-fxtrade.oanda.com"
# Forex pairs trade 24/5 (Sun 5PM ET to Fri 5PM ET)
# Map our timeframes to OANDA granularity
TF_MAP = {
'1m': 'M1', '5m': 'M5', '15m': 'M15',
'1h': 'H1', '4h': 'H4', '1d': 'D',
}
def __init__(self, config: Dict):
self.config = config
self.api_token = config['api_token']
self.account_id = config['account_id']
self.base_url = self.PRACTICE_URL if config.get('practice', True) else self.LIVE_URL
self.session = requests.Session()
self.session.headers.update({
'Authorization': f'Bearer {self.api_token}',
'Content-Type': 'application/json',
})
# Verify connection
try:
acct = self._get(f'/v3/accounts/{self.account_id}/summary')
balance = acct['account']['balance']
currency = acct['account']['currency']
logger.info(f"OANDA connected (PRACTICE) - Balance: {currency} {balance}")
except Exception as e:
logger.error(f"OANDA connection failed: {e}")
raise
def _get(self, path: str, params: Dict = None) -> Dict:
resp = self.session.get(f'{self.base_url}{path}', params=params, timeout=15)
resp.raise_for_status()
return resp.json()
def _post(self, path: str, data: Dict) -> Dict:
resp = self.session.post(f'{self.base_url}{path}', json=data, timeout=15)
resp.raise_for_status()
return resp.json()
def _put(self, path: str, data: Dict) -> Dict:
resp = self.session.put(f'{self.base_url}{path}', json=data, timeout=15)
resp.raise_for_status()
return resp.json()
# --- Account / Portfolio ---
def get_account(self) -> Dict:
data = self._get(f'/v3/accounts/{self.account_id}/summary')
return data['account']
def get_portfolio(self) -> Dict:
acct = self.get_account()
nav = float(acct['NAV'])
balance = float(acct['balance'])
unrealized_pl = float(acct['unrealizedPL'])
pl = float(acct['pl'])
return {
'equity': nav,
'cash': balance,
'buying_power': float(acct.get('marginAvailable', balance)),
'portfolio_value': nav,
'long_market_value': nav - balance,
'day_pnl': unrealized_pl,
'day_pnl_pct': (unrealized_pl / balance * 100) if balance > 0 else 0,
}
def get_positions(self) -> List[Dict]:
data = self._get(f'/v3/accounts/{self.account_id}/openPositions')
positions = []
for p in data.get('positions', []):
# OANDA separates long/short
long_units = int(p['long']['units']) if p['long']['units'] != '0' else 0
short_units = abs(int(p['short']['units'])) if p['short']['units'] != '0' else 0
if long_units > 0:
unrealized = float(p['long']['unrealizedPL'])
avg_price = float(p['long']['averagePrice'])
# Estimate current price from avg + pnl
current_price = avg_price + (unrealized / long_units) if long_units else avg_price
positions.append({
'symbol': p['instrument'],
'qty': long_units,
'market_value': long_units * current_price,
'cost_basis': long_units * avg_price,
'unrealized_pl': unrealized,
'unrealized_plpc': (unrealized / (long_units * avg_price) * 100)
if avg_price > 0 else 0,
'current_price': current_price,
'avg_entry_price': avg_price,
})
if short_units > 0:
unrealized = float(p['short']['unrealizedPL'])
avg_price = float(p['short']['averagePrice'])
current_price = avg_price - (unrealized / short_units) if short_units else avg_price
positions.append({
'symbol': p['instrument'],
'qty': -short_units,
'market_value': short_units * current_price,
'cost_basis': short_units * avg_price,
'unrealized_pl': unrealized,
'unrealized_plpc': (unrealized / (short_units * avg_price) * 100)
if avg_price > 0 else 0,
'current_price': current_price,
'avg_entry_price': avg_price,
})
return positions
def is_market_open(self) -> bool:
"""Forex is open 24/5 - closed Saturday and most of Sunday"""
now = datetime.utcnow()
# Closed: Friday 22:00 UTC to Sunday 22:00 UTC (roughly)
if now.weekday() == 5: # Saturday
return False
if now.weekday() == 6 and now.hour < 22: # Sunday before 22:00
return False
if now.weekday() == 4 and now.hour >= 22: # Friday after 22:00
return False
return True
def get_market_hours(self) -> Dict:
return {
'is_open': self.is_market_open(),
'next_open': None,
'next_close': None,
}
# --- Orders ---
def place_market_order(self, symbol: str, qty, side: str) -> Dict:
"""Place a market order. qty is in units (not lots)."""
units = int(qty) if side == 'buy' else -int(qty)
data = {
'order': {
'type': 'MARKET',
'instrument': symbol,
'units': str(units),
'timeInForce': 'FOK',
}
}
try:
result = self._post(f'/v3/accounts/{self.account_id}/orders', data)
fill = result.get('orderFillTransaction', {})
order_id = fill.get('id', result.get('orderCreateTransaction', {}).get('id', ''))
logger.info(f"OANDA {side.upper()} {abs(units)} {symbol}")
return {
'id': order_id,
'symbol': symbol,
'qty': abs(units),
'side': side,
'type': 'market',
'status': 'filled' if fill else 'pending',
'fill_price': float(fill.get('price', 0)) if fill else 0,
}
except Exception as e:
logger.error(f"OANDA order error: {e}")
raise
def place_limit_order(self, symbol: str, qty, side: str, limit_price: float) -> Dict:
units = int(qty) if side == 'buy' else -int(qty)
data = {
'order': {
'type': 'LIMIT',
'instrument': symbol,
'units': str(units),
'price': f'{limit_price:.5f}',
'timeInForce': 'GTC',
}
}
result = self._post(f'/v3/accounts/{self.account_id}/orders', data)
order = result.get('orderCreateTransaction', {})
return {
'id': order.get('id', ''),
'symbol': symbol,
'qty': abs(units),
'side': side,
'type': 'limit',
'limit_price': limit_price,
'status': 'pending',
}
def cancel_order(self, order_id: str) -> bool:
try:
self._put(f'/v3/accounts/{self.account_id}/orders/{order_id}/cancel', {})
return True
except Exception:
return False
def get_orders(self, status: str = "open") -> List[Dict]:
state = 'PENDING' if status == 'open' else 'ALL'
data = self._get(f'/v3/accounts/{self.account_id}/orders', {'state': state})
return [
{
'id': o['id'],
'symbol': o.get('instrument', ''),
'qty': abs(int(o.get('units', 0))),
'side': 'buy' if int(o.get('units', 0)) > 0 else 'sell',
'type': o.get('type', '').lower(),
'status': o.get('state', '').lower(),
'created_at': o.get('createTime', ''),
}
for o in data.get('orders', [])
]
# --- Market Data ---
def get_bars(self, symbol: str, timeframe: str = "1d", limit: int = 100) -> List[Dict]:
gran = self.TF_MAP.get(timeframe, 'H1')
params = {'granularity': gran, 'count': min(limit, 5000)}
try:
data = self._get(f'/v3/instruments/{symbol}/candles', params)
return self._parse_candles(data.get('candles', []))
except Exception as e:
logger.error(f"OANDA bars error for {symbol}: {e}")
return []
def fetch_bars_range(self, symbol: str, timeframe: str = "1h",
start=None, end=None) -> List[Dict]:
gran = self.TF_MAP.get(timeframe, 'H1')
params = {'granularity': gran, 'price': 'M'}
if start:
if isinstance(start, datetime):
params['from'] = start.strftime('%Y-%m-%dT%H:%M:%SZ')
else:
params['from'] = str(start)
if end:
if isinstance(end, datetime):
params['to'] = end.strftime('%Y-%m-%dT%H:%M:%SZ')
else:
params['to'] = str(end)
if 'from' not in params:
params['count'] = 500
try:
data = self._get(f'/v3/instruments/{symbol}/candles', params)
return self._parse_candles(data.get('candles', []))
except Exception as e:
logger.error(f"OANDA bars range error for {symbol}: {e}")
return []
def get_latest_price(self, symbol: str) -> Optional[float]:
try:
data = self._get(f'/v3/instruments/{symbol}/candles',
{'granularity': 'M1', 'count': 1, 'price': 'M'})
candles = data.get('candles', [])
if candles:
return float(candles[-1]['mid']['c'])
return None
except Exception as e:
logger.error(f"OANDA price error for {symbol}: {e}")
return None
def close_position(self, symbol: str) -> bool:
"""Close entire position for an instrument"""
try:
# Close long
try:
self._put(f'/v3/accounts/{self.account_id}/positions/{symbol}/close',
{'longUnits': 'ALL'})
except Exception:
pass
# Close short
try:
self._put(f'/v3/accounts/{self.account_id}/positions/{symbol}/close',
{'shortUnits': 'ALL'})
except Exception:
pass
logger.info(f"Closed OANDA position: {symbol}")
return True
except Exception as e:
logger.error(f"Error closing OANDA position {symbol}: {e}")
return False
# --- Helpers ---
def _parse_candles(self, candles: List[Dict]) -> List[Dict]:
"""Convert OANDA candles to our standard format"""
result = []
for c in candles:
if not c.get('complete', True) and len(candles) > 1:
continue # Skip incomplete candles unless it's the only one
mid = c.get('mid', {})
ts = c.get('time', '')
try:
dt = datetime.fromisoformat(ts.replace('Z', '+00:00'))
ts_ms = int(dt.timestamp() * 1000)
except (ValueError, AttributeError):
continue
result.append({
'timestamp': ts_ms,
'open': float(mid.get('o', 0)),
'high': float(mid.get('h', 0)),
'low': float(mid.get('l', 0)),
'close': float(mid.get('c', 0)),
'volume': int(c.get('volume', 0)),
})
return result
def get_tradeable_instruments(self) -> List[str]:
"""Get list of available forex instruments"""
data = self._get(f'/v3/accounts/{self.account_id}/instruments')
return [i['name'] for i in data.get('instruments', [])
if i.get('type') == 'CURRENCY']
verify.sh
+114
View File
@@ -0,0 +1,114 @@
#!/bin/bash
# BIGGFISH Installation Verification Script
echo "🐟 BIGGFISH Installation Verification"
echo "======================================"
echo ""
# Check Python version
echo "📍 Checking Python version..."
python3 --version
if [ $? -ne 0 ]; then
echo "❌ Python 3 not found. Please install Python 3.9+"
exit 1
fi
echo "✅ Python OK"
echo ""
# Check directory structure
echo "📍 Checking directory structure..."
required_dirs=(
"src/trading"
"src/research"
"src/strategies"
"src/reporting"
"config"
"data/stocks"
"data/reports"
"data/trades"
"logs"
"tests"
)
all_good=true
for dir in "${required_dirs[@]}"; do
if [ -d "$dir" ]; then
echo " ✅ $dir"
else
echo " ❌ $dir - MISSING"
all_good=false
fi
done
if [ "$all_good" = false ]; then
echo "❌ Some directories are missing"
exit 1
fi
echo "✅ Directory structure OK"
echo ""
# Check config file
echo "📍 Checking configuration..."
if [ ! -f "config/config.json" ]; then
echo "⚠️ config/config.json not found"
echo " Run: cp config/config.example.json config/config.json"
echo " Then add your Alpaca API keys"
else
echo "✅ config.json exists"
# Check if keys are set
if grep -q "YOUR_ALPACA" config/config.json; then
echo "⚠️ Please update config.json with your Alpaca API keys"
else
echo "✅ API keys appear to be configured"
fi
fi
echo ""
# Check Python packages
echo "📍 Checking Python dependencies..."
if pip list | grep -q alpaca-py; then
echo "✅ alpaca-py installed"
else
echo "❌ alpaca-py not installed"
echo " Run: pip install -r requirements.txt"
fi
if pip list | grep -q yfinance; then
echo "✅ yfinance installed"
else
echo "❌ yfinance not installed"
echo " Run: pip install -r requirements.txt"
fi
if pip list | grep -q loguru; then
echo "✅ loguru installed"
else
echo "❌ loguru not installed"
echo " Run: pip install -r requirements.txt"
fi
echo ""
# Final status
echo "======================================"
echo "🎯 Next Steps:"
echo ""
if [ ! -f "config/config.json" ]; then
echo "1. Copy config: cp config/config.example.json config/config.json"
echo "2. Edit config.json and add your Alpaca paper trading API keys"
echo "3. Install packages: pip install -r requirements.txt"
echo "4. Test connection: python src/cli.py market"
elif grep -q "YOUR_ALPACA" config/config.json 2>/dev/null; then
echo "1. Edit config.json and add your Alpaca paper trading API keys"
echo "2. Test connection: python src/cli.py market"
echo "3. Run your first scan: python src/cli.py scan"
else
echo "✅ You're ready to go!"
echo ""
echo "Try these commands:"
echo " python src/cli.py market # Check market status"
echo " python src/cli.py scan # Scan for opportunities"
echo " python src/cli.py status # Check portfolio"
echo " python src/main.py # Start the trading system"
fi
echo ""