188 lines
5.9 KiB
Markdown
188 lines
5.9 KiB
Markdown
# Pokemon Stock Monitor - Project Guide
|
||
|
||
## Overview
|
||
A Pokemon TCG stock monitoring system that tracks retail sites (Target, PokemonCenter, GameStop, Best Buy, Walmart) for restocks and new drops, sending Discord notifications when products become available.
|
||
|
||
## Architecture
|
||
|
||
```
|
||
User -> Chrome Extension (PokemonCenter API interception)
|
||
-> Python Monitor (main.py) -> Scrapers -> ProductTracker -> Discord Notifications
|
||
-> Flask Dashboard (localhost:5000) -> REST API
|
||
```
|
||
|
||
## Directory Structure
|
||
|
||
### Core Application
|
||
- `main.py` - Entry point, scheduling loop, orchestrates scraping
|
||
- `config.py` - All configuration (intervals, URLs, Discord webhook, browser settings)
|
||
|
||
### Source Code (`src/`)
|
||
- `browser.py` - Playwright/Chrome browser management
|
||
- `product_tracker.py` - Detects new products and restocks, manages state
|
||
- `discord_notifier.py` - Sends Discord webhook notifications
|
||
- `discord_bot.py` - Interactive Discord bot commands
|
||
- `database.py` - SQLite database for historical tracking
|
||
- `favorites.py` - User favorites/watchlist management
|
||
- `scraper_state.py` - Persists scraper state between runs
|
||
- `store_locator.py` - Target store location lookup
|
||
|
||
### Scrapers (`scrapers/`)
|
||
- `base.py` - `Product` dataclass, `BaseScraper` abstract class
|
||
- `pokemoncenter.py` - PokemonCenter scraper (needs stealth browser)
|
||
- `target.py` - Target.com scraper
|
||
- `gamestop.py` - GameStop.com scraper (light bot protection)
|
||
- `bestbuy.py` - BestBuy.com scraper (moderate bot protection)
|
||
- `walmart.py` - Walmart.com scraper (aggressive bot protection, may need stealth)
|
||
|
||
### Chrome Extension (`chrome-extension/`)
|
||
- `manifest.json` - Extension config (MV3)
|
||
- `background.js` - Service worker, manages alarms and notifications
|
||
- `content.js` - Injected into PokemonCenter pages
|
||
- `api-interceptor.js` - Intercepts PokemonCenter API responses
|
||
- `popup.html/js` - Extension popup UI
|
||
|
||
### Dashboard (`dashboard/`)
|
||
- `app.py` - Flask app, routes
|
||
- `api.py` - REST API endpoints (`/api/*`)
|
||
- `templates/index.html` - Main template
|
||
- `static/app.js` - Frontend JavaScript
|
||
|
||
### Data (`data/`)
|
||
- `products.json` - Tracked products state
|
||
- `scraper_state.json` - Persisted scraper state
|
||
- `known_skus.json` - Known product SKUs
|
||
- `proxies.json` - Proxy configuration
|
||
- `user_locations.json` - User store locations
|
||
|
||
### Tools (`tools/`)
|
||
Development and debugging utilities:
|
||
- `api_monitor.py`, `backend_monitor.py` - API monitoring
|
||
- `stealth_browser.py` - Anti-detection browser
|
||
- `capture_api_calls.py` - HAR capture for debugging
|
||
|
||
## Key Classes
|
||
|
||
### `Product` (scrapers/base.py)
|
||
```python
|
||
@dataclass
|
||
class Product:
|
||
name: str
|
||
url: str
|
||
price: Optional[str]
|
||
in_stock: bool
|
||
image_url: Optional[str]
|
||
site: str
|
||
product_id: Optional[str]
|
||
```
|
||
|
||
### `ProductTracker` (src/product_tracker.py)
|
||
- `process_products(products)` -> `(new_products, restocked_products)`
|
||
- `get_stats()` -> dashboard statistics
|
||
|
||
### `BaseScraper` (scrapers/base.py)
|
||
- `scrape_category_page(url)` -> `List[Product]`
|
||
- `check_product_stock(url)` -> `(in_stock, price)`
|
||
|
||
## Common Tasks
|
||
|
||
### Add a new retailer scraper
|
||
1. Create `scrapers/newsite.py` extending `BaseScraper`
|
||
2. Implement `scrape_category_page()` and `check_product_stock()`
|
||
3. Add to `scrapers/__init__.py`
|
||
4. Add config in `config.py` (URLs, enable flag)
|
||
5. Add check function in `main.py`
|
||
|
||
### Modify Discord notifications
|
||
- Webhook format: `src/discord_notifier.py`
|
||
- Bot commands: `src/discord_bot.py`
|
||
|
||
### Change check intervals or URLs
|
||
- Edit `config.py`
|
||
|
||
### Dashboard changes
|
||
- Routes: `dashboard/app.py`
|
||
- API: `dashboard/api.py`
|
||
- Frontend: `dashboard/static/app.js`
|
||
|
||
### Chrome extension changes
|
||
- Popup UI: `chrome-extension/popup.html`, `popup.js`
|
||
- API interception: `chrome-extension/api-interceptor.js`
|
||
- Background logic: `chrome-extension/background.js`
|
||
|
||
## Conventions
|
||
- Python 3.13+
|
||
- Logging via `logging` module (configured in main.py)
|
||
- Dataclasses for data structures
|
||
- Flask for web/API
|
||
- Playwright for browser automation
|
||
- SQLite for persistence (`stats.db`)
|
||
|
||
## Running
|
||
|
||
```bash
|
||
# Main monitor
|
||
python main.py
|
||
|
||
# Dashboard only
|
||
python dashboard/app.py
|
||
|
||
# Discord bot
|
||
python src/discord_bot.py
|
||
```
|
||
|
||
## Testing
|
||
```bash
|
||
python -m pytest tests/
|
||
```
|
||
|
||
## Auto-Buy System
|
||
|
||
### Architecture
|
||
- `src/buyers/base_buyer.py` — Abstract base, browser launch, shared utilities
|
||
- `src/buyers/target_buyer.py` — Target checkout flow
|
||
- `src/buyers/bestbuy_buyer.py` — BestBuy checkout flow
|
||
- `src/buyers/gamestop_buyer.py` — GameStop checkout flow (Selenium)
|
||
- `src/profile_store.py` — AES-256 encrypted user checkout profiles
|
||
- `data/browser_states/<user_id>/<site>.json` — Saved browser sessions per user
|
||
|
||
### Multi-User Browser Session Setup (Target)
|
||
|
||
Target blocks automated logins with a server-side error. Each user must save a browser session once by logging in manually. Sessions are stored per-user per-site.
|
||
|
||
**One-time setup per user:**
|
||
```bash
|
||
# User 1 logs into Target
|
||
python test_buy.py --setup target --user-id 1
|
||
|
||
# User 2 logs into Target
|
||
python test_buy.py --setup target --user-id 2
|
||
```
|
||
|
||
A Chrome window opens on the Target account page. The user logs in normally, then presses Enter in the terminal. Session saved to `data/browser_states/<user_id>/target.json`.
|
||
|
||
**Sessions expire** when the Target login cookie expires (typically 30–90 days). Re-run `--setup` for that user if checkout starts failing.
|
||
|
||
**The same pattern applies to BestBuy and GameStop:**
|
||
```bash
|
||
python test_buy.py --setup bestbuy --user-id 1
|
||
python test_buy.py --setup gamestop --user-id 1
|
||
```
|
||
|
||
### Dry-Run Testing
|
||
```bash
|
||
# Non-interactive (set TEST_USER_ID and TEST_PROFILE_PASSWORD in .env)
|
||
python test_buy.py target https://www.target.com/p/...
|
||
|
||
# Interactive (prompts for user selection and profile password)
|
||
python test_buy.py target https://www.target.com/p/...
|
||
```
|
||
|
||
### Enabling Real Purchases
|
||
In `config.py`:
|
||
```python
|
||
AUTO_BUY_ENABLED = True
|
||
AUTO_BUY_DRY_RUN = False # default True — must explicitly disable
|
||
AUTO_BUY_MAX_PRICE = 60.00 # price ceiling per item
|
||
```
|