mmcghen
8d382e723f
- Added Walmart scraper to scrape product data from Walmart.com, including category pages and product details. - Introduced a stealth browser module to handle bot protection and improve scraping reliability. - Created a SQLite database for tracking product history, price changes, stock events, and user favorites. - Developed a Discord bot for user interaction, allowing location setting and stock checking at local stores. - Implemented a favorites system to manage priority products and categories with custom notification settings. - Added news aggregation module to fetch and analyze Pokemon TCG news from various sources. - Created tools for API discovery and monitoring, including a backend monitor for detecting new products. - Added unit tests for database operations, product filtering, and API endpoints to ensure functionality. - Enhanced existing modules with improved error handling and logging for better maintainability.
138 lines
4.1 KiB
Markdown
138 lines
4.1 KiB
Markdown
# Pokemon Stock Monitor - Project Guide
|
|
|
|
## Overview
|
|
A Pokemon TCG stock monitoring system that tracks retail sites (Target, PokemonCenter, GameStop, Best Buy, Walmart) for restocks and new drops, sending Discord notifications when products become available.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
User -> Chrome Extension (PokemonCenter API interception)
|
|
-> Python Monitor (main.py) -> Scrapers -> ProductTracker -> Discord Notifications
|
|
-> Flask Dashboard (localhost:5000) -> REST API
|
|
```
|
|
|
|
## Directory Structure
|
|
|
|
### Core Application
|
|
- `main.py` - Entry point, scheduling loop, orchestrates scraping
|
|
- `config.py` - All configuration (intervals, URLs, Discord webhook, browser settings)
|
|
|
|
### Source Code (`src/`)
|
|
- `browser.py` - Playwright/Chrome browser management
|
|
- `product_tracker.py` - Detects new products and restocks, manages state
|
|
- `discord_notifier.py` - Sends Discord webhook notifications
|
|
- `discord_bot.py` - Interactive Discord bot commands
|
|
- `database.py` - SQLite database for historical tracking
|
|
- `favorites.py` - User favorites/watchlist management
|
|
- `scraper_state.py` - Persists scraper state between runs
|
|
- `store_locator.py` - Target store location lookup
|
|
|
|
### Scrapers (`scrapers/`)
|
|
- `base.py` - `Product` dataclass, `BaseScraper` abstract class
|
|
- `pokemoncenter.py` - PokemonCenter scraper (needs stealth browser)
|
|
- `target.py` - Target.com scraper
|
|
- `gamestop.py` - GameStop.com scraper (light bot protection)
|
|
- `bestbuy.py` - BestBuy.com scraper (moderate bot protection)
|
|
- `walmart.py` - Walmart.com scraper (aggressive bot protection, may need stealth)
|
|
|
|
### Chrome Extension (`chrome-extension/`)
|
|
- `manifest.json` - Extension config (MV3)
|
|
- `background.js` - Service worker, manages alarms and notifications
|
|
- `content.js` - Injected into PokemonCenter pages
|
|
- `api-interceptor.js` - Intercepts PokemonCenter API responses
|
|
- `popup.html/js` - Extension popup UI
|
|
|
|
### Dashboard (`dashboard/`)
|
|
- `app.py` - Flask app, routes
|
|
- `api.py` - REST API endpoints (`/api/*`)
|
|
- `templates/index.html` - Main template
|
|
- `static/app.js` - Frontend JavaScript
|
|
|
|
### Data (`data/`)
|
|
- `products.json` - Tracked products state
|
|
- `scraper_state.json` - Persisted scraper state
|
|
- `known_skus.json` - Known product SKUs
|
|
- `proxies.json` - Proxy configuration
|
|
- `user_locations.json` - User store locations
|
|
|
|
### Tools (`tools/`)
|
|
Development and debugging utilities:
|
|
- `api_monitor.py`, `backend_monitor.py` - API monitoring
|
|
- `stealth_browser.py` - Anti-detection browser
|
|
- `capture_api_calls.py` - HAR capture for debugging
|
|
|
|
## Key Classes
|
|
|
|
### `Product` (scrapers/base.py)
|
|
```python
|
|
@dataclass
|
|
class Product:
|
|
name: str
|
|
url: str
|
|
price: Optional[str]
|
|
in_stock: bool
|
|
image_url: Optional[str]
|
|
site: str
|
|
product_id: Optional[str]
|
|
```
|
|
|
|
### `ProductTracker` (src/product_tracker.py)
|
|
- `process_products(products)` -> `(new_products, restocked_products)`
|
|
- `get_stats()` -> dashboard statistics
|
|
|
|
### `BaseScraper` (scrapers/base.py)
|
|
- `scrape_category_page(url)` -> `List[Product]`
|
|
- `check_product_stock(url)` -> `(in_stock, price)`
|
|
|
|
## Common Tasks
|
|
|
|
### Add a new retailer scraper
|
|
1. Create `scrapers/newsite.py` extending `BaseScraper`
|
|
2. Implement `scrape_category_page()` and `check_product_stock()`
|
|
3. Add to `scrapers/__init__.py`
|
|
4. Add config in `config.py` (URLs, enable flag)
|
|
5. Add check function in `main.py`
|
|
|
|
### Modify Discord notifications
|
|
- Webhook format: `src/discord_notifier.py`
|
|
- Bot commands: `src/discord_bot.py`
|
|
|
|
### Change check intervals or URLs
|
|
- Edit `config.py`
|
|
|
|
### Dashboard changes
|
|
- Routes: `dashboard/app.py`
|
|
- API: `dashboard/api.py`
|
|
- Frontend: `dashboard/static/app.js`
|
|
|
|
### Chrome extension changes
|
|
- Popup UI: `chrome-extension/popup.html`, `popup.js`
|
|
- API interception: `chrome-extension/api-interceptor.js`
|
|
- Background logic: `chrome-extension/background.js`
|
|
|
|
## Conventions
|
|
- Python 3.13+
|
|
- Logging via `logging` module (configured in main.py)
|
|
- Dataclasses for data structures
|
|
- Flask for web/API
|
|
- Playwright for browser automation
|
|
- SQLite for persistence (`stats.db`)
|
|
|
|
## Running
|
|
|
|
```bash
|
|
# Main monitor
|
|
python main.py
|
|
|
|
# Dashboard only
|
|
python dashboard/app.py
|
|
|
|
# Discord bot
|
|
python src/discord_bot.py
|
|
```
|
|
|
|
## Testing
|
|
```bash
|
|
python -m pytest tests/
|
|
```
|