Add deduplication and normalize names/URLs
Add product name cleanup and URL normalization across backend, scrapers, and frontend; introduce product/event deduplication. Key changes: - Clean up SVG/CSS garbage, trailing "X Reviews", mojibake and extra spaces in product names (api, scrapers, JS). - Normalize URLs (lowercase host, strip tracking/query params, trim trailing slashes) in scrapers and Database and use normalized URL for comparisons/storage. - Database: use normalized URLs when creating/getting products, prevent duplicate stock events within 1 hour, adjust recent events query to avoid duplicate products. - Implement deduplication routines in Database: deduplicate_products (by product_id and normalized/clean name) and deduplicate_events (remove duplicate events within same hour). - API: add POST endpoints /products/deduplicate and /events/deduplicate to trigger deduplication and return stats. - Frontend: add UI buttons and handlers to call deduplication endpoints and display results. These changes reduce duplicate product records/events caused by minor URL/name variations and stray CSS/svg artifacts.
This commit is contained in:
@@ -356,6 +356,16 @@
|
||||
<button id="checkBrokenUrlsBtn" class="btn-secondary">Check Broken URLs</button>
|
||||
<button id="cleanupBrokenUrlsBtn" class="btn-danger">Cleanup Broken URLs</button>
|
||||
<p id="brokenUrlsResult" class="help-text"></p>
|
||||
|
||||
<h3>Duplicate Products</h3>
|
||||
<p class="help-text">Remove duplicate products from Target, GameStop, and other stores</p>
|
||||
<button id="deduplicateBtn" class="btn-secondary">Remove Duplicates</button>
|
||||
<p id="deduplicateResult" class="help-text"></p>
|
||||
|
||||
<h3>Duplicate Events</h3>
|
||||
<p class="help-text">Remove duplicate events from the activity feed (same product/type within 1 hour)</p>
|
||||
<button id="deduplicateEventsBtn" class="btn-secondary">Clean Up Events</button>
|
||||
<p id="deduplicateEventsResult" class="help-text"></p>
|
||||
</div>
|
||||
</section>
|
||||
</main>
|
||||
|
||||
Reference in New Issue
Block a user