mirror of
https://github.com/mendableai/firecrawl.git
synced 2025-12-29 16:16:46 +00:00
411ecdf04b
* feat: Add crawl delay functionality with per-crawl concurrency limiting (FIR-249) Co-Authored-By: mogery@sideguide.dev <mogery@sideguide.dev> * fix: Skip crawl delay in test environment to fix CI tests Co-Authored-By: mogery@sideguide.dev <mogery@sideguide.dev> * refactor: Use crawlerOptions.delay instead of separate fields Co-Authored-By: mogery@sideguide.dev <mogery@sideguide.dev> * refactor: Rename crawlDelay to delay in type definitions for uniformity Co-Authored-By: mogery@sideguide.dev <mogery@sideguide.dev> * refactor: Fix crawl concurrency implementation based on PR feedback Co-Authored-By: mogery@sideguide.dev <mogery@sideguide.dev> * refactor: Simplify if/else structure in queue-jobs.ts based on PR feedback Co-Authored-By: mogery@sideguide.dev <mogery@sideguide.dev> * human fixes * test: Add tests for crawl delay functionality Co-Authored-By: mogery@sideguide.dev <mogery@sideguide.dev> * test: Move crawl delay tests to existing crawl.test.ts file Co-Authored-By: mogery@sideguide.dev <mogery@sideguide.dev> * fix: Ensure sitemapped URLs are added to crawl concurrency queue and update crawl status endpoint Co-Authored-By: mogery@sideguide.dev <mogery@sideguide.dev> * dbg * fix: Ensure jobs with crawl delay are properly added to BullMQ Co-Authored-By: mogery@sideguide.dev <mogery@sideguide.dev> * fix: Remove duplicate job addition to BullMQ for jobs with crawl delay Co-Authored-By: mogery@sideguide.dev <mogery@sideguide.dev> * fixes * warning for devin * test: Simplify crawl delay test as requested in PR feedback Co-Authored-By: mogery@sideguide.dev <mogery@sideguide.dev> * bump delay test timeout * fix operation order * bump further??? * fix: broken on self-host * Update apps/api/src/services/queue-jobs.ts Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix: import --------- Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> Co-authored-by: mogery@sideguide.dev <mogery@sideguide.dev> Co-authored-by: Gergő Móricz <mo.geryy@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>