yujunjun/firecrawl
1
0
Fork 0
mirror of https://github.com/mendableai/firecrawl.git synced 2025-08-04 23:03:05 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,537 Commits 686 Branches 22 Tags
Commit Graph

2271 Commits

Author SHA1 Message Date
Gergő Móricz
d497284b40 feat(api/queue): auto-remove completed jobs after 25 hours 2024-09-12 19:47:22 +02:00
Nicolas
620b02f9ca Nick: 2024-09-12 12:51:14 -04:00
Nicolas
e1171ade1b Merge branch 'main' into mog/fix-status-job-get 2024-09-12 11:35:36 -04:00
Nicolas
4cd1065ae2 Update rate-limiter.ts 2024-09-11 14:03:34 -04:00
Gergő Móricz
97ffabff3a fix(v1): converting bad docs always gives null 2024-09-10 21:21:20 +02:00
Gergő Móricz
83a165db0f fix(v0/scrape): ensure url is string 2024-09-10 21:18:53 +02:00
Gergő Móricz
ad1a6fbc74 fix(v1/map): handle invalid URLs gracefully 2024-09-10 19:41:01 +02:00
Gergő Móricz
4ebc35c9dd fix(crawl-status): add success: true 2024-09-10 18:59:09 +02:00
Nicolas
a7406031fa Merge branch 'main' of https://github.com/mendableai/firecrawl 2024-09-10 06:53:30 -03:00
Nicolas
a17e1cac92 Rate bump 2024-09-10 06:53:24 -03:00
Gergo Moricz
b4dbf75537 fix(v1): check if url is string in blocklistMiddleware 
Fixes FIRECRAWL-SCRAPER-JS-9Z
 2024-09-10 10:25:14 +02:00
Gergo Moricz
26f2095de6 fix(v1): proper Invalid URL handling 2024-09-10 09:24:23 +02:00
Gergo Moricz
f8fbc71f91 fix(supabase-jobs): do not use RPCs 
RPCs are more failure-prone for this use case than regular queries are.
 2024-09-10 09:20:18 +02:00
Gergo Moricz
a6bcf7b438 fix(v0/crawl-status): don't crash on big crawls when requesting jobs from supabase 2024-09-10 08:53:40 +02:00
Nicolas
17e419a7fb Nick: 2024-09-09 21:06:23 -03:00
Nicolas
ca9a781eb7 Update index.ts 2024-09-09 12:27:55 -03:00
Nicolas
22a5e85899 Update index.ts 2024-09-09 12:26:55 -03:00
Nicolas
60a15d00eb Update types.ts 2024-09-08 16:39:12 -03:00
Nicolas
fbdfa1256b Update credit_billing.ts 2024-09-08 13:07:10 -03:00
Nicolas
48c665519e Update credit_billing.ts 2024-09-07 13:42:45 -03:00
Nicolas
32097fa29d Merge branch 'main' of https://github.com/mendableai/firecrawl 2024-09-07 13:12:52 -03:00
Nicolas
5758af3291 Update website_params.ts 2024-09-07 13:12:46 -03:00
Nicolas
79870e7305 Update excludeTags.ts 2024-09-06 20:15:26 -03:00
Nicolas
a0f9ab2be7 Update map.ts 2024-09-06 20:14:47 -03:00
Nicolas
f5b84e15e1 Update sitemap.ts 2024-09-05 17:52:27 -03:00
rafaelsideguide
82d6bf4ec8 feat(go-parser): singleton 2024-09-05 16:14:21 -03:00
Nicolas
cb630bfc34 Update fireEngine.ts 2024-09-05 14:24:10 -03:00
rafaelsideguide
8c1097e9e1 fix: pageOptions 2024-09-05 14:16:31 -03:00
rafaelsideguide
b301ffc922 added missing variables 2024-09-05 13:57:26 -03:00
Nicolas
c6f1d80992 Update crawl.ts 2024-09-05 13:03:43 -03:00
Nicolas
eb03a81152 Update crawl-status.ts 2024-09-05 12:55:04 -03:00
Sebastjan Prachovskij
503e83e83e
Add SearchApi to search 
Add support for engines, improve status code error

Remove changes in package, add engine to env params

Improve description in env example

Remove unnecessary empty line

Improve text
 2024-09-05 18:36:59 +03:00
Nicolas
34adf43200 Merge branch 'main' into feat/go-html-to-md-parser 2024-09-05 12:13:14 -03:00
Nicolas
82cb80c817 Update map.ts 2024-09-04 23:46:18 -03:00
Nicolas
a0113dac37 Update credit_billing.ts 2024-09-04 16:54:20 -03:00
Nicolas
b7b99b53da Merge branch 'main' into nsc/check-credits-optimization 2024-09-04 16:39:18 -03:00
rafaelsideguide
85b824e122 test: what about false false? 2024-09-04 16:35:32 -03:00
rafaelsideguide
78edf13ec6 test: usedbauth envs wth 2024-09-04 16:31:42 -03:00
rafaelsideguide
cb8571abad fix: enforced dotenv config 2024-09-04 15:57:57 -03:00
Nicolas
5ecb243693 Nick: 2024-09-04 15:19:45 -03:00
rafaelsideguide
3f462eabe9 fix(cicd): 2024-09-04 12:27:46 -03:00
rafaelsideguide
364ba9f901 fix(cicd): mendable->firecrawl and waitfor 2024-09-04 12:05:02 -03:00
rafaelsideguide
ad950a6c9d fixed controller res and tests 2024-09-04 11:29:32 -03:00
rafaelsideguide
74ac8915cd details 2024-09-04 09:41:38 -03:00
rafaelsideguide
d836ba6782 added log to check response on cicd 2024-09-04 09:35:56 -03:00
Nicolas
049a11187d Nick: 2024-09-03 21:09:32 -03:00
Nicolas
3072d4a333 Nick: fixed .sort coupons and sentry to withAuth 2024-09-03 21:02:41 -03:00
Nicolas
7561fd279f Nick: debug the billing email system for free credits 2024-09-03 17:08:12 -03:00
rafaelsideguide
c5e1d77a82 added invalid html tests 2024-09-03 15:21:45 -03:00
rafaelsideguide
d60fa6e084 fixed dockerfile and function name. it's working 2024-09-03 14:08:07 -03:00