yujunjun/firecrawl
1
0
Fork 0
mirror of https://github.com/mendableai/firecrawl.git synced 2025-07-25 18:05:12 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,138 Commits 652 Branches 22 Tags
Commit Graph

1997 Commits

Author SHA1 Message Date
Nicolas
c6cad942ab Nick: errors -> warn 2025-03-13 13:07:51 -04:00
Nicolas
7ec278a908 Nick: fixes 2025-03-13 13:05:12 -04:00
Nicolas
134de67a3b
(fix/map) Map failed to filter by path if indexed (#1333) 
* Nick:

* Update map.ts

* Update map.ts
 2025-03-13 12:48:56 -04:00
Gergő Móricz
f87e11712c fix: don't log bull secret 2025-03-13 17:30:37 +01:00
Loris
c53df8650a
Update searxng.ts (#1319) 
remove default categories. otherwise we cannot use engines
 2025-03-13 17:21:44 +01:00
Nicolas
d54af97754 Update rate-limiter.ts 2025-03-12 21:57:50 -04:00
Nicolas
b87b0d9b7f Merge branch 'nsc/concurrent-browsers' 2025-03-12 20:21:26 -04:00
Nicolas
e886189394 Update search.ts 2025-03-12 20:21:19 -04:00
Nicolas
1e6b484add
(feat/pricing) Concurrent Browsers - Improve rate limits (#1331) 
* Update rate-limiter.ts

* Update rate-limiter.ts
 2025-03-12 15:23:09 -04:00
Nicolas
d438b235f6 Update rate-limiter.ts 2025-03-12 15:22:28 -04:00
Gergő Móricz
71b6b83ec2
tally rework api switchover (#1328) 
* tally rework api switchover

* fix and send logs

* temp: force main instance while RPCs propagate

* Revert "temp: force main instance while RPCs propagate"

This reverts commit 4c93379cfa64efd60eb4767dd8eced1bdd302531.
 2025-03-12 20:10:33 +01:00
Nicolas
9edbdc987f Update rate-limiter.ts 2025-03-12 14:17:22 -04:00
Gergő Móricz
7cf2e52fe6
feat(crawl): add maxDiscoveryDepth (#1329) 2025-03-12 18:46:57 +01:00
Gergő Móricz
d855f5a567 fix(test/snips/scrape): allow more time for stealth proxy 2025-03-12 18:22:40 +01:00
Gergő Móricz
337bc261f7 temp: remove flakey ad-block tests 2025-03-12 18:17:52 +01:00
Abdul Rahman ArM
e9804d25f9
fix: resolve circular JSON structure error in search function (#1330) 
Modified error handlers to avoid logging entire response objects that contain circular references
 2025-03-12 18:13:53 +01:00
Nicolas
4cc33e1e8d Nick: added team_id_o 2025-03-10 22:30:26 -04:00
Gergő Móricz
0c457e6f2f adjust acuc balance 2025-03-07 20:34:43 +01:00
rafaelmmiller
3786ad7f5d restart 2025-03-07 16:19:59 -03:00
Nicolas
1ced546bd0 Update rate-limiter.ts 2025-03-06 19:17:02 -03:00
Nicolas
f45b3c0155 Merge branch 'main' into fix/preview_token 2025-03-06 19:02:29 -03:00
Nicolas
b66e0ccb7d Nick: 2025-03-06 18:36:01 -03:00
rafaelmmiller
55a4a3d768 preview-token 2025-03-06 18:16:07 -03:00
Gergő Móricz
39b1390289 fix: crawl 2025-03-06 21:57:42 +01:00
Gergő Móricz
a1e6c13b67 move crawl to read replica 2025-03-06 21:57:42 +01:00
Nicolas
982b3da27d Update auth.ts 2025-03-06 17:53:41 -03:00
Nicolas
5a149a1c30 Revert "fix(auth): always use replica for acuc" 
This reverts commit 57b313608cb6e9f3c123e4754c653fef8dfacbdf.
 2025-03-06 17:53:28 -03:00
Nicolas
783fad90dd Nick: more read replicas 2025-03-06 17:45:51 -03:00
Nicolas
949fb68bdd Merge branch 'main' of https://github.com/mendableai/firecrawl 2025-03-06 17:36:57 -03:00
Nicolas
4c4d51e60b Revert "Update runWebScraper.ts" 
This reverts commit 40eacfacd4c5a782f722fd85e3935f4a1741b27a.
 2025-03-06 17:36:08 -03:00
Gergő Móricz
57b313608c fix(auth): always use replica for acuc 2025-03-06 21:35:30 +01:00
Gergő Móricz
7b05512054 fix(credit_billing): teams check 2025-03-06 21:30:57 +01:00
Nicolas
ae010a76c1 Update blocklist.ts 2025-03-06 17:11:00 -03:00
Gergő Móricz
e6c3f20944 fix(preview): temporarily disable 2025-03-06 21:08:12 +01:00
Nicolas
72d894c2a3 Update rate-limiter.ts 2025-03-06 17:06:42 -03:00
Nicolas
1de5a2c589 Update batch_billing.ts 2025-03-06 16:57:23 -03:00
Nicolas
eba28a4210 Merge branch 'main' of https://github.com/mendableai/firecrawl 2025-03-06 16:41:22 -03:00
Nicolas
40eacfacd4 Update runWebScraper.ts 2025-03-06 16:41:21 -03:00
Gergő Móricz
e1cfe1da48
feat(crawl): includes/excludes fixes (FIR-1300) (#1303) 
* feat(crawl): includes/excludes fixes pt. 1

* fix(snips): billing tests

* drop tha logs

* fix(ci): add replica url

* feat(crawl): drop initial scrape if it's not included

* feat(ci): more verbose logging

* fix crawl path in test

* fix(ci): wait for api

* fix(snips/scrape/ad): test for more pixels

* feat(js-sdk/crawl): add regexOnFullURL
 2025-03-06 17:05:15 +01:00
Gergő Móricz
f8df18ed6a feat(feng-check): run on chrome-cdp 2025-03-05 18:53:10 +01:00
Gergő Móricz
c22c87ab0a fix(crawler): check for more strings 2025-03-05 18:48:05 +01:00
Eric Ciarla
5a1886936c
Truncate llmstxt cache based on maxurls limit & improve maxurls handling (#1285) 
* init

* Update generate-llmstxt-service.ts
 2025-03-03 18:37:33 -03:00
Gergő Móricz
1beadf39f0 fix 2025-03-03 21:06:01 +01:00
Gergő Móricz
6dce8e9ddf asd 2025-03-03 21:05:45 +01:00
Gergő Móricz
8620bf3d5e fix(auth): split load evenly between two instances 2025-03-03 20:53:51 +01:00
Gergő Móricz
67ee2662de feat(auth): force acuc on read replica 2025-03-03 20:36:54 +01:00
Nicolas
64af3ba728 Revert "Update auth.ts" 
This reverts commit fea249c568c4a8d63bcc2e560fd3655e73350be6.
 2025-03-03 16:26:42 -03:00
Nicolas
9eb25786dc Update deep-research-service.ts 2025-03-02 18:16:20 -03:00
Nicolas
30c1e92a2e Update deep-research-service.ts 2025-03-02 17:37:15 -03:00
Nicolas
22d4f0de38
(feat/deep-research) Alpha prep + Improvements (#1284) 
* Update index.ts

* Nick:

* Nick: topic -> query

* Update deep-research.ts

* Nick: bump

* Nick: add onSource callback
 2025-03-02 17:33:21 -03:00