Gergő Móricz
58564bceb8
Revert "temp: debugging"
...
wrong branch
This reverts commit 97e6c28cf96fe837d3934b126e3b5e86fa10df2b.
2025-02-18 11:28:35 +01:00
Gergő Móricz
97e6c28cf9
temp: debugging
2025-02-18 11:28:12 +01:00
Gergő Móricz
5c4021e8cf
fix(scrapeURL/fire-engine): screenshot broken hotfix
2025-02-17 18:59:16 +01:00
Nicolas
b0534d0767
Merge branch 'main' of https://github.com/mendableai/firecrawl
2025-02-17 12:42:59 -03:00
Nicolas
03da3c36a6
Update auth.ts
2025-02-17 12:42:57 -03:00
Gergő Móricz
445b906af1
fix(scrapeURL/fire-engine): perform format screenshot after specified actions ( #1192 )
2025-02-17 12:35:55 -03:00
Gergő Móricz
7ecbff3b20
fix(map): do not remove query parameters from results (FIR-1015) ( #1191 )
...
* fix(map): do not remove query parameters from results
* feat(map): add tests for query parameter handling
2025-02-17 12:34:58 -03:00
Gergő Móricz
1491b5b141
fix(scrapeURL/sb): enforce timeout (FIR-980) ( #1183 )
...
* fix(scrapeURL/scrapingbee): enforce timeout
* fix(scrapeURL/sb): types
* fix the test
* fixup: remove nix files
2025-02-16 15:55:03 +01:00
Gergő Móricz
fd8b38902a
fix(crawl-status): consider concurrency limited jobs as prioritized ( #1184 )
2025-02-16 15:52:17 +01:00
Nicolas
7ac2b99210
Update auth.ts
2025-02-14 11:12:02 -03:00
Nicolas
ce9d3d32f6
Nick: adjust for old compatibility s2
2025-02-14 11:09:09 -03:00
Gergő Móricz
a4cf8f439c
Revert "(feat/extract) Multi-entity prompt improvements ( #1181 )"
...
This reverts commit 584221a106216dfcc25de14c16720224cad237e1.
2025-02-14 11:19:16 +01:00
Gergő Móricz
d9c99e5892
fix(v1/batch/scrape): use scrape rate limit ( #1182 )
2025-02-13 17:32:58 -03:00
Gergő Móricz
73e7884df4
fix(queue-worker/crawl): only report successful page count in num_docs ( #1179 )
2025-02-13 13:14:24 -03:00
Nicolas
584221a106
(feat/extract) Multi-entity prompt improvements ( #1181 )
...
* Update build-prompts.ts
* Update build-prompts.ts
* Update build-prompts.ts
* Update build-prompts.ts
2025-02-13 13:11:35 -03:00
Nicolas
73ecaf74c2
feat(v1/extract) Show sources out of __experimental ( #1180 )
...
* Nick:
* Nick:
* Nick:
2025-02-13 12:41:45 -03:00
Gergő Móricz
1d9a0b9653
feat(v1/checkCredits): say "tokens" instead of "credits" if out of tokens ( #1178 )
...
* feat(v1/checkCredits): say "tokens" instead of "credits" if out of tokens
* feat(v1/checkCredits): better instructions for upgrade
2025-02-13 12:14:13 -03:00
Carter LaSalle
78094e2fe4
fix(html-transformer): Update free_string function parameter type ( #1163 )
...
Correct the parameter type to use libc::c_char for better type consistency
2025-02-11 15:11:29 +01:00
Nicolas
6b313ad0b0
Update types.ts
2025-02-10 17:42:12 -03:00
Gergő Móricz
bc787de62a
fix(blocklist): behavioural fixes
2025-02-10 19:10:20 +01:00
Gergő Móricz
290dd033d3
Update apps/api/src/controllers/v1/types.ts
2025-02-10 09:31:23 +01:00
Ademílson F. Tonato
843cec971d
[FIR-796] feat(api/types): Add action and wait time validation for scrape requests
...
- Implement max actions limit (15 actions)
- Add total wait time calculation for actions
- Increase max waitFor time to 60 seconds
- Refactor scrape options schema to include validation
2025-02-07 21:16:38 +00:00
Rafael Miller
ac5c88bffb
added scrapeOptions to extract ( #1133 )
2025-02-07 13:38:08 -03:00
Gergő Móricz
892f3a41f3
fix(scrape): allow getting valid JSON via rawHtml (FIR-852) ( #1138 )
...
* fix(scrape): allow getting valid JSON via rawHtml
* fix(scrape/test):
2025-02-06 18:35:28 +01:00
Nicolas
401c18761a
Merge branch 'main' of https://github.com/mendableai/firecrawl
2025-02-06 12:31:31 -03:00
Nicolas
394a31064d
Update types.ts
2025-02-06 12:31:19 -03:00
Ademílson Tonato
996352c24c
Merge pull request #1117 from mendableai/feat/blocked-domains
...
feat(blocklist): Improve URL blocking with tldts parsing
2025-02-06 15:23:58 +00:00
Gergő Móricz
34fe360dc1
fix(crawl-redis/generateURLPermutations): dedupe index.html/index.php/slash/bare URL ends (FIR-827) ( #1134 )
...
* fix(crawl-redis/generateURLPermutations): dedupe index.html/index.php/slash/bare URL ends
* fix(crawl-redis/tests): adjust tests
2025-02-05 15:44:56 +01:00
Nick Roth
aa1e820d52
Fix corepack and self hosting setup ( #1131 )
...
* Update Dockerfile
Config corepack and pnpm
* Update Dockerfile
Try using this approach: https://github.com/pnpm/pnpm/issues/9029#issuecomment-2631400936
* Add ulimits command to docker compose
* Remove ulimit from entrypoint
* Add ulimit back to docker-entrypoint.sh
This implements the proposed approach to not fail when the ulimit command can't run due to the environment
* Resolve signature mismatch
Resolves issue encountered when using docker compose up on a clean environment on an intel mac. This just adds the clean command to avoid leveraging any cached data before updating apt-get and installing ca-certificates
---------
Co-authored-by: Nick Roth <nroth@dealnews.com>
Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
2025-02-05 10:19:11 +01:00
Thomas Kosmas
73d19aa01b
fix(dockerfile): Temporarily resolve corepack issue
2025-02-05 02:42:16 +02:00
Gergő Móricz
1e7d42e8af
fix(concurrency-limit): allow for big crawls
2025-02-04 19:26:30 +01:00
Ademílson F. Tonato
bcd74498e3
test(blocklist): Enhance URL blocking test coverage with decrypted domain validation
2025-01-31 18:23:38 +00:00
Rafael Miller
8d7e8c4f50
added cached scrapes to extract ( #1107 )
...
* added cached scrapes to extract
* dont save if exists
* no duplicates
* experimental tag
* Update requests.http
---------
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
2025-01-31 13:58:52 -03:00
Ademílson F. Tonato
24e8aaf6b5
feat(blocklist): Improve URL blocking with tldts parsing
2025-01-31 13:14:48 +00:00
Móricz Gergő
8a8d7d645f
fix(concurrency): proper job timeouting
2025-01-31 11:22:10 +01:00
Nicolas
e026917698
Nick: disable auto rech for extract requests
2025-01-31 01:10:59 -03:00
Nicolas
6bfd24d903
Nick: waitFor fixes
2025-01-30 23:23:03 -03:00
Nicolas
5819f60688
Update types.ts
2025-01-30 23:21:47 -03:00
Nicolas
26c61b96f6
Update queue-jobs.ts
2025-01-30 22:58:08 -03:00
Rafael Miller
948f7866df
Added instructions for empty string to extract prompts ( #1114 )
...
* added instructions to extract prompts
* getting back the null for not strings
2025-01-30 16:11:55 -03:00
Gergő Móricz
a6722d4a95
feat(webhook): add event picker
2025-01-30 18:43:37 +01:00
Nicolas
7c0b3ad098
(feat/conc) Move fully to a concurrency limit system ( #1045 )
...
* Nick: conc limits init
* Nick: test suite plans
* fix(email_notification): move expiry check to redis
* fix(email_notification): add db check in case redis resets
* Update rate-limiter.ts
* Update queue-jobs.ts
* Create concurrency-limit.test.ts
* Update concurrency-limit.test.ts
* Create queue-concurrency-integration.test.ts
* Update queue-concurrency-integration.test.ts
---------
Co-authored-by: Móricz Gergő <mo.geryy@gmail.com>
2025-01-30 13:47:29 -03:00
Nicolas
e2c3932dd2
Update auto_charge.ts
2025-01-30 12:47:08 -03:00
Nicolas
a585340740
Merge branch 'main' of https://github.com/mendableai/firecrawl
2025-01-30 12:24:38 -03:00
Nicolas
ec2c0f671c
Added more safety guards to auto-rech
2025-01-30 12:24:37 -03:00
Gergő Móricz
71878cf4d9
fix(cc): hotfix
2025-01-30 16:07:39 +01:00
Móricz Gergő
a7eb2f7c6a
fix(crawler/rust): dedupe
2025-01-30 08:16:51 +01:00
Nicolas
c88176a596
Update blocklist.ts
2025-01-29 22:23:50 -03:00
Gergő Móricz
86f05a07ba
feat(github/ci): connect to tailscale (FIR-748) ( #1112 )
...
* feat(github/ci): connect to tailscale
* fix(tests/snips): adjust timeouts
2025-01-29 16:59:43 +01:00
Gergő Móricz
aaa16ee388
feat(v0): store v0 users (team ID) in Redis for collection ( #1111 )
2025-01-29 12:43:55 -03:00
