yujunjun/firecrawl
1
0
Fork 0
mirror of https://github.com/mendableai/firecrawl.git synced 2025-07-19 23:16:51 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,537 Commits 633 Branches 22 Tags
Commit Graph

3537 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Gergő Móricz
97e6c28cf9 temp: debugging 2025-02-18 11:28:12 +01:00
Gergő Móricz
5c4021e8cf fix(scrapeURL/fire-engine): screenshot broken hotfix 2025-02-17 18:59:16 +01:00
Nicolas
b0534d0767 Merge branch 'main' of https://github.com/mendableai/firecrawl 2025-02-17 12:42:59 -03:00
Nicolas
03da3c36a6 Update auth.ts 2025-02-17 12:42:57 -03:00
Gergő Móricz
445b906af1
fix(scrapeURL/fire-engine): perform format screenshot after specified actions (#1192) 2025-02-17 12:35:55 -03:00
Gergő Móricz
7ecbff3b20
fix(map): do not remove query parameters from results (FIR-1015) (#1191) 
* fix(map): do not remove query parameters from results

* feat(map): add tests for query parameter handling
 2025-02-17 12:34:58 -03:00
Eric Ciarla
5cb2918368 update name of example 2025-02-16 18:46:10 -05:00
Gergő Móricz
1491b5b141
fix(scrapeURL/sb): enforce timeout (FIR-980) (#1183) 
* fix(scrapeURL/scrapingbee): enforce timeout

* fix(scrapeURL/sb): types

* fix the test

* fixup: remove nix files
 2025-02-16 15:55:03 +01:00
Gergő Móricz
fd8b38902a
fix(crawl-status): consider concurrency limited jobs as prioritized (#1184) 2025-02-16 15:52:17 +01:00
Nicolas
7ac2b99210 Update auth.ts v1.4.4 2025-02-14 11:12:02 -03:00
Nicolas
ce9d3d32f6 Nick: adjust for old compatibility s2 2025-02-14 11:09:09 -03:00
Gergő Móricz
a4cf8f439c Revert "(feat/extract) Multi-entity prompt improvements (#1181)" 
This reverts commit 584221a106216dfcc25de14c16720224cad237e1.
 2025-02-14 11:19:16 +01:00
Gergő Móricz
d9c99e5892
fix(v1/batch/scrape): use scrape rate limit (#1182) 2025-02-13 17:32:58 -03:00
dolonfly
254580a284
fix: relative url 2 full url use error base url (#584) 
according to ： https://developer.mozilla.org/en-US/docs/Web/API/URL/URL 

the second  parameter:  A string representing the base URL to use in cases where url is a relative reference

so need use the pageUrl , not url origin

eg: 
when pageUrl = http://example.com/a/b/c/d.html and tag a  href is '../e.html'
then 
correct: new URL('../e.html','http://example.com/a/b/c/d.html')
incorrect: new URL('../e.html','http://example.com')
 2025-02-13 20:38:47 +01:00
Gergő Móricz
73e7884df4
fix(queue-worker/crawl): only report successful page count in num_docs (#1179) 2025-02-13 13:14:24 -03:00
Nicolas
584221a106
(feat/extract) Multi-entity prompt improvements (#1181) 
* Update build-prompts.ts

* Update build-prompts.ts

* Update build-prompts.ts

* Update build-prompts.ts
 2025-02-13 13:11:35 -03:00
Nicolas
73ecaf74c2
feat(v1/extract) Show sources out of __experimental (#1180) 
* Nick:

* Nick:

* Nick:
 2025-02-13 12:41:45 -03:00
Gergő Móricz
1d9a0b9653
feat(v1/checkCredits): say "tokens" instead of "credits" if out of tokens (#1178) 
* feat(v1/checkCredits): say "tokens" instead of "credits" if out of tokens

* feat(v1/checkCredits): better instructions for upgrade
 2025-02-13 12:14:13 -03:00
Eric Ciarla
582bbf8da4
Merge pull request #1175 from mayooear/feat/gemini-sub-links 
fix: update gemini library. extract pdf links from scraped content
 2025-02-12 12:00:00 -05:00
mayo
32f9897670 fix: update gemini library. extract pdf links from markdowncontent 2025-02-12 16:43:59 +00:00
Eric Ciarla
a1b7d6e696
Merge pull request #1173 from mayooear/feat/gemini-sub-links 2025-02-11 22:11:49 -05:00
mayo
cb3bc5e445 Add detection of PDF/image sub-links and extract text via Gemini 2025-02-12 02:52:20 +00:00
Eric Ciarla
c67b052f4a Update gemini-2.0-crawler.py 2025-02-11 13:10:48 -05:00
Carter LaSalle
78094e2fe4
fix(html-transformer): Update free_string function parameter type (#1163) 
Correct the parameter type to use libc::c_char for better type consistency
 2025-02-11 15:11:29 +01:00
Luke Martin
52c0f78d4c
chore: add Serper and Search API envs to docker-compose (#1147) 2025-02-11 11:11:16 -03:00
Eric Ciarla
ef9aad8b8a
Merge pull request #1161 from aparupganguly/feature/gemini-2.0-crawler 
Implemented Gemini 2.0 crawler
 2025-02-11 08:13:06 -05:00
Nicolas
6b313ad0b0
Update types.ts 2025-02-10 17:42:12 -03:00
Gergő Móricz
bc787de62a fix(blocklist): behavioural fixes 2025-02-10 19:10:20 +01:00
Ademílson Tonato
3608256569
Merge pull request #1146 from mendableai/feat/wait-validation 
[FIR-796] feat(api/types): Add action and wait time validation for scrape requests
 2025-02-10 14:45:47 +00:00
Gergő Móricz
290dd033d3
Update apps/api/src/controllers/v1/types.ts 2025-02-10 09:31:23 +01:00
Ademílson F. Tonato
843cec971d
[FIR-796] feat(api/types): Add action and wait time validation for scrape requests 
- Implement max actions limit (15 actions)
- Add total wait time calculation for actions
- Increase max waitFor time to 60 seconds
- Refactor scrape options schema to include validation
 2025-02-07 21:16:38 +00:00
Aparup Ganguly
f33fa5bfc6 Initial commit 2025-02-08 01:24:21 +05:30
Eric Ciarla
2b7b7400f6
Merge pull request #1144 from aparupganguly/feature/o3-mini-job-resource 
Feature/o3 mini job resource extractor
v1.4.3 		2025-02-07 11:39:29 -05:00
Rafael Miller
ac5c88bffb
added scrapeOptions to extract (#1133) 2025-02-07 13:38:08 -03:00
Eric Ciarla
42f4f7ef09
Merge pull request #1139 from aparupganguly/feature/o3-mini-review-summarizer 
Implemented prodcut reviews summarizer using o3 mini
 2025-02-07 11:37:57 -05:00
Aparup Ganguly
46f05a750d Job Resources Analyzer 2025-02-07 20:32:52 +05:30
Gergő Móricz
892f3a41f3
fix(scrape): allow getting valid JSON via rawHtml (FIR-852) (#1138) 
* fix(scrape): allow getting valid JSON via rawHtml

* fix(scrape/test):
 2025-02-06 18:35:28 +01:00
Nicolas
401c18761a Merge branch 'main' of https://github.com/mendableai/firecrawl 2025-02-06 12:31:31 -03:00
Nicolas
394a31064d Update types.ts 2025-02-06 12:31:19 -03:00
Ademílson Tonato
996352c24c
Merge pull request #1117 from mendableai/feat/blocked-domains 
feat(blocklist): Improve URL blocking with tldts parsing
 2025-02-06 15:23:58 +00:00
Nicolas
fe276060cf Update pyproject.toml 2025-02-06 12:19:32 -03:00
Gergő Móricz
844ba39f74 fix(sdks): MIT license 2025-02-06 16:07:50 +01:00
Aparup Ganguly
310ef66506 Implemented prodcut reviews summarizer using o3 mini 2025-02-06 20:13:35 +05:30
Gergő Móricz
34fe360dc1
fix(crawl-redis/generateURLPermutations): dedupe index.html/index.php/slash/bare URL ends (FIR-827) (#1134) 
* fix(crawl-redis/generateURLPermutations): dedupe index.html/index.php/slash/bare URL ends

* fix(crawl-redis/tests): adjust tests
 2025-02-05 15:44:56 +01:00
Nick Roth
aa1e820d52
Fix corepack and self hosting setup (#1131) 
* Update Dockerfile

Config corepack and pnpm

* Update Dockerfile

Try using this approach: https://github.com/pnpm/pnpm/issues/9029#issuecomment-2631400936

* Add ulimits command to docker compose

* Remove ulimit from entrypoint

* Add ulimit back to docker-entrypoint.sh

This implements the proposed approach to not fail when the ulimit command can't run due to the environment

* Resolve signature mismatch

Resolves issue encountered when using docker compose up on a clean environment on an intel mac. This just adds the clean command to avoid leveraging any cached data before updating apt-get and installing ca-certificates

---------

Co-authored-by: Nick Roth <nroth@dealnews.com>
Co-authored-by: Gergő Móricz <mo.geryy@gmail.com>
 2025-02-05 10:19:11 +01:00
Thomas Kosmas
73d19aa01b fix(dockerfile): Temporarily resolve corepack issue 2025-02-05 02:42:16 +02:00
Eric Ciarla
b0f9e73376 Update o3-mini_web_crawler.py 2025-02-04 16:35:30 -05:00
Gergő Móricz
1e7d42e8af fix(concurrency-limit): allow for big crawls 2025-02-04 19:26:30 +01:00
Eric Ciarla
1951b60881
Merge pull request #1130 from aparupganguly/feature/o3-company-researcher 
Updated Model Parameters (o3-mini_company_researcher)
 2025-02-04 09:46:16 -05:00
Aparup Ganguly
7aa2db25a3 Updated Model Parameters 2025-02-04 17:54:44 +05:30