yujunjun/firecrawl
1
0
Fork 0
mirror of https://github.com/mendableai/firecrawl.git synced 2025-10-22 21:44:10 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,801 Commits 871 Branches 28 Tags
Commit Graph

125 Commits

Author SHA1 Message Date
Gergő Móricz
5733b82e9d
fix(scrapeURL/fire-engine): default to separate US-generic proxy list if no location is specified (FIR-728) (#1104) 
* feat(location/country): default to us-generic

* add tests + fix mock
 2025-01-29 08:23:36 +01:00
Móricz Gergő
a2d94b525f feat: rewrite html transformer in rust 2025-01-25 09:41:33 +01:00
Gergő Móricz
b005450a34
port most of cheerio stuff to rust (#1089) 2025-01-24 22:04:54 +01:00
Móricz Gergő
bee2b2873e fix(sitemap): better ordering 2025-01-23 08:58:18 +01:00
Nicolas
498558d358 Nick: formatting done 2025-01-22 18:47:44 -03:00
Nicolas
994e1eb502 Nick: rm logs 2025-01-22 17:27:48 -03:00
Nicolas
56f048aeff Reapply "Nick:" 
This reverts commit 4b4385c520c7223cf79ebba981dded8ffaefde11.
 2025-01-22 17:26:32 -03:00
Nicolas
4b4385c520 Revert "Nick:" 
This reverts commit 6718ce89085339eaaceb1e88a0aa45ecff3216ac.
 2025-01-22 17:26:09 -03:00
Nicolas
6718ce8908 Nick: 2025-01-22 17:25:48 -03:00
Nicolas
04916f17e2 Nick: bug fixes + acuc fixes + cache fixes 2025-01-21 19:17:06 -03:00
Nicolas
1ca50e6e8f Update llmExtract.ts 2025-01-19 22:18:51 -03:00
Nicolas
d786949639 Reapply "Merge pull request #1068 from mendableai/nsc/llm-usage-extract" 
This reverts commit 8b17af40018688c34f95727ceaec289b02ab2023.
 2025-01-19 22:04:12 -03:00
Nicolas
8b17af4001 Revert "Merge pull request #1068 from mendableai/nsc/llm-usage-extract" 
This reverts commit 406f28c04aff2ba3ae65f483627da13f02943cc3, reversing
changes made to 34ad9ec25d73f37deb1e3adec2315a121ec52f0e.
 2025-01-19 22:00:28 -03:00
Nicolas
406f28c04a
Merge pull request #1068 from mendableai/nsc/llm-usage-extract 
(feat/extract) - LLMs usage analysis + billing
 2025-01-19 21:36:33 -03:00
Nicolas
81c347f538 Update llmExtract.ts 2025-01-18 22:49:03 -03:00
Nicolas
34b40f6a23 Nick: 2025-01-18 17:17:42 -03:00
Nicolas
260a726f37 Merge branch 'main' into nsc/llm-usage-extract 2025-01-17 23:02:12 -03:00
Móricz Gergő
faf58dfca7 fix(removeUnwantedElements): post-includeTags excludeTags 
Fixes #700
 2025-01-17 12:41:00 +01:00
Gergő Móricz
6c94db7ed0 fix(html,markdown): always get absolute links 2025-01-16 16:56:13 +01:00
Gergő Móricz
e824303d87 feat(html): always pick largest image from srcset 2025-01-16 16:51:33 +01:00
Nicolas
4db023280d Nick: introduce llm-usage cost analysis 2025-01-15 21:01:29 -03:00
Nicolas
957eea4113 Nick: extract without a schema should work as expected 2025-01-14 11:37:00 -03:00
Nicolas
033e9bbf29 Nick: __experimental_streamSteps 2025-01-14 01:45:50 -03:00
Nicolas
5e5b5ee0e2
(feat/extract) New re-ranker + multi entity extraction (#1061) 
* agent that decides if splits schema or not

* split and merge properties done

* wip

* wip

* changes

* ch

* array merge working!

* comment

* wip

* dereferentiate schema

* dereference schemas

* Nick: new re-ranker

* Create llm-links.txt

* Nick: format

* Update extraction-service.ts

* wip: cooking schema mix and spread functions

* wip

* wip getting there!!!

* nick:

* moved functions to helpers

* nick:

* cant reproduce the error anymore

* error handling all scrapes failed

* fix

* Nick: added the sitemap index

* Update sitemap-index.ts

* Update map.ts

* deduplicate and merge arrays

* added error handler for object transformations

* Update url-processor.ts

* Nick:

* Nick: fixes

* Nick: big improvements to rerank of multi-entity

* Nick: working

* Update reranker.ts

* fixed transformations for nested objs

* fix merge nulls

* Nick: fixed error piping

* Update queue-worker.ts

* Update extraction-service.ts

* Nick: format

* Update queue-worker.ts

* Update pnpm-lock.yaml

* Update queue-worker.ts

---------

Co-authored-by: rafaelmmiller <150964962+rafaelsideguide@users.noreply.github.com>
Co-authored-by: Thomas Kosmas <thomas510111@gmail.com>
 2025-01-13 22:30:15 -03:00
Gergő Móricz
5c62bb1195
feat: new snips test framework (FIR-414) (#1033) 
* feat: new snips test framework

* Update mock.ts

---------

Co-authored-by: Nicolas <nicolascamara29@gmail.com>
 2025-01-13 20:50:47 +01:00
Nicolas
f4d10c5031 Nick: formatting fixes 2025-01-10 18:35:10 -03:00
Gergő Móricz
d1f3b96388 feat: add scrapeId in document.metadata 2025-01-09 20:52:12 +01:00
Gergő Móricz
97bf54214f fix(scrapeURL/loop): re-add is long enough check with lt 0 2025-01-09 18:43:50 +01:00
Móricz Gergő
3c614a2e5c fix(scrapeURL/engines/pdf,docx): support authorization 2025-01-09 10:03:27 +01:00
Móricz Gergő
363021ea78 feat(crawl): ensure url trimming 2025-01-08 12:35:42 +01:00
Móricz Gergő
977a3e13c5 fix(scrapeURL): remove short content check 2025-01-08 11:23:25 +01:00
Nicolas
aef040b41e Nick: from cache fixes 2025-01-03 23:07:15 -03:00
Nicolas
e8a9d8ddcd Merge branch 'main' of https://github.com/mendableai/firecrawl 2025-01-03 22:55:42 -03:00
Nicolas
05e845a971 Update cache.ts 2025-01-03 22:55:38 -03:00
Nicolas
a4f7c38834 Nick: fixed 2025-01-03 22:15:23 -03:00
Nicolas
6b2e1cbb28 Nick: cache /extract scrapes 2025-01-03 21:19:40 -03:00
Nicolas
c9d91af86f Merge branch 'main' into nsc/semantic-index-extract 2025-01-02 15:26:40 -03:00
Nicolas
c3fd13a82b Nick: fixed re-ranker and enabled url cache of 2hrs 2024-12-31 18:06:07 -03:00
Nicolas
07f4b714af Update removeUnwantedElements.ts 2024-12-31 15:23:02 -03:00
Nicolas
e6da214aeb Nick: async background index 2024-12-30 21:42:01 -03:00
Gergő Móricz
4d1f92f4c8 fix(scrapeURL/fetch): block loopback and link-local IPs 2024-12-29 17:35:14 +01:00
Nicolas
e255301005 Update index.ts 2024-12-27 21:31:29 -03:00
Nicolas
1eca61bffb Update index.ts 2024-12-27 20:59:18 -03:00
Nicolas
f9d55efba8 Update index.ts 2024-12-27 20:54:26 -03:00
Nicolas
b8d7f9f257 Nick: we are using runpod 2024-12-27 19:59:05 -03:00
Nicolas
5fcf3fa97e Merge branch 'main' into mog/mineru 2024-12-27 19:53:09 -03:00
Nicolas
05d5f84d87
Merge pull request #1018 from mendableai/feat/add-favicon-metadata 
[FIR-37] feat: extract and return favicon URL during scraping
 2024-12-27 17:44:03 -03:00
Nicolas
eba5fda9a1
Merge pull request #955 from mendableai/rafa/fix-default-on-schema-llm-extract 
fixed optional+default bug on llm schema
 2024-12-27 16:33:04 -03:00
Ademílson F. Tonato
a4cf814f70 feat: return favicon url when scraping 2024-12-27 19:18:53 +00:00
Gergő Móricz
4772951313 feat(scrapeURL/fire-engine): explicitly delete job after scrape 2024-12-27 16:44:41 +01:00