yujunjun/firecrawl
1
0
Fork 0
mirror of https://github.com/mendableai/firecrawl.git synced 2025-08-15 20:27:28 +00:00
Code Issues Packages Projects Releases Wiki Activity
2,778 Commits 712 Branches 22 Tags
Commit Graph

474 Commits

Author SHA1 Message Date
Gergő Móricz
b005450a34
port most of cheerio stuff to rust (#1089) 2025-01-24 22:04:54 +01:00
Móricz Gergő
6557365149 feat(sitemap): change sitemap logging 2025-01-23 12:06:50 +01:00
Móricz Gergő
434a435a4b fix(sitemap): increase limit to 20 2025-01-23 11:29:49 +01:00
Móricz Gergő
1e28ba291e fix(sitemap): increase limit 2025-01-23 09:21:38 +01:00
Móricz Gergő
bee2b2873e fix(sitemap): better ordering 2025-01-23 08:58:18 +01:00
Móricz Gergő
3761eb17a7 feat(sitemap): reenable fallback to tlsclient 2025-01-23 08:43:13 +01:00
Móricz Gergő
72198123cb fix(crawler): move sitemap deduplication to deeper in the process 2025-01-23 08:10:46 +01:00
Móricz Gergő
aa2c369060 feat(sitemap): propagate crawlid 2025-01-23 07:19:00 +01:00
Móricz Gergő
a922aac805 fix(crawler): dumb sitemap limit 2025-01-23 07:10:07 +01:00
Móricz Gergő
51a0e233e3 fix(sitemap): temporarily disable tlsclient 2025-01-23 06:56:15 +01:00
Nicolas
498558d358 Nick: formatting done 2025-01-22 18:47:44 -03:00
Gergő Móricz
ed929221ab feat(sitemap): switch around engine order 2025-01-22 19:10:27 +01:00
Nicolas
3604f2a3ae Nick: misc improvements 2025-01-21 16:57:45 -03:00
Nicolas
c7b219169b Nick: fixed crawl maps index dedup 2025-01-21 16:22:27 -03:00
Nicolas
92b8d97be3 Nick: 2025-01-19 13:09:29 -03:00
Nicolas
513f61a2d1 Nick: map improvements 2025-01-19 12:33:44 -03:00
Nicolas
2e5785d8d9 Nick: fetch sitemap timeout param 2025-01-19 11:40:13 -03:00
Gergő Móricz
e6531278f6 feat(v1): crawl/batch scrape errors route 2025-01-17 17:12:04 +01:00
Gergő Móricz
5992c57158 fix(crawler): bad urls from sitemap 2025-01-17 17:07:44 +01:00
Gergő Móricz
23bb172592 fix(crawler): recognize sitemaps in robots.txt 2025-01-17 15:45:52 +01:00
Gergő Móricz
ec039dcb8f fix(blocklist): unblock 2025-01-15 18:54:26 +01:00
Nicolas
5e5b5ee0e2
(feat/extract) New re-ranker + multi entity extraction (#1061) 
* agent that decides if splits schema or not

* split and merge properties done

* wip

* wip

* changes

* ch

* array merge working!

* comment

* wip

* dereferentiate schema

* dereference schemas

* Nick: new re-ranker

* Create llm-links.txt

* Nick: format

* Update extraction-service.ts

* wip: cooking schema mix and spread functions

* wip

* wip getting there!!!

* nick:

* moved functions to helpers

* nick:

* cant reproduce the error anymore

* error handling all scrapes failed

* fix

* Nick: added the sitemap index

* Update sitemap-index.ts

* Update map.ts

* deduplicate and merge arrays

* added error handler for object transformations

* Update url-processor.ts

* Nick:

* Nick: fixes

* Nick: big improvements to rerank of multi-entity

* Nick: working

* Update reranker.ts

* fixed transformations for nested objs

* fix merge nulls

* Nick: fixed error piping

* Update queue-worker.ts

* Update extraction-service.ts

* Nick: format

* Update queue-worker.ts

* Update pnpm-lock.yaml

* Update queue-worker.ts

---------

Co-authored-by: rafaelmmiller <150964962+rafaelsideguide@users.noreply.github.com>
Co-authored-by: Thomas Kosmas <thomas510111@gmail.com>
 2025-01-13 22:30:15 -03:00
Nicolas
f4d10c5031 Nick: formatting fixes 2025-01-10 18:35:10 -03:00
Móricz Gergő
363021ea78 feat(crawl): ensure url trimming 2025-01-08 12:35:42 +01:00
Gergő Móricz
ccfada98ca various queue fixes 2025-01-07 19:15:23 +01:00
Móricz Gergő
35d1d85978 fix(crawler): also take the hostname of the base url when determining isInternalLink 2025-01-07 09:29:58 +01:00
Gergő Móricz
71a8f7452c fix(WebScraper/sitemap): await urlsHandler to fix race condition 2024-12-30 16:09:22 +01:00
Nicolas
8ae34a0d31 Nick: rm .xml from isFile 2024-12-30 11:57:01 -03:00
Gergő Móricz
0421f81020
Sitemap fixes (#1010) 
* sitemap fixes iter 1

* feat(sitemap): dedupe improvements

---------

Co-authored-by: Nicolas <nicolascamara29@gmail.com>
 2024-12-27 19:59:26 +01:00
Nicolas
4451c4f671 Nick: 2024-12-26 13:51:20 -03:00
Gergő Móricz
e97ee4a4be fix(WebScraper/tryGetSitemap): deduplicate sitemap links list 2024-12-15 22:33:36 +01:00
Nicolas
588f747ee8 chore: formatting 2024-12-15 02:54:49 -03:00
Nicolas
4987880b32 Nick: random fixes 2024-12-15 02:52:06 -03:00
Eric Ciarla
a2998d4499 Hash Urls 2024-12-12 16:10:10 -05:00
Nicolas
8a1c404918 Nick: revert trailing comma 2024-12-11 19:51:08 -03:00
Nicolas
00335e2ba9 Nick: fixed prettier 2024-12-11 19:46:11 -03:00
Gergő Móricz
f877fbfb8f fix(WebCrawler/isFile): add .wav 2024-12-10 23:24:53 +01:00
Gergő Móricz
845c2744a9 feat(app): add extra crawl logging (app-side only for now) 2024-12-05 20:50:36 +01:00
rafaelmmiller
5ddb7eb922 parameter 2024-11-29 16:44:54 -03:00
rafaelmmiller
b69c6f9f95 added library.tiktok to allowedKeywords 2024-11-27 10:10:43 -03:00
Gergő Móricz
b1eaecfdb0 fix 2 2024-11-20 20:19:16 +01:00
Gergő Móricz
e2ddc6c65c fix handling of badly formatted URLs 2024-11-20 20:18:40 +01:00
Gergő Móricz
79a75e088a feat(crawl): allowSubdomain 2024-11-19 18:38:59 +01:00
Gergő Móricz
350d00d27a fix(crawler): treat XML files as sitemaps (temporarily) 2024-11-15 20:09:20 +01:00
Nicolas
7f084c6c43 Nick: 2024-11-14 17:44:32 -05:00
Gergő Móricz
86a78a03cb fix(sitemap): scrape with tlsclient 2024-11-14 18:56:32 +01:00
Gergő Móricz
0310cd2afa fix(crawl): redirect rebase 2024-11-13 21:38:44 +01:00
Gergő Móricz
fbabc779f5
fix(crawler): relative URL handling on non-start pages (#893) 
* fix(crawler): relative URL handling on non-start pages

* fix(crawl): further fixing
 2024-11-12 18:20:53 +01:00
Gergő Móricz
68c9615f2d fix(crawl/maxDepth): fix maxDepth behaviour 2024-11-11 22:02:17 +01:00
Gergő Móricz
8d467c8ca7
WebScraper refactor into scrapeURL (#714) 
* feat: use strictNullChecking

* feat: switch logger to Winston

* feat(scrapeURL): first batch

* fix(scrapeURL): error swallow

* fix(scrapeURL): add timeout to EngineResultsTracker

* fix(scrapeURL): report unexpected error to sentry

* chore: remove unused modules

* feat(transfomers/coerce): warn when a format's response is missing

* feat(scrapeURL): feature flag priorities, engine quality sorting, PDF and DOCX support

* (add note)

* feat(scrapeURL): wip readme

* feat(scrapeURL): LLM extract

* feat(scrapeURL): better warnings

* fix(scrapeURL/engines/fire-engine;playwright): fix screenshot

* feat(scrapeURL): add forceEngine internal option

* feat(scrapeURL/engines): scrapingbee

* feat(scrapeURL/transformars): uploadScreenshot

* feat(scrapeURL): more intense tests

* bunch of stuff

* get rid of WebScraper (mostly)

* adapt batch scrape

* add staging deploy workflow

* fix yaml

* fix logger issues

* fix v1 test schema

* feat(scrapeURL/fire-engine/chrome-cdp): remove wait inserts on actions

* scrapeURL: v0 backwards compat

* logger fixes

* feat(scrapeurl): v0 returnOnlyUrls support

* fix(scrapeURL/v0): URL leniency

* fix(batch-scrape): ts non-nullable

* fix(scrapeURL/fire-engine/chromecdp): fix wait action

* fix(logger): remove error debug key

* feat(requests.http): use dotenv expression

* fix(scrapeURL/extractMetadata): extract custom metadata

* fix crawl option conversion

* feat(scrapeURL): Add retry logic to robustFetch

* fix(scrapeURL): crawl stuff

* fix(scrapeURL): LLM extract

* fix(scrapeURL/v0): search fix

* fix(tests/v0): grant larger response size to v0 crawl status

* feat(scrapeURL): basic fetch engine

* feat(scrapeURL): playwright engine

* feat(scrapeURL): add url-specific parameters

* Update readme and examples

* added e2e tests for most parameters. Still a few actions, location and iframes to be done.

* fixed type

* Nick:

* Update scrape.ts

* Update index.ts

* added actions and base64 check

* Nick: skipTls feature flag?

* 403

* todo

* todo

* fixes

* yeet headers from url specific params

* add warning when final engine has feature deficit

* expose engine results tracker for ScrapeEvents implementation

* ingest scrape events

* fixed some tests

* comment

* Update index.test.ts

* fixed rawHtml

* Update index.test.ts

* update comments

* move geolocation to global f-e option, fix removeBase64Images

* Nick:

* trim url-specific params

* Update index.ts

---------

Co-authored-by: Eric Ciarla <ericciarla@yahoo.com>
Co-authored-by: rafaelmmiller <8574157+rafaelmmiller@users.noreply.github.com>
Co-authored-by: Nicolas <nicolascamara29@gmail.com>
 2024-11-07 20:57:33 +01:00