yujunjun/firecrawl
1
0
Fork 0
mirror of https://github.com/mendableai/firecrawl.git synced 2025-12-05 03:21:29 +00:00
Code Issues Packages Projects Releases Wiki Activity
1,340 Commits 986 Branches 30 Tags
Commit Graph

806 Commits

Author SHA1 Message Date
rafaelsideguide
74ea820bc6 fix: url and check for metadata 2024-08-22 18:32:19 -03:00
rafaelsideguide
5f60a55967 workflow and npm now running v1 tests 2024-08-22 15:28:49 -03:00
rafaelsideguide
30e809966f Merge remote-tracking branch 'origin/v1/python-sdk' into v1-webscraper 2024-08-22 15:18:05 -03:00
rafaelsideguide
a37681bdff fix: replace jest, removed map for v0 2024-08-22 15:16:46 -03:00
rafaelsideguide
7473b74021 fix: html and rawlhtmls for pdfs 2024-08-22 15:15:45 -03:00
Nicolas
d2521612b4 Update .gitignore 2024-08-22 14:15:19 -03:00
rafaelsideguide
b1d61d8557 Merge remote-tracking branch 'origin/v1-webscraper' into v1/python-sdk 2024-08-22 13:39:09 -03:00
rafaelsideguide
fe2e8c0b7a includehtml fix 2024-08-21 15:54:00 -03:00
rafaelsideguide
e9d6ca197e tests passing now 2024-08-20 20:00:41 -03:00
Gergő Móricz
5818236659 fix: remove rawHtml properly 2024-08-20 22:51:12 +02:00
Gergő Móricz
1368f9a87f fix: treat existing screenshot as a scraper success condition 2024-08-20 22:24:18 +02:00
Nicolas
0c48c8a436 Nick: billing for map 2024-08-20 16:43:46 -03:00
Nicolas
39388cdc35 Update crawl.ts 2024-08-20 14:41:43 -03:00
Nicolas
674adee144 Merge branch 'v1-webscraper' of https://github.com/mendableai/firecrawl into v1-webscraper 2024-08-20 14:41:05 -03:00
Nicolas
b36faeaf54 Nick: 2024-08-20 14:39:52 -03:00
Gergő Móricz
cf32893c2e add strict enforcement + move crawlerOptions to top-level in /crawl 2024-08-20 19:31:26 +02:00
Gergő Móricz
70d50b3640 fix(queue-worker): move dotenv config up 2024-08-20 19:25:19 +02:00
Nicolas
c5ad4dedeb Update crawl.ts 2024-08-20 14:19:20 -03:00
Nicolas
de0dc20a02 Update credit_billing.ts 2024-08-20 14:18:14 -03:00
Nicolas
e200ec9e12 Nick: 2024-08-20 12:24:14 -03:00
Nicolas
55dad82df1 Nick: fixed map search 2024-08-20 12:17:53 -03:00
Nicolas
27903247b6 Nick: map tests and fixes 2024-08-20 12:04:08 -03:00
Nicolas
3dc298be54 Nick: 2x rate limits for standard and growth for /scrape 2024-08-19 13:52:54 -03:00
rafaelsideguide
72461ce9a6 Update index.test.ts 2024-08-19 13:29:52 -03:00
rafaelsideguide
fd7fdc1d52 added blocklist middleware 2024-08-19 13:28:54 -03:00
Nicolas
ff84f1fe5e Update map.ts 2024-08-16 20:42:36 -04:00
Nicolas
4314313477 Update map.ts 2024-08-16 19:56:18 -04:00
Nicolas
af9a0a6f0b Update map.ts 2024-08-16 19:56:03 -04:00
Nicolas
ba5279eafc Nick: all tests passing 2024-08-16 19:55:44 -04:00
Nicolas
5205c5f005 Update map.ts 2024-08-16 19:37:00 -04:00
Nicolas
0c05d096a9 Merge branch 'v1-webscraper' of https://github.com/mendableai/firecrawl into v1-webscraper 2024-08-16 19:33:58 -04:00
Nicolas
ab48353226 Nick: /map almost good 2024-08-16 19:33:57 -04:00
Gergő Móricz
eb84673b06 feat: crawl status websocket WIP 2024-08-17 01:04:14 +02:00
Gergő Móricz
e2a6ef26d3 mount v1Router under v1 path 2024-08-16 23:48:50 +02:00
Gergő Móricz
4c1b74dab3 fix(map): remove robots.txt 2024-08-16 23:46:10 +02:00
Gergő Móricz
803577eeba feat(crawl): webhook 2024-08-16 23:42:48 +02:00
rafaelsideguide
086ba6280b fixed markdown format 2024-08-16 18:39:13 -03:00
Gergő Móricz
aabfaf0ac5 clean up crawl-status, fix db ddos 2024-08-16 23:29:39 +02:00
rafaelsideguide
e5b807ccc4 Merge branch 'v1-webscraper' of https://github.com/mendableai/firecrawl into v1-webscraper 2024-08-16 17:57:31 -03:00
rafaelsideguide
7a61325500 map + search + scrape markdown bug 2024-08-16 17:57:11 -03:00
Gergő Móricz
5896153d19 fix: crawl status and redis fixes 2024-08-16 22:52:48 +02:00
Gergő Móricz
3fcb21930e remove log 2024-08-16 22:48:23 +02:00
Gergő Móricz
f20328bdbb crawl status and document stuff 2024-08-16 22:48:05 +02:00
Nicolas
0c057bb649 Update index.test.ts 2024-08-16 16:45:10 -04:00
Nicolas
b32464558a Update index.test.ts 2024-08-16 16:41:09 -04:00
Nicolas
5bac7988a6 Update index.test.ts 2024-08-16 16:08:38 -04:00
Nicolas
290c7ee936 Update index.test.ts 2024-08-16 16:06:46 -04:00
Nicolas
23a033fe61 Nick: fixes and more e2e tests 2024-08-16 16:03:35 -04:00
Nicolas
37ae9a9043 Update index.test.ts 2024-08-16 14:17:43 -04:00
Nicolas
200ce8e2ce Merge branch 'v1-webscraper' of https://github.com/mendableai/firecrawl into v1-webscraper 2024-08-16 14:16:35 -04:00