1448 Commits

Author	SHA1	Message	Date
Nicolas	4bb46ed152	Nick: extract prompt fixes and limit the number of urls	2024-12-01 20:29:03 -03:00
rafaelmmiller	5ddb7eb922	parameter	2024-11-29 16:44:54 -03:00
Gergő Móricz	42980c899d	fix(scrapeURL/fire-engine): fast fail on chrome error	2024-11-28 18:41:48 +01:00
Móricz Gergő	60ea97c51c	fix(log_job): infinite loop	2024-11-28 08:49:03 +01:00
rafaelmmiller	943bbae88d	fixed nested data inside extract	2024-11-27 18:29:37 -03:00
Nicolas	53e0cb6b19	Merge branch 'main' of https://github.com/mendableai/firecrawl	2024-11-27 12:47:12 -03:00
Nicolas	02cd5bcfa4	Nick: bumped the status rl	2024-11-27 12:47:11 -03:00
rafaelmmiller	b69c6f9f95	added library.tiktok to allowedKeywords	2024-11-27 10:10:43 -03:00
Nicolas	6c33b978f3	Merge pull request #915 from mendableai/nsc/new-extract ... Extract (beta)	2024-11-26 10:02:09 -08:00
Nicolas	5522d6af7d	Update extract.ts	2024-11-26 15:01:42 -03:00
Gergő Móricz	d3a9d29288	return bug	2024-11-26 18:04:09 +01:00
Gergő Móricz	e217952434	fix(crawl): finish crawl even if last one fails	2024-11-26 16:28:45 +01:00
Gergő Móricz	f395c5b008	fix(crawl): failed behaviour	2024-11-26 16:25:48 +01:00
Nicolas	8a26f08b14	Update extract.ts	2024-11-24 20:37:58 -08:00
Nicolas	2513efc971	Update extract.ts	2024-11-24 20:31:38 -08:00
Nicolas	a18614cd00	Update queue-jobs.ts	2024-11-24 19:48:57 -08:00
Nicolas	18b864eace	Update index.ts	2024-11-24 19:48:13 -08:00
Nicolas	d817aa744f	Update v1.ts	2024-11-24 19:46:31 -08:00
Nicolas	30def84c0a	Nick: scrape timeout + warnings	2024-11-24 19:44:51 -08:00
Nicolas	b693c6c23b	Update extract.ts	2024-11-24 19:36:18 -08:00
Nicolas	95bea6a391	Nick: re-ranker safety + unit tests	2024-11-24 19:34:56 -08:00
rafaelmmiller	24724e958e	added new etier	2024-11-21 13:45:30 -03:00
Nicolas	aa26dbe74e	Nick: map e2e tests	2024-11-20 17:03:04 -08:00
Nicolas	6fbfeafe38	Nick: fixed map settings	2024-11-20 16:51:13 -08:00
Nicolas	aaddbdc1bc	Update map.ts	2024-11-20 16:47:07 -08:00
Nicolas	5f4c8da109	Update pnpm-lock.yaml	2024-11-20 16:44:52 -08:00
Nicolas	42922c68d6	Update package.json	2024-11-20 16:44:40 -08:00
Nicolas	93e106d321	Update v0.ts	2024-11-20 16:43:02 -08:00
Nicolas	3eaa3b38ab	Nick: formatting	2024-11-20 16:42:42 -08:00
Nicolas	c78dae178b	Merge branch 'main' into nsc/new-extract	2024-11-20 16:41:13 -08:00
Nicolas	945183ffbd	Update extract.ts	2024-11-20 16:40:55 -08:00
Nicolas	d196b9d93d	Update extract.ts	2024-11-20 13:16:36 -08:00
Nicolas	9512d81e05	Update extract.ts	2024-11-20 13:15:52 -08:00
Nicolas	3de4997f4d	Loggin num tokens	2024-11-20 13:09:46 -08:00
Nicolas	769f08c10d	Billing and log for extract	2024-11-20 13:08:09 -08:00
Nicolas	0e4e9a3b37	Nick:	2024-11-20 13:01:36 -08:00
Nicolas	09dd5136b7	Update build-document.ts	2024-11-20 12:51:16 -08:00
Nicolas	67a2989874	Nick: fixes	2024-11-20 12:48:10 -08:00
Nicolas	28696da6b2	Nick: gpt-4o	2024-11-20 12:25:50 -08:00
Nicolas	d49f62fb56	Nick: extract fixes	2024-11-20 11:50:14 -08:00
Gergő Móricz	b1eaecfdb0	fix 2	2024-11-20 20:19:16 +01:00
Gergő Móricz	e2ddc6c65c	fix handling of badly formatted URLs	2024-11-20 20:18:40 +01:00
Gergő Móricz	ba6f29cdda	crawl fix, again	2024-11-20 19:55:35 +01:00
Gergő Móricz	b468bb4014	crawl fixes	2024-11-20 19:48:01 +01:00
Nicolas	c9b0a80522	Nick:	2024-11-20 10:23:44 -08:00
Nicolas	103c3f28e6	Update rate-limiter.ts	2024-11-19 17:51:31 -08:00
Gergő Móricz	79a75e088a	feat(crawl): allowSubdomain	2024-11-19 18:38:59 +01:00
rafaelmmiller	2fb8a3c8dc	fix schema	2024-11-19 10:04:42 -03:00
rafaelmmiller	53134b7c85	Rafa: removed throw error and added map to requests	2024-11-19 09:34:52 -03:00
rafaelmmiller	36cf49c959	Merge remote-tracking branch 'origin/main' into nsc/new-extract	2024-11-19 09:34:08 -03:00