Rafael Miller 8d7e8c4f50

added cached scrapes to extract (#1107) ...

* added cached scrapes to extract

* dont save if exists

* no duplicates

* experimental tag

* Update requests.http

---------

Co-authored-by: Nicolas <nicolascamara29@gmail.com>

2025-01-31 13:58:52 -03:00

..

sharedLibs

chore(go-html-to-md): Update html-to-markdown dependency

2025-01-28 18:36:01 +02:00

src

added cached scrapes to extract (#1107)

2025-01-31 13:58:52 -03:00

utils

feat(queue-worker): fix redirect slipping

2024-12-15 20:16:52 +01:00

.dockerignore

…

.env.example

WebScraper refactor into scrapeURL (#714)

2024-11-07 20:57:33 +01:00

.env.local

Add SearchApi to search

2024-09-05 18:36:59 +03:00

.gitattributes

…

.gitignore

Update .gitignore

2024-12-23 18:42:05 -03:00

.prettierrc

Nick: revert trailing comma

2024-12-11 19:51:08 -03:00

docker-entrypoint.sh

Nick:

2025-01-19 13:09:29 -03:00

Dockerfile

port most of cheerio stuff to rust (#1089)

2025-01-24 22:04:54 +01:00

fly.staging.toml

fix(fly): change proxy limits

2024-08-14 18:52:40 +02:00

fly.toml

fix(fly): change proxy limits

2024-08-14 18:52:40 +02:00

jest.config.js

…

jest.setup.js

WebScraper refactor into scrapeURL (#714)

2024-11-07 20:57:33 +01:00

openapi-v0.json

add zod, create middleware, update openapi declaration, add crawl logic

2024-08-15 23:30:33 +02:00

openapi.json

add zod, create middleware, update openapi declaration, add crawl logic

2024-08-15 23:30:33 +02:00

package.json

update bullmq

2025-01-24 18:56:03 +01:00

pnpm-lock.yaml

update bullmq

2025-01-24 18:56:03 +01:00

requests.http

added cached scrapes to extract (#1107)

2025-01-31 13:58:52 -03:00

tsconfig.json

revert to pdf parse

2024-12-17 12:12:22 -05:00

v1-openapi.json

docs: update cancel crawl response

2025-01-24 16:16:17 +00:00