Gergő Móricz b03670a8b7 feat: parse PDFs on fc side and reject if too long for timeout (FIR-2083) (#1592) ...

* feat: pdf-parser, implementation in scrapeURL

* use pdf-parser for page count instead of mu

* fix(pdf-parser): bindings

* feat(scrapeURL/pdf): adjust MILLISECONDS_PER_PAGE

* implement post-runsync polling and fix

* fix(Dockerfile): copy in the pdf-parser source code

* fix(scrapeURL/pdf): better error for timeout below 0

2025-05-23 13:45:53 +02:00

..

__tests__

ACUC: Dynamic Limits (FIR-1641) (#1434)

2025-04-10 18:49:23 +02:00

deep-research

FIR-2006: Fix maxUrls and timeLimit parameters in Deep Research API (#1569)

2025-05-20 18:39:56 -03:00

extract

feat(api/extract): show extract as origin for scrapes originating from it (#1584)

2025-05-21 21:45:44 -03:00

generate-llmstxt

feat(api/extract): show extract as origin for scrapes originating from it (#1584)

2025-05-21 21:45:44 -03:00

LLM-extraction

(feat/ai-sdk) Migrate to AI-SDK (#1220)

2025-02-20 22:48:58 +01:00

batch-process.ts

Nick: revert trailing comma

2024-12-11 19:51:08 -03:00

cache.ts

Update cache.ts

2025-01-23 02:37:04 -03:00

canonical-url.test.ts

Fix LLMs.txt cache bug with subdomains and add bypass option (#1557)

2025-05-16 16:29:09 -03:00

canonical-url.ts

Nick: url normalization

2025-01-03 23:54:03 -03:00

concurrency-limit.ts

fix(concurrency-limit): rework cc queue to work by time not priority (#1526)

2025-05-08 12:40:13 +02:00

crawl-redis.test.ts

fix(crawl-redis/generateURLPermutations): dedupe index.html/index.php/slash/bare URL ends (FIR-827) (#1134)

2025-02-05 15:44:56 +01:00

crawl-redis.ts

feat(acuc): propagate team flags (FIR-1879) (#1522)

2025-05-08 20:23:35 +02:00

custom-error.ts

Nick: revert trailing comma

2024-12-11 19:51:08 -03:00

default-values.ts

Nick: revert trailing comma

2024-12-11 19:51:08 -03:00

entities.ts

feat(scrape/actions/click): add all parameter (FIR-1443) (#1342)

2025-03-17 20:47:17 +01:00

gcs-jobs.ts

[feat] Implement GCS storage option for scrape results across controllers an… (#1500)

2025-04-29 15:15:44 -03:00

gcs-pdf-cache.ts

feat: parse PDFs on fc side and reject if too long for timeout (FIR-2083) (#1592)

2025-05-23 13:45:53 +02:00

generic-ai.ts

create openAI provider using base url parameter (#1480)

2025-05-12 20:43:07 +02:00

html-to-markdown.ts

Update html-to-markdown.ts

2025-01-20 00:20:20 -03:00

html-transformer.ts

fix(scrape): allow getting valid JSON via rawHtml (FIR-852) (#1138)

2025-02-06 18:35:28 +01:00

job-priority.ts

extract concurrency hotfix

2025-04-11 20:38:54 +02:00

logger.ts

Nick: formatting fixes

2025-01-10 18:35:10 -03:00

map-cosine.ts

Nick: revert trailing comma

2024-12-11 19:51:08 -03:00

parse-mode.ts

…

parseApi.ts

Nick: fixed prettier

2024-12-11 19:46:11 -03:00

pdf-parser.ts

feat: parse PDFs on fc side and reject if too long for timeout (FIR-2083) (#1592)

2025-05-23 13:45:53 +02:00

ranker.test.ts

Nick: revert trailing comma

2024-12-11 19:51:08 -03:00

ranker.ts

(feat/ai-sdk) Migrate to AI-SDK (#1220)

2025-02-20 22:48:58 +01:00

scrape-events.ts

NIck: rm scrape events

2025-04-23 14:19:25 -04:00

strings.ts

Nick: blocklist string

2024-12-20 18:09:49 -03:00

supabase-jobs.ts

fix(v1): scrape-status with GCS

2025-04-15 17:42:38 -07:00

timeout.ts

Nick: fixed prettier

2024-12-11 19:46:11 -03:00

validate-country.ts

Nick: revert trailing comma

2024-12-11 19:51:08 -03:00

validateUrl.test.ts

Nick: revert trailing comma

2024-12-11 19:51:08 -03:00

validateUrl.ts

fix(map): do not remove query parameters from results (FIR-1015) (#1191)

2025-02-17 12:34:58 -03:00

withAuth.ts

fix: merge mock success data

2024-12-26 17:56:30 +08:00