Commit Graph

Select branches

Hide Pull Requests

main

1.4.1

v.1.3.0

v1.0.0

v1.1.0

v1.1.1

v1.10.0

v1.11.0

v1.12.0

v1.13.0

v1.14.0

v1.2.0

v1.2.1

v1.4.0

v1.4.2

v1.4.3

v1.4.4

v1.5.0

v1.6.0

v1.7.0

v1.8.0

v1.9.0

Mono Color

• ac875b8ad9 docs: kubernetes simple update (#1772) main Gergő Móricz 2025-07-07 16:46:13 +02:00
• 13335a339d feat: test RunPod MU new version (#1771) Thomas Kosmas 2025-07-07 15:41:47 +03:00
• 7cac330cb8 revert: resume A/B testing Gergő Móricz 2025-07-05 16:22:44 +02:00
• 2add2d132e temp: stop A/B testing Gergő Móricz 2025-07-04 23:27:38 +02:00
• 87baa23197 Nick: reverting ssl changes to py sdk Nicolas 2025-07-04 17:57:09 -03:00
• 8922d7eb5d Add created_at field to /crawl/active endpoint response (#1718) 3 people 2025-07-04 20:18:29 +00:00
• 9cc32c2821 feat(html-transformer, scrapeURL): omce support (#1764) Gergő Móricz 2025-07-04 22:16:07 +02:00
• 58081963e7 fix(logger): correct method names in logger.child calls (#1731) ahnafatef 2025-07-05 01:48:13 +06:00
• 80799557b8 fix(zdr): missing zdr clause on runWebScraper logger Gergő Móricz 2025-07-04 20:55:47 +02:00
• 8f36f7c0c8 Update SELF_HOST.md Gergő Móricz 2025-07-04 19:24:20 +02:00
• b28d4f72f8 Make worker Express server port configurable via environment variable (#1748) 4 people 2025-07-04 19:13:45 +02:00
• 61059d689d chore(ab): ramp up v1.14.0 Gergő Móricz 2025-07-04 15:36:57 +02:00
• 5d40664384 feat(scrapeURL/fire-engine): start AB testing (#1763) Gergő Móricz 2025-07-04 15:29:40 +02:00
• 82e76a7405 fix: replace hardcoded search billing with calculateCreditsToBeBilled (#1714) 4 people 2025-07-03 12:41:56 -04:00
• 251e42c2a3 feat: add waitFor validation rule to enforce waitFor <= timeout/2 (#1751) 4 people 2025-07-03 18:24:52 +02:00
• b5b6ca0a32 Add end-to-end test for deep research functionality (ENG-2627) (#1760) 3 people 2025-07-03 17:34:02 +02:00
• 46850d1e8d Implement fallback mechanism for onlyMainContent scraping (ENG-2499) (#1759) 3 people 2025-07-03 17:33:38 +02:00
• 88549c26ad html-transformer improvements (#1762) Gergő Móricz 2025-07-03 17:33:28 +02:00
• 8d81209ffd fix(html-transformer): check base tag when resolving relative URLs (#1761) Gergő Móricz 2025-07-03 16:54:41 +02:00
• de1bc96aa0 Implement JavaScript-only base href handling (ENG-2302) (#1756) 3 people 2025-07-02 23:14:53 +02:00
• 4c1af8b96e Make error message for backwards crawling more clear (#1758) Micah Stairs 2025-07-02 17:08:54 -04:00
• 6d8e771822 fix(api/deep-research): bad default values leading to bug on self-host (#1754) Gergő Móricz 2025-07-02 22:57:56 +02:00
• a9f4b1f207 Add URL depth validation to crawl requests (ENG-2617) (#1753) 3 people 2025-07-02 22:30:27 +02:00
• de4094b19c Add health check endpoint for Playwright service (ENG-2585) (#1752) 3 people 2025-07-02 21:52:36 +02:00
• 179737104d bugfix zero_data_retention param and certifi dependency (#1749) Rafael Miller 2025-07-02 14:04:21 -03:00
• 8b54a37b19 Merge pull request #1715 from mendableai/feat/integration-param-pythonsdk Ademílson Tonato 2025-07-02 14:49:45 +01:00
• 72b9074ecd Fix queue worker liveness endpoint for self-hosted environments (#1747) 3 people 2025-07-02 15:27:37 +02:00
• 30b7e17327 feat(firecrawl): add integration parameter support and enhance kwargs handling Ademílson Tonato 2025-06-25 22:08:38 +01:00
• dcbed186a9 Add local environment configuration to docker-compose services (#1742) NicolasandCursor Agent 2025-07-01 19:27:01 -03:00
• e3dc2e87db chore: bump js sdk Gergő Móricz 2025-07-01 20:29:00 +02:00
• 4506b21185 feat(api): zero data retention (ENG-2376) (#1687) Gergő MóriczandNicolas 2025-07-01 20:07:26 +02:00
• 1f1f733011 fix(map): pass timeout to sitemap fetch (#1741) Gergő Móricz 2025-07-01 19:26:43 +02:00
• ec298f58b6 Update search.ts Nicolas 2025-07-01 13:43:00 -03:00
• 3e09f9fb8a Nick: init (#1740) Nicolas 2025-07-01 11:39:45 -03:00
• 6c2f432d49 feat(crawl-status): better creditsUsed field (#1738) Gergő Móricz 2025-07-01 16:34:30 +02:00
• ebf98e3c16 feat(queue-worker): decrease job lock duration to pick up jobs on dead workers faster (#1737) Gergő Móricz 2025-07-01 16:27:38 +02:00
• 400d497fca feat(scrapeURL): ask user to increase timeout if there's a DOM.getDocument or queryAXTree error (#1739) Gergő Móricz 2025-07-01 16:25:49 +02:00
• 1816cfc4c8 feat: implement IDN support with Punycode encoding (#1735) 3 people 2025-07-01 15:39:34 +02:00
• 8a282e3fc8 fix(auto_charge): bad hourly counter logic (#1736) Gergő Móricz 2025-07-01 15:37:37 +02:00
• b4eedce3e0 (feat/ledger) Ledger events (#1728) NicolasandGergő Móricz 2025-06-30 12:48:06 -03:00
• 13f012c583 add pdf prefetch log for debugging (ENG-2542) (#1734) Gergő Móricz 2025-06-30 17:37:31 +02:00
• 9162952744 proxy used improvement (#1727) Gergő Móricz 2025-06-30 17:37:19 +02:00
• 9b95a17c0d fix json format on search (#1729) Gergő Móricz 2025-06-30 17:17:52 +02:00
• 17ff8be67b Nick; (#1726) v1.13.0 Nicolas 2025-06-27 12:02:15 -03:00
• 57b8e66bc8 feat(api/worker): liveness check in queueing -- don't take jobs when the worker is dying (#1725) Gergő Móricz 2025-06-27 16:51:40 +02:00
• c4adc687ea Update index.ts Nicolas 2025-06-27 11:42:26 -03:00
• caec228f60 Nick: version bump Nicolas 2025-06-27 11:33:36 -03:00
• fa5b96c521 Add parsePDF parameter to JS SDK (#1720) 4 people 2025-06-27 11:30:49 -03:00
• 070d1c1d98 Fix unreachable allowSubdomains code in crawler filterURL method (#1719) 3 people 2025-06-27 11:28:58 -03:00
• 2b87ea6599 feat: improve DNS resolution error message (#1724) Gergő Móricz 2025-06-27 16:27:25 +02:00
• 55d5c1f41d feat(scrapeURL/skipTlsVerification): improve error message (#1723) Gergő Móricz 2025-06-27 16:27:04 +02:00
• 9ed26e1e07 feat(sdk/python): add pdf action (ENG-2515) (#1722) Gergő MóriczandNicolas 2025-06-27 16:26:27 +02:00
• d8796e4536 feat: Screenshot quality (#1721) Nicolas 2025-06-27 11:07:14 -03:00
• 9a5d40c3cf Allow international URLs to pass validation (#1717) Micah Stairs 2025-06-26 13:16:42 -04:00
• 1919799bed feat(python-sdk): add parsePDF parameter support (#1713) 4 people 2025-06-26 16:34:43 +00:00
• 89e57ace3c Add temporary exception for Faire team ID to bypass job expiration (#1716) 4 people 2025-06-26 13:42:34 +00:00
• f4714f4849 fix(js-sdk/extract): use same zod fallback logic (#1711) Gergő Móricz 2025-06-25 19:59:58 +02:00
• 3d04c2087e fix(api): cached acuc didn't have the is_extract flag set (#1712) Gergő Móricz 2025-06-25 16:43:53 +02:00
• bc9065810d fix(concurrency-limit): overlogging (#1709) Gergő Móricz 2025-06-24 19:01:32 +02:00
• cc3afa2578 fix(concurrency-limit): scan instead of taking jobs (#1708) Gergő Móricz 2025-06-24 18:32:22 +02:00
• ae94edd43e feat(api/ci): idmux (#1707) Gergő Móricz 2025-06-24 15:36:05 +02:00
• 86603de664 fix(api): instantiate Storage only once (#1706) Gergő Móricz 2025-06-24 00:18:07 +02:00
• 11f469488e fix(api/batch/scrape): maxConcurrency field support when using ignoreInvalidURLs (#1705) Gergő Móricz 2025-06-23 21:44:55 +02:00
• e7a62dd490 fix(api): pdf bug + testing bugs (#1704) Gergő Móricz 2025-06-23 19:57:27 +02:00
• fe9057559b fix(v1): check credits variable scope collision (#1703) Gergő Móricz 2025-06-23 19:19:01 +02:00
• e3948ae5b1 feat(api): pdf action + housekeeping (#1702) Gergő Móricz 2025-06-23 19:03:35 +02:00
• 78a3579d6e feat: add relevanceai as part of the integrations Ademílson Tonato 2025-06-23 16:41:19 +01:00
• 439619ffc6 fix(api/v1/crawl/ongoing): only crawls, no batch scrape (#1701) Gergő Móricz 2025-06-23 16:33:02 +02:00
• 1fdf95913d feat(api): optimize job count query and improve error handling (#1700) Gergő Móricz 2025-06-23 16:18:55 +02:00
• c31172493e fix(api): handle errors better in redis-less crawl status (#1699) Gergő Móricz 2025-06-23 15:56:20 +02:00
• 66cde50a2a fix(api): enhance error handler with optional ACUC data (#1698) Gergő Móricz 2025-06-23 15:42:38 +02:00
• e06ec2d047 fix(api): improve error logging with structured error object (#1697) Gergő Móricz 2025-06-23 15:26:15 +02:00
• 7ed19c0ac0 feat(scrapeURL): separate URL rewrites to different function Gergő Móricz 2025-06-21 02:02:29 +02:00
• 9174e0c8a0 fix(api): CI (#1692) Gergő Móricz 2025-06-20 22:57:23 +02:00
• 2082243cb5 feat(scrape): support Google Slides (#1693) Meet Soni 2025-06-21 01:28:45 +05:30
• 4b03ffca36 fix(search): respect parsePDF in pricing (#1690) Gergő Móricz 2025-06-20 21:15:14 +02:00
• 125e1ada45 feat(scrapeURL): support cookies in safeFetch (#1688) Gergő Móricz 2025-06-20 20:43:04 +02:00
• 3f0b8b8e27 Remove old cache mechanisms (redis cache, PDF cache, crawl maps, etc.) (FIR-2266) (#1667) Gergő Móricz 2025-06-20 19:40:28 +02:00
• 363afb8048 Nick: updated openapi specs Nicolas 2025-06-20 14:30:37 -03:00
• 80f7177473 Nick: bump version v1.12.0 Nicolas 2025-06-20 12:05:15 -03:00
• 09aabbedb5 feat: add followInternalLinks parameter as semantic replacement for allowBackwardLinks (#1684) 4 people 2025-06-20 12:02:23 -03:00
• f939428264 feat(scrape): support Google Docs (FIR-1365) (#1686) Gergő Móricz 2025-06-20 11:42:41 +02:00
• f8983fffb7 Concurrency limit refactor + maxConcurrency parameter (FIR-2191) (#1643) Gergő Móricz 2025-06-20 10:45:36 +02:00
• a8e3c29664 feat(scrape, extract): creditsUsed, tokensUsed fields (FIR-2336) (#1683) Gergő Móricz 2025-06-18 21:49:20 +02:00
• fbd81b4168 fix(scrape): log FIRE-1 credits billed on failures properly (FIR-2331) (#1682) Gergő Móricz 2025-06-18 21:47:58 +02:00
• ebc1de9d60 feat(crawl-status): refactor to work after a redis flush (#1664) Gergő Móricz 2025-06-18 18:58:04 +02:00
• cd2e0f868c Add deployment type field to bug report template (#1681) 3 people 2025-06-18 12:26:15 -03:00
• 199115c7be stop testing new mu Thomas Kosmas 2025-06-18 00:48:50 +03:00
• f46f845efc fix: send the request to new mu version before the main one to achieve better sync Thomas Kosmas 2025-06-17 20:37:45 +03:00
• ee7b29b3f6 feat: Test mu v3 (#1678) Thomas Kosmas 2025-06-17 20:13:19 +03:00
• 5ca8e2e98e feat(index): store short titles and descriptions (#1677) Gergő Móricz 2025-06-17 19:09:07 +02:00
• 9710bdffc0 Improve URL filtering error messages with specific denial reasons (FIR-2352) (#1676) 3 people 2025-06-17 19:00:29 +02:00
• c6482eaf2d Nick: prevent additional logging on /extract scrapes Nicolas 2025-06-13 18:17:17 -03:00
• ea321b4936 fix search test timeouts Gergő Móricz 2025-06-13 17:42:55 +02:00
• 38c5795282 feat(vertex): fix vertex ai provider bug and update model references to use "gemini-2.5-pro" (#1668) Thomas Kosmas 2025-06-13 18:29:03 +03:00
• 0bf23071ff feat(index): add domain splitting for improved map querying (#1666) v1.11.0 Gergő Móricz 2025-06-13 15:22:45 +02:00
• 07224b8cd4 feat: use index in search and extract (#1660) Gergő Móricz 2025-06-13 12:30:28 +02:00
• f296342731 feat(index): remove unused columns (#1662) Gergő Móricz 2025-06-12 16:51:40 +02:00
• 89e42b1137 fix(api): remove query parameter sanitization that was breaking extracts (#1661) Gergő Móricz 2025-06-12 15:37:45 +02:00
• 3c03d07051 feat: add credits_billed everywhere (FIR-2286) (#1655) Gergő Móricz 2025-06-11 23:06:55 +02:00