Skip to content

Commit 7790d30

Browse files
committed
fix(providers): keep every attachment that works today working
Auditing the routing change for backwards compatibility turned up two bands it silently broke. Lowering the single inline cap made the upload path mandatory above ~6 MB, but every large-file path reads its bytes back out of cloud object storage. A deployment without it — local dev, any disk-backed self-host — inlines those files as base64 today and would have started failing outright with "requires cloud file storage". Split the one number in two: the inline ceiling stays at 10 MiB, and a separate threshold marks where an upload becomes *preferable* because the base64 copy no longer fits the payload store. Where no upload path is reachable, base64 hydration now runs to the inline ceiling as before, and a missing cloud-storage backend leaves the file for the inline path instead of throwing. The two strategies also cross over at different sizes now. `files-api` carries every type the provider already accepts, so it takes over at the lower threshold. `remote-url` only fetches images and PDFs, so switching early would have started rejecting 6-10 MB text documents that inline fine today; it takes over only once inlining is genuinely impossible. Revert the Groq ceilings. Its published "20MB" governs a request carrying an image URL, and on this path the body holds only the URL, so it cannot bind on the files maxBytes guards. Groq documents no limit on the image it fetches, so tightening the per-file cap to 20,000,000 and summing raw bytes against the request cap would both reject uploads that work today on no documented basis.
1 parent 2ba228a commit 7790d30

6 files changed

Lines changed: 107 additions & 40 deletions

File tree

apps/sim/executor/handlers/agent/agent-handler.ts

Lines changed: 17 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -54,9 +54,11 @@ import { resolveVertexCredential } from '@/executor/utils/vertex-credential'
5454
import { executeProviderRequest } from '@/providers'
5555
import {
5656
INLINE_ATTACHMENT_THRESHOLD_BYTES,
57+
LARGE_FILE_PATH_THRESHOLD_BYTES,
5758
shouldUseLargeFilePath,
5859
supportsFileAttachments,
5960
} from '@/providers/attachments'
61+
import { canUseProviderLargeFilePath } from '@/providers/file-attachments.server'
6062
import { isAutoModel, SIM_AUTO_MODEL_ID } from '@/providers/models'
6163
import { getProviderFromModel, transformBlockTool } from '@/providers/utils'
6264
import type { SerializedBlock } from '@/serializer/types'
@@ -946,6 +948,15 @@ export class AgentBlockHandler implements BlockHandler {
946948
const requestId = ctx.executionId || ctx.workflowId || 'agent-files'
947949
const nextMessages = [...messages]
948950

951+
/**
952+
* Stop hydrating base64 early only where an upload can actually take over. Where it cannot —
953+
* an inline-only provider, or any deployment without cloud storage — base64 stays the only
954+
* delivery path, so it has to be hydrated all the way to the inline ceiling.
955+
*/
956+
const inlineMaxBytes = canUseProviderLargeFilePath(providerId)
957+
? LARGE_FILE_PATH_THRESHOLD_BYTES
958+
: INLINE_ATTACHMENT_THRESHOLD_BYTES
959+
949960
for (let messageIndex = 0; messageIndex < messages.length; messageIndex++) {
950961
const message = messages[messageIndex]
951962
if (!message.files?.length) {
@@ -963,16 +974,17 @@ export class AgentBlockHandler implements BlockHandler {
963974
allowLargeValueWorkflowScope: ctx.allowLargeValueWorkflowScope,
964975
userId: ctx.userId,
965976
logger,
966-
maxBytes: INLINE_ATTACHMENT_THRESHOLD_BYTES,
977+
maxBytes: inlineMaxBytes,
967978
})
968979

969980
const missingFile = hydratedFiles.find(
970-
(file) => !file.base64 && !shouldUseLargeFilePath(file, providerId)
981+
(file) =>
982+
!file.base64 &&
983+
!(canUseProviderLargeFilePath(providerId) && shouldUseLargeFilePath(file, providerId))
971984
)
972985
if (missingFile) {
973-
const inlineMB = (INLINE_ATTACHMENT_THRESHOLD_BYTES / (1024 * 1024)).toFixed(0)
974-
const oversized =
975-
Number.isFinite(missingFile.size) && missingFile.size > INLINE_ATTACHMENT_THRESHOLD_BYTES
986+
const inlineMB = (inlineMaxBytes / (1024 * 1024)).toFixed(0)
987+
const oversized = Number.isFinite(missingFile.size) && missingFile.size > inlineMaxBytes
976988
throw new Error(
977989
oversized
978990
? `File "${missingFile.name}" (${(missingFile.size / (1024 * 1024)).toFixed(2)}MB) exceeds the ${inlineMB}MB inline attachment limit, and provider "${providerId}" has no large-file upload path for it.`

apps/sim/providers/attachments.test.ts

Lines changed: 22 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ import {
1616
getProviderFileStrategy,
1717
INLINE_ATTACHMENT_THRESHOLD_BYTES,
1818
inferAttachmentMimeType,
19+
LARGE_FILE_PATH_THRESHOLD_BYTES,
1920
prepareProviderAttachments,
2021
shouldUseLargeFilePath,
2122
} from '@/providers/attachments'
@@ -288,13 +289,27 @@ describe('provider attachments', () => {
288289

289290
describe('provider large-file capability', () => {
290291
/**
291-
* Guards the regression where the inline cap (10 MB) sat above what the payload store could
292-
* hold once base64 inflated it, so every 6-10 MB attachment died with "Execution memory limit
293-
* exceeded" instead of taking the provider's large-file path.
292+
* Guards the regression where every 6-10 MB attachment died with "Execution memory limit
293+
* exceeded": past this size the base64 copy no longer fits the payload store, so an upload
294+
* has to take over wherever one is reachable.
294295
*/
295-
it('keeps the inline cap inside the payload store ceiling once base64-encoded', () => {
296-
const encodedBytes = Math.ceil(INLINE_ATTACHMENT_THRESHOLD_BYTES / 3) * 4
296+
it('starts preferring an upload before base64 outgrows the payload store', () => {
297+
const encodedBytes = Math.ceil(LARGE_FILE_PATH_THRESHOLD_BYTES / 3) * 4
297298
expect(encodedBytes).toBeLessThanOrEqual(LARGE_VALUE_THRESHOLD_BYTES)
299+
expect(LARGE_FILE_PATH_THRESHOLD_BYTES).toBeLessThan(INLINE_ATTACHMENT_THRESHOLD_BYTES)
300+
})
301+
302+
/**
303+
* A `remote-url` provider only fetches images and PDFs, so it must not take over from base64
304+
* early — text documents in the 6-10 MB band inline fine today and would start failing.
305+
*/
306+
it('crosses over to an upload at different sizes for files-api and remote-url', () => {
307+
const midBand = { size: LARGE_FILE_PATH_THRESHOLD_BYTES + 1, type: 'text/plain' }
308+
expect(shouldUseLargeFilePath(midBand, 'openai')).toBe(true)
309+
expect(shouldUseLargeFilePath(midBand, 'anthropic')).toBe(false)
310+
311+
const aboveInline = { size: INLINE_ATTACHMENT_THRESHOLD_BYTES + 1, type: 'application/pdf' }
312+
expect(shouldUseLargeFilePath(aboveInline, 'anthropic')).toBe(true)
298313
})
299314

300315
it('reports per-provider strategy and ceiling, defaulting others to inline', () => {
@@ -316,7 +331,7 @@ describe('provider large-file capability', () => {
316331

317332
it('routes only oversized files on capable providers to the large-file path', () => {
318333
const small = { ...imageFile, size: 1024 }
319-
const large = { ...imageFile, size: INLINE_ATTACHMENT_THRESHOLD_BYTES + 1 }
334+
const large = { ...imageFile, size: LARGE_FILE_PATH_THRESHOLD_BYTES + 1 }
320335
expect(shouldUseLargeFilePath(small, 'openai')).toBe(false)
321336
expect(shouldUseLargeFilePath(large, 'openai')).toBe(true)
322337
expect(shouldUseLargeFilePath(large, 'bedrock')).toBe(false)
@@ -325,7 +340,7 @@ describe('provider large-file capability', () => {
325340
it('does not expose generated source through a remote-url large-file path', () => {
326341
const generated = {
327342
...pdfFile,
328-
size: INLINE_ATTACHMENT_THRESHOLD_BYTES + 1,
343+
size: LARGE_FILE_PATH_THRESHOLD_BYTES + 1,
329344
type: 'text/x-python-pdf',
330345
}
331346
expect(shouldUseLargeFilePath(generated, 'openai')).toBe(true)

apps/sim/providers/attachments.ts

Lines changed: 19 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@ import type { UserFile } from '@/executor/types'
1515
import {
1616
getProviderFileAttachment,
1717
INLINE_ATTACHMENT_MAX_BYTES,
18+
LARGE_FILE_PATH_THRESHOLD_BYTES,
1819
type ProviderFileAttachmentStrategy,
1920
} from '@/providers/models'
2021
import type { ProviderId } from '@/providers/types'
@@ -75,13 +76,12 @@ type ProviderFormattedMessage = {
7576
[key: string]: unknown
7677
}
7778

78-
/**
79-
* Files at or below this size are inlined as base64; larger files take the provider's
80-
* large-file path. Sized to the execution payload store, not to any provider — see
81-
* {@link INLINE_ATTACHMENT_MAX_BYTES}.
82-
*/
79+
/** Largest file that can be carried as inline base64 when no upload path is available. */
8380
export const INLINE_ATTACHMENT_THRESHOLD_BYTES = INLINE_ATTACHMENT_MAX_BYTES
8481

82+
/** Re-exported so callers choosing a hydration cap do not reach into `models.ts` directly. */
83+
export { LARGE_FILE_PATH_THRESHOLD_BYTES }
84+
8585
export type ProviderFileStrategy = ProviderFileAttachmentStrategy
8686

8787
/** Large-file delivery strategy for a provider, sourced from its `models.ts` definition. */
@@ -90,9 +90,17 @@ export function getProviderFileStrategy(providerId: ProviderId | string): Provid
9090
}
9191

9292
/**
93-
* True when an oversized file has a safe provider path. Remote URLs point at the
94-
* primary storage object, so source-backed documents can only use artifact-aware
95-
* Files API uploads.
93+
* True when a file should be delivered through the provider's large-file path rather than as
94+
* inline base64.
95+
*
96+
* The two strategies cross over at different sizes on purpose. `files-api` carries every type
97+
* this provider already accepts, so it takes over as soon as base64 stops being cacheable. A
98+
* `remote-url` provider only fetches images and PDFs, so switching early would start rejecting
99+
* text documents that inline fine today; it therefore only takes over once inlining is no longer
100+
* possible at all.
101+
*
102+
* Remote URLs point at the primary storage object, so source-backed generated documents can only
103+
* use artifact-aware Files API uploads.
96104
*/
97105
export function shouldUseLargeFilePath(
98106
file: Pick<UserFile, 'size' | 'type'>,
@@ -101,7 +109,9 @@ export function shouldUseLargeFilePath(
101109
const strategy = getProviderFileAttachment(providerId).strategy
102110
if (strategy === 'inline') return false
103111
if (strategy === 'remote-url' && isGeneratedDocumentSourceType(file.type)) return false
104-
return Number.isFinite(file.size) && file.size > INLINE_ATTACHMENT_THRESHOLD_BYTES
112+
const threshold =
113+
strategy === 'files-api' ? LARGE_FILE_PATH_THRESHOLD_BYTES : INLINE_ATTACHMENT_THRESHOLD_BYTES
114+
return Number.isFinite(file.size) && file.size > threshold
105115
}
106116

107117
const PDF_MIME_TYPE = 'application/pdf'

apps/sim/providers/file-attachments.server.test.ts

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -128,6 +128,23 @@ describe('OpenAI large-file attachment lifecycle', () => {
128128
expect(request.messages?.[0].files?.[0].remoteUrl).toBeUndefined()
129129
})
130130

131+
/**
132+
* Local and disk-backed deployments have no cloud storage, so the upload path cannot read the
133+
* bytes back. These files inline as base64 today and must keep doing so rather than hard-fail.
134+
*/
135+
it('leaves the file for the inline path when cloud storage is unavailable', async () => {
136+
mockHasCloudStorage.mockReturnValue(false)
137+
const request = makeRequest(CSV_BYTES)
138+
139+
await attachLargeFileRemoteUrls(request, 'openai')
140+
await uploadLargeFilesToProvider(request, 'openai')
141+
142+
expect(fetch).not.toHaveBeenCalled()
143+
const file = request.messages?.[0].files?.[0]
144+
expect(file?.remoteUrl).toBeUndefined()
145+
expect(file?.providerFileId).toBeUndefined()
146+
})
147+
131148
it('rejects a request whose attachments together exceed the combined ceiling', async () => {
132149
const request = makeRequest(30 * 1024 * 1024)
133150
const [first] = request.messages?.[0].files ?? []

apps/sim/providers/file-attachments.server.ts

Lines changed: 13 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -33,12 +33,21 @@ function* iterateRequestFiles(messages: Message[] | undefined): Generator<UserFi
3333
}
3434
}
3535

36+
/**
37+
* True when this deployment can actually deliver an oversized attachment through the provider's
38+
* large-file path. A provider strategy alone is not enough — every large-file path reads the
39+
* bytes back out of cloud object storage, so a deployment without it has to keep inlining.
40+
*/
41+
export function canUseProviderLargeFilePath(providerId: ProviderId | string): boolean {
42+
return getProviderFileStrategy(providerId) !== 'inline' && StorageService.hasCloudStorage()
43+
}
44+
3645
/**
3746
* Resolves every attachment that exceeds the inline threshold on a large-file-capable
3847
* provider to a short-lived signed URL on `file.remoteUrl`. `remote-url` providers send it
3948
* to the model directly; for `files-api` providers it marks the file for upload (the bytes
40-
* are read from storage at upload time). Requires cloud storage — a large file (already past
41-
* the inline base64 cap) cannot be sent without it, so the request fails with a clear error.
49+
* are read from storage at upload time). Every large-file path needs cloud storage to read the
50+
* bytes back, so without it the file is left for the inline base64 path instead.
4251
*
4352
* Runs for every request in {@link executeProviderRequest} (after the API key resolves), so
4453
* the server-only handle fields are first cleared on every file for every provider — a forged
@@ -74,11 +83,9 @@ export async function attachLargeFileRemoteUrls(
7483

7584
if (!StorageService.hasCloudStorage()) {
7685
logger.warn(
77-
`[${requestId}] "${file.name}" exceeds the inline limit for "${providerId}" but cloud storage is unavailable`
78-
)
79-
throw new Error(
80-
`File "${file.name}" exceeds the inline attachment limit and requires cloud file storage, which is not configured`
86+
`[${requestId}] Sending "${file.name}" inline for "${providerId}": the large-file path needs cloud storage, which is not configured`
8187
)
88+
continue
8289
}
8390

8491
if (!request.userId) {

apps/sim/providers/models.ts

Lines changed: 19 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -143,16 +143,20 @@ export interface ProviderFileAttachment {
143143
strategy: ProviderFileAttachmentStrategy
144144
}
145145

146+
/** Inline base64 attachment cap, also the fallback limit for providers without a large-file path. */
147+
export const INLINE_ATTACHMENT_MAX_BYTES = 10 * 1024 * 1024
148+
146149
/**
147-
* Inline base64 attachment cap, also the fallback limit for providers without a large-file path.
150+
* Size above which an attachment should prefer the provider's Files API over base64, when the
151+
* deployment can reach one.
148152
*
149-
* Bounded by the execution payload store rather than by any provider. Base64 inflates bytes by
150-
* 4/3 and a single stored value may not exceed {@link LARGE_VALUE_THRESHOLD_BYTES}, so the
151-
* largest raw file whose base64 still fits is three quarters of that ceiling. A larger cap does
152-
* not send a bigger file — it fails the run with "Execution memory limit exceeded" partway
153-
* through hydration instead of routing the file to the provider's large-file path.
153+
* Set by the execution payload store, not by any provider. Base64 inflates bytes by 4/3 and a
154+
* single stored value may not exceed {@link LARGE_VALUE_THRESHOLD_BYTES}, so past three quarters
155+
* of that ceiling the encoded copy no longer fits the cache. Inlining still succeeds above this
156+
* point — the cache write is skipped, not fatal — but it carries a needlessly large encoded
157+
* payload, so an upload is preferred wherever one is available.
154158
*/
155-
export const INLINE_ATTACHMENT_MAX_BYTES = Math.floor(LARGE_VALUE_THRESHOLD_BYTES / 4) * 3
159+
export const LARGE_FILE_PATH_THRESHOLD_BYTES = Math.floor(LARGE_VALUE_THRESHOLD_BYTES / 4) * 3
156160

157161
const DEFAULT_FILE_ATTACHMENT: ProviderFileAttachment = {
158162
maxBytes: INLINE_ATTACHMENT_MAX_BYTES,
@@ -2443,12 +2447,14 @@ export const PROVIDER_DEFINITIONS: Record<string, ProviderDefinition> = {
24432447
},
24442448
groq: {
24452449
id: 'groq',
2446-
/** "Maximum allowed size for a request containing an image URL as input is 20MB." */
2447-
fileAttachment: {
2448-
maxBytes: 20_000_000,
2449-
perRequestMaxBytes: 20_000_000,
2450-
strategy: 'remote-url',
2451-
},
2450+
/**
2451+
* Left at the pre-existing ceiling: Groq's published "20MB" governs a request carrying an
2452+
* image URL, and on this path the request body holds only the URL, so it cannot bind on the
2453+
* files this guards. Groq documents no ceiling on the image it fetches, so both tightening
2454+
* the per-file cap and summing raw bytes against the request cap would reject uploads that
2455+
* work today on no documented basis.
2456+
*/
2457+
fileAttachment: { maxBytes: 20 * 1024 * 1024, strategy: 'remote-url' },
24522458
name: 'Groq',
24532459
description: "Groq's LLM models with high-performance inference",
24542460
defaultModel: 'groq/llama-3.3-70b-versatile',

0 commit comments

Comments
 (0)