From 0d2f0f1ea72b97d9968f31085a754baef167777b Mon Sep 17 00:00:00 2001 From: Yang Jun Date: Sat, 11 Jul 2026 15:29:52 +0800 Subject: [PATCH] feat: remove memoryLimit option (#910) Co-authored-by: Cursor --- docs/source/tutorials/security-model.md | 30 +++------ src/context/context.ts | 7 +-- src/filters/array.ts | 18 ------ src/filters/base64.ts | 3 - src/filters/crypto.ts | 2 - src/filters/date.ts | 12 ++-- src/filters/html.ts | 4 -- src/filters/misc.ts | 84 +++++++++---------------- src/filters/string.ts | 27 -------- src/liquid-options.ts | 6 -- src/render/expression.ts | 1 - src/util/strftime.ts | 12 ++-- test/e2e/issues.spec.ts | 7 --- test/integration/filters/date.spec.ts | 26 +------- test/integration/liquid/dos.spec.ts | 81 +----------------------- 15 files changed, 51 insertions(+), 269 deletions(-) diff --git a/docs/source/tutorials/security-model.md b/docs/source/tutorials/security-model.md index 5d65a0196..ff35d2571 100644 --- a/docs/source/tutorials/security-model.md +++ b/docs/source/tutorials/security-model.md @@ -2,7 +2,7 @@ title: Security Model --- -LiquidJS provides DoS-oriented limits (`parseLimit`, `renderLimit`, `memoryLimit`) to reduce risk. This page summarizes those limits, [`ownPropertyOnly`][ownPropertyOnly], custom [`Drop`][drop] usage, and the security boundary to assume in production. +LiquidJS provides DoS-oriented limits (`parseLimit`, `renderLimit`) to reduce risk. This page summarizes those limits, [`ownPropertyOnly`][ownPropertyOnly], custom [`Drop`][drop] usage, and the security boundary to assume in production. ## Security boundary @@ -12,11 +12,14 @@ The built-in limits are cooperative safeguards, not strict runtime isolation. - They do **not** sandbox JavaScript execution. - They should be combined with process/container limits and request timeouts for defense in depth. +LiquidJS does **not** enforce memory or CPU budgets inside the engine. Major template engines take the same approach: byte-level heap tracking is unreliable in garbage-collected runtimes (non-deterministic GC, accounting overhead) and does not map cleanly to real process memory. [Jinja2](https://jinja.palletsprojects.com/en/stable/sandbox/) relies on `sys.setrecursionlimit`, `SandboxedEnvironment` for access control, and advises OS/process limits (`ulimit`, cgroups). [Twig](https://twig.symfony.com/doc/3.x/api.html#security-policy) documents a `SecurityPolicy` for tags/filters/methods and explicitly leaves resource limits to PHP (`memory_limit`, execution timeouts). Handlebars and EJS provide no render budgets; Node.js users typically combine [`vm.Script` timeouts](https://nodejs.org/api/vm.html) or [`worker_threads`](https://nodejs.org/api/worker_threads.html) with process isolation for untrusted templates. + +For LiquidJS in production, prefer **external** controls: Node.js `vm` or worker threads (or packages such as [`isolated-vm`](https://www.npmjs.com/package/isolated-vm) when stronger isolation is required), separate processes or containers, OS/container memory and CPU quotas, and request timeouts — not in-engine heap tracking. + ## Limits at a glance - [parseLimit][parseLimit]: limit total template size per `parse()` call. - [renderLimit][renderLimit]: limit total render time per `render()` call. -- [memoryLimit][memoryLimit]: cooperatively limit memory-sensitive allocations counted by LiquidJS. ## Limit details @@ -40,25 +43,7 @@ Render time is checked on a per-template basis (before rendering each template). `renderLimit` is not a hard CPU limiter. It is checked between template renders, so compute-intensive filters/tags/user-defined functions or deeply nested template execution between checks can still cause DoS. -### memoryLimit - -`memoryLimit` only limits operations that LiquidJS explicitly counts. - -- Counted: memory-sensitive LiquidJS operations that call internal memory accounting. -- Not guaranteed counted: arbitrary user object behavior such as custom `toValue()`/`toString()` chains, or other host-side code that allocates outside LiquidJS accounting points. - -In other words, `memoryLimit` limits what LiquidJS counts, not every byte your process may allocate. - -Even with a small number of templates and iterations, memory usage can grow exponentially. In the following example, memory doubles with each iteration: - -```liquid -{% assign array = "1,2,3" | split: "," %} -{% for i in (1..32) %} - {% assign array = array | concat: array %} -{% endfor %} -``` - -As [JavaScript uses GC to manage memory](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Memory_management), `memoryLimit` may not reflect the actual memory footprint. +Memory-heavy templates (for example exponential `concat` in a loop) are not capped by LiquidJS. Mitigate them with process/container memory limits, output size checks after render, or template restrictions — the same pattern Jinja2 and Twig recommend for heap and CPU. ## `ownPropertyOnly` and scope data @@ -74,14 +59,13 @@ If you run an online service, avoid rendering fully user-defined templates whene - Prefer curated templates or a restricted template subset. - If user-defined templates are required, isolate rendering (worker/process/container), enforce OS/container memory and CPU limits, and apply request rate limits. -- Treat `parseLimit`/`renderLimit`/`memoryLimit` as one layer in a broader DoS defense strategy. +- Treat `parseLimit` and `renderLimit` as one layer in a broader DoS defense strategy. For heavy single-template operations, process-level isolation is still recommended (for example with [paralleljs][paralleljs]). [paralleljs]: https://www.npmjs.com/package/paralleljs [parseLimit]: /api/interfaces/LiquidOptions.html#parseLimit [renderLimit]: /api/interfaces/LiquidOptions.html#renderLimit -[memoryLimit]: /api/interfaces/LiquidOptions.html#memoryLimit [ownPropertyOnly]: /api/interfaces/LiquidOptions.html#ownPropertyOnly [renderOwnPropertyOnly]: /api/interfaces/RenderOptions.html#ownPropertyOnly [strictVariables]: /api/interfaces/LiquidOptions.html#strictVariables diff --git a/src/context/context.ts b/src/context/context.ts index 9dcb711ea..19b36d522 100644 --- a/src/context/context.ts +++ b/src/context/context.ts @@ -36,16 +36,14 @@ export class Context { */ public strictVariables: boolean; public ownPropertyOnly: boolean; - public memoryLimit: Limiter; public renderLimit: Limiter; - public constructor (env: object = {}, opts: NormalizedFullOptions = defaultOptions, renderOptions: RenderOptions = {}, { memoryLimit, renderLimit }: { [key: string]: Limiter } = {}) { + public constructor (env: object = {}, opts: NormalizedFullOptions = defaultOptions, renderOptions: RenderOptions = {}, { renderLimit }: { renderLimit?: Limiter } = {}) { this.sync = !!renderOptions.sync this.opts = opts this.globals = renderOptions.globals ?? opts.globals this.environments = isObject(env) ? env : Object(env) this.strictVariables = renderOptions.strictVariables ?? this.opts.strictVariables this.ownPropertyOnly = renderOptions.ownPropertyOnly ?? opts.ownPropertyOnly - this.memoryLimit = memoryLimit ?? new Limiter('memory alloc', renderOptions.memoryLimit ?? opts.memoryLimit) this.renderLimit = renderLimit ?? new Limiter('template render', getPerformance().now() + (renderOptions.renderLimit ?? opts.renderLimit)) } public getRegister (key: string, defaultValue: T = undefined as T): T { @@ -109,8 +107,7 @@ export class Context { strictVariables: this.strictVariables, ownPropertyOnly: this.ownPropertyOnly }, { - renderLimit: this.renderLimit, - memoryLimit: this.memoryLimit + renderLimit: this.renderLimit }) } private findScope (key: string | number) { diff --git a/src/filters/array.ts b/src/filters/array.ts index bf16b5951..2a5cb8a0e 100644 --- a/src/filters/array.ts +++ b/src/filters/array.ts @@ -8,9 +8,6 @@ import { EmptyDrop } from '../drop' export const join = argumentsToValue(function (this: FilterImpl, v: any[], arg: string) { const array = toArray(v) const sep = isNil(arg) ? ' ' : stringify(arg) - let outputSize = sep.length * Math.max(array.length - 1, 0) - for (let i = 0; i < array.length; i++) outputSize += String(array[i]).length - this.context.memoryLimit.use(outputSize) return Array.prototype.join.call(array, sep) }) export const last = argumentsToValue(function (this: FilterImpl, v: any) { @@ -21,14 +18,12 @@ export const first = argumentsToValue(function (this: FilterImpl, v: any) { }) export const reverse = argumentsToValue(function (this: FilterImpl, v: any[]) { const array = toArray(v) - this.context.memoryLimit.use(array.length) return [...array].reverse() }) function * sortBy (this: FilterImpl, arr: T[], property: string | undefined, comparator: (a: unknown, b: unknown) => number): IterableIterator { const values: [T, unknown][] = [] const array = toArray(arr) - this.context.memoryLimit.use(array.length) for (const item of array) { values.push([ item, @@ -51,7 +46,6 @@ export const size = (v: string | any[]) => v?.length || 0 export function * map (this: FilterImpl, arr: Scope[], property: string): IterableIterator { const results = [] const array = toArray(arr) - this.context.memoryLimit.use(array.length) for (const item of array) { results.push(yield this.context._getFromScope(item, stringify(property), false)) } @@ -70,14 +64,12 @@ export function * sum (this: FilterImpl, arr: Scope[], property?: string): Itera export function compact (this: FilterImpl, arr: T[]) { const array = toArray(arr) - this.context.memoryLimit.use(array.length) return Array.prototype.filter.call(array, x => !isNil(toValue(x))) } export function concat (this: FilterImpl, v: T1[], arg: T2[] = []): (T1 | T2)[] { const lhs = toArray(v) const rhs = toArray(arg) - this.context.memoryLimit.use(lhs.length + rhs.length) return Array.prototype.concat.call(lhs, rhs) } @@ -87,7 +79,6 @@ export function push (this: FilterImpl, v: T[], arg: T): T[] { export function unshift (this: FilterImpl, v: T[], arg: T): T[] { const array = toArray(v) - this.context.memoryLimit.use(array.length) const clone = [...array] clone.unshift(arg) return clone @@ -95,7 +86,6 @@ export function unshift (this: FilterImpl, v: T[], arg: T): T[] { export function pop (this: FilterImpl, v: T[]): T[] { const array = toArray(v) - this.context.memoryLimit.use(array.length) const clone = [...array] clone.pop() return clone @@ -103,7 +93,6 @@ export function pop (this: FilterImpl, v: T[]): T[] { export function shift (this: FilterImpl, v: T[]): T[] { const array = toArray(v) - this.context.memoryLimit.use(array.length) const clone = [...array] clone.shift() return clone @@ -114,7 +103,6 @@ export function slice (this: FilterImpl, v: T[] | string, begin: number, leng if (isNil(v)) return [] if (!isArray(v)) v = stringify(v) begin = begin < 0 ? v.length + begin : begin - this.context.memoryLimit.use(length) return isArray(v) ? Array.prototype.slice.call(v, begin, begin + length) : String.prototype.slice.call(v, begin, begin + length) @@ -133,7 +121,6 @@ function expectedMatcher (this: FilterImpl, expected: any): (v: any) => boolean function * filter (this: FilterImpl, include: boolean, arr: T[], property: string, expected: any): IterableIterator { const values: unknown[] = [] arr = toArray(arr) - this.context.memoryLimit.use(arr.length) const token = new Tokenizer(stringify(property)).readScopeValue() for (const item of arr) { values.push(yield evalToken(token, this.context.spawn(item))) @@ -146,7 +133,6 @@ function * filter_exp (this: FilterImpl, include: boolean, arr const filtered: unknown[] = [] const keyTemplate = new Value(stringify(exp), this.liquid) const array = toArray(arr) - this.context.memoryLimit.use(array.length) for (const item of array) { this.context.push({ [itemName]: item }) const value = yield keyTemplate.value(this.context) @@ -176,7 +162,6 @@ export function * group_by (this: FilterImpl, arr: T[], proper const map = new Map() arr = toEnumerable(arr) const token = new Tokenizer(stringify(property)).readScopeValue() - this.context.memoryLimit.use(arr.length) for (const item of arr) { const key = yield evalToken(token, this.context.spawn(item)) if (!map.has(key)) map.set(key, []) @@ -189,7 +174,6 @@ export function * group_by_exp (this: FilterImpl, arr: T[], it const map = new Map() const keyTemplate = new Value(stringify(exp), this.liquid) arr = toEnumerable(arr) - this.context.memoryLimit.use(arr.length) for (const item of arr) { this.context.push({ [itemName]: item }) const key = yield keyTemplate.value(this.context) @@ -253,7 +237,6 @@ export function * find_exp (this: FilterImpl, arr: T[], itemNa export function uniq (this: FilterImpl, arr: T[]): T[] { arr = toArray(arr) - this.context.memoryLimit.use(arr.length) return [...new Set(arr)] } @@ -261,7 +244,6 @@ export function sample (this: FilterImpl, v: T[] | string, count = 1): T | st v = toValue(v) if (isNil(v)) return [] if (!isArray(v)) v = stringify(v) - this.context.memoryLimit.use(v.length) const shuffled = [...v].sort(() => Math.random() - 0.5) if (count === 1) return shuffled[0] return shuffled.slice(0, count) diff --git a/src/filters/base64.ts b/src/filters/base64.ts index 29ef84e52..5324c7473 100644 --- a/src/filters/base64.ts +++ b/src/filters/base64.ts @@ -10,16 +10,13 @@ import { base64Encode, base64Decode } from './base64-impl' export function base64_encode (this: FilterImpl, value: string | Buffer): string { if (typeof Buffer !== 'undefined' && Buffer.isBuffer(value)) { - this.context.memoryLimit.use(value.byteLength) return value.toString('base64') } const str = stringify(value) - this.context.memoryLimit.use(str.length) return base64Encode(str) } export function base64_decode (this: FilterImpl, value: string): string { const str = stringify(value) - this.context.memoryLimit.use(str.length) return base64Decode(str) } diff --git a/src/filters/crypto.ts b/src/filters/crypto.ts index 7210523bd..26caebd26 100644 --- a/src/filters/crypto.ts +++ b/src/filters/crypto.ts @@ -10,13 +10,11 @@ import { sha256 as sha256Impl, hmacSha256 as hmacSha256Impl } from './crypto-imp export function sha256 (this: FilterImpl, value: unknown): string | Promise { const str = stringify(value) - this.context.memoryLimit.use(str.length) return sha256Impl(str) } export function hmac_sha256 (this: FilterImpl, value: unknown, key: unknown): string | Promise { const str = stringify(value) const keyStr = stringify(key) - this.context.memoryLimit.use(str.length + keyStr.length) return hmacSha256Impl(str, keyStr) } diff --git a/src/filters/date.ts b/src/filters/date.ts index 07665c957..e2567421b 100644 --- a/src/filters/date.ts +++ b/src/filters/date.ts @@ -3,14 +3,11 @@ import { FilterImpl } from '../template' import { NormalizedFullOptions } from '../liquid-options' export function date (this: FilterImpl, v: string | Date, format?: string, timezoneOffset?: number | string) { - const size = ((v as string)?.length ?? 0) + ((timezoneOffset as string)?.length ?? 0) - this.context.memoryLimit.use(size) const date = parseDate(v, this.context.opts, timezoneOffset) if (!date) return v format = toValue(format) format = isNil(format) ? this.context.opts.dateFormat : stringify(format) - this.context.memoryLimit.use(format.length) - return strftime(date, format, this.context.memoryLimit) + return strftime(date, format) } export function date_to_xmlschema (this: FilterImpl, v: string | Date) { @@ -32,14 +29,13 @@ export function date_to_long_string (this: FilterImpl, v: string | Date, type?: function stringify_date (this: FilterImpl, v: string | Date, month_type: string, type?: string, style?: string) { const date = parseDate(v, this.context.opts) if (!date) return v - const ml = this.context.memoryLimit if (type === 'ordinal') { const d = date.getDate() return style === 'US' - ? strftime(date, `${month_type} ${d}%q, %Y`, ml) - : strftime(date, `${d}%q ${month_type} %Y`, ml) + ? strftime(date, `${month_type} ${d}%q, %Y`) + : strftime(date, `${d}%q ${month_type} %Y`) } - return strftime(date, `%d ${month_type} %Y`, ml) + return strftime(date, `%d ${month_type} %Y`) } function parseDate (v: string | Date, opts: NormalizedFullOptions, timezoneOffset?: number | string): LiquidDate | undefined { diff --git a/src/filters/html.ts b/src/filters/html.ts index b52f1b3da..01cd67d65 100644 --- a/src/filters/html.ts +++ b/src/filters/html.ts @@ -18,7 +18,6 @@ const unescapeMap: Record = { export function escape (this: FilterImpl, str: string) { str = stringify(str) - this.context.memoryLimit.use(str.length) return str.replace(/&|<|>|"|'/g, m => escapeMap[m]) } @@ -28,7 +27,6 @@ export function xml_escape (this: FilterImpl, str: string) { function unescape (this: FilterImpl, str: string) { str = stringify(str) - this.context.memoryLimit.use(str.length) return str.replace(/&(amp|lt|gt|#34|#39);/g, m => unescapeMap[m]) } @@ -38,7 +36,6 @@ export function escape_once (this: FilterImpl, str: string) { export function newline_to_br (this: FilterImpl, v: string) { const str = stringify(v) - this.context.memoryLimit.use(str.length) return str.replace(/\r?\n/gm, '
\n') } @@ -46,7 +43,6 @@ export function newline_to_br (this: FilterImpl, v: string) { // equivalent is O(n^2) in V8 on unclosed openers. export function strip_html (this: FilterImpl, v: string) { const str = stringify(v) - this.context.memoryLimit.use(str.length) const blocks = new Map([[''], [''], [''], ['<', '>']]) let out = '' let i = 0 diff --git a/src/filters/misc.ts b/src/filters/misc.ts index 447376550..33f428953 100644 --- a/src/filters/misc.ts +++ b/src/filters/misc.ts @@ -1,68 +1,42 @@ -import { isFalsy } from '../render/boolean' -import { identify, isArray, isString, toValue } from '../util/underscore' -import { FilterImpl } from '../template' - -function chargeJsonReplacerValue (memoryLimit: { use(count: number): void }, val: unknown) { - if (typeof val === 'string') { - memoryLimit.use(val.length) - } else if (val === null || typeof val === 'number' || typeof val === 'boolean') { - memoryLimit.use(JSON.stringify(val).length) - } else if (Array.isArray(val)) { - memoryLimit.use(val.length + 1) - } else if (typeof val === 'object') { - memoryLimit.use(2) - } -} - +import { isFalsy } from '../render/boolean' +import { identify, isArray, isString, toValue } from '../util/underscore' +import { FilterImpl } from '../template' function defaultFilter (this: FilterImpl, value: T1, defaultValue: T2, ...args: Array<[string, any]>): T1 | T2 { - value = toValue(value) - if (isArray(value) || isString(value)) return value.length ? value : defaultValue - if (value === false && (new Map(args)).get('allow_false')) return false as T1 + value = toValue(value) + if (isArray(value) || isString(value)) return value.length ? value : defaultValue + if (value === false && (new Map(args)).get('allow_false')) return false as T1 return isFalsy(value, this.context) ? defaultValue : value -} - +} function json (this: FilterImpl, value: any, space = 0) { - const memoryLimit = this.context.memoryLimit - return JSON.stringify(value, (_key, val) => { - chargeJsonReplacerValue(memoryLimit, val) - return val - }, space) -} - + return JSON.stringify(value, undefined, space) +} function inspect (this: FilterImpl, value: any, space = 0) { - const memoryLimit = this.context.memoryLimit - const ancestors: object[] = [] + const ancestors: object[] = [] return JSON.stringify(value, function (this: unknown, _key: unknown, value: any) { if (typeof value !== 'object' || value === null) { - chargeJsonReplacerValue(memoryLimit, value) return value - } - // `this` is the object that value is contained in, i.e., its direct parent. - while (ancestors.length > 0 && ancestors[ancestors.length - 1] !== this) ancestors.pop() + } + // `this` is the object that value is contained in, i.e., its direct parent. + while (ancestors.length > 0 && ancestors[ancestors.length - 1] !== this) ancestors.pop() if (ancestors.includes(value)) { - memoryLimit.use('[Circular]'.length) return '[Circular]' - } - ancestors.push(value) - chargeJsonReplacerValue(memoryLimit, value) + } + ancestors.push(value) return value }, space) -} - +} function to_integer (value: any) { return Number(value) -} - -const raw = { - raw: true, - handler: identify -} - -export default { - default: defaultFilter, - raw, - jsonify: json, - to_integer, - json, - inspect -} +} +const raw = { + raw: true, + handler: identify +} +export default { + default: defaultFilter, + raw, + jsonify: json, + to_integer, + json, + inspect +} diff --git a/src/filters/string.ts b/src/filters/string.ts index d0c59708e..3d19ace6a 100644 --- a/src/filters/string.ts +++ b/src/filters/string.ts @@ -22,7 +22,6 @@ export function append (this: FilterImpl, v: string, arg: string) { assert(arguments.length === 2, 'append expect 2 arguments') const lhs = stringify(v) const rhs = stringify(arg) - this.context.memoryLimit.use(lhs.length + rhs.length) return lhs + rhs } @@ -30,16 +29,13 @@ export function prepend (this: FilterImpl, v: string, arg: string) { assert(arguments.length === 2, 'prepend expect 2 arguments') const lhs = stringify(v) const rhs = stringify(arg) - this.context.memoryLimit.use(lhs.length + rhs.length) return rhs + lhs } export function lstrip (this: FilterImpl, v: string, chars?: string) { const str = stringify(v) - this.context.memoryLimit.use(str.length) if (chars) { chars = stringify(chars) - this.context.memoryLimit.use(chars.length) for (let i = 0, set = new Set(chars); i < str.length; i++) { if (!set.has(str[i])) return str.slice(i) } @@ -50,34 +46,29 @@ export function lstrip (this: FilterImpl, v: string, chars?: string) { export function downcase (this: FilterImpl, v: string) { const str = stringify(v) - this.context.memoryLimit.use(str.length) return str.toLowerCase() } export function upcase (this: FilterImpl, v: string) { const str = stringify(v) - this.context.memoryLimit.use(str.length) return stringify(str).toUpperCase() } export function remove (this: FilterImpl, v: string, arg: string) { const str = stringify(v) arg = stringify(arg) - this.context.memoryLimit.use(str.length + arg.length) return str.split(arg).join('') } export function remove_first (this: FilterImpl, v: string, l: string) { const str = stringify(v) l = stringify(l) - this.context.memoryLimit.use(str.length + l.length) return str.replace(l, '') } export function remove_last (this: FilterImpl, v: string, l: string) { const str = stringify(v) const pattern = stringify(l) - this.context.memoryLimit.use(str.length + pattern.length) const index = str.lastIndexOf(pattern) if (index === -1) return str return str.substring(0, index) + str.substring(index + pattern.length) @@ -85,10 +76,8 @@ export function remove_last (this: FilterImpl, v: string, l: string) { export function rstrip (this: FilterImpl, str: string, chars?: string) { str = stringify(str) - this.context.memoryLimit.use(str.length) if (chars) { chars = stringify(chars) - this.context.memoryLimit.use(chars.length) for (let i = str.length - 1, set = new Set(chars); i >= 0; i--) { if (!set.has(str[i])) return str.slice(0, i + 1) } @@ -99,7 +88,6 @@ export function rstrip (this: FilterImpl, str: string, chars?: string) { export function split (this: FilterImpl, v: string, arg: string) { const str = stringify(v) - this.context.memoryLimit.use(str.length) const arr = str.split(stringify(arg)) // align to ruby split, which is the behavior of shopify/liquid // see: https://ruby-doc.org/core-2.4.0/String.html#method-i-split @@ -109,10 +97,8 @@ export function split (this: FilterImpl, v: string, arg: string) { export function strip (this: FilterImpl, v: string, chars?: string) { const str = stringify(v) - this.context.memoryLimit.use(str.length) if (chars) { const set = new Set(stringify(chars)) - this.context.memoryLimit.use(set.size) let i = 0 let j = str.length - 1 while (set.has(str[i])) i++ @@ -124,13 +110,11 @@ export function strip (this: FilterImpl, v: string, chars?: string) { export function strip_newlines (this: FilterImpl, v: string) { const str = stringify(v) - this.context.memoryLimit.use(str.length) return str.replace(/\r?\n/gm, '') } export function capitalize (this: FilterImpl, str: string) { str = stringify(str) - this.context.memoryLimit.use(str.length) return str.charAt(0).toUpperCase() + str.slice(1).toLowerCase() } @@ -139,8 +123,6 @@ export function replace (this: FilterImpl, v: string, pattern: string, replaceme pattern = stringify(pattern) replacement = stringify(replacement) const parts = str.split(pattern) - const outputSize = str.length + (parts.length - 1) * (replacement.length - pattern.length) - this.context.memoryLimit.use(outputSize) return parts.join(replacement) } @@ -148,7 +130,6 @@ export function replace_first (this: FilterImpl, v: string, arg1: string, arg2: const str = stringify(v) arg1 = stringify(arg1) arg2 = stringify(arg2) - this.context.memoryLimit.use(str.length + arg1.length + arg2.length) return str.replace(arg1, () => arg2) } @@ -156,7 +137,6 @@ export function replace_last (this: FilterImpl, v: string, arg1: string, arg2: s const str = stringify(v) const pattern = stringify(arg1) const replacement = stringify(arg2) - this.context.memoryLimit.use(str.length + pattern.length + replacement.length) const index = str.lastIndexOf(pattern) if (index === -1) return str return str.substring(0, index) + replacement + str.substring(index + pattern.length) @@ -165,7 +145,6 @@ export function replace_last (this: FilterImpl, v: string, arg1: string, arg2: s export function truncate (this: FilterImpl, v: string, l = 50, o = '...') { const str = stringify(v) o = stringify(o) - this.context.memoryLimit.use(str.length + o.length) if (str.length <= l) return v return str.substring(0, l - o.length) + o } @@ -173,7 +152,6 @@ export function truncate (this: FilterImpl, v: string, l = 50, o = '...') { export function truncatewords (this: FilterImpl, v: string, words = 15, o = '...') { const str = stringify(v) o = stringify(o) - this.context.memoryLimit.use(str.length + o.length) const arr = str.split(/\s+/) if (words <= 0) words = 1 let ret = arr.slice(0, words).join(' ') @@ -183,13 +161,11 @@ export function truncatewords (this: FilterImpl, v: string, words = 15, o = '... export function normalize_whitespace (this: FilterImpl, v: string) { const str = stringify(v) - this.context.memoryLimit.use(str.length) return str.replace(/\s+/g, ' ') } export function number_of_words (this: FilterImpl, input: string, mode?: 'cjk' | 'auto') { const str = stringify(input) - this.context.memoryLimit.use(str.length) input = str.trim() if (!input) return 0 switch (mode) { @@ -209,9 +185,6 @@ export function number_of_words (this: FilterImpl, input: string, mode?: 'cjk' | export function array_to_sentence_string (this: FilterImpl, array: unknown[], connector = 'and') { connector = stringify(connector) - let outputSize = connector.length + array.length * 2 - for (let i = 0; i < array.length; i++) outputSize += stringify(array[i]).length - this.context.memoryLimit.use(outputSize) switch (array.length) { case 0: return '' diff --git a/src/liquid-options.ts b/src/liquid-options.ts index f41ba1ec8..03531b0be 100644 --- a/src/liquid-options.ts +++ b/src/liquid-options.ts @@ -89,8 +89,6 @@ export interface LiquidOptions { parseLimit?: number; /** For DoS handling, limit total time (in ms) for each `render()` call. */ renderLimit?: number; - /** For DoS handling, limit new objects creation, including array concat/join/strftime, etc. A typical PC can handle 1e9 (1G) memory without issue. */ - memoryLimit?: number; } export interface RenderOptions { @@ -114,8 +112,6 @@ export interface RenderOptions { templateLimit?: number; /** For DoS handling, limit total time (in ms) for each `render()` call. */ renderLimit?: number; - /** For DoS handling, limit new objects creation, including array concat/join/strftime, etc. A typical PC can handle 1e9 (1G) memory without issue.. */ - memoryLimit?: number; } export interface RenderFileOptions extends RenderOptions { @@ -161,7 +157,6 @@ export interface NormalizedFullOptions extends NormalizedOptions { operators: Operators; parseLimit: number; renderLimit: number; - memoryLimit: number; } export const defaultOptions: NormalizedFullOptions = { @@ -194,7 +189,6 @@ export const defaultOptions: NormalizedFullOptions = { lenientIf: false, globals: {}, operators: defaultOperators, - memoryLimit: Infinity, parseLimit: Infinity, renderLimit: Infinity } diff --git a/src/render/expression.ts b/src/render/expression.ts index 62041ae17..a05aca395 100644 --- a/src/render/expression.ts +++ b/src/render/expression.ts @@ -67,7 +67,6 @@ export function evalQuotedToken (token: QuotedToken) { function * evalRangeToken (token: RangeToken, ctx: Context) { const low: number = yield evalToken(token.lhs, ctx) const high: number = yield evalToken(token.rhs, ctx) - ctx.memoryLimit.use(high - low + 1) return range(+low, +high + 1) } diff --git a/src/util/strftime.ts b/src/util/strftime.ts index 8bf8b09d1..732b50dd1 100644 --- a/src/util/strftime.ts +++ b/src/util/strftime.ts @@ -1,13 +1,11 @@ import { changeCase, padStart, padEnd } from './underscore' import { LiquidDate } from './liquid-date' -import type { Limiter } from './limiter' const rFormat = /%([-_0^#:]+)?(\d+)?([EO])?(.)/ interface FormatOptions { flags: Record; width?: string; modifier?: string; - memoryLimit?: Pick; } // prototype extensions @@ -99,7 +97,6 @@ const formatCodes: Record = { N: (d: LiquidDate, opts: FormatOptions) => { const width = Number(opts.width) || 9 const str = String(d.getMilliseconds()).slice(0, width) - opts.memoryLimit?.use(width - str.length) return padEnd(str, width, '0') }, p: (d: LiquidDate) => (d.getHours() < 12 ? 'AM' : 'PM'), @@ -123,25 +120,25 @@ const formatCodes: Record = { } formatCodes.h = formatCodes.b -export function strftime (d: LiquidDate, formatStr: string, memoryLimit?: Pick) { +export function strftime (d: LiquidDate, formatStr: string) { let output = '' let remaining = formatStr let match while ((match = rFormat.exec(remaining))) { output += remaining.slice(0, match.index) remaining = remaining.slice(match.index + match[0].length) - output += format(d, match, memoryLimit) + output += format(d, match) } return output + remaining } -function format (d: LiquidDate, match: RegExpExecArray, memoryLimit?: Pick) { +function format (d: LiquidDate, match: RegExpExecArray) { const [input, flagStr = '', width, modifier, conversion] = match const convert = formatCodes[conversion] if (!convert) return input const flags: Record = {} for (const flag of flagStr) flags[flag] = true - let ret = String(convert(d, { flags, width, modifier, memoryLimit })) + let ret = String(convert(d, { flags, width, modifier })) let padChar = padSpaceChars.has(conversion) ? ' ' : '0' let padWidth = Number(width) || padWidths[conversion] || 0 if (flags['^']) ret = ret.toUpperCase() @@ -149,6 +146,5 @@ function format (d: LiquidDate, match: RegExpExecArray, memoryLimit?: Pick { - const engine = new Liquid({ - memoryLimit: 1e5 - }) - const tpl = `{% for i in (1..1000000000) %} {{'a'}} {% endfor %}` - expect(() => engine.parseAndRenderSync(tpl)).toThrow('memory alloc limit exceeded, line:1, col:1') - }) it('group_by_exp fails with object as input #785', () => { const site = { tags: { diff --git a/test/integration/filters/date.spec.ts b/test/integration/filters/date.spec.ts index 992876906..3d3a683d4 100644 --- a/test/integration/filters/date.spec.ts +++ b/test/integration/filters/date.spec.ts @@ -204,31 +204,11 @@ describe('filters/date', function () { return test('{{ "1990-12-31T23:00:00Z" | date: "%Y-%m-%dT%H:%M:%S" }}', '1991-01-01T04:30:00', undefined, optsWithDateFormat) }) }) - describe('strftime width / memoryLimit', () => { - it('should charge memoryLimit for huge numeric strftime widths', () => { - const liquid = new Liquid({ memoryLimit: 500 }) - expect(() => liquid.parseAndRenderSync('{{ d | date: f }}', { d: 'now', f: '%5000000d' })) - .toThrow('memory alloc limit exceeded') - }) - it('should charge memoryLimit for array format PoC', () => { - const liquid = new Liquid({ memoryLimit: 50, renderLimit: 1e9 }) - expect(() => liquid.parseAndRenderSync('{{ d | date: f }}', { d: 'now', f: ['a'.repeat(2000000)] })) - .toThrow('memory alloc limit exceeded') - }) - it('should charge memoryLimit for object toString format PoC', () => { - const liquid = new Liquid({ memoryLimit: 50, renderLimit: 1e9 }) - const huge = 'a'.repeat(2000000) - const f = { toString: () => huge } - expect(() => liquid.parseAndRenderSync('{{ d | date: f }}', { d: 'now', f })) - .toThrow('memory alloc limit exceeded') - }) - it('should honor numeric strftime pad width when memoryLimit allows', () => { - const liquid = new Liquid({ memoryLimit: 1e7 }) + describe('strftime width', () => { + it('should honor numeric strftime pad width', () => { + const liquid = new Liquid() const out = liquid.parseAndRenderSync('{{ d | date: f }}', { d: 'now', f: '%5000d' }) expect(out.length).toBe(5000) - const tight = new Liquid({ memoryLimit: 100 }) - expect(() => tight.parseAndRenderSync('{{ d | date: f }}', { d: 'now', f: '%5000d' })) - .toThrow('memory alloc limit exceeded') }) }) }) diff --git a/test/integration/liquid/dos.spec.ts b/test/integration/liquid/dos.spec.ts index 65d78adb5..28dbfb856 100644 --- a/test/integration/liquid/dos.spec.ts +++ b/test/integration/liquid/dos.spec.ts @@ -49,93 +49,16 @@ describe('DoS related', function () { await expect(liquid.parseAndRender('{% render "small" %}')).resolves.toBe('12345') }) it('should enforce renderLimit when for body has no template nodes', () => { - const liquid = new Liquid({ memoryLimit: 1e9, renderLimit: 1 }) + const liquid = new Liquid({ renderLimit: 1 }) expect(() => liquid.parseAndRenderSync('{%- for i in (1..5000000) -%}{%- endfor -%}', {})) .toThrow('template render limit exceeded') }) it('should enforce renderLimit when tablerow body has no template nodes', () => { - const liquid = new Liquid({ memoryLimit: 1e9, renderLimit: 1 }) + const liquid = new Liquid({ renderLimit: 1 }) expect(() => liquid.parseAndRenderSync('{%- tablerow i in (1..1000000) cols:1 -%}{%- endtablerow -%}', {})) .toThrow('template render limit exceeded') }) }) - describe('#memoryLimit', () => { - it('should throw for too many array creation in filters', async () => { - const array = Array(1e3).fill(0) - const liquid = new Liquid({ memoryLimit: 100 }) - await expect(liquid.parseAndRender('{{ array | slice: 0, 3 | join }}', { array })).resolves.toBe('0 0 0') - await expect(liquid.parseAndRender('{{ array | slice: 0, 300 | join }}', { array })).rejects.toThrow('memory alloc limit exceeded, line:1, col:1') - }) - it('should support reset when calling render', async () => { - const array = Array(1e3).fill(0) - const liquid = new Liquid({ memoryLimit: 100 }) - await expect(liquid.parseAndRender('{{ array | slice: 0, 300 | join }}', { array })).rejects.toThrow('memory alloc limit exceeded, line:1, col:1') - await expect(liquid.parseAndRender('{{ array | slice: 0, 300 | join }}', { array }, { memoryLimit: 1e3 })).resolves.toBe(Array(300).fill(0).join(' ')) - }) - it('should throw for too many array iteration in tags', async () => { - const array = ['a'] - const liquid = new Liquid({ memoryLimit: 100 }) - const src = '{% for i in (1..count) %}{% assign array = array | concat: array %}{% endfor %}{{ array | join }}' - await expect(liquid.parseAndRender(src, { array, count: 3 })).resolves.toBe('a a a a a a a a') - await expect(liquid.parseAndRender(src, { array, count: 100 })).rejects.toThrow('memory alloc limit exceeded, line:1, col:26') - }) - it('should charge pop allocation to memoryLimit', async () => { - const array = Array(1e3).fill(0) - const liquid = new Liquid({ memoryLimit: 100 }) - await expect(liquid.parseAndRender('{{ array | pop | size }}', { array })).rejects.toThrow('memory alloc limit exceeded') - }) - it('should charge sample allocation to memoryLimit', async () => { - const array = Array(1e3).fill(0) - const liquid = new Liquid({ memoryLimit: 100 }) - await expect(liquid.parseAndRender('{{ array | sample: 1 | size }}', { array })).rejects.toThrow('memory alloc limit exceeded') - }) - it('should charge join by produced output size, not element count', () => { - const array = ['a'.repeat(100), 'b'.repeat(100)] - const liquid = new Liquid({ memoryLimit: 100 }) - expect(() => liquid.parseAndRenderSync('{{ array | join: "" }}', { array })) - .toThrow('memory alloc limit exceeded') - }) - it('should allow join within memoryLimit', () => { - const array = ['a'.repeat(20), 'b'.repeat(20)] - const liquid = new Liquid({ memoryLimit: 100 }) - expect(liquid.parseAndRenderSync('{{ array | join: "" }}', { array })).toBe('a'.repeat(20) + 'b'.repeat(20)) - }) - it('should prevent concat doubling from bypassing join memoryLimit', () => { - const liquid = new Liquid({ memoryLimit: 1e4 }) - const src = '{%- assign a = s | split: "NOSEP" -%}' + - '{%- assign a = a | concat: a -%}{%- assign a = a | concat: a -%}{%- assign a = a | concat: a -%}' + - '{{ a | join: "" | size }}' - expect(() => liquid.parseAndRenderSync(src, { s: 'a'.repeat(5000) })) - .toThrow('memory alloc limit exceeded') - }) - it('should charge array_to_sentence_string by produced output size', () => { - const array = ['a'.repeat(100), 'b'.repeat(100), 'c'.repeat(100)] - const liquid = new Liquid({ memoryLimit: 100 }) - expect(() => liquid.parseAndRenderSync('{{ array | array_to_sentence_string }}', { array })) - .toThrow('memory alloc limit exceeded') - }) - it('should charge json serialization of concat-doubled arrays', () => { - const liquid = new Liquid({ memoryLimit: 1e4 }) - const src = '{%- assign a = s | split: "NOSEP" -%}' + - '{%- assign a = a | concat: a -%}{%- assign a = a | concat: a -%}{%- assign a = a | concat: a -%}' + - '{{ a | json | size }}' - expect(() => liquid.parseAndRenderSync(src, { s: 'a'.repeat(5000) })) - .toThrow('memory alloc limit exceeded') - }) - it('should charge inspect serialization of concat-doubled arrays', () => { - const liquid = new Liquid({ memoryLimit: 1e4 }) - const src = '{%- assign a = s | split: "NOSEP" -%}' + - '{%- assign a = a | concat: a -%}{%- assign a = a | concat: a -%}{%- assign a = a | concat: a -%}' + - '{{ a | inspect | size }}' - expect(() => liquid.parseAndRenderSync(src, { s: 'a'.repeat(5000) })) - .toThrow('memory alloc limit exceeded') - }) - it('should charge strip_html input length to memoryLimit', () => { - const liquid = new Liquid({ memoryLimit: 100 }) - expect(() => liquid.parseAndRenderSync('{{ s | strip_html }}', { s: 'a'.repeat(200) })) - .toThrow('memory alloc limit exceeded') - }) - }) describe('strip_html ReDoS', () => { // Regression for O(n^2) backtracking on unclosed `