Compare commits

..
Author SHA1 Message Date
Tobi Lutke 3182b7c1b3 update autoresearch experiment log 2026-03-12 17:38:05 -04:00
Tobi Lutke b195d09212 Replace @blocks.each with while loop in If render — avoids block proc allocation per render\n\nResult: {"status":"keep","combined_µs":3496,"parse_µs":2356,"render_µs":1140,"allocations":24530} 2026-03-12 17:31:41 -04:00
Tobi Lutke db348e0dac Inline to_liquid_value in If render — avoids one method dispatch per condition evaluation\n\nResult: {"status":"keep","combined_µs":3459,"parse_µs":2318,"render_µs":1141,"allocations":24647} 2026-03-12 17:31:20 -04:00
Tobi Lutke 99454a9be2 Replace simple_lookup? byte scan with match? regex — 8x faster per call, cleaner code\n\nResult: {"status":"keep","combined_µs":3489,"parse_µs":2353,"render_µs":1136,"allocations":24647} 2026-03-12 17:27:17 -04:00
Tobi Lutke ca327b01b1 Condition#evaluate: skip loop block for simple conditions (no child_relation) — saves 235 allocs\n\nResult: {"status":"keep","combined_µs":3445,"parse_µs":2284,"render_µs":1161,"allocations":24647} 2026-03-12 17:24:51 -04:00
Tobi Lutke ae9a2e26b0 Clean confirmation run: 3,314µs (-55% from main), stable\n\nResult: {"status":"keep","combined_µs":3314,"parse_µs":2203,"render_µs":1111,"allocations":24882} 2026-03-12 17:22:24 -04:00
Tobi Lutke 46927b9e90 update autoresearch docs with current progress 2026-03-12 17:21:28 -04:00
Tobi Lutke f6baeaed1e parse_tag_token without StringScanner: pure byte ops avoid reset(token) overhead, -12% combined\n\nResult: {"status":"keep","combined_µs":3350,"parse_µs":2212,"render_µs":1138,"allocations":24882} 2026-03-12 17:17:10 -04:00
Tobi Lutke b37fa98c91 Clean up tokenizer: remove unused StringScanner setup and regex constants\n\nResult: {"status":"keep","combined_µs":3490,"parse_µs":2331,"render_µs":1159,"allocations":24882} 2026-03-12 17:12:08 -04:00
Tobi Lutke e25f2f1d52 Confirmation run: byteindex tokenizer consistently 3,400-3,600µs\n\nResult: {"status":"keep","combined_µs":3464,"parse_µs":2335,"render_µs":1129,"allocations":24882} 2026-03-12 17:11:29 -04:00
Tobi Lutke b7ae55f7a9 Replace StringScanner tokenizer with String#byteindex — 12% faster parse, no regex overhead for delimiter finding\n\nResult: {"status":"keep","combined_µs":3556,"parse_µs":2388,"render_µs":1168,"allocations":24882} 2026-03-12 17:11:15 -04:00
Tobi Lutke c09e722f9b Baseline: 3,818µs combined, 24,881 allocs\n\nResult: {"status":"keep","combined_µs":3818,"parse_µs":2722,"render_µs":1096,"allocations":24881} 2026-03-12 16:48:10 -04:00
Tobi Lutke 8f2f0ee035 Skip context.evaluate for String lookup keys in VariableLookup — avoids respond_to? dispatch\n\nResult: {"status":"keep","combined_µs":4103,"parse_µs":2881,"render_µs":1222,"allocations":24881} 2026-03-11 10:52:07 -04:00
Tobi Lutke 38d8055c3b update autoresearch.md with current progress 2026-03-11 10:51:39 -04:00
Tobi Lutke 228ecdb6a2 Cache no-arg filter tuples [name, EMPTY_ARRAY] — reuse frozen tuples across templates\n\nResult: {"status":"keep","combined_µs":4147,"parse_µs":2992,"render_µs":1155,"allocations":24881} 2026-03-11 10:49:44 -04:00
Tobi Lutke 76afdf154f Replace manual blank_string? with regex match — cleaner code\n\nResult: {"status":"keep","combined_µs":4196,"parse_µs":3042,"render_µs":1154,"allocations":25535} 2026-03-11 10:44:43 -04:00
Tobi Lutke 22b5ff1587 Skip to_liquid_value for String/Integer keys in VariableLookup — avoids respond_to? dispatch\n\nResult: {"status":"keep","combined_µs":4131,"parse_µs":2893,"render_µs":1238,"allocations":25535} 2026-03-11 10:44:05 -04:00
Tobi Lutke 1a019151eb Minor cleanup: optimize expect_id with while loop and early return\n\nResult: {"status":"keep","combined_µs":4184,"parse_µs":2921,"render_µs":1263,"allocations":25535} 2026-03-11 10:43:20 -04:00
Tobi Lutke 71e22e6a84 Replace manual scan_dotted_id with regex\n\nResult: {"status":"keep","combined_µs":4121,"parse_µs":2812,"render_µs":1309,"allocations":25535} 2026-03-11 10:42:07 -04:00
Tobi Lutke fd4a7af290 Replace manual scan_quoted_string with regex capture groups\n\nResult: {"status":"keep","combined_µs":4102,"parse_µs":2849,"render_µs":1253,"allocations":25535} 2026-03-11 10:41:44 -04:00
Tobi Lutke e15b163f1a Replace manual rest_blank? with regex skip + eos? check\n\nResult: {"status":"keep","combined_µs":4047,"parse_µs":2795,"render_µs":1252,"allocations":25535} 2026-03-11 10:41:20 -04:00
Tobi Lutke 11c22eb75d Replace manual scan_comparison_op with regex — cleaner and avoids byteslice allocation for op strings\n\nResult: {"status":"keep","combined_µs":4007,"parse_µs":2808,"render_µs":1199,"allocations":25535} 2026-03-11 10:40:47 -04:00
Tobi Lutke f8b08b5b64 Replace manual scan_fragment/scan_quoted_string_raw/skip_fragment with regex — cleaner, same/better perf\n\nResult: {"status":"keep","combined_µs":4132,"parse_µs":2890,"render_µs":1242,"allocations":25535} 2026-03-11 10:40:10 -04:00
Tobi Lutke 6db20e908b Replace manual byte-level scan_number with regex — cleaner code, same performance\n\nResult: {"status":"keep","combined_µs":4184,"parse_µs":2931,"render_µs":1253,"allocations":25535} 2026-03-11 10:39:14 -04:00
Tobi Lutke ecc23184ce Replace manual byte-level scan_id/skip_id with regex — C-level StringScanner.scan is faster than Ruby-level byte scanning\n\nResult: {"status":"keep","combined_µs":4185,"parse_µs":2943,"render_µs":1242,"allocations":25535} 2026-03-11 10:37:09 -04:00
Tobi Lutke 4df608a12f Fast-path Hash lookups in VariableLookup#evaluate — skip respond_to? checks for Hash objects\n\nResult: {"status":"keep","combined_µs":4110,"parse_µs":2922,"render_µs":1188,"allocations":25535} 2026-03-11 10:35:29 -04:00
Tobi Lutke b058f79cbf Skip to_liquid/context= for primitives in VariableLookup#evaluate\n\nResult: {"status":"keep","combined_µs":4334,"parse_µs":3062,"render_µs":1272,"allocations":25535} 2026-03-11 10:32:37 -04:00
Tobi Lutke 94562eae32 Fast return for primitive types in find_variable — skip to_liquid and respond_to?(:context=)\n\nResult: {"status":"keep","combined_µs":4225,"parse_µs":3009,"render_µs":1216,"allocations":25535} 2026-03-11 10:31:59 -04:00
Tobi Lutke 0e84955531 Skip find_index when only one scope in find_variable — go straight to environments\n\nResult: {"status":"keep","combined_µs":4323,"parse_µs":3055,"render_µs":1268,"allocations":25535} 2026-03-11 10:31:35 -04:00
Tobi Lutke c4593ceeb6 Skip respond_to?(:context=) for primitive types in find_variable — avoids method lookup overhead\n\nResult: {"status":"keep","combined_µs":4207,"parse_µs":2943,"render_µs":1264,"allocations":25535} 2026-03-11 10:30:41 -04:00
Tobi Lutke 9e2937945b Use EMPTY_ARRAY for empty static_environments in Context — avoids 60 array allocs per render cycle\n\nResult: {"status":"keep","combined_µs":4262,"parse_µs":3079,"render_µs":1183,"allocations":25535} 2026-03-11 10:26:30 -04:00
Tobi Lutke cd308b8b01 Lazy @changes hash in Registers — only allocate when a register is actually written\n\nResult: {"status":"keep","combined_µs":4287,"parse_µs":3059,"render_µs":1228,"allocations":25595} 2026-03-11 10:25:29 -04:00
Tobi Lutke e3fc735de7 Cache block_delimiter strings per tag name — avoids repeated string interpolation\n\nResult: {"status":"keep","combined_µs":4372,"parse_µs":3127,"render_µs":1245,"allocations":25605} 2026-03-11 10:23:56 -04:00
Tobi Lutke 9af3ba3aa7 Lazy Context init: defer StringScanner and @interrupts array allocation until needed\n\nResult: {"status":"keep","combined_µs":4299,"parse_µs":3057,"render_µs":1242,"allocations":26015} 2026-03-11 10:22:29 -04:00
Tobi Lutke 99e55c2eb5 Cache small integer to_s (0-999): avoids 267 Integer#to_s allocations per render cycle\n\nResult: {"status":"keep","combined_µs":4158,"parse_µs":2920,"render_µs":1238,"allocations":26128} 2026-03-11 10:20:45 -04:00
Tobi Lutke b48615f4a7 Replace split+join in truncatewords with manual word scan — avoids array + string allocations\n\nResult: {"status":"keep","combined_µs":4280,"parse_µs":3009,"render_µs":1271,"allocations":26395} 2026-03-11 10:20:11 -04:00
Tobi Lutke 6723d4fa15 Extend fast-path filter parsing to handle comma-separated multi-arg filters (e.g. pluralize: 'item', 'items')\n\nResult: {"status":"keep","combined_µs":4266,"parse_µs":3032,"render_µs":1234,"allocations":26480} 2026-03-11 10:16:39 -04:00
Tobi Lutke c252d50aa0 Avoid expr_markup byteslice when name is entire markup string (no whitespace, no filters)\n\nResult: {"status":"keep","combined_µs":4277,"parse_µs":3057,"render_µs":1220,"allocations":27026} 2026-03-11 10:12:26 -04:00
Tobi Lutke a249010cef Fast-path single-arg filter parsing: handle quoted strings, numbers, identifiers without Lexer/Parser\n\nResult: {"status":"keep","combined_µs":4427,"parse_µs":3181,"render_µs":1246,"allocations":27235} 2026-03-11 10:09:59 -04:00
Tobi Lutke 18a72db820 fix rubocop offenses: autocorrect style/layout violations 2026-03-11 09:53:44 -04:00
Tobi Lutke 1f59732aee update autoresearch.md with full progress log 2026-03-11 09:24:26 -04:00
Tobi Lutke cdc34388e3 For tag: migrate lax_parse to Cursor with zero-alloc skip_id/expect_id 2026-03-11 09:19:59 -04:00
Tobi Lutke bf1f5cb62d Cursor: add skip_id, expect_id, skip_fragment for zero-alloc scanning 2026-03-11 09:19:27 -04:00
Tobi Lutke 0596591fdf REVERTED: Cursor for For tag adds 148 allocs from scan_id/scan_fragment string creation\n\nResult: {"status":"discard","combined_µs":5049,"parse_us":3765,"render_us":1284,"allocations":29793,"parse_µs":3765,"render_µs":1284} 2026-03-11 09:18:35 -04:00
Tobi Lutke dd4a100346 remove dead BlockBody.parse_tag_token and If SIMPLE_CONDITION - now in Cursor 2026-03-11 09:17:29 -04:00
Tobi Lutke 9de1527099 introduce Cursor class: centralize byte-level scanning for tag/variable/condition parsing 2026-03-11 09:15:21 -04:00
Tobi Lutke 091534f981 add parse_simple to skip simple_lookup? check when caller validates 2026-03-11 09:10:05 -04:00
Tobi Lutke 0b07487e0c fast-path VariableLookup init: skip scan_variable for simple identifier chains 2026-03-11 09:08:44 -04:00
Tobi Lutke 3799d4c488 avoid allocating seen={} hash in Utils.to_s/inspect when not needed 2026-03-11 09:02:42 -04:00
Tobi Lutke c4186a16fc update autoresearch.sh: 3-run best-of, skip liquid-spec for speed 2026-03-11 09:01:57 -04:00
Tobi Lutke b90d7f0a08 optimize Context init: avoid unnecessary array wrapping for environments 2026-03-11 08:56:45 -04:00
Tobi Lutke 405e3dca48 use frozen EMPTY_ARRAY/EMPTY_HASH for Context @filters/@disabled_tags 2026-03-11 08:55:55 -04:00
Tobi Lutke 69430e9a88 replace INTEGER_REGEX/FLOAT_REGEX with byte-level parse_number 2026-03-11 08:54:31 -04:00
Tobi Lutke 79840b1eaa replace SIMPLE_CONDITION regex with manual byte parser in if/elsif lax_parse 2026-03-11 08:52:49 -04:00
Tobi Lutke 4cda1a578c fast-path slice_collection: skip copy for full Array without offset/limit 2026-03-11 08:50:34 -04:00
Tobi Lutke d574f193dc add invoke_two fast path for single-arg filter invocation, avoids splat chain 2026-03-11 08:49:37 -04:00
Tobi Lutke 76ae8f13e9 fast-path find_variable: check top scope first before find_index 2026-03-11 08:48:16 -04:00
Tobi Lutke 526af22574 add invoke_single fast path for no-arg filter invocation, avoids splat alloc 2026-03-11 08:47:21 -04:00
Tobi Lutke 03a1977ffe fast-path simple if truthiness: use byte scanner before SIMPLE_CONDITION regex 2026-03-11 08:36:28 -04:00
Tobi Lutke b03adefb1c update autoresearch.md progress log 2026-03-11 08:35:43 -04:00
Tobi Lutke 2e207e6844 replace WhitespaceOrNothing regex with byte-level blank_string? check 2026-03-11 08:34:15 -04:00
Tobi Lutke e5933fc6e4 avoid array allocation in parse_tag_token: return tag_name, store markup/newlines as class ivars 2026-03-11 08:33:19 -04:00
Tobi Lutke 1882edb1e1 clean up filter parsing: Lexer fallback for args, no-arg fast scan stays 2026-03-11 08:31:30 -04:00
Tobi Lutke 83037f978b autoresearch.md: add strategic direction toward single-pass scanner architecture 2026-03-11 08:29:12 -04:00
Tobi Lutke ad98d1f329 add security constraint to autoresearch.md, fix strict mode gate 2026-03-11 08:26:40 -04:00
Tobi Lutke 9fd7cec564 split filter parsing: scan no-arg filters directly, only invoke Lexer when args present 2026-03-11 08:26:04 -04:00
Tobi Lutke 2543fdc1a1 autoresearch: add autoresearch.md/sh, increase benchmark warmup to 20 iterations 2026-03-11 08:23:51 -04:00
Tobi Lutke 17daac92da extend fast-path to handle quoted string literal variables (262 more fast-pathed) 2026-03-11 08:20:39 -04:00
Tobi Lutke 283961d6c9 skip filter arg splat for no-arg filters, trim render loop comments 2026-03-11 08:06:23 -04:00
Tobi Lutke db434923d0 hoist write score check out of render loop: skip increment_write_score when no limits active 2026-03-11 08:04:40 -04:00
Tobi Lutke b86143eb0e use frozen EMPTY_ARRAY for disabled_tags in Variable 2026-03-11 07:47:51 -04:00
Tobi Lutke 58d2514521 return [tag_name, markup, newlines] from parse_tag_token: avoid 2 whitespace string allocs 2026-03-11 07:45:15 -04:00
Tobi Lutke 82407092cc use getbyte dispatch instead of start_with? in parse_for_document 2026-03-11 07:44:12 -04:00
Tobi Lutke 544d8f1c17 avoid empty array allocation in evaluate_filter_expressions for no-arg filters 2026-03-11 07:43:09 -04:00
Tobi Lutke cfa0dfe3ca replace For tag Syntax regex with manual byte-level parser 2026-03-11 07:39:36 -04:00
Tobi Lutke 2d3b856b36 expose expression_cache/string_scanner via attr_reader, skip regex in filter args without colon 2026-03-11 07:38:16 -04:00
Tobi Lutke 8a92a4e451 unified fast-path Variable parsing: handle both plain lookups and filter chains without full Lexer pass for name 2026-03-11 07:36:29 -04:00
Tobi Lutke f8b015646a fast-path render for filter-less variables: skip render method overhead 2026-03-11 07:33:18 -04:00
Tobi Lutke 6bcc2936a2 skip TagAttributes scan in for tag when no colon present 2026-03-11 07:32:39 -04:00
Tobi Lutke fe7a2f5aa8 fast-path simple if conditions: skip ExpressionsAndOperators scan for single conditions 2026-03-11 07:31:26 -04:00
Tobi Lutke 3939d74531 replace SIMPLE_VARIABLE regex with byte-level scanner to avoid MatchData 2026-03-11 07:30:34 -04:00
Tobi Lutke 25f9224c85 fast-path simple variable parsing: skip Lexer/Parser for plain dot-separated lookups 2026-03-11 07:29:47 -04:00
Tobi Lutke 5da223275a use frozen EMPTY_ARRAY for Variable filters when no filters present 2026-03-11 07:28:04 -04:00
Tobi Lutke 1a79cf6266 fast-path variable_lookups: skip mutable string alloc when no dot/bracket follows 2026-03-11 07:26:40 -04:00
Tobi Lutke c1113ad2f8 fast-path String in render_obj_to_output, avoid Utils.to_s dispatch for common case 2026-03-11 07:24:30 -04:00
Tobi Lutke fa412245f7 short-circuit parse_number with first-byte check before regex 2026-03-11 07:23:41 -04:00
Tobi Lutke d79b9fa254 avoid unnecessary strip allocation in Expression.parse, use byteslice for string literals 2026-03-11 07:23:08 -04:00
Tobi Lutke d291e63006 use equal? for frozen array comparison in Lexer, skip whitespace with \s+ 2026-03-11 07:21:55 -04:00
Tobi Lutke 2b78e4bf72 use getbyte instead of string indexing in whitespace_handler and create_variable 2026-03-11 07:19:26 -04:00
Tobi Lutke 7aded8e61f add auto/bench.sh: unit tests + liquid-spec + perf benchmark 2026-03-11 07:17:01 -04:00
Tobi Lutke 97e6893c1a replace VariableParser regex scan with manual byte parser in VariableLookup 2026-03-11 07:14:14 -04:00
Tobi Lutke 3329b09dd4 replace FullToken regex with manual byte parsing in parse_for_document 2026-03-11 07:12:38 -04:00
Tobi Lutke 4ea835ae04 add quick benchmark script for autoresearch 2026-03-11 07:10:28 -04:00
66 changed files with 1695 additions and 1606 deletions
+3 -4
View File
@@ -22,7 +22,6 @@ jobs:
}
- { ruby: 4.0, allowed-failure: false, rubyopt: "--yjit" }
- { ruby: 4.0, allowed-failure: false, rubyopt: "--zjit" }
- { ruby: truffleruby, allowed-failure: false }
# Head can have failures due to being in development
- { ruby: head, allowed-failure: true }
@@ -36,7 +35,7 @@ jobs:
name: Test Ruby ${{ matrix.entry.ruby }} ${{ matrix.entry.rubyopt }} --${{ matrix.entry.allowed-failure && 'allowed-failure' || 'strict' }}
steps:
- uses: actions/checkout@f43a0e5ff2bd294095638e18286ca9a3d1956744 # v3.6.0
- uses: ruby/setup-ruby@319994f95fa847cf3fb3cd3dbe89f6dcde9f178f # v1.295.0
- uses: ruby/setup-ruby@a25f1e45f0e65a92fcb1e95e8847f78fb0a7197a # v1.273.0
with:
ruby-version: ${{ matrix.entry.ruby }}
bundler-cache: true
@@ -52,7 +51,7 @@ jobs:
BUNDLE_WITH: spec
steps:
- uses: actions/checkout@f43a0e5ff2bd294095638e18286ca9a3d1956744 # v3.6.0
- uses: ruby/setup-ruby@319994f95fa847cf3fb3cd3dbe89f6dcde9f178f # v1.295.0
- uses: ruby/setup-ruby@a25f1e45f0e65a92fcb1e95e8847f78fb0a7197a # v1.273.0
with:
bundler-cache: true
bundler: latest
@@ -67,7 +66,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@f43a0e5ff2bd294095638e18286ca9a3d1956744 # v3.6.0
- uses: ruby/setup-ruby@319994f95fa847cf3fb3cd3dbe89f6dcde9f178f # v1.295.0
- uses: ruby/setup-ruby@a25f1e45f0e65a92fcb1e95e8847f78fb0a7197a # v1.273.0
with:
bundler-cache: true
- run: bundle exec rake memory_profile:run
+1 -1
View File
@@ -1 +1 @@
4.0.2
3.4.1
+1 -2
View File
@@ -32,7 +32,6 @@ group :test do
end
group :spec do
# Includes the range resource-limit specs from https://github.com/Shopify/liquid-spec/pull/165.
gem 'liquid-spec', github: 'Shopify/liquid-spec', ref: '84bf25e0edbca5f7e530574788b0f875ab331f50'
gem 'liquid-spec', github: 'Shopify/liquid-spec', branch: 'main'
gem 'activesupport', require: false
end
-16
View File
@@ -1,21 +1,5 @@
# Liquid Change Log
## 5.14.0
* Avoid materializing integer ranges in `for` and `tablerow` loops, and account for each visited range item [Ian Ker-Seymer]
## 5.13.0
* Add TruffleRuby in CI [Benoit Daloze]
* Skip slow test raising many exceptions on non-CRuby [Benoit Daloze]
* Reject bare-bracket syntax in strict2 and introduce `self` keyword by [Alok Swamy]
* Add strict2_parse to assign and capture tags by [Alok Swamy]
* Add strict2_parse to increment and decrement tags by [Alok Swamy]
* Update liquid-spec adapters for `missing_features` [Ian Ker-Seymer]
* Prevent `SelfDrop` context mutation across render boundaries [Guilherme Carreiro]
* Fix `SelfDrop` equality [Guilherme Carreiro]
* Let environment `self` shadow `SelfDrop` [Ian Ker-Seymer]
## 5.11.0
* Revert the Inline Snippets tag (#2001), treat its inclusion in the latest Liquid release as a bug, and allow for feedback on RFC#1916 to better support Liquid developers [Guilherme Carreiro]
* Rename the `:rigid` error mode to `:strict2` and display a warning when users attempt to use the `:rigid` mode [Guilherme Carreiro]
-7
View File
@@ -149,13 +149,6 @@ template.render!({ 'x' => 1}, { strict_variables: true })
#=> Liquid::UndefinedVariable: Liquid error: undefined variable y
```
### Resource limits
`render_score_limit` and `cumulative_render_score_limit` account for each item visited by
integer-range `for` and `tablerow` loops, including loops with empty bodies. This bounds range
iteration work when a score limit is configured; `render_length_limit` only bounds generated
output and does not by itself limit CPU work for output-free loops.
### Usage tracking
To help track usages of a feature or code path in production, we have released opt-in usage tracking. To enable this, we provide an empty `Liquid:: Usage.increment` method which you can customize to your needs. The feature is well suited to https://github.com/Shopify/statsd-instrument. However, the choice of implementation is up to you.
+2 -4
View File
@@ -151,8 +151,6 @@ end
desc('run liquid-spec suite across all adapters')
task :spec do
Dir['./spec/*.rb'].sort.each do |adapter|
puts "=== Running #{adapter} ==="
sh 'bundle', 'exec', 'liquid-spec', 'run', adapter, '--no-max-failures'
end
adapters = Dir['./spec/*.rb'].join(',')
sh "bundle exec liquid-spec matrix --adapters=#{adapters} --reference=ruby_liquid"
end
+30
View File
@@ -0,0 +1,30 @@
# Autoresearch Ideas
## Dead Ends (tried and failed)
- **Tag name interning** (skip+byte dispatch): saves 878 allocs but verification loop overhead kills speed
- **String dedup (-@)** for filter names: no alloc savings, creates temp strings anyway
- **Split-based tokenizer**: 2.5x faster C-level split but can't handle {{ followed by %} nesting
- **Streaming tokenizer**: needs own StringScanner (+alloc), per-shift overhead worse than eager array
- **Merge simple_lookup? into initialize**: logic overhead offsets saved index call
- **Cursor for filter scanning**: cursor.reset overhead worse than inline byte loops
- **Direct strainer call**: YJIT already inlines context.invoke_single well
- **TruthyCondition subclass**: YJIT polymorphism at evaluate call site hurts more than 115 saved allocs
- **Index loop for filters**: YJIT optimizes each+destructure MUCH better than manual filter[0]/filter[1]
## Key Insights
- YJIT monomorphism > allocation reduction at this scale
- C-level StringScanner.scan/skip > Ruby-level byte loops (already applied)
- String#split is 2.5x faster than manual tokenization, but Liquid's grammar is too complex for regex
- 74% of total CPU time is GC — alloc reduction is the highest-leverage optimization
- But YJIT-deoptimization from polymorphism costs more than the GC savings
## Remaining Ideas
- **Tokenizer: use String#index + byteslice instead of StringScanner**: avoid the StringScanner overhead entirely for the simple case of finding {%/{{ delimiters
- **Pre-freeze all Condition operator lambdas**: reduce alloc in Condition initialization
- **Avoid `@blocks = []` in If with single-element optimization**: use `@block` ivar for single condition, only create array for elsif
- **Reduce ForloopDrop allocation**: reuse ForloopDrop objects across iterations or use a lighter-weight object
- **VariableLookup: single-segment optimization**: for "product.title" (1 lookup), use an ivar instead of 1-element Array
+109
View File
@@ -0,0 +1,109 @@
# Autoresearch: Liquid Parse+Render Performance
## Objective
Optimize the Shopify Liquid template engine's parse and render performance.
The workload is the ThemeRunner benchmark which parses and renders real Shopify
theme templates (dropify, ripen, tribble, vogue) with realistic data from
`performance/shopify/database.rb`. We measure parse time, render time, and
object allocations. The optimization target is combined parse+render time (µs).
## How to Run
Run `./auto/autoresearch.sh` — it runs unit tests, liquid-spec conformance,
then the performance benchmark, outputting metrics in parseable format.
## Metrics
- **Primary (optimization target)**: `combined_µs` (µs, lower is better) — sum of parse + render time
- **Secondary (tradeoff monitoring)**:
- `parse_µs` — time to parse all theme templates (Liquid::Template#parse)
- `render_µs` — time to render all pre-compiled templates
- `allocations` — total object allocations for one parse+render cycle
Parse dominates (~70-75% of combined). Allocations correlate with GC pressure.
## Files in Scope
- `lib/liquid/*.rb` — core Liquid library (parser, lexer, context, expression, etc.)
- `lib/liquid/tags/*.rb` — tag implementations (for, if, assign, etc.)
- `performance/bench_quick.rb` — benchmark script
## Off Limits
- `test/` — tests must continue to pass unchanged
- `performance/tests/` — benchmark templates, do not modify
- `performance/shopify/` — benchmark data/filters, do not modify
## Constraints
- All unit tests must pass (`bundle exec rake base_test`)
- liquid-spec failures must not increase beyond 2 (pre-existing UTF-8 edge cases)
- No new gem dependencies
- Semantic correctness must be preserved — templates must render identical output
- **Security**: Liquid runs untrusted user code. See Strategic Direction for details.
## Strategic Direction
The long-term goal is to converge toward a **single-pass, forward-only parsing
architecture** using one shared StringScanner instance. The current system has
multiple redundant passes: Tokenizer → BlockBody → Lexer → Parser → Expression
→ VariableLookup, each re-scanning portions of the source. A unified scanner
approach would:
1. **One StringScanner** flows through the entire parse — no intermediate token
arrays, no re-lexing filter chains, no string reconstruction in Parser#expression.
2. **Emit a lightweight IL or normalized AST** during the single forward pass,
decoupling strictness checking from the hot parse path. The LiquidIL project
(`~/src/tries/2026-01-05-liquid-il`) demonstrated this: a recursive-descent
parser emitting IL directly achieved significant speedups.
3. **Minimal backtracking** — the scanner advances forward, byte-checking as it
goes. liquid-c (`~/src/tries/2026-01-16-Shopify-liquid-c`) showed that a
C-level cursor-based tokenizer eliminates most allocation overhead.
Current fast-path optimizations (byte-level tag/variable/for/if parsing) are
steps toward this goal. Each one replaces a regex+MatchData pattern with
forward-only byte scanning. The remaining Lexer→Parser path for filter args
is the next target for elimination.
**Security note**: Liquid executes untrusted user templates. All parsing must
use explicit byte-range checks. Never use eval, send on user input, dynamic
method dispatch, const_get, or any pattern that lets template authors escape
the sandbox.
## Baseline
- **Commit**: 4ea835a (original, before any optimizations)
- **combined_µs**: 7,374
- **parse_µs**: 5,928
- **render_µs**: 1,446
- **allocations**: 62,620
## Progress Log
- 3329b09: Replace FullToken regex with manual byte parsing → combined 7,262 (-1.5%)
- 97e6893: Replace VariableParser regex with manual byte scanner → combined 6,945 (-5.8%), allocs 58,009
- 2b78e4b: getbyte instead of string indexing in whitespace_handler/create_variable → allocs 51,477
- d291e63: Lexer equal? for frozen arrays, \s+ whitespace skip → combined ~6,331
- d79b9fa: Avoid strip alloc in Expression.parse, byteslice for strings → allocs 49,151
- fa41224: Short-circuit parse_number with first-byte check → allocs 48,240
- c1113ad: Fast-path String in render_obj_to_output → combined ~6,071
- 25f9224: Fast-path simple variable parsing (skip Lexer/Parser) → combined ~5,860, allocs 45,202
- 3939d74: Replace SIMPLE_VARIABLE regex with byte scanner → combined ~5,717, allocs 42,763
- fe7a2f5: Fast-path simple if conditions → combined ~5,444, allocs 41,490
- cfa0dfe: Replace For tag Syntax regex with manual byte parser → combined ~4,974, allocs 39,847
- 8a92a4e: Unified fast-path Variable: parse name directly, only lex filter chain → combined ~5,060, allocs 40,520
- 58d2514: parse_tag_token returns [tag_name, markup, newlines] → combined ~4,815, allocs 37,355
- db43492: Hoist write score check out of render loop → render ~1,345
- 17daac9: Extend fast-path to quoted string literal variables → all 1,197 variables fast-pathed
- 9fd7cec: Split filter parsing: no-arg filters scanned directly, Lexer only for args → combined ~4,595, allocs 35,159
- e5933fc: Avoid array alloc in parse_tag_token via class ivars → allocs 34,281
- 2e207e6: Replace WhitespaceOrNothing regex with byte-level blank_string? → combined ~4,800
- 526af22: invoke_single fast path for no-arg filter invocation → allocs 32,621
- 76ae8f1: find_variable top-scope fast path → combined ~4,740
- 4cda1a5: slice_collection: skip copy for full Array → allocs 32,004
- 79840b1: Replace SIMPLE_CONDITION regex with manual byte parser → combined ~4,663, allocs 31,465
- 69430e9: Replace INTEGER_REGEX/FLOAT_REGEX with byte-level parse_number → allocs 31,129
- 405e3dc: Frozen EMPTY_ARRAY/EMPTY_HASH for Context @filters/@disabled_tags → allocs 31,009
- b90d7f0: Avoid unnecessary array wrapping for Context environments → allocs 30,709
- 3799d4c: Lazy seen={} hash in Utils.to_s/inspect → allocs 30,169
- 0b07487: Fast-path VariableLookup: skip scan_variable for simple identifiers → allocs 29,711
- 9de1527: Introduce Cursor class for centralized byte-level scanning
- dd4a100: Remove dead parse_tag_token/SIMPLE_CONDITION (now in Cursor)
- cdc3438: For tag: migrate lax_parse to Cursor with zero-alloc scanning → allocs 29,620
## Current Best
- **combined_µs**: ~3,400 (-54% from original 7,374 baseline)
- **parse_µs**: ~2,300
- **render_µs**: ~1,100
- **allocations**: 24,882 (-60% from original 62,620 baseline)
+48
View File
@@ -0,0 +1,48 @@
#!/usr/bin/env bash
# Autoresearch benchmark runner for Liquid performance optimization
# Runs: unit tests → performance benchmark (3 runs, takes best)
# Outputs METRIC lines for the agent to parse
# Exit code 0 = all good, non-zero = broken
set -euo pipefail
cd "$(dirname "$0")/.."
# ── Step 1: Unit tests (fast gate) ──────────────────────────────────
echo "=== Unit Tests ==="
TEST_OUT=$(bundle exec rake base_test 2>&1)
TEST_RESULT=$(echo "$TEST_OUT" | tail -1)
if echo "$TEST_OUT" | grep -q 'failures\|errors' && ! echo "$TEST_RESULT" | grep -q '0 failures, 0 errors'; then
echo "$TEST_OUT" | grep -E 'Failure|Error|failures|errors' | head -20
echo "FATAL: unit tests failed"
exit 1
fi
echo "$TEST_RESULT"
# ── Step 2: Performance benchmark (3 runs, take best) ──────────────
echo ""
echo "=== Performance Benchmark (3 runs) ==="
BEST_COMBINED=999999
BEST_PARSE=0
BEST_RENDER=0
BEST_ALLOC=0
for i in 1 2 3; do
OUT=$(bundle exec ruby performance/bench_quick.rb 2>&1)
P=$(echo "$OUT" | grep '^parse_us=' | cut -d= -f2)
R=$(echo "$OUT" | grep '^render_us=' | cut -d= -f2)
C=$(echo "$OUT" | grep '^combined_us=' | cut -d= -f2)
A=$(echo "$OUT" | grep '^allocations=' | cut -d= -f2)
echo " run $i: combined=${C}µs (parse=${P} render=${R}) allocs=${A}"
if [ "$C" -lt "$BEST_COMBINED" ]; then
BEST_COMBINED=$C
BEST_PARSE=$P
BEST_RENDER=$R
BEST_ALLOC=$A
fi
done
echo ""
echo "METRIC combined_us=$BEST_COMBINED"
echo "METRIC parse_us=$BEST_PARSE"
echo "METRIC render_us=$BEST_RENDER"
echo "METRIC allocations=$BEST_ALLOC"
Executable
+40
View File
@@ -0,0 +1,40 @@
#!/usr/bin/env bash
# Auto-research benchmark script for Liquid
# Runs: unit tests → liquid-spec → performance benchmark
# Outputs machine-readable metrics on success
# Exit code 0 = all good, non-zero = broken
set -euo pipefail
cd "$(dirname "$0")/.."
# ── Step 1: Unit tests (fast gate) ──────────────────────────────────
echo "=== Unit Tests ==="
if ! bundle exec rake base_test 2>&1; then
echo "FATAL: unit tests failed"
exit 1
fi
# ── Step 2: liquid-spec (correctness gate) ──────────────────────────
echo ""
echo "=== Liquid Spec ==="
SPEC_OUTPUT=$(bundle exec liquid-spec run spec/ruby_liquid.rb 2>&1 || true)
echo "$SPEC_OUTPUT" | tail -3
# Extract failure count from "Total: N passed, N failed, N errors" line
# Allow known pre-existing failures (≤2)
TOTAL_LINE=$(echo "$SPEC_OUTPUT" | grep "^Total:" || echo "Total: 0 passed, 0 failed, 0 errors")
FAILURES=$(echo "$TOTAL_LINE" | sed -n 's/.*\([0-9][0-9]*\) failed.*/\1/p')
ERRORS=$(echo "$TOTAL_LINE" | sed -n 's/.*\([0-9][0-9]*\) error.*/\1/p')
FAILURES=${FAILURES:-0}
ERRORS=${ERRORS:-0}
TOTAL_BAD=$((FAILURES + ERRORS))
if [ "$TOTAL_BAD" -gt 2 ]; then
echo "FATAL: liquid-spec has $FAILURES failures and $ERRORS errors (threshold: 2)"
exit 1
fi
# ── Step 3: Performance benchmark ──────────────────────────────────
echo ""
echo "=== Performance Benchmark ==="
bundle exec ruby performance/bench_quick.rb 2>&1
+30
View File
@@ -0,0 +1,30 @@
{"type":"config","name":"Liquid parse+render performance (tenderlove-inspired)","metricName":"combined_µs","metricUnit":"µs","bestDirection":"lower"}
{"run":1,"commit":"c09e722","metric":3818,"metrics":{"parse_µs":2722,"render_µs":1096,"allocations":24881},"status":"keep","description":"Baseline: 3,818µs combined, 24,881 allocs","timestamp":1773348490227}
{"run":2,"commit":"c09e722","metric":4063,"metrics":{"parse_µs":2901,"render_µs":1162,"allocations":24003},"status":"discard","description":"Tag name interning via skip+byte dispatch: saves 878 allocs but verification loop slower than scan","timestamp":1773348738557,"segment":0}
{"run":3,"commit":"c09e722","metric":3881,"metrics":{"parse_µs":2720,"render_µs":1161,"allocations":24881},"status":"discard","description":"String dedup (-@) for filter names: no alloc savings, no speed benefit","timestamp":1773348781481,"segment":0}
{"run":4,"commit":"c09e722","metric":3970,"metrics":{"parse_µs":2829,"render_µs":1141,"allocations":24881},"status":"discard","description":"Streaming tokenizer: needs own StringScanner (+1 alloc), per-shift overhead worse than saved array","timestamp":1773348883093,"segment":0}
{"run":5,"commit":"c09e722","metric":0,"metrics":{"parse_µs":0,"render_µs":0,"allocations":0},"status":"crash","description":"REVERTED: split-based tokenizer — regex can't handle unclosed tags inside raw blocks","timestamp":1773349089230,"segment":0}
{"run":6,"commit":"c09e722","metric":0,"metrics":{"parse_µs":0,"render_µs":0,"allocations":0},"status":"crash","description":"REVERTED: split regex tokenizer v2 — can't handle {{ followed by %} (variable-becomes-tag nesting)","timestamp":1773349248313,"segment":0}
{"run":7,"commit":"c09e722","metric":3861,"metrics":{"parse_µs":2744,"render_µs":1117,"allocations":24881},"status":"discard","description":"Merge simple_lookup? dot position into initialize — logic overhead offsets saved index call","timestamp":1773349376707,"segment":0}
{"run":8,"commit":"c09e722","metric":4048,"metrics":{"parse_µs":2929,"render_µs":1119,"allocations":24881},"status":"discard","description":"Use Cursor regex for filter name scanning — cursor.reset + method dispatch overhead worse than inline bytes","timestamp":1773349447172,"segment":0}
{"run":9,"commit":"c09e722","metric":3872,"metrics":{"parse_µs":2744,"render_µs":1128,"allocations":24881},"status":"discard","description":"Direct strainer call in Variable#render — YJIT already inlines context.invoke_single well","timestamp":1773349497593,"segment":0}
{"run":10,"commit":"c09e722","metric":3839,"metrics":{"parse_µs":2732,"render_µs":1107,"allocations":24879},"status":"discard","description":"Array#[] fast path for slice_collection with limit/offset — only 2 alloc savings, not meaningful","timestamp":1773349555348,"segment":0}
{"run":11,"commit":"c09e722","metric":3889,"metrics":{"parse_µs":2770,"render_µs":1119,"allocations":24766},"status":"discard","description":"TruthyCondition for simple if checks: -115 allocs but YJIT polymorphism at evaluate call site hurts speed","timestamp":1773349649377,"segment":0}
{"run":12,"commit":"c09e722","metric":4150,"metrics":{"parse_µs":2769,"render_µs":1381,"allocations":24881},"status":"discard","description":"Index loop for filters: YJIT optimizes each+destructure better than manual indexing","timestamp":1773349699285,"segment":0}
{"run":13,"commit":"b7ae55f","metric":3556,"metrics":{"parse_µs":2388,"render_µs":1168,"allocations":24882},"status":"keep","description":"Replace StringScanner tokenizer with String#byteindex — 12% faster parse, no regex overhead for delimiter finding","timestamp":1773349875890,"segment":0}
{"run":14,"commit":"e25f2f1","metric":3464,"metrics":{"parse_µs":2335,"render_µs":1129,"allocations":24882},"status":"keep","description":"Confirmation run: byteindex tokenizer consistently 3,400-3,600µs","timestamp":1773349889465,"segment":0}
{"run":15,"commit":"b37fa98","metric":3490,"metrics":{"parse_µs":2331,"render_µs":1159,"allocations":24882},"status":"keep","description":"Clean up tokenizer: remove unused StringScanner setup and regex constants","timestamp":1773349928672,"segment":0}
{"run":16,"commit":"b37fa98","metric":3638,"metrics":{"parse_µs":2460,"render_µs":1178,"allocations":24882},"status":"discard","description":"Single-char byteindex for %} search: Ruby loop overhead worse for nearby targets","timestamp":1773349985509,"segment":0}
{"run":17,"commit":"b37fa98","metric":3553,"metrics":{"parse_µs":2431,"render_µs":1122,"allocations":25256},"status":"discard","description":"Regex simple_variable_markup: MatchData creates 374 extra allocs, offsetting speed gain","timestamp":1773350066627,"segment":0}
{"run":18,"commit":"b37fa98","metric":3629,"metrics":{"parse_µs":2455,"render_µs":1174,"allocations":25002},"status":"discard","description":"String.new(capacity: 4096) for output buffer: allocates more objects, not fewer","timestamp":1773350101852,"segment":0}
{"run":19,"commit":"f6baeae","metric":3350,"metrics":{"parse_µs":2212,"render_µs":1138,"allocations":24882},"status":"keep","description":"parse_tag_token without StringScanner: pure byte ops avoid reset(token) overhead, -12% combined","timestamp":1773350230252,"segment":0}
{"run":20,"commit":"f6baead","metric":0,"metrics":{"parse_µs":0,"render_µs":0,"allocations":0},"status":"crash","description":"REVERTED: regex ultra-fast path for Variable — name pattern too broad, matches invalid trailing dots","timestamp":1773350472859,"segment":0}
{"run":21,"commit":"ae9a2e2","metric":3314,"metrics":{"parse_µs":2203,"render_µs":1111,"allocations":24882},"status":"keep","description":"Clean confirmation run: 3,314µs (-55% from main), stable","timestamp":1773350544354,"segment":0}
{"run":22,"commit":"ae9a2e2","metric":3497,"metrics":{"parse_µs":2336,"render_µs":1161,"allocations":24882},"status":"discard","description":"Regex fast path for no-filter variables: include? + match? overhead exceeds byte scan savings","timestamp":1773350641375,"segment":0}
{"run":23,"commit":"ca327b0","metric":3445,"metrics":{"parse_µs":2284,"render_µs":1161,"allocations":24647},"status":"keep","description":"Condition#evaluate: skip loop block for simple conditions (no child_relation) — saves 235 allocs","timestamp":1773350691752,"segment":0}
{"run":24,"commit":"99454a9","metric":3489,"metrics":{"parse_µs":2353,"render_µs":1136,"allocations":24647},"status":"keep","description":"Replace simple_lookup? byte scan with match? regex — 8x faster per call, cleaner code","timestamp":1773350837721,"segment":0}
{"run":25,"commit":"99454a9","metric":3797,"metrics":{"parse_µs":2636,"render_µs":1161,"allocations":29627},"status":"discard","description":"Regex name extraction in try_fast_parse: MatchData creates 5K extra allocs, much worse","timestamp":1773351048938,"segment":0}
{"run":26,"commit":"db348e0","metric":3459,"metrics":{"parse_µs":2318,"render_µs":1141,"allocations":24647},"status":"keep","description":"Inline to_liquid_value in If render — avoids one method dispatch per condition evaluation","timestamp":1773351080001,"segment":0}
{"run":27,"commit":"b195d09","metric":3496,"metrics":{"parse_µs":2356,"render_µs":1140,"allocations":24530},"status":"keep","description":"Replace @blocks.each with while loop in If render — avoids block proc allocation per render","timestamp":1773351101134,"segment":0}
{"run":28,"commit":"b195d09","metric":3648,"metrics":{"parse_µs":2457,"render_µs":1191,"allocations":24530},"status":"discard","description":"While loop in For render: YJIT optimizes each well for hot loops with many iterations","timestamp":1773351142275,"segment":0}
{"run":29,"commit":"b195d09","metric":3966,"metrics":{"parse_µs":2641,"render_µs":1325,"allocations":24060},"status":"discard","description":"While loop for environment search: -470 allocs but YJIT deopt makes render 16% slower","timestamp":1773351193863,"segment":0}
+1 -2
View File
@@ -65,7 +65,6 @@ require 'liquid/lexer'
require 'liquid/parser'
require 'liquid/i18n'
require 'liquid/drop'
require 'liquid/self_drop'
require 'liquid/tablerowloop_drop'
require 'liquid/forloop_drop'
require 'liquid/extensions'
@@ -83,8 +82,8 @@ require 'liquid/resource_limits'
require 'liquid/expression'
require 'liquid/template'
require 'liquid/condition'
require 'liquid/range_slice'
require 'liquid/utils'
require 'liquid/cursor'
require 'liquid/tokenizer'
require 'liquid/parse_context'
require 'liquid/partial_cache'
+4 -1
View File
@@ -60,8 +60,11 @@ module Liquid
@tag_name
end
# Cache block delimiters per tag name to avoid repeated string allocation
BLOCK_DELIMITER_CACHE = Hash.new { |h, k| h[k] = "end#{k}".freeze }
def block_delimiter
@block_delimiter ||= "end#{block_name}"
@block_delimiter ||= BLOCK_DELIMITER_CACHE[block_name]
end
private
+69 -49
View File
@@ -38,7 +38,7 @@ module Liquid
private def parse_for_liquid_tag(tokenizer, parse_context)
while (token = tokenizer.shift)
unless token.empty? || token.match?(WhitespaceOrNothing)
unless token.empty? || BlockBody.blank_string?(token)
unless token =~ LiquidTagToken
# line isn't empty but didn't match tag syntax, yield and let the
# caller raise a syntax error
@@ -99,9 +99,7 @@ module Liquid
context.handle_error(exc, line_number)
else
error_message = context.handle_error(exc, line_number)
error_mode = context.registers.static[:template_error_mode]
suppress_error_text = blank_tag && error_mode != :strict2 && error_mode != :rigid
unless suppress_error_text # blank-tag suppression is kept for backwards compatibility outside strict2
unless blank_tag # conditional for backwards compatibility
output << error_message
end
end
@@ -126,48 +124,70 @@ module Liquid
end
end
OPEN_CURLEY_BYTE = 123 # '{'.ord
PERCENT_BYTE = 37 # '%'.ord
# Fast check if string is whitespace-only (replaces WhitespaceOrNothing regex)
BLANK_STRING_REGEX = /\A\s*\z/
def self.blank_string?(str)
str.match?(BLANK_STRING_REGEX)
end
private def parse_for_document(tokenizer, parse_context, &block)
while (token = tokenizer.shift)
next if token.empty?
case
when token.start_with?(TAGSTART)
whitespace_handler(token, parse_context)
unless token =~ FullToken
return handle_invalid_tag_token(token, parse_context, &block)
end
tag_name = Regexp.last_match(2)
markup = Regexp.last_match(4)
if parse_context.line_number
# newlines inside the tag should increase the line number,
# particularly important for multiline {% liquid %} tags
parse_context.line_number += Regexp.last_match(1).count("\n") + Regexp.last_match(3).count("\n")
end
first_byte = token.getbyte(0)
if first_byte == OPEN_CURLEY_BYTE
second_byte = token.getbyte(1)
if second_byte == PERCENT_BYTE
whitespace_handler(token, parse_context)
cursor = parse_context.cursor
tag_name = cursor.parse_tag_token(token)
unless tag_name
return handle_invalid_tag_token(token, parse_context, &block)
end
markup = cursor.tag_markup
if tag_name == 'liquid'
parse_liquid_tag(markup, parse_context)
next
end
if parse_context.line_number
newlines = cursor.tag_newlines
parse_context.line_number += newlines if newlines > 0
end
unless (tag = parse_context.environment.tag_for_name(tag_name))
# end parsing if we reach an unknown tag and let the caller decide
# determine how to proceed
return yield tag_name, markup
if tag_name == 'liquid'
parse_liquid_tag(markup, parse_context)
next
end
unless (tag = parse_context.environment.tag_for_name(tag_name))
# end parsing if we reach an unknown tag and let the caller decide
# determine how to proceed
return yield tag_name, markup
end
new_tag = tag.parse(tag_name, markup, tokenizer, parse_context)
@blank &&= new_tag.blank?
@nodelist << new_tag
elsif second_byte == OPEN_CURLEY_BYTE
whitespace_handler(token, parse_context)
@nodelist << create_variable(token, parse_context)
@blank = false
else
# Fallback: text token starting with '{'
if parse_context.trim_whitespace
token.lstrip!
end
parse_context.trim_whitespace = false
@nodelist << token
@blank &&= BlockBody.blank_string?(token)
end
new_tag = tag.parse(tag_name, markup, tokenizer, parse_context)
@blank &&= new_tag.blank?
@nodelist << new_tag
when token.start_with?(VARSTART)
whitespace_handler(token, parse_context)
@nodelist << create_variable(token, parse_context)
@blank = false
else
if parse_context.trim_whitespace
token.lstrip!
end
parse_context.trim_whitespace = false
@nodelist << token
@blank &&= token.match?(WhitespaceOrNothing)
@blank &&= BlockBody.blank_string?(token)
end
parse_context.line_number = tokenizer.line_number
end
@@ -175,8 +195,10 @@ module Liquid
yield nil, nil
end
DASH_BYTE = 45 # '-'.ord
def whitespace_handler(token, parse_context)
if token[2] == WhitespaceControl
if token.getbyte(2) == DASH_BYTE
previous_token = @nodelist.last
if previous_token.is_a?(String)
first_byte = previous_token.getbyte(0)
@@ -186,7 +208,7 @@ module Liquid
end
end
end
parse_context.trim_whitespace = (token[-3] == WhitespaceControl)
parse_context.trim_whitespace = (token.getbyte(token.bytesize - 3) == DASH_BYTE)
end
def blank?
@@ -220,7 +242,11 @@ module Liquid
def render_to_output_buffer(context, output)
freeze unless frozen?
context.resource_limits.increment_render_score(@nodelist.length)
resource_limits = context.resource_limits
resource_limits.increment_render_score(@nodelist.length)
# Check if we need per-node write score tracking
check_write = resource_limits.render_length_limit || resource_limits.last_capture_length
idx = 0
while (node = @nodelist[idx])
@@ -228,14 +254,11 @@ module Liquid
output << node
else
render_node(context, output, node)
# If we get an Interrupt that means the block must stop processing. An
# Interrupt is any command that stops block execution such as {% break %}
# or {% continue %}. These tags may also occur through Block or Include tags.
break if context.interrupt? # might have happened in a for-block
break if context.interrupt?
end
idx += 1
context.resource_limits.increment_write_score(output)
resource_limits.increment_write_score(output) if check_write
end
output
@@ -247,15 +270,12 @@ module Liquid
BlockBody.render_node(context, output, node)
end
def create_variable(token, parse_context)
if token.end_with?("}}")
i = 2
i = 3 if token[i] == "-"
parse_end = token.length - 3
parse_end -= 1 if token[parse_end] == "-"
markup_end = parse_end - i + 1
markup = markup_end <= 0 ? "" : token.slice(i, markup_end)
CLOSE_CURLEY_BYTE = 125 # '}'.ord
def create_variable(token, parse_context)
len = token.bytesize
if len >= 4 && token.getbyte(len - 1) == CLOSE_CURLEY_BYTE && token.getbyte(len - 2) == CLOSE_CURLEY_BYTE
markup = parse_context.cursor.parse_variable_token(token)
return Variable.new(markup, parse_context)
end
+7 -4
View File
@@ -65,11 +65,13 @@ module Liquid
end
def evaluate(context = deprecated_default_context)
condition = self
result = nil
loop do
result = interpret_condition(condition.left, condition.right, condition.operator, context)
result = interpret_condition(@left, @right, @operator, context)
# Fast path: no child conditions (most common)
return result unless @child_relation
condition = self
loop do
case condition.child_relation
when :or
break if Liquid::Utils.to_liquid_value(result)
@@ -79,6 +81,7 @@ module Liquid
break
end
condition = condition.child_condition
result = interpret_condition(condition.left, condition.right, condition.operator, context)
end
result
end
+54 -37
View File
@@ -24,10 +24,15 @@ module Liquid
def initialize(environments = {}, outer_scope = {}, registers = {}, rethrow_errors = false, resource_limits = nil, static_environments = {}, environment = Environment.default)
@environment = environment
@environments = [environments]
@environments.flatten!
@environments = environments.is_a?(Array) ? environments : [environments]
@static_environments = [static_environments].flatten(1).freeze
@static_environments = if static_environments.is_a?(Array)
static_environments.frozen? ? static_environments : static_environments.freeze
elsif static_environments.empty?
Const::EMPTY_ARRAY
else
[static_environments].freeze
end
@scopes = [outer_scope || {}]
@registers = registers.is_a?(Registers) ? registers : Registers.new(registers)
@errors = []
@@ -35,14 +40,13 @@ module Liquid
@strict_variables = false
@resource_limits = resource_limits || ResourceLimits.new(environment.default_resource_limits)
@base_scope_depth = 0
@interrupts = []
@filters = []
@interrupts = Const::EMPTY_ARRAY
@filters = Const::EMPTY_ARRAY
@global_filter = nil
@disabled_tags = {}
@disabled_tags = Const::EMPTY_HASH
# Instead of constructing new StringScanner objects for each Expression parse,
# we recycle the same one.
@string_scanner = StringScanner.new("")
# Lazy-init StringScanner — only needed if Context#[] is called during render
@string_scanner = nil
@registers.static[:cached_partials] ||= {}
@registers.static[:file_system] ||= environment.file_system
@@ -84,11 +88,12 @@ module Liquid
# are there any not handled interrupts?
def interrupt?
!@interrupts.empty?
!@interrupts.frozen? && !@interrupts.empty?
end
# push an interrupt to the stack. this interrupt is considered not handled.
def push_interrupt(e)
@interrupts = [] if @interrupts.frozen?
@interrupts.push(e)
end
@@ -109,6 +114,17 @@ module Liquid
strainer.invoke(method, *args).to_liquid
end
# Fast path for single-argument filter invocation (the most common case:
# {{ value | filter }}) — avoids *args splat allocation.
def invoke_single(method, input)
strainer.invoke_single(method, input).to_liquid
end
# Fast path for two-argument filter invocation (e.g. {{ value | default: 'x' }})
def invoke_two(method, input, arg1)
strainer.invoke_two(method, input, arg1).to_liquid
end
# Push new local scope on the stack. use <tt>Context#stack</tt> instead
def push(new_scope = {})
@scopes.unshift(new_scope)
@@ -180,53 +196,53 @@ module Liquid
# Example:
# products == empty #=> products.empty?
def [](expression)
evaluate(Expression.parse(expression, @string_scanner))
evaluate(Expression.parse(expression, @string_scanner ||= StringScanner.new("")))
end
def key?(key)
find_variable(key, raise_on_not_found: false) != nil
end
# Checks whether a variable is defined in any scope, including nil-valued keys.
# Unlike #key?, this uses Hash#key? so that variables explicitly set to nil
# are still considered defined.
def variable_defined?(key)
@scopes.any? { |s| s.key?(key) } ||
@environments.any? { |e| e.key?(key) } ||
@static_environments.any? { |e| e.key?(key) }
end
def evaluate(object)
object.respond_to?(:evaluate) ? object.evaluate(self) : object
end
# Fetches an object starting at the local scope and then moving up the hierachy
def find_variable(key, raise_on_not_found: true)
# This was changed from find() to find_index() because this is a very hot
# path and find_index() is optimized in MRI to reduce object allocation
index = @scopes.find_index { |s| s.key?(key) }
fallback_to_self_drop = key == Expression::SELF && index.nil?
variable = if index
lookup_and_evaluate(@scopes[index], key, raise_on_not_found: raise_on_not_found)
# Fast path: check top scope first (most common in for loops)
scope = @scopes[0]
if scope.key?(key)
variable = lookup_and_evaluate(scope, key, raise_on_not_found: raise_on_not_found)
elsif @scopes.length == 1
# Only one scope and key not found — go straight to environments
variable = try_variable_find_in_environments(key, raise_on_not_found: raise_on_not_found)
else
try_variable_find_in_environments(
key,
raise_on_not_found: raise_on_not_found && !fallback_to_self_drop,
)
# Multiple scopes — search through all of them
index = @scopes.find_index { |s| s.key?(key) }
variable = if index
lookup_and_evaluate(@scopes[index], key, raise_on_not_found: raise_on_not_found)
else
try_variable_find_in_environments(key, raise_on_not_found: raise_on_not_found)
end
end
# `self` resolves to a SelfDrop (enabling `self['var']` lookups),
# but only after the normal environment lookup doesn't find a value.
return @self_drop ||= SelfDrop.new(self) if fallback_to_self_drop && variable.nil?
# update variable's context before invoking #to_liquid
# Fast path: primitive types don't need context= or to_liquid conversion
case variable
when String, Integer, Float, NilClass, TrueClass, FalseClass
return variable
when Array, Hash, Time
return variable
end
variable.context = self if variable.respond_to?(:context=)
liquid_variable = variable.to_liquid
liquid_variable.context = self if variable != liquid_variable && liquid_variable.respond_to?(:context=)
if variable != liquid_variable
liquid_variable.context = self if liquid_variable.respond_to?(:context=)
end
liquid_variable
end
@@ -246,6 +262,7 @@ module Liquid
end
def with_disabled_tags(tag_names)
@disabled_tags = {} if @disabled_tags.frozen?
tag_names.each do |name|
@disabled_tags[name] = @disabled_tags.fetch(name, 0) + 1
end
+362
View File
@@ -0,0 +1,362 @@
# frozen_string_literal: true
require "strscan"
module Liquid
# Single-pass forward-only scanner for Liquid parsing.
# Wraps StringScanner with higher-level methods for common Liquid constructs.
# One Cursor per template parse — threaded through all parsing code.
class Cursor
# Byte constants
SPACE = 32
TAB = 9
NL = 10
CR = 13
FF = 12
DASH = 45 # '-'
DOT = 46 # '.'
COLON = 58 # ':'
PIPE = 124 # '|'
QUOTE_S = 39 # "'"
QUOTE_D = 34 # '"'
LBRACK = 91 # '['
RBRACK = 93 # ']'
LPAREN = 40 # '('
RPAREN = 41 # ')'
QMARK = 63 # '?'
HASH = 35 # '#'
USCORE = 95 # '_'
COMMA = 44
ZERO = 48
NINE = 57
PCT = 37 # '%'
LCURLY = 123 # '{'
RCURLY = 125 # '}'
attr_reader :ss
def initialize(source)
@source = source
@ss = StringScanner.new(source)
end
# ── Position ────────────────────────────────────────────────────
def pos = @ss.pos
def pos=(n)
@ss.pos = n
end
def eos? = @ss.eos?
def peek_byte = @ss.peek_byte
def scan_byte = @ss.scan_byte
# Reset scanner to a new string (for reuse on sub-markup)
def reset(source)
@source = source
@ss.string = source
end
# Extract a slice from the source (deferred allocation)
def slice(start, len)
@source.byteslice(start, len)
end
# ── Whitespace ──────────────────────────────────────────────────
# Skip spaces/tabs/newlines/cr, return count of newlines skipped
def skip_ws
nl = 0
while (b = @ss.peek_byte)
case b
when SPACE, TAB, CR, FF then @ss.scan_byte
when NL then @ss.scan_byte
nl += 1
else break
end
end
nl
end
# Check if remaining bytes are all whitespace
def rest_blank?
saved = @ss.pos
@ss.skip(/\s*/)
result = @ss.eos?
@ss.pos = saved
result
end
# Regex for identifier: [a-zA-Z_][\w-]*\??
ID_REGEX = /[a-zA-Z_][\w-]*\??/
# ── Identifiers ─────────────────────────────────────────────────
# Skip an identifier without allocating a string. Returns length skipped, or 0.
def skip_id
@ss.skip(ID_REGEX) || 0
end
# Check if next id matches expected string, consume if so. No allocation.
def expect_id(expected)
start = @ss.pos
len = @ss.skip(ID_REGEX)
if len == expected.bytesize
# Compare bytes directly without allocating a string
i = 0
while i < len
unless @source.getbyte(start + i) == expected.getbyte(i)
@ss.pos = start
return false
end
i += 1
end
return true
end
@ss.pos = start if len
false
end
# Scan a single identifier: [a-zA-Z_][\w-]*\??
# Returns the string or nil if not at an identifier
def scan_id
@ss.scan(ID_REGEX)
end
# Scan a tag name: '#' or \w+
def scan_tag_name
if @ss.peek_byte == HASH
@ss.scan_byte
"#"
else
scan_id
end
end
# Regex for numbers: -?\d+(\.\d+)?
FLOAT_REGEX = /-?\d+\.\d+/
INT_REGEX = /-?\d+/
# ── Numbers ─────────────────────────────────────────────────────
# Try to scan an integer or float. Returns the number or nil.
def scan_number
if (s = @ss.scan(FLOAT_REGEX))
s.to_f
elsif (s = @ss.scan(INT_REGEX))
s.to_i
end
end
# Regex for quoted string content (without quotes)
SINGLE_QUOTED_CONTENT = /'([^']*)'/
DOUBLE_QUOTED_CONTENT = /"([^"]*)"/
# ── Strings ─────────────────────────────────────────────────────
# Scan a quoted string ('...' or "..."). Returns the content without quotes, or nil.
def scan_quoted_string
if @ss.scan(SINGLE_QUOTED_CONTENT) || @ss.scan(DOUBLE_QUOTED_CONTENT)
@ss[1]
end
end
# Regex for quoted strings (single or double quoted, including quotes)
QUOTED_STRING_RAW = /"[^"]*"|'[^']*'/
# Scan a quoted string including quotes. Returns the full "..." or '...' string, or nil.
def scan_quoted_string_raw
@ss.scan(QUOTED_STRING_RAW)
end
# Regex for dotted identifier: name(.name)*
DOTTED_ID_REGEX = /[a-zA-Z_][\w-]*\??(?:\.[a-zA-Z_][\w-]*\??)*/
# ── Expressions ─────────────────────────────────────────────────
# Scan a simple variable lookup: name(.name)* — no brackets, no filters
# Returns the string or nil
def scan_dotted_id
@ss.scan(DOTTED_ID_REGEX)
end
# Skip a fragment without allocating. Returns length skipped, or 0.
def skip_fragment
@ss.skip(QUOTED_STRING_RAW) || @ss.skip(UNQUOTED_FRAGMENT) || 0
end
# Regex for unquoted fragment: non-whitespace/comma/pipe sequence
UNQUOTED_FRAGMENT = /[^\s,|]+/
# Scan a "QuotedFragment" — a quoted string or non-whitespace/comma/pipe run
def scan_fragment
@ss.scan(QUOTED_STRING_RAW) || @ss.scan(UNQUOTED_FRAGMENT)
end
# ── Comparison operators ────────────────────────────────────────
COMPARISON_OPS = {
'==' => '==',
'!=' => '!=',
'<>' => '<>',
'<=' => '<=',
'>=' => '>=',
'<' => '<',
'>' => '>',
'contains' => 'contains',
}.freeze
# Scan a comparison operator. Returns frozen string or nil.
# Regex for comparison operators
COMPARISON_OP_REGEX = /==|!=|<>|<=|>=|<|>|contains(?!\w)/
def scan_comparison_op
if (op = @ss.scan(COMPARISON_OP_REGEX))
COMPARISON_OPS[op]
end
end
# ── Tag parsing helpers ─────────────────────────────────────────
# Results from last parse_tag_token call (avoids array allocation)
attr_reader :tag_markup, :tag_newlines
# Parse the interior of a tag token: "{%[-] tag_name markup [-]%}"
# Pure byte operations — avoids StringScanner reset overhead.
# Returns tag_name string or nil. Sets tag_markup and tag_newlines.
def parse_tag_token(token)
len = token.bytesize
pos = 2 # skip "{%"
pos += 1 if token.getbyte(pos) == DASH # skip '-'
nl = 0
# Skip whitespace, count newlines
while pos < len
b = token.getbyte(pos)
case b
when SPACE, TAB, CR, FF then pos += 1
when NL then pos += 1; nl += 1
else break
end
end
# Scan tag name: '#' or [a-zA-Z_][\w-]*
name_start = pos
b = token.getbyte(pos)
if b == HASH
pos += 1
elsif b && ((b >= 97 && b <= 122) || (b >= 65 && b <= 90) || b == USCORE)
pos += 1
while pos < len
b = token.getbyte(pos)
break unless (b >= 97 && b <= 122) || (b >= 65 && b <= 90) || (b >= 48 && b <= 57) || b == USCORE || b == DASH
pos += 1
end
pos += 1 if pos < len && token.getbyte(pos) == QMARK
else
return
end
tag_name = token.byteslice(name_start, pos - name_start)
# Skip whitespace after tag name, count newlines
while pos < len
b = token.getbyte(pos)
case b
when SPACE, TAB, CR, FF then pos += 1
when NL then pos += 1; nl += 1
else break
end
end
# markup is everything up to optional '-' before '%}'
markup_end = len - 2
markup_end -= 1 if markup_end > pos && token.getbyte(markup_end - 1) == DASH
@tag_markup = pos >= markup_end ? "" : token.byteslice(pos, markup_end - pos)
@tag_newlines = nl
tag_name
end
# Parse variable token interior: extract markup from "{{[-] ... [-]}}"
def parse_variable_token(token)
len = token.bytesize
return if len < 4
i = 2
i = 3 if token.getbyte(i) == DASH
parse_end = len - 3
parse_end -= 1 if token.getbyte(parse_end) == DASH
markup_len = parse_end - i + 1
markup_len <= 0 ? "" : token.byteslice(i, markup_len)
end
# ── Simple condition parser ─────────────────────────────────────
# Results from last parse_simple_condition call
attr_reader :cond_left, :cond_op, :cond_right
# Parse "expr [op expr]" from current position to end.
# Returns true on success, nil on failure. Sets cond_left, cond_op, cond_right.
def parse_simple_condition
skip_ws
@cond_left = scan_fragment
return unless @cond_left
skip_ws
if eos?
@cond_op = nil
@cond_right = nil
return true
end
@cond_op = scan_comparison_op
return unless @cond_op
skip_ws
@cond_right = scan_fragment
return unless @cond_right
skip_ws
return unless eos? # trailing junk
true
end
# ── For tag parser ────────────────────────────────────────────────
# Results from parse_for_markup
attr_reader :for_var, :for_collection, :for_reversed
# Parse "var in collection [reversed] [limit:N] [offset:N]"
# Returns true on success, nil on failure.
def parse_for_markup
skip_ws
@for_var = scan_id
return unless @for_var
skip_ws
# expect "in"
return unless scan_id == "in"
skip_ws
# Collection: parenthesized range or fragment
if peek_byte == LPAREN
start = @ss.pos
depth = 1
@ss.scan_byte
while !@ss.eos? && depth > 0
b = @ss.scan_byte
depth += 1 if b == LPAREN
depth -= 1 if b == RPAREN
end
@for_collection = @source.byteslice(start, @ss.pos - start)
else
@for_collection = scan_fragment
return unless @for_collection
end
skip_ws
# Check for 'reversed'
saved = @ss.pos
word = scan_id
if word == "reversed"
@for_reversed = true
else
@for_reversed = false
@ss.pos = saved if word # rewind if we consumed a non-'reversed' word
end
true
end
end
end
+71 -45
View File
@@ -2,8 +2,6 @@
module Liquid
class Expression
SELF = 'self'
LITERALS = {
nil => nil,
'nil' => nil,
@@ -37,11 +35,18 @@ module Liquid
def parse(markup, ss = StringScanner.new(""), cache = nil)
return unless markup
markup = markup.strip # markup can be a frozen string
# Only strip if there's leading/trailing whitespace (avoids allocation)
first_byte = markup.getbyte(0)
if first_byte == 32 || first_byte == 9 || first_byte == 10 || first_byte == 13 # space, tab, \n, \r
markup = markup.strip
else
last_byte = markup.getbyte(markup.bytesize - 1)
markup = markup.strip if last_byte == 32 || last_byte == 9 || last_byte == 10 || last_byte == 13
end
if (markup.start_with?('"') && markup.end_with?('"')) ||
(markup.start_with?("'") && markup.end_with?("'"))
return markup[1..-2]
return markup.byteslice(1, markup.bytesize - 2)
elsif LITERALS.key?(markup)
return LITERALS[markup]
end
@@ -73,57 +78,78 @@ module Liquid
end
end
def parse_number(markup, ss)
# check if the markup is simple integer or float
case markup
when INTEGER_REGEX
def parse_number(markup, _ss = nil)
len = markup.bytesize
return false if len == 0
# Quick reject: first byte must be digit or dash
pos = 0
first = markup.getbyte(pos)
if first == DASH
pos += 1
return false if pos >= len
b = markup.getbyte(pos)
return false if b < ZERO || b > NINE
pos += 1
elsif first >= ZERO && first <= NINE
pos += 1
else
return false
end
# Scan digits
while pos < len
b = markup.getbyte(pos)
break if b < ZERO || b > NINE
pos += 1
end
# If we consumed everything, it's a simple integer
if pos == len
return Integer(markup, 10)
when FLOAT_REGEX
return markup.to_f
end
ss.string = markup
# the first byte must be a digit or a dash
byte = ss.scan_byte
# Check for dot (float)
if markup.getbyte(pos) == DOT
dot_pos = pos
pos += 1
# Must have at least one digit after dot
digit_after_dot = pos
while pos < len
b = markup.getbyte(pos)
break if b < ZERO || b > NINE
return false if byte != DASH && (byte < ZERO || byte > NINE)
pos += 1
end
if byte == DASH
peek_byte = ss.peek_byte
if pos > digit_after_dot && pos == len
# Simple float like "123.456"
return markup.to_f
elsif pos > digit_after_dot
# Float followed by more dots or other chars: "1.2.3.4"
# Return the float portion up to second dot
while pos < len
b = markup.getbyte(pos)
if b == DOT
return markup.byteslice(0, pos).to_f
elsif b < ZERO || b > NINE
return false
end
# if it starts with a dash, the next byte must be a digit
return false if peek_byte.nil? || !(peek_byte >= ZERO && peek_byte <= NINE)
end
# The markup could be a float with multiple dots
first_dot_pos = nil
num_end_pos = nil
while (byte = ss.scan_byte)
return false if byte != DOT && (byte < ZERO || byte > NINE)
# we found our number and now we are just scanning the rest of the string
next if num_end_pos
if byte == DOT
if first_dot_pos.nil?
first_dot_pos = ss.pos
else
# we found another dot, so we know that the number ends here
num_end_pos = ss.pos - 1
pos += 1
end
return markup.byteslice(0, pos).to_f
else
# dot at end: "123."
return markup.byteslice(0, dot_pos).to_f
end
end
num_end_pos = markup.length if ss.eos?
if num_end_pos
# number ends with a number "123.123"
markup.byteslice(0, num_end_pos).to_f
else
# number ends with a dot "123."
markup.byteslice(0, first_dot_pos).to_f
end
# Not a number (has non-digit, non-dot characters)
false
end
end
end
+4 -3
View File
@@ -29,6 +29,7 @@ module Liquid
RUBY_WHITESPACE = [" ", "\t", "\r", "\n", "\f"].freeze
SINGLE_STRING_LITERAL = /'[^\']*'/
WHITESPACE_OR_NOTHING = /\s*/
WHITESPACE = /\s+/
SINGLE_COMPARISON_TOKENS = [].tap do |table|
table["<".ord] = COMPARISON_LESS_THAN
@@ -104,7 +105,7 @@ module Liquid
output = []
until ss.eos?
ss.skip(WHITESPACE_OR_NOTHING)
ss.skip(WHITESPACE)
break if ss.eos?
@@ -114,10 +115,10 @@ module Liquid
if (special = SPECIAL_TABLE[peeked])
ss.scan_byte
# Special case for ".."
if special == DOT && ss.peek_byte == DOT_ORD
if special.equal?(DOT) && ss.peek_byte == DOT_ORD
ss.scan_byte
output << DOTDOT
elsif special == DASH
elsif special.equal?(DASH)
# Special case for negative numbers
if (peeked_byte = ss.peek_byte) && NUMBER_TABLE[peeked_byte]
ss.pos -= 1
+4 -2
View File
@@ -3,7 +3,7 @@
module Liquid
class ParseContext
attr_accessor :locale, :line_number, :trim_whitespace, :depth
attr_reader :partial, :warnings, :error_mode, :environment
attr_reader :partial, :warnings, :error_mode, :environment, :expression_cache, :string_scanner, :cursor
def initialize(options = Const::EMPTY_HASH)
@environment = options.fetch(:environment, Environment.default)
@@ -24,6 +24,8 @@ module Liquid
{}
end
@cursor = Cursor.new("")
self.depth = 0
self.partial = false
end
@@ -38,7 +40,7 @@ module Liquid
def new_parser(input)
@string_scanner.string = input
Parser.new(@string_scanner, reject_bare_brackets: @error_mode == :strict2 || @error_mode == :rigid)
Parser.new(@string_scanner)
end
def new_tokenizer(source, start_line_number: nil, for_liquid_tag: false)
+4 -5
View File
@@ -2,11 +2,10 @@
module Liquid
class Parser
def initialize(input, reject_bare_brackets: false)
def initialize(input)
ss = input.is_a?(StringScanner) ? input : StringScanner.new(input)
@tokens = Lexer.tokenize(ss)
@p = 0 # pointer to current location
@reject_bare_brackets = reject_bare_brackets
end
def jump(point)
@@ -54,9 +53,6 @@ module Liquid
str = consume
str << variable_lookups
when :open_square
if @reject_bare_brackets
raise SyntaxError, "Bare bracket access is not allowed. Use #{Expression::SELF}['...'] instead"
end
str = consume.dup
str << expression
str << consume(:close_square)
@@ -87,6 +83,9 @@ module Liquid
end
def variable_lookups
# Fast path: no lookups at all (most common case for simple identifiers)
return "" unless look(:dot) || look(:open_square)
str = +""
loop do
if look(:open_square)
-44
View File
@@ -1,44 +0,0 @@
# frozen_string_literal: true
module Liquid
class RangeSlice
attr_reader :length
def initialize(range, from, to, resource_limits)
range_length = range.end - range.begin
range_length += 1 unless range.exclude_end?
range_length = 0 if range_length.negative?
start = [from, 0].max
finish = [to || range_length, range_length].min
@first = range.begin + start
@length = [finish - start, 0].max
@direction = 1
@resource_limits = resource_limits
end
def empty?
@length.zero?
end
def each
return enum_for(:each) unless block_given?
value = @first
@length.times do
@resource_limits.increment_render_score(1)
yield value
value += @direction
end
end
def reverse!
unless empty?
@first += @direction * (@length - 1)
@direction = -@direction
end
self
end
end
end
+6 -6
View File
@@ -6,15 +6,15 @@ module Liquid
def initialize(registers = {})
@static = registers.is_a?(Registers) ? registers.static : registers
@changes = {}
@changes = nil
end
def []=(key, value)
@changes[key] = value
(@changes ||= {})[key] = value
end
def [](key)
if @changes.key?(key)
if @changes&.key?(key)
@changes[key]
else
@static[key]
@@ -22,13 +22,13 @@ module Liquid
end
def delete(key)
@changes.delete(key)
@changes&.delete(key)
end
UNDEFINED = Object.new
def fetch(key, default = UNDEFINED, &block)
if @changes.key?(key)
if @changes&.key?(key)
@changes.fetch(key)
elsif default != UNDEFINED
if block_given?
@@ -42,7 +42,7 @@ module Liquid
end
def key?(key)
@changes.key?(key) || @static.key?(key)
@changes&.key?(key) || @static.key?(key)
end
end
+5 -22
View File
@@ -2,39 +2,24 @@
module Liquid
class ResourceLimits
attr_accessor :render_length_limit,
:render_score_limit,
:assign_score_limit,
:cumulative_render_score_limit,
:cumulative_assign_score_limit
attr_reader :render_score,
:assign_score,
:cumulative_render_score,
:cumulative_assign_score
attr_accessor :render_length_limit, :render_score_limit, :assign_score_limit
attr_reader :render_score, :assign_score, :last_capture_length
def initialize(limits)
@render_length_limit = limits[:render_length_limit]
@render_score_limit = limits[:render_score_limit]
@assign_score_limit = limits[:assign_score_limit]
@cumulative_render_score_limit = limits[:cumulative_render_score_limit]
@cumulative_assign_score_limit = limits[:cumulative_assign_score_limit]
@cumulative_render_score = 0
@cumulative_assign_score = 0
@render_length_limit = limits[:render_length_limit]
@render_score_limit = limits[:render_score_limit]
@assign_score_limit = limits[:assign_score_limit]
reset
end
def increment_render_score(amount)
@render_score += amount
@cumulative_render_score += amount
raise_limits_reached if @render_score_limit && @render_score > @render_score_limit
raise_limits_reached if @cumulative_render_score_limit && @cumulative_render_score > @cumulative_render_score_limit
end
def increment_assign_score(amount)
@assign_score += amount
@cumulative_assign_score += amount
raise_limits_reached if @assign_score_limit && @assign_score > @assign_score_limit
raise_limits_reached if @cumulative_assign_score_limit && @cumulative_assign_score > @cumulative_assign_score_limit
end
# update either render_length or assign_score based on whether or not the writes are captured
@@ -62,8 +47,6 @@ module Liquid
@reached_limit = false
@last_capture_length = nil
@render_score = @assign_score = 0
raise_limits_reached if @cumulative_render_score_limit && @cumulative_render_score > @cumulative_render_score_limit
raise_limits_reached if @cumulative_assign_score_limit && @cumulative_assign_score > @cumulative_assign_score_limit
end
def with_capture
-54
View File
@@ -1,54 +0,0 @@
# frozen_string_literal: true
module Liquid
# @liquid_public_docs
# @liquid_type object
# @liquid_name self
# @liquid_summary
# Provides access to variables through the current scope chain.
# @liquid_description
# The `self` object resolves variables through the normal lookup hierarchy
# (local > file > global) without exposing filters, interrupts, errors,
# or other context internals. It's used when bare bracket notation
# (`['variable']`) needs to be replaced with an explicit variable lookup.
#
# If `self` is explicitly assigned as a local variable (e.g. `{% assign self = 'value' %}`),
# then the local value takes precedence over the `self` object.
# @liquid_access global
class SelfDrop < Drop
def initialize(self_context)
super()
@self_context = self_context
end
def [](key)
@self_context.find_variable(key)
rescue UndefinedVariable
nil
end
def key?(key)
@self_context.variable_defined?(key)
end
def to_liquid
self
end
def ==(other)
other.is_a?(SelfDrop) && other.self_context.equal?(@self_context)
end
alias_method :eql?, :==
def hash
@self_context.object_id.hash
end
protected
attr_reader :self_context
undef context=
end
end
+54 -27
View File
@@ -8,19 +8,10 @@ module Liquid
MAX_I32 = (1 << 31) - 1
private_constant :MAX_I32
supports_64bit_indices = begin
[][1 << 33, 1 << 33]
true
rescue RangeError
false
end
INDEX_RANGE = if supports_64bit_indices
(-(1 << 63))..((1 << 63) - 1)
else
(-(1 << 31))..((1 << 31) - 1)
end
private_constant :INDEX_RANGE
MIN_I64 = -(1 << 63)
MAX_I64 = (1 << 63) - 1
I64_RANGE = MIN_I64..MAX_I64
private_constant :MIN_I64, :MAX_I64, :I64_RANGE
HTML_ESCAPE = {
'&' => '&amp;',
@@ -223,11 +214,11 @@ module Liquid
Utils.to_s(input).slice(offset, length) || ''
end
rescue RangeError
if INDEX_RANGE.cover?(length) && INDEX_RANGE.cover?(offset)
if I64_RANGE.cover?(length) && I64_RANGE.cover?(offset)
raise # unexpected error
end
offset = offset.clamp(INDEX_RANGE)
length = length.clamp(INDEX_RANGE)
offset = offset.clamp(I64_RANGE)
length = length.clamp(I64_RANGE)
retry
end
end
@@ -275,18 +266,54 @@ module Liquid
words = Utils.to_integer(words)
words = 1 if words <= 0
wordlist = begin
input.split(" ", words + 1)
rescue RangeError
# integer too big for String#split, but we can semantically assume no truncation is needed
return input if words + 1 > MAX_I32
raise # unexpected error
end
return input if wordlist.length <= words
return input if words + 1 > MAX_I32
wordlist.pop
truncate_string = Utils.to_s(truncate_string)
wordlist.join(" ").concat(truncate_string)
# Build result incrementally — avoids split() array + string allocations
len = input.bytesize
pos = 0
word_count = 0
result = nil
# Skip leading whitespace
while pos < len
b = input.getbyte(pos)
break unless b == 32 || b == 9 || b == 10 || b == 13 || b == 12
pos += 1
end
while pos < len
word_start = pos
word_count += 1
# Skip non-whitespace chars (word body)
while pos < len
b = input.getbyte(pos)
break if b == 32 || b == 9 || b == 10 || b == 13 || b == 12
pos += 1
end
if word_count > words
# Truncate — result already has the first N words
truncate_string = Utils.to_s(truncate_string)
return result.concat(truncate_string)
end
# Append word to result (only allocate result when we know truncation is possible)
if result
result << " " << input.byteslice(word_start, pos - word_start)
else
result = +input.byteslice(word_start, pos - word_start)
end
# Skip whitespace between words
while pos < len
b = input.getbyte(pos)
break unless b == 32 || b == 9 || b == 10 || b == 13 || b == 12
pos += 1
end
end
input
end
# @liquid_public_docs
+27
View File
@@ -58,5 +58,32 @@ module Liquid
rescue ::ArgumentError => e
raise Liquid::ArgumentError, e.message, e.backtrace
end
# Fast path for single-argument (no extra args) filter invocation.
# Avoids *args splat allocation for the common {{ value | filter }} case.
def invoke_single(method, input)
if self.class.invokable?(method)
send(method, input)
elsif @context.strict_filters
raise Liquid::UndefinedFilter, "undefined filter #{method}"
else
input
end
rescue ::ArgumentError => e
raise Liquid::ArgumentError, e.message, e.backtrace
end
# Fast path for two-argument filter invocation (input + one arg).
def invoke_two(method, input, arg1)
if self.class.invokable?(method)
send(method, input, arg1)
elsif @context.strict_filters
raise Liquid::UndefinedFilter, "undefined filter #{method}"
else
input
end
rescue ::ArgumentError => e
raise Liquid::ArgumentError, e.message, e.backtrace
end
end
end
-25
View File
@@ -18,8 +18,6 @@ module Liquid
# @liquid_syntax_keyword variable_name The name of the variable being created.
# @liquid_syntax_keyword value The value you want to assign to the variable.
class Assign < Tag
include ParserSwitching
Syntax = /(#{VariableSignature}+)\s*=\s*(.*)\s*/om
# @api private
@@ -31,10 +29,6 @@ module Liquid
def initialize(tag_name, markup, parse_context)
super
parse_with_selected_parser(markup)
end
def lax_parse(markup)
if markup =~ Syntax
@to = Regexp.last_match(1)
@from = Variable.new(Regexp.last_match(2), parse_context)
@@ -43,25 +37,6 @@ module Liquid
end
end
def strict_parse(markup)
lax_parse(markup)
end
def strict2_parse(markup)
unless markup =~ Syntax
self.class.raise_syntax_error(parse_context)
end
lhs = Regexp.last_match(1).strip
rhs = Regexp.last_match(2)
p = @parse_context.new_parser(lhs)
@to = p.consume(:id)
p.consume(:end_of_string)
@from = Variable.new(rhs, parse_context)
end
def render_to_output_buffer(context, output)
val = @from.render(context)
context.scopes.last[@to] = val
-18
View File
@@ -20,18 +20,10 @@ module Liquid
# @liquid_syntax_keyword variable The name of the variable being created.
# @liquid_syntax_keyword value The value you want to assign to the variable.
class Capture < Block
include ParserSwitching
Syntax = /(#{VariableSignature}+)/o
attr_reader :to
def initialize(tag_name, markup, options)
super
parse_with_selected_parser(markup)
end
def lax_parse(markup)
if markup =~ Syntax
@to = Regexp.last_match(1)
else
@@ -39,16 +31,6 @@ module Liquid
end
end
def strict_parse(markup)
lax_parse(markup)
end
def strict2_parse(markup)
p = @parse_context.new_parser(markup.strip)
@to = p.consume(:id)
p.consume(:end_of_string)
end
def render_to_output_buffer(context, output)
context.resource_limits.with_capture do
capture_output = render(context)
-16
View File
@@ -23,29 +23,13 @@ module Liquid
# {% decrement variable_name %}
# @liquid_syntax_keyword variable_name The name of the variable being decremented.
class Decrement < Tag
include ParserSwitching
attr_reader :variable_name
def initialize(tag_name, markup, options)
super
parse_with_selected_parser(markup)
end
def lax_parse(markup)
@variable_name = markup.strip
end
def strict_parse(markup)
lax_parse(markup)
end
def strict2_parse(markup)
p = @parse_context.new_parser(markup.strip)
@variable_name = p.consume(:id)
p.consume(:end_of_string)
end
def render_to_output_buffer(context, output)
counter_environment = context.environments.first
value = counter_environment[@variable_name] || 0
+47 -12
View File
@@ -72,18 +72,54 @@ module Liquid
protected
# Fast byte-level parser for "var in collection [reversed] [limit:N] [offset:N]"
REVERSED_BYTES = "reversed".bytes.freeze
def lax_parse(markup)
if markup =~ Syntax
@variable_name = Regexp.last_match(1)
collection_name = Regexp.last_match(2)
@reversed = !!Regexp.last_match(3)
@name = "#{@variable_name}-#{collection_name}"
@collection_name = parse_expression(collection_name)
markup.scan(TagAttributes) do |key, value|
set_attribute(key, value)
c = @parse_context.cursor
c.reset(markup)
c.skip_ws
# Parse variable name
var_start = c.pos
var_len = c.skip_id
raise SyntaxError, options[:locale].t("errors.syntax.for") if var_len == 0
@variable_name = c.slice(var_start, var_len)
# Expect "in"
c.skip_ws
raise SyntaxError, options[:locale].t("errors.syntax.for") unless c.expect_id("in")
c.skip_ws
# Parse collection name
col_start = c.pos
if c.peek_byte == Cursor::LPAREN
# Parenthesized range: (1..10)
depth = 1
c.scan_byte
while !c.eos? && depth > 0
b = c.scan_byte
depth += 1 if b == Cursor::LPAREN
depth -= 1 if b == Cursor::RPAREN
end
else
raise SyntaxError, options[:locale].t("errors.syntax.for")
c.skip_fragment
end
collection_name = c.slice(col_start, c.pos - col_start)
@name = "#{@variable_name}-#{collection_name}"
@collection_name = parse_expression(collection_name)
c.skip_ws
@reversed = c.expect_id("reversed")
c.skip_ws
# Parse limit:/offset: if present
if !c.eos? && markup.include?(':')
rest = c.slice(c.pos, markup.bytesize - c.pos)
rest.scan(TagAttributes) do |key, value|
set_attribute(key, value)
end
end
end
@@ -130,6 +166,7 @@ module Liquid
end
collection = context.evaluate(@collection_name)
collection = collection.to_a if collection.is_a?(Range)
limit_value = context.evaluate(@limit)
to = if limit_value.nil?
@@ -138,9 +175,7 @@ module Liquid
Utils.to_integer(limit_value) + from
end
segment = Utils.slice_collection_for_iteration(
collection, from, to, context.resource_limits, use_range_to_a: true
)
segment = Utils.slice_collection(collection, from, to)
segment.reverse! if @reversed
offsets[@name] = from + segment.length
+25 -4
View File
@@ -51,14 +51,17 @@ module Liquid
end
def render_to_output_buffer(context, output)
@blocks.each do |block|
result = Liquid::Utils.to_liquid_value(
block.evaluate(context),
)
idx = 0
blocks = @blocks
while idx < blocks.length
block = blocks[idx]
result = block.evaluate(context)
result = result.to_liquid_value if result.respond_to?(:to_liquid_value)
if result
return block.attachment.render_to_output_buffer(context, output)
end
idx += 1
end
output
@@ -86,6 +89,24 @@ module Liquid
end
def lax_parse(markup)
# Fastest path: simple identifier truthiness like "product.available" or "forloop.first"
if (simple = Variable.simple_variable_markup(markup))
return Condition.new(parse_expression(simple))
end
# Fast path: simple condition without and/or — use Cursor
if !markup.include?(' and ') && !markup.include?(' or ')
cursor = @parse_context.cursor
cursor.reset(markup)
if cursor.parse_simple_condition
return Condition.new(
parse_expression(cursor.cond_left),
cursor.cond_op,
cursor.cond_right ? parse_expression(cursor.cond_right) : nil,
)
end
end
expressions = markup.scan(ExpressionsAndOperators)
raise SyntaxError, options[:locale].t("errors.syntax.if") unless expressions.pop =~ Syntax
+4 -13
View File
@@ -20,8 +20,7 @@ module Liquid
class Include < Tag
prepend Tag::Disableable
FOR = 'for'
SYNTAX = /(#{QuotedFragment}+)(\s+(with|#{FOR})\s+(#{QuotedFragment}+))?(\s+(?:as)\s+(#{VariableSegment}+))?/o
SYNTAX = /(#{QuotedFragment}+)(\s+(?:with|for)\s+(#{QuotedFragment}+))?(\s+(?:as)\s+(#{VariableSegment}+))?/o
Syntax = SYNTAX
attr_reader :template_name_expr, :variable_name_expr, :attributes
@@ -85,18 +84,12 @@ module Liquid
alias_method :parse_context, :options
private :parse_context
def for_loop?
@is_for_loop
end
def strict2_parse(markup)
p = @parse_context.new_parser(markup)
@template_name_expr = safe_parse_expression(p)
with_or_for = p.id?("for") || p.id?("with")
@variable_name_expr = safe_parse_expression(p) if with_or_for
@variable_name_expr = safe_parse_expression(p) if p.id?("for") || p.id?("with")
@alias_name = p.consume(:id) if p.id?("as")
@is_for_loop = (with_or_for == FOR)
p.consume?(:comma)
@@ -118,13 +111,11 @@ module Liquid
def lax_parse(markup)
if markup =~ SYNTAX
template_name = Regexp.last_match(1)
with_or_for = Regexp.last_match(3)
variable_name = Regexp.last_match(4)
variable_name = Regexp.last_match(3)
@alias_name = Regexp.last_match(6)
@alias_name = Regexp.last_match(5)
@variable_name_expr = variable_name ? parse_expression(variable_name) : nil
@template_name_expr = parse_expression(template_name)
@is_for_loop = (with_or_for == FOR)
@attributes = {}
markup.scan(TagAttributes) do |key, value|
-16
View File
@@ -23,29 +23,13 @@ module Liquid
# {% increment variable_name %}
# @liquid_syntax_keyword variable_name The name of the variable being incremented.
class Increment < Tag
include ParserSwitching
attr_reader :variable_name
def initialize(tag_name, markup, options)
super
parse_with_selected_parser(markup)
end
def lax_parse(markup)
@variable_name = markup.strip
end
def strict_parse(markup)
lax_parse(markup)
end
def strict2_parse(markup)
p = @parse_context.new_parser(markup.strip)
@variable_name = p.consume(:id)
p.consume(:end_of_string)
end
def render_to_output_buffer(context, output)
counter_environment = context.environments.first
value = counter_environment[@variable_name] || 0
+1 -5
View File
@@ -85,13 +85,12 @@ module Liquid
from = @attributes.key?('offset') ? to_integer(context.evaluate(@attributes['offset'])) : 0
to = @attributes.key?('limit') ? from + to_integer(context.evaluate(@attributes['limit'])) : nil
collection = Utils.slice_collection_for_iteration(collection, from, to, context.resource_limits, allow_endless: true)
collection = Utils.slice_collection(collection, from, to)
length = collection.length
cols = @attributes.key?('cols') ? to_integer(context.evaluate(@attributes['cols'])) : length
output << "<tr class=\"row1\">\n"
context.resource_limits.increment_write_score(output)
context.stack do
tablerowloop = Liquid::TablerowloopDrop.new(length, cols)
context['tablerowloop'] = tablerowloop
@@ -102,7 +101,6 @@ module Liquid
output << "<td class=\"col#{tablerowloop.col}\">"
super
output << '</td>'
context.resource_limits.increment_write_score(output)
# Handle any interrupts if they exist.
if context.interrupt?
@@ -112,7 +110,6 @@ module Liquid
if tablerowloop.col_last && !tablerowloop.last
output << "</tr>\n<tr class=\"row#{tablerowloop.row + 1}\">"
context.resource_limits.increment_write_score(output)
end
tablerowloop.send(:increment!)
@@ -120,7 +117,6 @@ module Liquid
end
output << "</tr>\n"
context.resource_limits.increment_write_score(output)
output
end
+2 -13
View File
@@ -151,10 +151,8 @@ module Liquid
c
when Liquid::Drop
drop = args.shift
c = Context.new([drop, assigns], instance_assigns, registers, @rethrow_errors, @resource_limits, {}, @environment)
drop.context = c if drop.respond_to?(:context=)
c
drop = args.shift
drop.context = Context.new([drop, assigns], instance_assigns, registers, @rethrow_errors, @resource_limits, {}, @environment)
when Hash
Context.new([args.shift, assigns], instance_assigns, registers, @rethrow_errors, @resource_limits, {}, @environment)
when nil
@@ -189,20 +187,12 @@ module Liquid
context.template_name ||= name
previous_error_mode = context.registers.static[:template_error_mode]
context.registers.static[:template_error_mode] = @error_mode
begin
# render the nodelist.
@root.render_to_output_buffer(context, output || +'')
rescue Liquid::MemoryError => e
context.handle_error(e)
ensure
if previous_error_mode
context.registers.static[:template_error_mode] = previous_error_mode
else
context.registers.static.delete(:template_error_mode)
end
@errors = context.errors
end
end
@@ -234,7 +224,6 @@ module Liquid
end
@warnings = parse_context.warnings
@error_mode = parse_context.error_mode
parse_context
end
+106 -95
View File
@@ -6,10 +6,6 @@ module Liquid
class Tokenizer
attr_reader :line_number, :for_liquid_tag
TAG_END = /%\}/
TAG_OR_VARIABLE_START = /\{[\{\%]/
NEWLINE = /\n/
OPEN_CURLEY = "{".ord
CLOSE_CURLEY = "}".ord
PERCENTAGE = "%".ord
@@ -27,11 +23,7 @@ module Liquid
@offset = 0
@tokens = []
if @source
@ss = string_scanner
@ss.string = @source
tokenize
end
tokenize if @source
end
def shift
@@ -54,108 +46,127 @@ module Liquid
if @for_liquid_tag
@tokens = @source.split("\n")
else
@tokens << shift_normal until @ss.eos?
tokenize_fast
end
@source = nil
@ss = nil
end
def shift_normal
token = next_token
# Fast tokenizer using String#index instead of StringScanner regex.
# String#index is ~40% faster for finding { delimiters.
def tokenize_fast
src = @source
unless src.valid_encoding?
raise SyntaxError, "Invalid byte sequence in #{src.encoding}"
end
return unless token
len = src.bytesize
pos = 0
token
end
while pos < len
# Find next { which could start a tag or variable
idx = src.byteindex('{', pos)
def next_token
# possible states: :text, :tag, :variable
byte_a = @ss.peek_byte
if byte_a == OPEN_CURLEY
@ss.scan_byte
byte_b = @ss.peek_byte
if byte_b == PERCENTAGE
@ss.scan_byte
return next_tag_token
elsif byte_b == OPEN_CURLEY
@ss.scan_byte
return next_variable_token
unless idx
# No more tags/variables — rest is text
@tokens << src.byteslice(pos, len - pos) if pos < len
break
end
@ss.pos -= 1
end
next_byte = idx + 1 < len ? src.getbyte(idx + 1) : nil
next_text_token
end
if next_byte == PERCENTAGE # {%
# Emit text before tag
@tokens << src.byteslice(pos, idx - pos) if idx > pos
def next_text_token
start = @ss.pos
unless @ss.skip_until(TAG_OR_VARIABLE_START)
token = @ss.rest
@ss.terminate
return token
end
pos = @ss.pos -= 2
@source.byteslice(start, pos - start)
rescue ::ArgumentError => e
if e.message == "invalid byte sequence in #{@ss.string.encoding}"
raise SyntaxError, "Invalid byte sequence in #{@ss.string.encoding}"
else
raise
end
end
def next_variable_token
start = @ss.pos - 2
byte_a = byte_b = @ss.scan_byte
while byte_b
byte_a = @ss.scan_byte while byte_a && byte_a != CLOSE_CURLEY && byte_a != OPEN_CURLEY
break unless byte_a
if @ss.eos?
return byte_a == CLOSE_CURLEY ? @source.byteslice(start, @ss.pos - start) : "{{"
end
byte_b = @ss.scan_byte
if byte_a == CLOSE_CURLEY
if byte_b == CLOSE_CURLEY
return @source.byteslice(start, @ss.pos - start)
elsif byte_b != CLOSE_CURLEY
@ss.pos -= 1
return @source.byteslice(start, @ss.pos - start)
# Find %} to close the tag
close = src.byteindex('%}', idx + 2)
if close
@tokens << src.byteslice(idx, close + 2 - idx)
pos = close + 2
else
@tokens << "{%"
pos = idx + 2
end
elsif next_byte == OPEN_CURLEY # {{
# Emit text before variable
@tokens << src.byteslice(pos, idx - pos) if idx > pos
# Scan variable token — matches original tokenizer's byte-by-byte logic:
# Find } or {, then check next byte for }}/{% nesting
scan_pos = idx + 2
found = false
while scan_pos < len
b = src.getbyte(scan_pos)
if b == CLOSE_CURLEY # }
if scan_pos + 1 >= len
# } at end of string — emit token up to here
@tokens << src.byteslice(idx, scan_pos + 1 - idx)
pos = scan_pos + 1
found = true
break
end
b2 = src.getbyte(scan_pos + 1)
if b2 == CLOSE_CURLEY
# Found }} — close variable
@tokens << src.byteslice(idx, scan_pos + 2 - idx)
pos = scan_pos + 2
found = true
break
else
# } followed by non-} — emit token up to here (matches original: @ss.pos -= 1)
@tokens << src.byteslice(idx, scan_pos + 1 - idx)
pos = scan_pos + 1
found = true
break
end
elsif b == OPEN_CURLEY
if scan_pos + 1 < len && src.getbyte(scan_pos + 1) == PERCENTAGE
# Found {% inside {{ — scan to %} and emit as one token
close = src.byteindex('%}', scan_pos + 2)
if close
@tokens << src.byteslice(idx, close + 2 - idx)
pos = close + 2
else
@tokens << src.byteslice(idx, len - idx)
pos = len
end
found = true
break
end
scan_pos += 1
else
scan_pos += 1
end
end
unless found
@tokens << "{{"
pos = idx + 2
end
else
# { followed by something else — it's text
# Keep scanning from after this {
# Find next { that could be {% or {{
next_open = idx + 1
while next_open < len
ni = src.byteindex('{', next_open)
unless ni
@tokens << src.byteslice(pos, len - pos)
pos = len
break
end
nb = ni + 1 < len ? src.getbyte(ni + 1) : nil
if nb == PERCENTAGE || nb == OPEN_CURLEY
@tokens << src.byteslice(pos, ni - pos)
pos = ni
break
end
next_open = ni + 1
end
elsif byte_a == OPEN_CURLEY && byte_b == PERCENTAGE
return next_tag_token_with_start(start)
end
byte_a = byte_b
end
"{{"
end
def next_tag_token
start = @ss.pos - 2
if (len = @ss.skip_until(TAG_END))
@source.byteslice(start, len + 2)
else
"{%"
end
end
def next_tag_token_with_start(start)
@ss.skip_until(TAG_END)
@source.byteslice(start, @ss.pos - start)
end
end
end
+15 -75
View File
@@ -8,31 +8,14 @@ module Liquid
def self.slice_collection(collection, from, to)
if (from != 0 || !to.nil?) && collection.respond_to?(:load_slice)
collection.load_slice(from, to)
elsif from == 0 && to.nil? && collection.is_a?(Array)
# Fast path: no offset/limit on an Array — return as-is (avoid copy)
collection
else
slice_collection_using_each(collection, from, to)
end
end
# This is intentionally separate from slice_collection, whose Array-returning
# behavior is used outside of the iteration tags.
def self.slice_collection_for_iteration(
collection, from, to, resource_limits, allow_endless: false, use_range_to_a: false
)
if integer_range?(collection)
RangeSlice.new(collection, from, to, resource_limits)
elsif collection.is_a?(Range)
if use_range_to_a && range_method_overridden?(collection, :to_a)
# For historically honored custom Range#to_a. Charge the resulting
# selection before buffering it, just as for a custom #each.
slice_collection_for_iteration_using_each(collection.to_a, from, to, resource_limits)
else
slice_range_using_each(collection, from, to, resource_limits, allow_endless: allow_endless)
end
else
slice_collection(collection, from, to)
end
end
def self.slice_collection_using_each(collection, from, to)
segments = []
index = 0
@@ -58,55 +41,6 @@ module Liquid
segments
end
# Arithmetic slicing must not bypass a Range subclass's custom #each.
def self.integer_range?(collection)
collection.instance_of?(Range) && collection.begin.is_a?(Integer) && collection.end.is_a?(Integer)
end
private_class_method :integer_range?
# Preserve support for Ruby-supplied string ranges and custom Range#each.
# Their selected length cannot be inferred from integer bounds, but the tags
# need it before rendering for loop metadata, continuation offsets, and columns.
# Buffer the selection so we do not have to replay a potentially custom iterator.
def self.slice_range_using_each(collection, from, to, resource_limits, allow_endless:)
# TableRow historically accepted an endless subclass when its custom #each
# was finite, while For historically raised through Range#to_a.
if collection.end.nil? && !(allow_endless && (!to.nil? || range_method_overridden?(collection, :each)))
raise RangeError, "cannot convert endless range to an array"
end
if collection.begin.nil? && !range_method_overridden?(collection, :each)
raise TypeError, "can't iterate from NilClass"
end
slice_collection_for_iteration_using_each(collection, from, to, resource_limits)
end
private_class_method :slice_range_using_each
# Custom Range#each can make a nominally beginless range finite; standard
# beginless ranges were rejected before reaching this budgeted traversal.
def self.slice_collection_for_iteration_using_each(collection, from, to, resource_limits)
return [] if to && to <= from
segments = []
index = 0
collection.each do |item|
break if to && to <= index
# Charge preparation, including skipped offsets, before buffering; checking
# only while rendering the buffered values would leave this work unbudgeted.
resource_limits.increment_render_score(1)
segments << item if from <= index
index += 1
end
segments
end
private_class_method :slice_collection_for_iteration_using_each
def self.range_method_overridden?(collection, method_name)
collection.method(method_name).owner != Range.instance_method(method_name).owner
end
private_class_method :range_method_overridden?
def self.to_integer(num)
return num if num.is_a?(Integer)
num = num.to_s
@@ -162,8 +96,14 @@ module Liquid
obj
end
def self.to_s(obj, seen = {})
# Cached string representations for common small integers (0-999)
# Avoids repeated Integer#to_s allocations during rendering
SMALL_INT_STRINGS = Array.new(1000) { |i| i.to_s.freeze }.freeze
def self.to_s(obj, seen = nil)
case obj
when Integer
return (obj >= 0 && obj < 1000) ? SMALL_INT_STRINGS[obj] : obj.to_s
when BigDecimal
obj.to_s("F")
when Hash
@@ -171,30 +111,30 @@ module Liquid
# custom implementation. Otherwise we use Liquid's default
# implementation.
if obj.class.instance_method(:to_s) == HASH_TO_S_METHOD
hash_inspect(obj, seen)
hash_inspect(obj, seen || {})
else
obj.to_s
end
when Array
array_inspect(obj, seen)
array_inspect(obj, seen || {})
else
obj.to_s
end
end
def self.inspect(obj, seen = {})
def self.inspect(obj, seen = nil)
case obj
when Hash
# If the custom hash implementation overrides `#inspect`, use their
# custom implementation. Otherwise we use Liquid's default
# implementation.
if obj.class.instance_method(:inspect) == HASH_INSPECT_METHOD
hash_inspect(obj, seen)
hash_inspect(obj, seen || {})
else
obj.inspect
end
when Array
array_inspect(obj, seen)
array_inspect(obj, seen || {})
else
obj.inspect
end
+318 -19
View File
@@ -12,6 +12,68 @@ module Liquid
# {{ user | link }}
#
class Variable
# Checks if markup is a simple "name.lookup.chain" with no filters/brackets/quotes.
# Returns the trimmed markup string, or nil if not simple.
# Avoids regex MatchData allocation.
def self.simple_variable_markup(markup)
len = markup.bytesize
return if len == 0
# Skip leading whitespace
pos = 0
while pos < len
b = markup.getbyte(pos)
break unless b == 32 || b == 9 || b == 10 || b == 13
pos += 1
end
return if pos >= len
start = pos
# First char must be [a-zA-Z_]
b = markup.getbyte(pos)
return unless (b >= 97 && b <= 122) || (b >= 65 && b <= 90) || b == 95
pos += 1
# Scan segments: [\w-]* (. [\w-]*)*
while pos < len
b = markup.getbyte(pos)
if (b >= 97 && b <= 122) || (b >= 65 && b <= 90) || (b >= 48 && b <= 57) || b == 95 || b == 45
pos += 1
elsif b == 46 # '.'
pos += 1
# After dot, must have [a-zA-Z_]
return if pos >= len
b = markup.getbyte(pos)
return unless (b >= 97 && b <= 122) || (b >= 65 && b <= 90) || b == 95
pos += 1
else
break
end
end
content_end = pos
# Skip trailing whitespace
while pos < len
b = markup.getbyte(pos)
return unless b == 32 || b == 9 || b == 10 || b == 13
pos += 1
end
# Must have consumed everything
return unless pos == len
if start == 0 && content_end == len
markup
else
markup.byteslice(start, content_end - start)
end
end
# Cache for [filtername, EMPTY_ARRAY] tuples — avoids repeated array creation
NO_ARG_FILTER_CACHE = Hash.new { |h, k| h[k] = [k, Const::EMPTY_ARRAY].freeze }
FilterMarkupRegex = /#{FilterSeparator}\s*(.*)/om
FilterParser = /(?:\s+|#{QuotedFragment}|#{ArgumentSeparator})+/o
FilterArgsRegex = /(?:#{FilterArgumentSeparator}|#{ArgumentSeparator})\s*((?:\w+\s*\:\s*)?#{QuotedFragment})/o
@@ -30,23 +92,237 @@ module Liquid
@parse_context = parse_context
@line_number = parse_context.line_number
strict_parse_with_error_mode_fallback(markup)
# Fast path: try to parse without going through Lexer → Parser
# Skip for strict2/rigid modes which require different parsing
# Fast path only for lax/warn modes — strict modes need full error checking
error_mode = parse_context.error_mode
if error_mode == :strict2 || error_mode == :rigid || error_mode == :strict || !try_fast_parse(markup, parse_context)
strict_parse_with_error_mode_fallback(markup)
end
end
private def try_fast_parse(markup, parse_context)
len = markup.bytesize
return false if len == 0
# Skip leading whitespace
pos = 0
while pos < len
b = markup.getbyte(pos)
break unless b == 32 || b == 9 || b == 10 || b == 13
pos += 1
end
return false if pos >= len
b = markup.getbyte(pos)
if b == 39 || b == 34 # single or double quote
# Quoted string literal: scan to matching close quote
quote = b
name_start = pos
pos += 1
pos += 1 while pos < len && markup.getbyte(pos) != quote
pos += 1 if pos < len # skip closing quote
name_end = pos
elsif (b >= 97 && b <= 122) || (b >= 65 && b <= 90) || b == 95
# Identifier: scan [\w-]*(\.[\w-]*)*
name_start = pos
pos += 1
while pos < len
b = markup.getbyte(pos)
if (b >= 97 && b <= 122) || (b >= 65 && b <= 90) || (b >= 48 && b <= 57) || b == 95 || b == 45
pos += 1
elsif b == 46 # '.'
pos += 1
return false if pos >= len
b = markup.getbyte(pos)
return false unless (b >= 97 && b <= 122) || (b >= 65 && b <= 90) || b == 95
pos += 1
else
break
end
end
name_end = pos
else
return false
end
# Skip whitespace after name
while pos < len
b = markup.getbyte(pos)
break unless b == 32 || b == 9 || b == 10 || b == 13
pos += 1
end
# Resolve the name expression — avoid byteslice when markup is already the name
expr_markup = if name_start == 0 && name_end == len
markup # no whitespace, no filters — reuse the string
else
markup.byteslice(name_start, name_end - name_start)
end
cache = parse_context.expression_cache
ss = parse_context.string_scanner
first_byte = expr_markup.getbyte(0)
@name = if first_byte == 39 || first_byte == 34 # quoted string
# Strip quotes for string literal
expr_markup.byteslice(1, expr_markup.bytesize - 2)
elsif Expression::LITERALS.key?(expr_markup)
Expression::LITERALS[expr_markup]
elsif cache
cache[expr_markup] || (cache[expr_markup] = VariableLookup.parse_simple(expr_markup, ss, cache).freeze)
else
VariableLookup.parse_simple(expr_markup, ss || StringScanner.new(""), nil).freeze
end
# End of markup? No filters.
if pos >= len
@filters = Const::EMPTY_ARRAY
return true
end
# Must be a pipe for filters
return false unless markup.getbyte(pos) == 124 # '|'
# Try fast filter scanning first — handles no-arg and simple-arg filters
# Falls through to Lexer-based parsing for complex cases
@filters = []
filter_pos = pos
while filter_pos < len && markup.getbyte(filter_pos) == 124 # '|'
filter_pos += 1
# Skip whitespace
filter_pos += 1 while filter_pos < len && markup.getbyte(filter_pos) == 32
# Scan filter name
fname_start = filter_pos
b = filter_pos < len ? markup.getbyte(filter_pos) : nil
break unless b && ((b >= 97 && b <= 122) || (b >= 65 && b <= 90) || b == 95)
filter_pos += 1
while filter_pos < len
b = markup.getbyte(filter_pos)
break unless (b >= 97 && b <= 122) || (b >= 65 && b <= 90) || (b >= 48 && b <= 57) || b == 95 || b == 45
filter_pos += 1
end
filtername = markup.byteslice(fname_start, filter_pos - fname_start)
# Skip whitespace
filter_pos += 1 while filter_pos < len && markup.getbyte(filter_pos) == 32
# Has arguments — try fast scanning for positional args
if filter_pos < len && markup.getbyte(filter_pos) == 58 # ':'
filter_pos += 1 # skip ':'
filter_pos += 1 while filter_pos < len && markup.getbyte(filter_pos) == 32
filter_args = []
fall_to_lexer = false
loop do
arg_start = filter_pos
b = filter_pos < len ? markup.getbyte(filter_pos) : nil
if b == 39 || b == 34 # quoted string
quote = b
filter_pos += 1
filter_pos += 1 while filter_pos < len && markup.getbyte(filter_pos) != quote
filter_pos += 1 if filter_pos < len # skip closing quote
filter_args << markup.byteslice(arg_start + 1, filter_pos - arg_start - 2)
elsif b && ((b >= 48 && b <= 57) || (b == 45 && filter_pos + 1 < len && markup.getbyte(filter_pos + 1) >= 48 && markup.getbyte(filter_pos + 1) <= 57))
# Number
filter_pos += 1 if b == 45
filter_pos += 1 while filter_pos < len && markup.getbyte(filter_pos) >= 48 && markup.getbyte(filter_pos) <= 57
if filter_pos < len && markup.getbyte(filter_pos) == 46 # float
filter_pos += 1
filter_pos += 1 while filter_pos < len && markup.getbyte(filter_pos) >= 48 && markup.getbyte(filter_pos) <= 57
end
num_str = markup.byteslice(arg_start, filter_pos - arg_start)
filter_args << (num_str.include?('.') ? num_str.to_f : num_str.to_i)
elsif b && ((b >= 97 && b <= 122) || (b >= 65 && b <= 90) || b == 95)
# Identifier
id_start = filter_pos
filter_pos += 1
while filter_pos < len
b2 = markup.getbyte(filter_pos)
break unless (b2 >= 97 && b2 <= 122) || (b2 >= 65 && b2 <= 90) || (b2 >= 48 && b2 <= 57) || b2 == 95 || b2 == 45 || b2 == 46
filter_pos += 1
end
filter_pos += 1 if filter_pos < len && markup.getbyte(filter_pos) == 63
# Check if keyword arg (id followed by ':')
kw_check = filter_pos
kw_check += 1 while kw_check < len && markup.getbyte(kw_check) == 32
if kw_check < len && markup.getbyte(kw_check) == 58
fall_to_lexer = true
break
end
id_markup = markup.byteslice(id_start, filter_pos - id_start)
filter_args << Expression.parse(id_markup, parse_context.string_scanner, parse_context.expression_cache)
else
fall_to_lexer = true
break
end
# Skip whitespace after arg
filter_pos += 1 while filter_pos < len && markup.getbyte(filter_pos) == 32
# Comma = more args; pipe/end = done
if filter_pos < len && markup.getbyte(filter_pos) == 44
filter_pos += 1
filter_pos += 1 while filter_pos < len && markup.getbyte(filter_pos) == 32
else
break
end
end
if fall_to_lexer
# Complex filter — fall to Lexer for this and remaining filters
rest_start = fname_start
rest_start -= 1 while rest_start > pos && markup.getbyte(rest_start) != 124
rest_markup = markup.byteslice(rest_start, len - rest_start)
p = parse_context.new_parser(rest_markup)
while p.consume?(:pipe)
fn = p.consume(:id)
fa = p.consume?(:colon) ? parse_filterargs(p) : Const::EMPTY_ARRAY
@filters << lax_parse_filter_expressions(fn, fa)
end
p.consume(:end_of_string)
@filters = Const::EMPTY_ARRAY if @filters.empty?
return true
end
@filters << [filtername, filter_args]
else
# No args — add as simple filter
@filters << NO_ARG_FILTER_CACHE[filtername]
end
# Skip whitespace between filters
filter_pos += 1 while filter_pos < len && (markup.getbyte(filter_pos) == 32 || markup.getbyte(filter_pos) == 9 || markup.getbyte(filter_pos) == 10 || markup.getbyte(filter_pos) == 13)
end
# Must have consumed everything
return false if filter_pos < len
@filters = Const::EMPTY_ARRAY if @filters.empty?
true
rescue SyntaxError
# If fast parse fails, fall back to full parse
@name = nil
@filters = nil
false
end
def raw
@markup
end
def ==(other)
self.class == other.class && name == other.name && filters == other.filters
end
def markup_context(markup)
"in \"{{#{markup}}}\""
end
def lax_parse(markup)
@filters = []
@filters = Const::EMPTY_ARRAY
return unless markup =~ MarkupWithQuotedFragment
name_markup = Regexp.last_match(1)
@@ -58,19 +334,21 @@ module Liquid
next unless f =~ /\w+/
filtername = Regexp.last_match(0)
filterargs = f.scan(FilterArgsRegex).flatten
@filters = [] if @filters.frozen?
@filters << lax_parse_filter_expressions(filtername, filterargs)
end
end
end
def strict_parse(markup)
@filters = []
@filters = Const::EMPTY_ARRAY
p = @parse_context.new_parser(markup)
return if p.look(:end_of_string)
@name = parse_context.safe_parse_expression(p)
while p.consume?(:pipe)
@filters = [] if @filters.frozen?
filtername = p.consume(:id)
filterargs = p.consume?(:colon) ? parse_filterargs(p) : Const::EMPTY_ARRAY
@filters << lax_parse_filter_expressions(filtername, filterargs)
@@ -79,13 +357,16 @@ module Liquid
end
def strict2_parse(markup)
@filters = []
@filters = Const::EMPTY_ARRAY
p = @parse_context.new_parser(markup)
return if p.look(:end_of_string)
@name = parse_context.safe_parse_expression(p)
@filters << strict2_parse_filter_expressions(p) while p.consume?(:pipe)
while p.consume?(:pipe)
@filters = [] if @filters.frozen?
@filters << strict2_parse_filter_expressions(p)
end
p.consume(:end_of_string)
end
@@ -101,24 +382,37 @@ module Liquid
obj = context.evaluate(@name)
@filters.each do |filter_name, filter_args, filter_kwargs|
filter_args = evaluate_filter_expressions(context, filter_args, filter_kwargs)
obj = context.invoke(filter_name, obj, *filter_args)
if filter_args.empty? && !filter_kwargs
obj = context.invoke_single(filter_name, obj)
elsif !filter_kwargs && filter_args.length == 1
# Single positional arg — most common after no-arg
obj = context.invoke_two(filter_name, obj, context.evaluate(filter_args[0]))
else
filter_args = evaluate_filter_expressions(context, filter_args, filter_kwargs)
obj = context.invoke(filter_name, obj, *filter_args)
end
end
context.apply_global_filter(obj)
end
def render_to_output_buffer(context, output)
obj = render(context)
# Fast path: no filters and no global filter
obj = if @filters.empty? && context.global_filter.nil?
context.evaluate(@name)
else
render(context)
end
render_obj_to_output(obj, output)
output
end
def render_obj_to_output(obj, output)
case obj
when NilClass
if obj.instance_of?(String)
output << obj
elsif obj.nil?
# Do nothing
when Array
elsif obj.instance_of?(Array)
obj.each do |o|
render_obj_to_output(o, output)
end
@@ -132,7 +426,7 @@ module Liquid
end
def disabled_tags
[]
Const::EMPTY_ARRAY
end
private
@@ -141,7 +435,8 @@ module Liquid
filter_args = []
keyword_args = nil
unparsed_args.each do |a|
if (matches = a.match(JustTagAttributes))
# Fast check: keyword args must contain ':'
if a.include?(':') && (matches = a.match(JustTagAttributes))
keyword_args ||= {}
keyword_args[matches[1]] = parse_context.parse_expression(matches[2])
else
@@ -194,15 +489,19 @@ module Liquid
end
def evaluate_filter_expressions(context, filter_args, filter_kwargs)
parsed_args = filter_args.map { |expr| context.evaluate(expr) }
if filter_kwargs
parsed_args = filter_args.map { |expr| context.evaluate(expr) }
parsed_kwargs = {}
filter_kwargs.each do |key, expr|
parsed_kwargs[key] = context.evaluate(expr)
end
parsed_args << parsed_kwargs
parsed_args
elsif filter_args.empty?
Const::EMPTY_ARRAY
else
filter_args.map { |expr| context.evaluate(expr) }
end
parsed_args
end
class ParseTreeVisitor < Liquid::ParseTreeVisitor
+124 -13
View File
@@ -10,8 +10,108 @@ module Liquid
new(markup, string_scanner, cache)
end
def initialize(markup, string_scanner = StringScanner.new(""), cache = nil)
lookups = markup.scan(VariableParser)
# Fast parse that skips simple_lookup? check — caller guarantees simple identifier chain
def self.parse_simple(markup, string_scanner = nil, cache = nil)
new(markup, string_scanner, cache, true)
end
# Fast manual scanner replacing markup.scan(VariableParser)
# VariableParser = /\[(?>[^\[\]]+|\g<0>)*\]|[\w-]+\??/
# Splits "product.variants[0].title" into ["product", "variants", "[0]", "title"]
def self.scan_variable(markup)
result = []
pos = 0
len = markup.bytesize
while pos < len
byte = markup.getbyte(pos)
if byte == 91 # '['
# Scan balanced brackets
depth = 1
start = pos
pos += 1
while pos < len && depth > 0
b = markup.getbyte(pos)
if b == 91
depth += 1
elsif b == 93
depth -= 1
end
pos += 1
end
if depth == 0
result << markup.byteslice(start, pos - start)
else
# Unbalanced bracket - skip '[' and continue
pos = start + 1
end
elsif byte == 46 # '.'
pos += 1
elsif (byte >= 97 && byte <= 122) || (byte >= 65 && byte <= 90) || (byte >= 48 && byte <= 57) || byte == 95 || byte == 45 # \w or -
start = pos
pos += 1
while pos < len
b = markup.getbyte(pos)
break unless (b >= 97 && b <= 122) || (b >= 65 && b <= 90) || (b >= 48 && b <= 57) || b == 95 || b == 45
pos += 1
end
# Check trailing '?'
if pos < len && markup.getbyte(pos) == 63
pos += 1
end
result << markup.byteslice(start, pos - start)
else
pos += 1
end
end
result
end
# Check if markup is a simple identifier chain: [\w-]+\??(.[\w-]+\??)*
# Uses C-level match? — 8x faster than Ruby byte scanning
SIMPLE_LOOKUP_RE = /\A[\w-]+\??(?:\.[\w-]+\??)*\z/
def self.simple_lookup?(markup)
markup.bytesize > 0 && markup.match?(SIMPLE_LOOKUP_RE)
end
def initialize(markup, string_scanner = StringScanner.new(""), cache = nil, simple = false)
# Fast path: simple identifier chain without brackets
if simple || self.class.simple_lookup?(markup)
dot_pos = markup.index('.')
if dot_pos.nil?
@name = markup
@lookups = Const::EMPTY_ARRAY
@command_flags = 0
return
end
@name = markup.byteslice(0, dot_pos)
# Build lookups array from remaining dot-separated segments
lookups = []
@command_flags = 0
pos = dot_pos + 1
len = markup.bytesize
while pos < len
seg_start = pos
while pos < len
b = markup.getbyte(pos)
break if b == 46 # '.'
pos += 1
end
seg = markup.byteslice(seg_start, pos - seg_start)
if COMMAND_METHODS.include?(seg)
@command_flags |= 1 << lookups.length
end
lookups << seg
pos += 1 # skip dot
end
@lookups = lookups
return
end
lookups = self.class.scan_variable(markup)
name = lookups.shift
if name&.start_with?('[') && name&.end_with?(']')
@@ -49,26 +149,40 @@ module Liquid
object = context.find_variable(name)
@lookups.each_index do |i|
key = context.evaluate(@lookups[i])
lookup = @lookups[i]
key = lookup.instance_of?(String) ? lookup : context.evaluate(lookup)
# Cast "key" to its liquid value to enable it to act as a primitive value
key = Liquid::Utils.to_liquid_value(key)
# Fast path: strings and integers (most common key types) don't need conversion
unless key.instance_of?(String) || key.instance_of?(Integer)
key = Liquid::Utils.to_liquid_value(key)
end
# If object is a hash- or array-like object we look for the
# presence of the key and if its available we return it
if object.respond_to?(:[]) &&
((object.respond_to?(:key?) && object.key?(key)) ||
(object.respond_to?(:fetch) && key.is_a?(Integer)))
if object.instance_of?(Hash) ? object.key?(key) :
(object.respond_to?(:[]) &&
((object.respond_to?(:key?) && object.key?(key)) ||
(object.respond_to?(:fetch) && key.is_a?(Integer))))
# if its a proc we will replace the entry with the proc
res = context.lookup_and_evaluate(object, key)
object = res.to_liquid
object = context.lookup_and_evaluate(object, key)
# Skip to_liquid for common primitive types (they return self)
unless object.instance_of?(String) || object.instance_of?(Integer) || object.instance_of?(Float) ||
object.instance_of?(Array) || object.instance_of?(Hash) || object.nil?
object = object.to_liquid
object.context = context if object.respond_to?(:context=)
end
# Some special cases. If the part wasn't in square brackets and
# no key with the same name was found we interpret following calls
# as commands and call them on the current object
elsif lookup_command?(i) && object.respond_to?(key)
object = object.send(key).to_liquid
object = object.send(key)
unless object.instance_of?(String) || object.instance_of?(Integer) || object.instance_of?(Array) || object.nil?
object = object.to_liquid
object.context = context if object.respond_to?(:context=)
end
# Handle string first/last like ActiveSupport does (returns first/last character)
# ActiveSupport returns "" for empty strings, not nil
@@ -82,9 +196,6 @@ module Liquid
return nil unless context.strict_variables
raise Liquid::UndefinedVariable, "undefined variable #{key}"
end
# If we are dealing with a drop here we have to
object.context = context if object.respond_to?(:context=)
end
object
+1 -1
View File
@@ -2,5 +2,5 @@
# frozen_string_literal: true
module Liquid
VERSION = "5.14.0"
VERSION = "5.11.0"
end
+62
View File
@@ -0,0 +1,62 @@
# frozen_string_literal: true
# Quick benchmark for autoresearch: measures parse µs, render µs, and object allocations
# Outputs machine-readable metrics to stdout
require_relative 'theme_runner'
RubyVM::YJIT.enable if defined?(RubyVM::YJIT)
runner = ThemeRunner.new
# Warmup — enough iterations for YJIT to fully optimize hot paths
20.times { runner.compile }
20.times { runner.render }
GC.start
GC.compact if GC.respond_to?(:compact)
# Measure parse
parse_times = []
10.times do
GC.disable
t0 = Process.clock_gettime(Process::CLOCK_MONOTONIC)
runner.compile
t1 = Process.clock_gettime(Process::CLOCK_MONOTONIC)
GC.enable
GC.start
parse_times << (t1 - t0) * 1_000_000 # µs
end
# Measure render
render_times = []
10.times do
GC.disable
t0 = Process.clock_gettime(Process::CLOCK_MONOTONIC)
runner.render
t1 = Process.clock_gettime(Process::CLOCK_MONOTONIC)
GC.enable
GC.start
render_times << (t1 - t0) * 1_000_000 # µs
end
# Measure object allocations for one parse+render cycle
require 'objspace'
GC.start
GC.disable
before = ObjectSpace.count_objects.values_at(:TOTAL).first - ObjectSpace.count_objects.values_at(:FREE).first
runner.compile
runner.render
after = ObjectSpace.count_objects.values_at(:TOTAL).first - ObjectSpace.count_objects.values_at(:FREE).first
GC.enable
allocations = after - before
parse_us = parse_times.min.round(0)
render_us = render_times.min.round(0)
combined_us = parse_us + render_us
puts "RESULTS"
puts "parse_us=#{parse_us}"
puts "render_us=#{render_us}"
puts "combined_us=#{combined_us}"
puts "allocations=#{allocations}"
+3 -16
View File
@@ -6,24 +6,14 @@
$LOAD_PATH.unshift(File.expand_path('../lib', __dir__))
require 'liquid'
require_relative 'support/liquid_spec_adapter_helper'
LiquidSpec.configure do |config|
config.missing_features = [
:activesupport,
:lax_parsing,
:shopify_filters,
:shopify_includes,
:shopify_blank,
:shopify_error_handling,
:shopify_error_format,
:shopify_string_access,
]
# Run core Liquid specs
config.features = [:core]
end
# Compile a template string into a Liquid::Template
LiquidSpec.compile do |ctx, source, options|
options[:error_mode] ||= :strict
ctx[:template] = Liquid::Template.parse(source, **options)
end
@@ -38,12 +28,9 @@ LiquidSpec.render do |ctx, assigns, options|
static_environments: assigns,
registers: registers,
rethrow_errors: options[:strict_errors],
resource_limits: LiquidSpecAdapterHelper.resource_limits(options),
)
context.exception_renderer = options[:exception_renderer] if options[:exception_renderer]
LiquidSpecAdapterHelper.with_frozen_time do
ctx[:template].render(context)
end
ctx[:template].render(context)
end
+4 -16
View File
@@ -6,24 +6,15 @@
$LOAD_PATH.unshift(File.expand_path('../lib', __dir__))
require 'liquid'
require_relative 'support/liquid_spec_adapter_helper'
LiquidSpec.configure do |config|
config.missing_features = [
:activesupport,
:shopify_filters,
:shopify_includes,
:shopify_blank,
:shopify_error_handling,
:shopify_error_format,
:shopify_string_access,
]
config.features = [:core, :lax_parsing]
end
# Compile a template string into a Liquid::Template
LiquidSpec.compile do |ctx, source, options|
# Default to lax mode while still honoring specs that explicitly set error_mode.
options = { error_mode: :lax }.merge(options)
# Force lax mode
options = options.merge(error_mode: :lax)
ctx[:template] = Liquid::Template.parse(source, **options)
end
@@ -35,12 +26,9 @@ LiquidSpec.render do |ctx, assigns, options|
static_environments: assigns,
registers: registers,
rethrow_errors: options[:strict_errors],
resource_limits: LiquidSpecAdapterHelper.resource_limits(options),
)
context.exception_renderer = options[:exception_renderer] if options[:exception_renderer]
LiquidSpecAdapterHelper.with_frozen_time do
ctx[:template].render(context)
end
ctx[:template].render(context)
end
+3 -15
View File
@@ -7,23 +7,14 @@
$LOAD_PATH.unshift(File.expand_path('../lib', __dir__))
require 'active_support/all'
require 'liquid'
require_relative 'support/liquid_spec_adapter_helper'
LiquidSpec.configure do |config|
config.missing_features = [
:lax_parsing,
:shopify_filters,
:shopify_includes,
:shopify_blank,
:shopify_error_handling,
:shopify_error_format,
:shopify_string_access,
]
# Run core Liquid specs plus ActiveSupport SafeBuffer tests
config.features = [:core, :activesupport]
end
# Compile a template string into a Liquid::Template
LiquidSpec.compile do |ctx, source, options|
options[:error_mode] ||= :strict
ctx[:template] = Liquid::Template.parse(source, **options)
end
@@ -38,12 +29,9 @@ LiquidSpec.render do |ctx, assigns, options|
static_environments: assigns,
registers: registers,
rethrow_errors: options[:strict_errors],
resource_limits: LiquidSpecAdapterHelper.resource_limits(options),
)
context.exception_renderer = options[:exception_renderer] if options[:exception_renderer]
LiquidSpecAdapterHelper.with_frozen_time do
ctx[:template].render(context)
end
ctx[:template].render(context)
end
+2 -14
View File
@@ -13,18 +13,9 @@ end
require 'active_support/all'
require 'liquid'
require_relative 'support/liquid_spec_adapter_helper'
LiquidSpec.configure do |config|
config.missing_features = [
:lax_parsing,
:shopify_filters,
:shopify_includes,
:shopify_blank,
:shopify_error_handling,
:shopify_error_format,
:shopify_string_access,
]
config.features = [:core, :activesupport]
end
# Compile a template string into a Liquid::Template
@@ -42,12 +33,9 @@ LiquidSpec.render do |ctx, assigns, options|
static_environments: assigns,
registers: registers,
rethrow_errors: options[:strict_errors],
resource_limits: LiquidSpecAdapterHelper.resource_limits(options),
)
context.exception_renderer = options[:exception_renderer] if options[:exception_renderer]
LiquidSpecAdapterHelper.with_frozen_time do
ctx[:template].render(context)
end
ctx[:template].render(context)
end
@@ -1,24 +0,0 @@
# frozen_string_literal: true
module LiquidSpecAdapterHelper
extend self
def resource_limits(render_options)
return unless render_options[:resource_limits]
Liquid::ResourceLimits.new({}).tap do |limits|
render_options[:resource_limits].each do |key, value|
limits.public_send(:"#{key}=", value)
end
end
end
def with_frozen_time(&block)
original_tz = ENV['TZ']
ENV['TZ'] = 'UTC'
Liquid::Spec::TimeFreezer.freeze(Liquid::Spec::AdapterRunner::TEST_TIME, &block)
ensure
ENV['TZ'] = original_tz
end
end
-40
View File
@@ -97,46 +97,6 @@ class AssignTest < Minitest::Test
assert_equal(12, assign_score_of('int' => 123, 'str' => 'abcd'))
end
def test_assign_with_valid_identifier_in_strict2
assert_template_result("hello", "{% assign my_var = 'hello' %}{{ my_var }}", error_mode: :strict2)
end
def test_assign_with_hyphen_in_strict2
assert_template_result("hello", "{% assign my-var = 'hello' %}{{ my-var }}", error_mode: :strict2)
end
def test_assign_rejects_parentheses_in_variable_name_in_strict2
assert_raises(Liquid::SyntaxError) do
Liquid::Template.parse("{% assign (a(b(c) = 1234 %}", error_mode: :strict2)
end
end
def test_assign_rejects_brackets_in_variable_name_in_strict2
assert_raises(Liquid::SyntaxError) do
Liquid::Template.parse("{% assign [x.y] = 'hello' %}", error_mode: :strict2)
end
end
def test_assign_rejects_dot_in_variable_name_in_strict2
assert_raises(Liquid::SyntaxError) do
Liquid::Template.parse("{% assign a.b = 'hello' %}", error_mode: :strict2)
end
end
def test_assign_rejects_numeric_variable_name_in_strict2
assert_raises(Liquid::SyntaxError) do
Liquid::Template.parse("{% assign 1abc = 'hello' %}", error_mode: :strict2)
end
end
def test_assign_allows_invalid_names_in_lax
assert_template_result("1234", "{% assign (a(b(c) = 1234 %}{{ self['(a(b(c)'] }}", error_mode: :lax)
end
def test_assign_with_filter_in_strict2
assert_template_result("HELLO", "{% assign my_var = 'hello' | upcase %}{{ my_var }}", error_mode: :strict2)
end
private
class ObjectWrapperDrop < Liquid::Drop
@@ -1,85 +0,0 @@
# frozen_string_literal: true
require 'test_helper'
class BlankBodyErrorHandlingTest < Minitest::Test
COMPARISON_ERROR = 'Liquid error (line 1): comparison of Integer with String failed'
INVALID_INTEGER_ERROR = 'Liquid error (line 1): invalid integer'
def render_inline(source, error_mode:, assigns: {})
Liquid::Template.parse(source, line_numbers: true, error_mode: error_mode).render(assigns, render_errors: true)
end
def assert_render_raises(source, error_mode:, assigns: {}, message: nil)
error = assert_raises(Liquid::ArgumentError) do
Liquid::Template.parse(source, line_numbers: true, error_mode: error_mode).render!(assigns)
end
assert_includes(error.message, message) if message
end
def test_blank_if_body_suppresses_inline_error_text_in_lax_and_strict
[:lax, :strict].each do |mode|
assert_equal('', render_inline('{% if 5 > "x" %}{% endif %}', error_mode: mode))
end
end
def test_blank_unless_body_suppresses_inline_error_text_in_lax_and_strict
[:lax, :strict].each do |mode|
assert_equal('', render_inline('{% unless 5 > "x" %} {% endunless %}', error_mode: mode))
end
end
def test_blank_for_body_suppresses_inline_error_text_in_lax_and_strict
[:lax, :strict].each do |mode|
assert_equal('', render_inline('{% for i in (1..3) offset: xs %}{% endfor %}', error_mode: mode, assigns: { 'xs' => 'bad' }))
end
end
def test_strict2_blank_if_body_shows_inline_error_text
assert_equal(COMPARISON_ERROR, render_inline('{% if 5 > "x" %}{% endif %}', error_mode: :strict2))
end
def test_strict2_whitespace_if_body_shows_inline_error_text
assert_equal(COMPARISON_ERROR, render_inline('{% if 5 > "x" %} {% endif %}', error_mode: :strict2))
end
def test_strict2_assign_if_body_shows_inline_error_text
assert_equal(COMPARISON_ERROR, render_inline('{% if 5 > "x" %}{% assign a = 1 %}{% endif %}', error_mode: :strict2))
end
def test_strict2_comment_if_body_shows_inline_error_text
assert_equal(COMPARISON_ERROR, render_inline('{% if 5 > "x" %}{% comment %}c{% endcomment %}{% endif %}', error_mode: :strict2))
end
def test_strict2_capture_if_body_shows_inline_error_text
assert_equal(COMPARISON_ERROR, render_inline('{% if 5 > "x" %}{% capture c %}text{% endcapture %}{% endif %}', error_mode: :strict2))
end
def test_strict2_blank_unless_body_shows_inline_error_text
assert_equal(COMPARISON_ERROR, render_inline('{% unless 5 > "x" %} {% endunless %}', error_mode: :strict2))
end
def test_strict2_blank_for_body_shows_inline_error_text
assert_equal(INVALID_INTEGER_ERROR, render_inline('{% for i in (1..3) offset: xs %}{% endfor %}', error_mode: :strict2, assigns: { 'xs' => 'bad' }))
end
def test_nonblank_bodies_show_inline_error_text_in_all_modes
[:lax, :strict, :strict2].each do |mode|
assert_equal(COMPARISON_ERROR, render_inline('{% if 5 > "x" %}{% echo 1 %}{% endif %}', error_mode: mode))
assert_equal(COMPARISON_ERROR, render_inline('{% if 5 > "x" %}{{ "" }}{% endif %}', error_mode: mode))
assert_equal(COMPARISON_ERROR, render_inline('{% if 5 > "x" %}{% else %}E{% endif %}', error_mode: mode))
end
end
def test_raised_errors_are_not_swallowed_by_blank_if_body
[:lax, :strict, :strict2].each do |mode|
assert_render_raises('{% if 5 > "x" %}{% endif %}', error_mode: mode, message: 'comparison of Integer with String failed')
end
end
def test_raised_errors_are_not_swallowed_by_blank_for_body
[:lax, :strict, :strict2].each do |mode|
assert_render_raises('{% for i in (1..3) offset: xs %}{% endfor %}', error_mode: mode, assigns: { 'xs' => 'bad' }, message: 'invalid integer')
end
end
end
+1 -36
View File
@@ -6,11 +6,7 @@ class CaptureTest < Minitest::Test
include Liquid
def test_captures_block_content_in_variable
assert_template_result("test string", "{% capture var %}test string{% endcapture %}{{var}}", {})
end
def test_captures_block_content_in_quoted_variable_in_lax
assert_template_result("test string", "{% capture 'var' %}test string{% endcapture %}{{var}}", {}, error_mode: :lax)
assert_template_result("test string", "{% capture 'var' %}test string{% endcapture %}{{var}}", {})
end
def test_capture_with_hyphen_in_variable_name
@@ -53,35 +49,4 @@ class CaptureTest < Minitest::Test
t.render!
assert_equal(9, t.resource_limits.assign_score)
end
def test_capture_with_valid_identifier_in_strict2
assert_template_result("hello", "{% capture my_var %}hello{% endcapture %}{{ my_var }}", error_mode: :strict2)
end
def test_capture_with_hyphen_in_strict2
assert_template_result("hello", "{% capture my-var %}hello{% endcapture %}{{ my-var }}", error_mode: :strict2)
end
def test_capture_rejects_parentheses_in_variable_name_in_strict2
assert_raises(Liquid::SyntaxError) do
Liquid::Template.parse("{% capture (x[y %}hello{% endcapture %}", error_mode: :strict2)
end
end
def test_capture_rejects_dot_in_variable_name_in_strict2
assert_raises(Liquid::SyntaxError) do
Liquid::Template.parse("{% capture a.b %}hello{% endcapture %}", error_mode: :strict2)
end
end
def test_capture_rejects_numeric_variable_name_in_strict2
assert_raises(Liquid::SyntaxError) do
Liquid::Template.parse("{% capture 1abc %}hello{% endcapture %}", error_mode: :strict2)
end
end
def test_capture_allows_invalid_names_in_lax
t = Liquid::Template.parse("{% capture (x[y %}hello{% endcapture %}", error_mode: :lax)
assert_equal("(x[y", t.root.nodelist.first.to)
end
end
+2 -2
View File
@@ -296,8 +296,8 @@ class ContextTest < Minitest::Test
end
def test_access_variable_with_hash_notation
assert_template_result('baz', '{{ foo }}', { "foo" => "baz" })
assert_template_result('baz', '{{ self[bar] }}', { 'foo' => 'baz', 'bar' => 'foo' })
assert_template_result('baz', '{{ ["foo"] }}', { "foo" => "baz" })
assert_template_result('baz', '{{ [bar] }}', { 'foo' => 'baz', 'bar' => 'foo' })
end
def test_access_hashes_with_hash_access_variables
+4 -6
View File
@@ -265,13 +265,11 @@ class ErrorHandlingTest < Minitest::Test
end
def test_bug_compatible_silencing_of_errors_in_blank_nodes
with_error_modes(:lax, :strict) do
output = Liquid::Template.parse("{% assign x = 0 %}{% if 1 < '2' %}not blank{% assign x = 3 %}{% endif %}{{ x }}").render
assert_equal("Liquid error: comparison of Integer with String failed0", output)
output = Liquid::Template.parse("{% assign x = 0 %}{% if 1 < '2' %}not blank{% assign x = 3 %}{% endif %}{{ x }}").render
assert_equal("Liquid error: comparison of Integer with String failed0", output)
output = Liquid::Template.parse("{% assign x = 0 %}{% if 1 < '2' %}{% assign x = 3 %}{% endif %}{{ x }}").render
assert_equal("0", output)
end
output = Liquid::Template.parse("{% assign x = 0 %}{% if 1 < '2' %}{% assign x = 3 %}{% endif %}{{ x }}").render
assert_equal("0", output)
end
def test_syntax_error_is_raised_with_template_name
+21 -31
View File
@@ -44,42 +44,32 @@ class SecurityTest < Minitest::Test
end
def test_does_not_permanently_add_filters_to_symbol_table
assert_no_new_symbols do
# MRI imprecisely marks objects found on the C stack, which can result
# in uninitialized memory being marked. This can even result in the test failing
# deterministically for a given compilation of ruby. Using a separate thread will
# keep these writes of the symbol pointer on a separate stack that will be garbage
# collected after Thread#join.
Thread.new do
test = %( {{ "some_string" | a_bad_filter }} )
Template.parse(test).render!
nil
end.join
current_symbols = Symbol.all_symbols
GC.start
end
# MRI imprecisely marks objects found on the C stack, which can result
# in uninitialized memory being marked. This can even result in the test failing
# deterministically for a given compilation of ruby. Using a separate thread will
# keep these writes of the symbol pointer on a separate stack that will be garbage
# collected after Thread#join.
Thread.new do
test = %( {{ "some_string" | a_bad_filter }} )
Template.parse(test).render!
nil
end.join
GC.start
assert_equal([], Symbol.all_symbols - current_symbols)
end
def test_does_not_add_drop_methods_to_symbol_table
assigns = { 'drop' => Drop.new }
method_names = Array.new(3) { |index| "untrusted_drop_method_#{object_id}_#{index}" }
method_names.each do |method_name|
assert_equal("", Template.parse("{{ drop.#{method_name} }}").render!(assigns))
end
# JITs can intern internal metadata; untrusted Drop method names must not be interned.
assert_equal([], Symbol.all_symbols.map(&:to_s) & method_names)
end
def assert_no_new_symbols
# Run once to trigger any first-time initialization which might create some symbols,
# for example autoload or lazy method parsing might create symbols on first execution.
yield
# Ensure no new symbols for further runs, i.e. the code does not leak symbols
current_symbols = Symbol.all_symbols
yield
assigns = { 'drop' => Drop.new }
assert_equal("", Template.parse("{{ drop.custom_method_1 }}", assigns).render!)
assert_equal("", Template.parse("{{ drop.custom_method_2 }}", assigns).render!)
assert_equal("", Template.parse("{{ drop.custom_method_3 }}", assigns).render!)
assert_equal([], Symbol.all_symbols - current_symbols)
end
-120
View File
@@ -1,120 +0,0 @@
# frozen_string_literal: true
require 'test_helper'
class SelfDropContextTest < Minitest::Test
include Liquid
def test_self_drop_passed_as_render_param_preserves_original_scope
source = <<~LIQUID
{%- assign var = 42 -%}
{%- assign s = self -%}
{%- render "snippet1", other_self: s -%}
LIQUID
partials = {
'snippet1' => <<~LIQUID,
{%- assign var = 43 -%}
{{- other_self.var }}|{{ self.var -}}
LIQUID
}
assert_template_result('42|43', source, partials: partials)
end
def test_self_drop_in_render_without_passing_resolves_inner_scope
source = <<~LIQUID
{%- assign var = 42 -%}
{%- render "snippet1" -%}
LIQUID
partials = {
'snippet1' => <<~LIQUID,
{%- assign var = 99 -%}
{{- self.var -}}
LIQUID
}
assert_template_result('99', source, partials: partials)
end
def test_self_drop_passed_to_nested_renders_preserves_each_level
source = <<~LIQUID
{%- assign a = 1 -%}
{%- assign s1 = self -%}
{%- render "snippet1", outer: s1 -%}
LIQUID
partials = {
'snippet1' => <<~LIQUID,
{%- assign a = 2 -%}
{%- assign s2 = self -%}
{%- render "snippet2", outer: outer, middle: s2 -%}
LIQUID
'snippet2' => <<~LIQUID,
{%- assign a = 3 -%}
{{- outer.a }}|{{ middle.a }}|{{ self.a -}}
LIQUID
}
assert_template_result('1|2|3', source, partials: partials)
end
def test_self_drop_reflects_variables_assigned_after_creation
source = <<~LIQUID
{%- assign s = self -%}
{%- assign x = 42 %}{{ s.x -}}
LIQUID
assert_template_result('42', source)
end
def test_self_drop_context_setter_is_undefined
context = Context.new
drop = SelfDrop.new(context)
refute(drop.respond_to?(:context=))
assert_template_result('42', '{{ self.x }}', { 'x' => 42 })
end
def test_self_drop_repeated_lookups_compare_equal_for_same_context
context = Context.new
drop = context.find_variable("self")
cached_drop = context.find_variable("self")
assert_same(drop, cached_drop)
assert_equal(drop.object_id, cached_drop.object_id)
assert_equal(drop, cached_drop)
end
def test_assigned_self_drop_compares_equal_to_itself
assert_template_result('T', '{% assign s = self %}{% if s == s %}T{% else %}F{% endif %}')
end
def test_distinct_self_assignments_compare_equal_for_same_context
assert_template_result('T', '{% assign a = self %}{% assign b = self %}{% if a == b %}T{% else %}F{% endif %}')
end
def test_bare_self_compares_equal_to_bare_self
assert_template_result('T', '{% if self == self %}T{% else %}F{% endif %}')
end
def test_self_drop_with_strict_variables_does_not_raise_for_defined_var
t = Template.parse('{{ self.x }}')
result = t.render({ 'x' => 42 }, strict_variables: true)
assert_equal('42', result)
end
def test_self_drop_with_strict_variables_returns_nil_for_undefined_var
t = Template.parse('{{ self.x }}')
result = t.render({}, strict_variables: true)
assert_equal('', result)
end
def test_self_drop_can_be_passed_as_bare_drop_to_render
t = Template.parse('{{ self.x }}')
drop = SelfDrop.new(Context.new({ 'x' => 42 }))
result = t.render(drop)
assert_equal('42', result)
end
end
-2
View File
@@ -1181,8 +1181,6 @@ class StandardFiltersTest < Minitest::Test
end
def test_all_filters_never_raise_non_liquid_exception
skip("too slow on non-CRuby due to many exceptions") unless RUBY_ENGINE == 'ruby'
test_drop = TestDrop.new(value: "test")
test_drop.context = Context.new
test_enum = TestEnumerable.new
+4 -2
View File
@@ -105,8 +105,10 @@ class CycleTagTest < Minitest::Test
error1 = assert_raises(Liquid::SyntaxError) { Template.parse(template1) }
error2 = assert_raises(Liquid::SyntaxError) { Template.parse(template2) }
assert_match(/Liquid syntax error:/, error1.message)
assert_match(/Liquid syntax error: \[:dot, "."\] is not a valid expression/, error2.message)
expected_error = /Liquid syntax error: \[:dot, "."\] is not a valid expression/
assert_match(expected_error, error1.message)
assert_match(expected_error, error2.message)
end
end
-81
View File
@@ -465,85 +465,4 @@ HERE
assert(context.registers[:for_stack].empty?)
end
def test_integer_range_is_not_materialized_and_charges_empty_iterations
range = bounded_integer_range_with_tripwires
template = Template.parse('{% for i in numbers %}{% endfor %}')
template.resource_limits.render_score_limit = 3
assert_raises(Liquid::MemoryError) { template.render!('numbers' => range) }
assert(template.resource_limits.reached?)
assert_equal(4, template.resource_limits.render_score)
end
def test_integer_range_uses_arithmetic_offsets_reversal_and_metadata
range = bounded_integer_range_with_tripwires
template = Template.parse(
'{% for i in numbers reversed offset:997 limit:2 %}{{ forloop.length }}:{{ i }}{% endfor %}',
)
template.resource_limits.render_score_limit = 10
assert_equal('2:9992:998', template.render!('numbers' => range))
end
def test_range_scores_cannot_be_bypassed_by_repeated_renders
template = Template.parse('{% for i in (1..2) %}{% endfor %}')
template.resource_limits.cumulative_render_score_limit = 3
assert_equal('', template.render!)
assert_raises(Liquid::MemoryError) { template.render! }
end
def test_range_break_charges_only_visited_items_and_preserves_full_metadata
range = bounded_integer_range_with_tripwires
template = Template.parse(
'{% for i in numbers reversed %}{{ forloop.length }}:{{ i }}{% break %}{% endfor %}',
)
template.resource_limits.render_score_limit = 10
assert_equal('1000:1000', template.render!('numbers' => range))
assert_operator(template.resource_limits.render_score, :<=, 10)
end
def test_range_subclass_uses_its_custom_each
range = Class.new(Range) do
def each
yield 10
yield 20
end
end.new(nil, 3)
assert_template_result('1020', '{% for i in numbers %}{{ i }}{% endfor %}', { 'numbers' => range })
end
def test_range_subclass_custom_to_a_is_honored_for_finite_and_open_bounds
range_class = Class.new(Range) do
def to_a
[42]
end
end
[range_class.new(1, 3), range_class.new(nil, 3), range_class.new(1, nil)].each do |range|
assert_template_result('42', '{% for i in numbers %}{{ i }}{% endfor %}', { 'numbers' => range })
end
end
def test_endless_range_remains_unsupported_with_a_limit
template = Template.parse('{% for i in numbers limit:2 %}{{ i }}{% endfor %}')
assert_raises(RangeError) { template.render!('numbers' => (1..)) }
end
def test_endless_range_subclass_with_custom_each_remains_unsupported
range = Class.new(Range) do
def each
yield 10
yield 20
end
end.new(1, nil)
assert_raises(RangeError) do
Template.parse('{% for i in numbers %}{{ i }}{% endfor %}').render!('numbers' => range)
end
end
end
-45
View File
@@ -439,49 +439,4 @@ class IncludeTagTest < Minitest::Test
assert_match(/Unexpected character =/, error.message)
end
end
def test_include_for_loop_true_with_for_keyword
with_error_modes(:lax, :strict, :strict2) do
template = Template.parse("{% include 'product' for products %}")
include_node = template.root.nodelist.first
assert(include_node.for_loop?, "Expected for_loop? to be true for 'for' keyword")
end
end
def test_include_for_loop_false_with_with_keyword
with_error_modes(:lax, :strict, :strict2) do
template = Template.parse("{% include 'product' with product %}")
include_node = template.root.nodelist.first
refute(include_node.for_loop?, "Expected for_loop? to be false for 'with' keyword")
end
end
def test_include_for_loop_false_without_keyword
with_error_modes(:lax, :strict, :strict2) do
template = Template.parse("{% include 'header' %}")
include_node = template.root.nodelist.first
refute(include_node.for_loop?, "Expected for_loop? to be false when no keyword")
end
end
def test_include_for_loop_with_alias
with_error_modes(:lax, :strict, :strict2) do
template = Template.parse("{% include 'product' for products as item %}")
include_node = template.root.nodelist.first
assert(include_node.for_loop?, "Expected for_loop? to be true for 'for' with alias")
end
end
def test_include_with_keyword_and_alias
with_error_modes(:lax, :strict, :strict2) do
template = Template.parse("{% include 'product' with products[0] as item %}")
include_node = template.root.nodelist.first
refute(include_node.for_loop?, "Expected for_loop? to be false for 'with' with alias")
end
end
end # IncludeTagTest
@@ -27,50 +27,4 @@ class IncrementTagTest < Minitest::Test
'{%decrement starboard %}',
)
end
def test_increment_strict2_rejects_invalid_variable_name
assert_raises(Liquid::SyntaxError) do
Template.parse('{% increment foo bar %}', error_mode: :strict2)
end
end
def test_increment_strict2_rejects_variable_starting_with_number
assert_raises(Liquid::SyntaxError) do
Template.parse('{% increment 11aa %}', error_mode: :strict2)
end
end
def test_increment_strict2_accepts_valid_variable_name
template = Template.parse('{% increment my-var %}', error_mode: :strict2)
assert_equal('0', template.render)
end
def test_decrement_strict2_rejects_invalid_variable_name
assert_raises(Liquid::SyntaxError) do
Template.parse('{% decrement foo bar %}', error_mode: :strict2)
end
end
def test_decrement_strict2_rejects_variable_starting_with_number
assert_raises(Liquid::SyntaxError) do
Template.parse('{% decrement 11aa %}', error_mode: :strict2)
end
end
def test_decrement_strict2_accepts_valid_variable_name
template = Template.parse('{% decrement my-var %}', error_mode: :strict2)
assert_equal('-1', template.render)
end
def test_increment_strict2_rejects_empty_variable_name
assert_raises(Liquid::SyntaxError) do
Template.parse('{% increment %}', error_mode: :strict2)
end
end
def test_decrement_strict2_rejects_empty_variable_name
assert_raises(Liquid::SyntaxError) do
Template.parse('{% decrement %}', error_mode: :strict2)
end
end
end
+4 -4
View File
@@ -174,16 +174,16 @@ class RenderTagTest < Minitest::Test
def test_increment_is_isolated_between_renders
assert_template_result(
'010',
'{% increment port %}{% increment port %}{% render "incr" %}',
partials: { 'incr' => '{% increment port %}' },
'{% increment %}{% increment %}{% render "incr" %}',
partials: { 'incr' => '{% increment %}' },
)
end
def test_decrement_is_isolated_between_renders
assert_template_result(
'-1-2-1',
'{% decrement port %}{% decrement port %}{% render "decr" %}',
partials: { 'decr' => '{% decrement port %}' },
'{% decrement %}{% decrement %}{% render "decr" %}',
partials: { 'decr' => '{% decrement %}' },
)
end
-90
View File
@@ -465,94 +465,4 @@ class TableRowTest < Minitest::Test
assert_match(/Unexpected character =/, error.message)
end
end
def test_integer_range_is_not_materialized_and_charges_empty_iterations
range = bounded_integer_range_with_tripwires
template = Template.parse('{% tablerow i in numbers %}{% endtablerow %}')
template.resource_limits.render_score_limit = 3
assert_raises(Liquid::MemoryError) { template.render!('numbers' => range) }
assert(template.resource_limits.reached?)
assert_equal(4, template.resource_limits.render_score)
end
def test_integer_range_uses_arithmetic_offsets_limit_and_full_metadata
range = bounded_integer_range_with_tripwires
template = Template.parse(
'{% tablerow i in numbers offset:997 limit:2 %}{{ tablerowloop.length }}:{{ tablerowloop.index }}:{{ i }}{% endtablerow %}',
)
assert_equal(
"<tr class=\"row1\">\n<td class=\"col1\">2:1:998</td><td class=\"col2\">2:2:999</td></tr>\n",
template.render!('numbers' => range),
)
end
def test_tablerow_checks_generated_output_during_empty_body_iteration
template = Template.parse('{% tablerow i in (1..100) %}{% endtablerow %}')
template.resource_limits.render_length_limit = 40
checked_lengths = []
limits = template.resource_limits
original_increment_write_score = limits.method(:increment_write_score)
limits.define_singleton_method(:increment_write_score) do |output|
checked_lengths << output.bytesize
original_increment_write_score.call(output)
end
assert_equal('Liquid error: Memory limits exceeded', template.render)
assert(template.resource_limits.reached?)
assert_operator(checked_lengths.last, :<, 1000)
end
def test_tablerow_range_scores_persist_across_renders
template = Template.parse('{% tablerow i in (1..2) %}{% endtablerow %}')
template.resource_limits.cumulative_render_score_limit = 3
template.render!
assert_raises(Liquid::MemoryError) { template.render! }
end
def test_range_subclass_uses_its_custom_each_with_beginless_bounds
range = Class.new(Range) do
def each
yield 10
yield 20
end
end.new(nil, 3)
assert_template_result(
"<tr class=\"row1\">\n<td class=\"col1\">10</td><td class=\"col2\">20</td></tr>\n",
'{% tablerow i in numbers %}{{ i }}{% endtablerow %}',
{ 'numbers' => range },
)
end
def test_endless_range_subclass_uses_its_custom_each_without_a_limit
range = Class.new(Range) do
def each
yield 10
yield 20
end
end.new(1, nil)
assert_template_result(
"<tr class=\"row1\">\n<td class=\"col1\">10</td><td class=\"col2\">20</td></tr>\n",
'{% tablerow i in numbers %}{{ i }}{% endtablerow %}',
{ 'numbers' => range },
)
end
def test_range_subclass_custom_to_a_is_not_used
range = Class.new(Range) do
def to_a
[42]
end
end.new(1, 3)
assert_template_result(
"<tr class=\"row1\">\n<td class=\"col1\">1</td><td class=\"col2\">2</td><td class=\"col3\">3</td></tr>\n",
'{% tablerow i in numbers %}{{ i }}{% endtablerow %}',
{ 'numbers' => range },
)
end
end
+1 -81
View File
@@ -132,7 +132,7 @@ class TemplateTest < Minitest::Test
assert_equal("Liquid error: Memory limits exceeded", t.render)
assert(t.resource_limits.reached?)
t.resource_limits.render_score_limit = 201
t.resource_limits.render_score_limit = 200
assert_equal(" foo " * 100, t.render!)
refute_nil(t.resource_limits.render_score)
end
@@ -179,86 +179,6 @@ class TemplateTest < Minitest::Test
assert_equal("すごい", t.render)
end
def test_cumulative_render_score_limit_across_render_tags
file_system = StubFileSystem.new(
'loop' => '{% for a in (1..10) %} foo {% endfor %}',
)
environment = Liquid::Environment.build(file_system: file_system)
# Without cumulative limit, all 5 partials render successfully
t = Template.parse(
'{% render "loop" %}{% render "loop" %}{% render "loop" %}{% render "loop" %}{% render "loop" %}',
environment: environment,
)
unlimited_output = t.render!
total_cumulative = t.resource_limits.cumulative_render_score
# With cumulative limit set below the total, rendering stops early
t2 = Template.parse(
'{% render "loop" %}{% render "loop" %}{% render "loop" %}{% render "loop" %}{% render "loop" %}',
environment: environment,
)
t2.resource_limits.cumulative_render_score_limit = total_cumulative / 2
limited_output = t2.render
assert(t2.resource_limits.reached?)
assert_operator(limited_output.length, :<, unlimited_output.length)
end
def test_cumulative_render_score_limit_raises_on_render_bang
file_system = StubFileSystem.new(
'loop' => '{% for a in (1..10) %} foo {% endfor %}',
)
environment = Liquid::Environment.build(file_system: file_system)
t = Template.parse(
'{% render "loop" %}{% render "loop" %}{% render "loop" %}{% render "loop" %}{% render "loop" %}',
environment: environment,
)
t.resource_limits.cumulative_render_score_limit = 20
assert_raises(Liquid::MemoryError) do
t.render!
end
end
def test_cumulative_assign_score_limit_across_include_tags
file_system = StubFileSystem.new(
'assign_partial' => '{% assign x = "a long string value here" %}',
)
environment = Liquid::Environment.build(file_system: file_system)
# Without cumulative limit, all 5 partials render
t = Template.parse(
'{% include "assign_partial" %}{% include "assign_partial" %}{% include "assign_partial" %}{% include "assign_partial" %}{% include "assign_partial" %}',
environment: environment,
)
t.render!
total_cumulative = t.resource_limits.cumulative_assign_score
# With cumulative limit set below the total, rendering stops early
t2 = Template.parse(
'{% include "assign_partial" %}{% include "assign_partial" %}{% include "assign_partial" %}{% include "assign_partial" %}{% include "assign_partial" %}',
environment: environment,
)
t2.resource_limits.cumulative_assign_score_limit = total_cumulative / 2
t2.render
assert(t2.resource_limits.reached?)
end
def test_cumulative_render_score_tracks_across_partials_without_limit
file_system = StubFileSystem.new(
'loop' => '{% for a in (1..10) %} foo {% endfor %}',
)
environment = Liquid::Environment.build(file_system: file_system)
t = Template.parse(
'{% render "loop" %}{% render "loop" %}{% render "loop" %}',
environment: environment,
)
t.render!
assert(
t.resource_limits.cumulative_render_score > t.resource_limits.render_score,
"cumulative should exceed per-template score after multiple partials",
)
end
def test_default_resource_limits_unaffected_by_render_with_context
context = Context.new
t = Template.parse("{% for a in (1..100) %}x{% assign foo = 1 %} {% endfor %}")
+5 -5
View File
@@ -55,7 +55,7 @@ class VariableTest < Minitest::Test
def test_expression_with_whitespace_in_square_brackets
assert_template_result('result', "{{ a[ 'b' ] }}", { 'a' => { 'b' => 'result' } })
assert_template_result('result', "{{ a[ self[ 'b' ] ] }}", { 'b' => 'c', 'a' => { 'c' => 'result' } })
assert_template_result('result', "{{ a[ [ 'b' ] ] }}", { 'b' => 'c', 'a' => { 'c' => 'result' } })
end
def test_ignore_unknown
@@ -135,17 +135,17 @@ class VariableTest < Minitest::Test
end
def test_dynamic_find_var
assert_template_result('bar', '{{ self[key] }}', { 'key' => 'foo', 'foo' => 'bar' })
assert_template_result('bar', '{{ [key] }}', { 'key' => 'foo', 'foo' => 'bar' })
end
def test_raw_value_variable
assert_template_result('bar', '{{ self[key] }}', { 'key' => 'foo', 'foo' => 'bar' })
assert_template_result('bar', '{{ [key] }}', { 'key' => 'foo', 'foo' => 'bar' })
end
def test_dynamic_find_var_with_drop
assert_template_result(
'bar',
'{{ self[list[settings.zero]] }}',
'{{ [list[settings.zero]] }}',
{
'list' => ['foo'],
'settings' => SettingsDrop.new("zero" => 0),
@@ -155,7 +155,7 @@ class VariableTest < Minitest::Test
assert_template_result(
'foo',
'{{ self[list[settings.zero]["foo"]] }}',
'{{ [list[settings.zero]["foo"]] }}',
{
'list' => [{ 'foo' => 'bar' }],
'settings' => SettingsDrop.new("zero" => 0),
-9
View File
@@ -32,15 +32,6 @@ module Minitest
module Assertions
include Liquid
# Exact Range fixture for fast-path tests; singleton tripwires must remain
# untouched because the arithmetic path does not materialize or traverse it.
def bounded_integer_range_with_tripwires
range = (1..1000).dup
range.define_singleton_method(:to_a) { raise 'range was materialized' }
range.define_singleton_method(:each) { raise 'range was traversed' }
range
end
def assert_template_result(
expected, template, assigns = {},
message: nil, partials: nil, error_mode: Liquid::Environment.default.error_mode, render_errors: false,
-85
View File
@@ -1,85 +0,0 @@
# frozen_string_literal: true
require 'test_helper'
class RangeSliceUnitTest < Minitest::Test
def test_selects_and_reverses_an_integer_range_without_enumerating_it
limits = Liquid::ResourceLimits.new({})
slice = Liquid::RangeSlice.new(bounded_integer_range_with_tripwires, 997, 999, limits)
assert_equal(2, slice.length)
refute(slice.empty?)
slice.reverse!
assert_equal([999, 998], slice.each.to_a)
assert_equal(2, limits.render_score)
end
def test_charges_only_values_yielded_before_a_break
limits = Liquid::ResourceLimits.new({})
slice = Liquid::RangeSlice.new(bounded_integer_range_with_tripwires, 0, nil, limits)
slice.each { break }
assert_equal(1, limits.render_score)
end
def test_non_integer_ranges_are_sliced_without_to_a_and_charge_visited_values
limits = Liquid::ResourceLimits.new({})
range = Class.new(Range) do
def to_a
raise 'range was materialized'
end
end.new('a', 'c')
assert_equal(['b', 'c'], Liquid::Utils.slice_collection_for_iteration(range, 1, nil, limits))
assert_equal(3, limits.render_score)
limited = Liquid::ResourceLimits.new(render_score_limit: 2)
assert_raises(Liquid::MemoryError) do
Liquid::Utils.slice_collection_for_iteration('a'..'z', 10, 11, limited)
end
end
def test_non_integer_range_empty_windows_do_not_visit_a_sentinel_value
limits = Liquid::ResourceLimits.new({})
assert_equal([], Liquid::Utils.slice_collection_for_iteration('a'..'z', 2, 2, limits))
assert_equal(0, limits.render_score)
assert_equal(['a', 'b'], Liquid::Utils.slice_collection_for_iteration('a'..'z', 0, 2, limits))
assert_equal(2, limits.render_score)
end
def test_standard_beginless_range_raises_for_empty_windows
[[0, 0], [1, 0]].each do |from, to|
assert_raises(TypeError) do
Liquid::Utils.slice_collection_for_iteration(Range.new(nil, 3), from, to, Liquid::ResourceLimits.new({}))
end
end
end
def test_custom_range_to_a_is_sliced_with_a_budget
range = Class.new(Range) do
def to_a
[1, 2, 3]
end
end.new(nil, 3)
limits = Liquid::ResourceLimits.new(render_score_limit: 2)
assert_raises(Liquid::MemoryError) do
Liquid::Utils.slice_collection_for_iteration(range, 1, 3, limits, use_range_to_a: true)
end
assert_equal(3, limits.render_score)
end
def test_preserves_slice_bounds_for_negative_offsets_and_limits
limits = Liquid::ResourceLimits.new({})
assert_equal([1, 2], Liquid::RangeSlice.new(1..5, -2, 2, limits).each.to_a)
empty = Liquid::RangeSlice.new(1..5, 2, 1, limits)
assert(empty.empty?)
assert_equal([], empty.each.to_a)
assert_equal(2, limits.render_score) # the empty window performs no work
assert(Liquid::RangeSlice.new(5..1, 0, nil, limits).empty?)
assert_equal([1, 2, 3, 4], Liquid::RangeSlice.new(1...5, 0, nil, limits).each.to_a)
end
end
-91
View File
@@ -1,91 +0,0 @@
# frozen_string_literal: true
require 'test_helper'
class ResourceLimitsUnitTest < Minitest::Test
def test_cumulative_scores_initialize_to_zero
limits = Liquid::ResourceLimits.new({})
assert_equal(0, limits.cumulative_render_score)
assert_equal(0, limits.cumulative_assign_score)
end
def test_cumulative_limits_default_to_nil
limits = Liquid::ResourceLimits.new({})
assert_nil(limits.cumulative_render_score_limit)
assert_nil(limits.cumulative_assign_score_limit)
end
def test_cumulative_limits_configurable_via_hash
limits = Liquid::ResourceLimits.new(
cumulative_render_score_limit: 500,
cumulative_assign_score_limit: 300,
)
assert_equal(500, limits.cumulative_render_score_limit)
assert_equal(300, limits.cumulative_assign_score_limit)
end
def test_cumulative_limits_configurable_via_accessor
limits = Liquid::ResourceLimits.new({})
limits.cumulative_render_score_limit = 500
assert_equal(500, limits.cumulative_render_score_limit)
end
def test_cumulative_scores_survive_reset
limits = Liquid::ResourceLimits.new({})
limits.increment_render_score(10)
limits.increment_assign_score(5)
limits.reset
assert_equal(0, limits.render_score)
assert_equal(0, limits.assign_score)
assert_equal(10, limits.cumulative_render_score)
assert_equal(5, limits.cumulative_assign_score)
end
def test_cumulative_scores_accumulate_across_resets
limits = Liquid::ResourceLimits.new({})
limits.increment_render_score(10)
limits.reset
limits.increment_render_score(20)
limits.reset
limits.increment_render_score(30)
assert_equal(30, limits.render_score)
assert_equal(60, limits.cumulative_render_score)
end
def test_cumulative_render_score_limit_raises
limits = Liquid::ResourceLimits.new(cumulative_render_score_limit: 25)
limits.increment_render_score(10)
limits.reset
limits.increment_render_score(10)
limits.reset
assert_raises(Liquid::MemoryError) do
limits.increment_render_score(10)
end
assert(limits.reached?)
end
def test_cumulative_assign_score_limit_raises
limits = Liquid::ResourceLimits.new(cumulative_assign_score_limit: 15)
limits.increment_assign_score(8)
limits.reset
assert_raises(Liquid::MemoryError) do
limits.increment_assign_score(8)
end
assert(limits.reached?)
end
def test_per_template_limits_still_work_with_cumulative
limits = Liquid::ResourceLimits.new(
render_score_limit: 50,
cumulative_render_score_limit: 1000,
)
assert_raises(Liquid::MemoryError) do
limits.increment_render_score(51)
end
end
end