Compare commits

..
Author SHA1 Message Date
Chris Pak 731f64deef Improves performance on hot parse and render paths
Splits the render loop in block_body.rb on the loop-invariant check_write
condition. The common case (no resource limits) now pays zero branch cost
per node.

Rewrites truncatewords in standardfilters.rb to scan word positions into a
flat int array and builds the result string only when truncation is confirmed.
No string allocation in the common no-truncation case beyond the array itself.

Simplifies rest_blank? in cursor.rb: replaces manual save/skip/restore of
StringScanner position with [email protected]?(/\S/). exist? does not advance
position; returns nil when no non-whitespace remains; handles EOS correctly.

Removes the nl newline counter from skip_ws in cursor.rb -- all callers
discarded the return value. NL now handled in the same when-branch as the
other whitespace bytes.
2026-04-04 22:14:19 -07:00
Chris Pak 84779f8a28 Removes dead code, tightens idioms, adds clarifying comments
Dead code removed:
  cursor.rb:     parse_for_markup + attr_reader :for_var/collection/reversed
                 (zero callers in lib/; method also incomplete -- no limit/offset)
  for.rb:        Syntax regex, REVERSED_BYTES constant (both unreferenced by
                 lax_parse and strict_parse)
  block_body.rb: BLANK_STRING_REGEX (exact duplicate of WhitespaceOrNothing)

Idioms:
  expression.rb:    parse_number returns nil on failure, not false
                    (Ruby convention for 'no result'; callers use if (num = parse_number))
  block_body.rb:    freeze (idempotent by spec); whitespace_handler marked private;
                    render_node gets rationale comment; redundant comments collapsed
  variable_lookup.rb: COMMAND_METHODS -> %w[]; initialize loop uses each_with_index;
                    removes redundant &. on second clause of &&
  for.rb:           strict2_parse -> alias_method; nil-guard if/else -> ternary;
                    render_else -> ternary; include? checks unconsumed rest only
  condition.rb:     loop/break chain -> while condition.child_relation
  utils.rb:         tightens slice_collection_using_each loop

Comments:
  variable.rb:         backward pipe-walk algorithm; SPACE-only whitespace asymmetry
  cursor.rb:           COMPARISON_OPS identity-map exists for frozen string interning
  if.rb:               include? pre-check is both correctness guard and perf gate
  for.rb:              cursor->regex fallback for limit:/offset: attributes
  strainer_template.rb: __LINE__+1 limitation in module_eval loop
2026-04-04 22:13:43 -07:00
Chris Pak ba11b85875 Decomposes, extracts, and names the important concepts
Decomposes the 211-line try_fast_parse monolith in variable.rb into four
named private methods -- fast_scan_name, fast_resolve_name, fast_scan_filters,
fast_scan_filter_args -- and simplifies simple_variable_markup from a 55-line
byte scanner to a 10-line regex with fast pre-checks.

Generates invoke_single/invoke_two via module_eval in strainer_template.rb
and context.rb instead of duplicating near-identical method bodies. The two
arity variants differ only in parameter lists; generating them makes the
pattern explicit and eliminates copy-paste drift.

Extracts identical 4-line text-token handling block in block_body.rb into
private append_text_token(token, parse_context). The block appeared in
both the stray-{ fallback and the plain-text branch of parse_for_document.

Extracts accessible?(object, key) predicate from the 4-line inline ternary
in variable_lookup.rb that checked hash/array key presence; extracts
liquidize(object, context) from the duplicated to_liquid + context= wiring
that appeared in both the key-found and command-method branches.

Extracts find_in_envs(envs, key, raise_on_not_found:) from
try_variable_find_in_environments in context.rb, which looped over
@environments then @static_environments with identical loop bodies.

Adds 28 fast-path equivalence tests for Variable (variable_fast_parse_test.rb).
2026-04-04 22:13:43 -07:00
Chris Pak 03e5e29b0b Adds ByteTables, moves cursor load order, consolidates byte constants
Adds ByteTables (lib/liquid/byte_tables.rb): four frozen 256-entry boolean
lookup arrays replacing inline byte-range comparisons throughout:
  IDENT_START, IDENT_CONT, DIGIT, WHITESPACE

Moves require 'liquid/cursor' to immediately after byte_tables in liquid.rb.
Cursor has zero Liquid dependencies; loading it early lets every subsequent
file reference Cursor:: constants directly.

Removes all local byte-constant definitions that duplicated Cursor::
  tokenizer.rb   OPEN_CURLEY / CLOSE_CURLEY / PERCENTAGE
  block_body.rb  OPEN_CURLEY_BYTE / PERCENT_BYTE / DASH_BYTE / CLOSE_CURLEY_BYTE
  expression.rb  DOT / DASH / ZERO / NINE / INTEGER_REGEX / FLOAT_REGEX

Replaces inline byte-range comparisons with ByteTables lookups in:
  cursor.rb, variable_lookup.rb, expression.rb, standardfilters.rb

Removes incidental dead code alongside the constant consolidation:
  tokenizer.rb: require 'strscan', unused string_scanner: param,
                @ss = nil, .to_s.to_str, 'tokenize if @source' guard
  block_body.rb: require 'English'
2026-04-04 22:09:14 -07:00
Chris Pak d9c42fd2eb Fixes infinite loop in tokenizer on trailing stray '{'
The stray-{ else branch in tokenize_fast had a nested while loop that could
exit via its condition (when next_open >= len) without advancing pos.
The outer while pos < len loop would then find the same { again forever.

Reproduced by: Liquid::Template.parse('a{') -- hangs indefinitely.

Replaces the nested scan loop with two String#byteindex calls to find the
next '{%' and '{{' directly, then takes the minimum. Always O(n), eliminates
the nested loop entirely, and impossible to leave pos stranded.

Adds regression tests covering the three inputs that previously hung
plus adjacent stray-brace cases.
2026-04-04 21:25:59 -07:00
Tobi LutkeandChris Pak bc60deb671 update autoresearch experiment log 2026-04-04 17:42:33 -07:00
Tobi LutkeandChris Pak 17b691ee32 Replace @blocks.each with while loop in If render — avoids block proc allocation per render\n\nResult: {"status":"keep","combined_µs":3496,"parse_µs":2356,"render_µs":1140,"allocations":24530} 2026-04-04 17:42:33 -07:00
Tobi LutkeandChris Pak de95af5414 Inline to_liquid_value in If render — avoids one method dispatch per condition evaluation\n\nResult: {"status":"keep","combined_µs":3459,"parse_µs":2318,"render_µs":1141,"allocations":24647} 2026-04-04 17:42:33 -07:00
Tobi LutkeandChris Pak 6faa62697b Replace simple_lookup? byte scan with match? regex — 8x faster per call, cleaner code\n\nResult: {"status":"keep","combined_µs":3489,"parse_µs":2353,"render_µs":1136,"allocations":24647} 2026-04-04 17:42:33 -07:00
Tobi LutkeandChris Pak f39fad8934 Condition#evaluate: skip loop block for simple conditions (no child_relation) — saves 235 allocs\n\nResult: {"status":"keep","combined_µs":3445,"parse_µs":2284,"render_µs":1161,"allocations":24647} 2026-04-04 17:42:33 -07:00
Tobi LutkeandChris Pak 1aa5854f44 Clean confirmation run: 3,314µs (-55% from main), stable\n\nResult: {"status":"keep","combined_µs":3314,"parse_µs":2203,"render_µs":1111,"allocations":24882} 2026-04-04 17:42:33 -07:00
Tobi LutkeandChris Pak feb7036664 update autoresearch docs with current progress 2026-04-04 17:42:33 -07:00
Tobi LutkeandChris Pak 111aeed8f1 parse_tag_token without StringScanner: pure byte ops avoid reset(token) overhead, -12% combined\n\nResult: {"status":"keep","combined_µs":3350,"parse_µs":2212,"render_µs":1138,"allocations":24882} 2026-04-04 17:42:33 -07:00
Tobi LutkeandChris Pak 9cbfae22b0 Clean up tokenizer: remove unused StringScanner setup and regex constants\n\nResult: {"status":"keep","combined_µs":3490,"parse_µs":2331,"render_µs":1159,"allocations":24882} 2026-04-04 17:42:33 -07:00
Tobi LutkeandChris Pak 89076dbf3a Confirmation run: byteindex tokenizer consistently 3,400-3,600µs\n\nResult: {"status":"keep","combined_µs":3464,"parse_µs":2335,"render_µs":1129,"allocations":24882} 2026-04-04 17:42:33 -07:00
Tobi LutkeandChris Pak f2c0fbfa0b Replace StringScanner tokenizer with String#byteindex — 12% faster parse, no regex overhead for delimiter finding\n\nResult: {"status":"keep","combined_µs":3556,"parse_µs":2388,"render_µs":1168,"allocations":24882} 2026-04-04 17:42:33 -07:00
Tobi LutkeandChris Pak 78550c0e0d Baseline: 3,818µs combined, 24,881 allocs\n\nResult: {"status":"keep","combined_µs":3818,"parse_µs":2722,"render_µs":1096,"allocations":24881} 2026-04-04 17:42:33 -07:00
Tobi LutkeandChris Pak 68d697d1c5 Skip context.evaluate for String lookup keys in VariableLookup — avoids respond_to? dispatch\n\nResult: {"status":"keep","combined_µs":4103,"parse_µs":2881,"render_µs":1222,"allocations":24881} 2026-04-04 17:42:33 -07:00
Tobi LutkeandChris Pak a206932c4a update autoresearch.md with current progress 2026-04-04 17:42:33 -07:00
Tobi LutkeandChris Pak 8a39c3aa33 Cache no-arg filter tuples [name, EMPTY_ARRAY] — reuse frozen tuples across templates\n\nResult: {"status":"keep","combined_µs":4147,"parse_µs":2992,"render_µs":1155,"allocations":24881} 2026-04-04 17:42:33 -07:00
Tobi LutkeandChris Pak 89fcea371e Replace manual blank_string? with regex match — cleaner code\n\nResult: {"status":"keep","combined_µs":4196,"parse_µs":3042,"render_µs":1154,"allocations":25535} 2026-04-04 17:42:33 -07:00
Tobi LutkeandChris Pak de8675f8fb Skip to_liquid_value for String/Integer keys in VariableLookup — avoids respond_to? dispatch\n\nResult: {"status":"keep","combined_µs":4131,"parse_µs":2893,"render_µs":1238,"allocations":25535} 2026-04-04 17:42:33 -07:00
Tobi LutkeandChris Pak e3ba4aa958 Minor cleanup: optimize expect_id with while loop and early return\n\nResult: {"status":"keep","combined_µs":4184,"parse_µs":2921,"render_µs":1263,"allocations":25535} 2026-04-04 17:42:33 -07:00
Tobi LutkeandChris Pak c3e55e92d9 Replace manual scan_dotted_id with regex\n\nResult: {"status":"keep","combined_µs":4121,"parse_µs":2812,"render_µs":1309,"allocations":25535} 2026-04-04 17:42:33 -07:00
Tobi LutkeandChris Pak 0924e59840 Replace manual scan_quoted_string with regex capture groups\n\nResult: {"status":"keep","combined_µs":4102,"parse_µs":2849,"render_µs":1253,"allocations":25535} 2026-04-04 17:42:33 -07:00
Tobi LutkeandChris Pak 8d1f030825 Replace manual rest_blank? with regex skip + eos? check\n\nResult: {"status":"keep","combined_µs":4047,"parse_µs":2795,"render_µs":1252,"allocations":25535} 2026-04-04 17:42:33 -07:00
Tobi LutkeandChris Pak 4896d6d497 Replace manual scan_comparison_op with regex — cleaner and avoids byteslice allocation for op strings\n\nResult: {"status":"keep","combined_µs":4007,"parse_µs":2808,"render_µs":1199,"allocations":25535} 2026-04-04 17:42:33 -07:00
Tobi LutkeandChris Pak edd8fabb3a Replace manual scan_fragment/scan_quoted_string_raw/skip_fragment with regex — cleaner, same/better perf\n\nResult: {"status":"keep","combined_µs":4132,"parse_µs":2890,"render_µs":1242,"allocations":25535} 2026-04-04 17:42:33 -07:00
Tobi LutkeandChris Pak 28ede9c539 Replace manual byte-level scan_number with regex — cleaner code, same performance\n\nResult: {"status":"keep","combined_µs":4184,"parse_µs":2931,"render_µs":1253,"allocations":25535} 2026-04-04 17:42:33 -07:00
Tobi LutkeandChris Pak 14ac591403 Replace manual byte-level scan_id/skip_id with regex — C-level StringScanner.scan is faster than Ruby-level byte scanning\n\nResult: {"status":"keep","combined_µs":4185,"parse_µs":2943,"render_µs":1242,"allocations":25535} 2026-04-04 17:42:33 -07:00
Tobi LutkeandChris Pak 3f9b8916b2 Fast-path Hash lookups in VariableLookup#evaluate — skip respond_to? checks for Hash objects\n\nResult: {"status":"keep","combined_µs":4110,"parse_µs":2922,"render_µs":1188,"allocations":25535} 2026-04-04 17:42:33 -07:00
Tobi LutkeandChris Pak 41de814045 Skip to_liquid/context= for primitives in VariableLookup#evaluate\n\nResult: {"status":"keep","combined_µs":4334,"parse_µs":3062,"render_µs":1272,"allocations":25535} 2026-04-04 17:42:33 -07:00
Tobi LutkeandChris Pak 91d9a509f9 Fast return for primitive types in find_variable — skip to_liquid and respond_to?(:context=)\n\nResult: {"status":"keep","combined_µs":4225,"parse_µs":3009,"render_µs":1216,"allocations":25535} 2026-04-04 17:42:33 -07:00
Tobi LutkeandChris Pak 37a4c19405 Skip find_index when only one scope in find_variable — go straight to environments\n\nResult: {"status":"keep","combined_µs":4323,"parse_µs":3055,"render_µs":1268,"allocations":25535} 2026-04-04 17:42:33 -07:00
Tobi LutkeandChris Pak dc35b765e8 Skip respond_to?(:context=) for primitive types in find_variable — avoids method lookup overhead\n\nResult: {"status":"keep","combined_µs":4207,"parse_µs":2943,"render_µs":1264,"allocations":25535} 2026-04-04 17:42:33 -07:00
Tobi LutkeandChris Pak 1dfdce82dc Use EMPTY_ARRAY for empty static_environments in Context — avoids 60 array allocs per render cycle\n\nResult: {"status":"keep","combined_µs":4262,"parse_µs":3079,"render_µs":1183,"allocations":25535} 2026-04-04 17:42:33 -07:00
Tobi LutkeandChris Pak bf16d291c9 Lazy @changes hash in Registers — only allocate when a register is actually written\n\nResult: {"status":"keep","combined_µs":4287,"parse_µs":3059,"render_µs":1228,"allocations":25595} 2026-04-04 17:42:33 -07:00
Tobi LutkeandChris Pak 2da3ae330a Cache block_delimiter strings per tag name — avoids repeated string interpolation\n\nResult: {"status":"keep","combined_µs":4372,"parse_µs":3127,"render_µs":1245,"allocations":25605} 2026-04-04 17:42:33 -07:00
Tobi LutkeandChris Pak 1800cffd3b Lazy Context init: defer StringScanner and @interrupts array allocation until needed\n\nResult: {"status":"keep","combined_µs":4299,"parse_µs":3057,"render_µs":1242,"allocations":26015} 2026-04-04 17:42:33 -07:00
Tobi LutkeandChris Pak 02764d28f7 Cache small integer to_s (0-999): avoids 267 Integer#to_s allocations per render cycle\n\nResult: {"status":"keep","combined_µs":4158,"parse_µs":2920,"render_µs":1238,"allocations":26128} 2026-04-04 17:42:33 -07:00
Tobi LutkeandChris Pak f08fe63042 Replace split+join in truncatewords with manual word scan — avoids array + string allocations\n\nResult: {"status":"keep","combined_µs":4280,"parse_µs":3009,"render_µs":1271,"allocations":26395} 2026-04-04 17:42:33 -07:00
Tobi LutkeandChris Pak 588966fd1c Extend fast-path filter parsing to handle comma-separated multi-arg filters (e.g. pluralize: 'item', 'items')\n\nResult: {"status":"keep","combined_µs":4266,"parse_µs":3032,"render_µs":1234,"allocations":26480} 2026-04-04 17:42:33 -07:00
Tobi LutkeandChris Pak 0e5edcc9da Avoid expr_markup byteslice when name is entire markup string (no whitespace, no filters)\n\nResult: {"status":"keep","combined_µs":4277,"parse_µs":3057,"render_µs":1220,"allocations":27026} 2026-04-04 17:42:33 -07:00
Tobi LutkeandChris Pak 3f10ac702c Fast-path single-arg filter parsing: handle quoted strings, numbers, identifiers without Lexer/Parser\n\nResult: {"status":"keep","combined_µs":4427,"parse_µs":3181,"render_µs":1246,"allocations":27235} 2026-04-04 17:42:33 -07:00
Tobi LutkeandChris Pak f31930555e fix rubocop offenses: autocorrect style/layout violations 2026-04-04 17:42:32 -07:00
Tobi LutkeandChris Pak f0ac941ac6 update autoresearch.md with full progress log 2026-04-04 17:42:32 -07:00
Tobi LutkeandChris Pak 19bf49c116 For tag: migrate lax_parse to Cursor with zero-alloc skip_id/expect_id 2026-04-04 17:42:32 -07:00
Tobi LutkeandChris Pak 5204033588 Cursor: add skip_id, expect_id, skip_fragment for zero-alloc scanning 2026-04-04 17:42:32 -07:00
Tobi LutkeandChris Pak b0c9f576af REVERTED: Cursor for For tag adds 148 allocs from scan_id/scan_fragment string creation\n\nResult: {"status":"discard","combined_µs":5049,"parse_us":3765,"render_us":1284,"allocations":29793,"parse_µs":3765,"render_µs":1284} 2026-04-04 17:42:32 -07:00
Tobi LutkeandChris Pak 343ae1da56 remove dead BlockBody.parse_tag_token and If SIMPLE_CONDITION - now in Cursor 2026-04-04 17:42:32 -07:00
Tobi LutkeandChris Pak f42593dc50 introduce Cursor class: centralize byte-level scanning for tag/variable/condition parsing 2026-04-04 17:42:32 -07:00
Tobi LutkeandChris Pak 80edd212a0 add parse_simple to skip simple_lookup? check when caller validates 2026-04-04 17:42:32 -07:00
Tobi LutkeandChris Pak 65d7568403 fast-path VariableLookup init: skip scan_variable for simple identifier chains 2026-04-04 17:42:32 -07:00
Tobi LutkeandChris Pak c2ba6b0676 avoid allocating seen={} hash in Utils.to_s/inspect when not needed 2026-04-04 17:42:32 -07:00
Tobi LutkeandChris Pak 92ca381fb1 update autoresearch.sh: 3-run best-of, skip liquid-spec for speed 2026-04-04 17:42:32 -07:00
Tobi LutkeandChris Pak 1032d57532 optimize Context init: avoid unnecessary array wrapping for environments 2026-04-04 17:42:32 -07:00
Tobi LutkeandChris Pak e1a0e7e716 use frozen EMPTY_ARRAY/EMPTY_HASH for Context @filters/@disabled_tags 2026-04-04 17:42:32 -07:00
Tobi LutkeandChris Pak 1f309b19a7 replace INTEGER_REGEX/FLOAT_REGEX with byte-level parse_number 2026-04-04 17:42:32 -07:00
Tobi LutkeandChris Pak c6617accc5 replace SIMPLE_CONDITION regex with manual byte parser in if/elsif lax_parse 2026-04-04 17:42:32 -07:00
Tobi LutkeandChris Pak cf062e1da2 fast-path slice_collection: skip copy for full Array without offset/limit 2026-04-04 17:42:32 -07:00
Tobi LutkeandChris Pak 06a718cd69 add invoke_two fast path for single-arg filter invocation, avoids splat chain 2026-04-04 17:42:32 -07:00
Tobi LutkeandChris Pak 50888426a1 fast-path find_variable: check top scope first before find_index 2026-04-04 17:42:32 -07:00
Tobi LutkeandChris Pak 1a29584f96 add invoke_single fast path for no-arg filter invocation, avoids splat alloc 2026-04-04 17:42:32 -07:00
Tobi LutkeandChris Pak 9d16eb3d05 fast-path simple if truthiness: use byte scanner before SIMPLE_CONDITION regex 2026-04-04 17:42:32 -07:00
Tobi LutkeandChris Pak f879183a23 update autoresearch.md progress log 2026-04-04 17:42:32 -07:00
Tobi LutkeandChris Pak 34971c0314 replace WhitespaceOrNothing regex with byte-level blank_string? check 2026-04-04 17:42:32 -07:00
Tobi LutkeandChris Pak f24ca042a2 avoid array allocation in parse_tag_token: return tag_name, store markup/newlines as class ivars 2026-04-04 17:42:32 -07:00
Tobi LutkeandChris Pak 4ea4350775 clean up filter parsing: Lexer fallback for args, no-arg fast scan stays 2026-04-04 17:42:32 -07:00
Tobi LutkeandChris Pak aedd2dded6 autoresearch.md: add strategic direction toward single-pass scanner architecture 2026-04-04 17:42:32 -07:00
Tobi LutkeandChris Pak e1f363c72c add security constraint to autoresearch.md, fix strict mode gate 2026-04-04 17:42:32 -07:00
Tobi LutkeandChris Pak 808dad6fab split filter parsing: scan no-arg filters directly, only invoke Lexer when args present 2026-04-04 17:42:32 -07:00
Tobi LutkeandChris Pak c82b6e58ef autoresearch: add autoresearch.md/sh, increase benchmark warmup to 20 iterations 2026-04-04 17:42:32 -07:00
Tobi LutkeandChris Pak b994f22983 extend fast-path to handle quoted string literal variables (262 more fast-pathed) 2026-04-04 17:42:32 -07:00
Tobi LutkeandChris Pak 62877e666c skip filter arg splat for no-arg filters, trim render loop comments 2026-04-04 17:42:32 -07:00
Tobi LutkeandChris Pak 48a2fae5f6 hoist write score check out of render loop: skip increment_write_score when no limits active 2026-04-04 17:42:32 -07:00
Tobi LutkeandChris Pak 27fcb3a4c9 use frozen EMPTY_ARRAY for disabled_tags in Variable 2026-04-04 17:33:53 -07:00
Tobi LutkeandChris Pak 6e9782c5cc return [tag_name, markup, newlines] from parse_tag_token: avoid 2 whitespace string allocs 2026-04-04 17:33:53 -07:00
Tobi LutkeandChris Pak da57336be9 use getbyte dispatch instead of start_with? in parse_for_document 2026-04-04 17:33:53 -07:00
Tobi LutkeandChris Pak 28f29433e4 avoid empty array allocation in evaluate_filter_expressions for no-arg filters 2026-04-04 17:33:53 -07:00
Tobi LutkeandChris Pak 4ff5523713 replace For tag Syntax regex with manual byte-level parser 2026-04-04 17:33:53 -07:00
Tobi LutkeandChris Pak f388c873de expose expression_cache/string_scanner via attr_reader, skip regex in filter args without colon 2026-04-04 17:33:53 -07:00
Tobi LutkeandChris Pak 543c1e1fe7 unified fast-path Variable parsing: handle both plain lookups and filter chains without full Lexer pass for name 2026-04-04 17:33:53 -07:00
Tobi LutkeandChris Pak 885c8df159 fast-path render for filter-less variables: skip render method overhead 2026-04-04 17:33:53 -07:00
Tobi LutkeandChris Pak 8f67d81023 skip TagAttributes scan in for tag when no colon present 2026-04-04 17:33:53 -07:00
Tobi LutkeandChris Pak 01f33e96e8 fast-path simple if conditions: skip ExpressionsAndOperators scan for single conditions 2026-04-04 17:33:53 -07:00
Tobi LutkeandChris Pak 9e6f93a494 replace SIMPLE_VARIABLE regex with byte-level scanner to avoid MatchData 2026-04-04 17:33:53 -07:00
Tobi LutkeandChris Pak dc6e9799e9 fast-path simple variable parsing: skip Lexer/Parser for plain dot-separated lookups 2026-04-04 17:33:53 -07:00
Tobi LutkeandChris Pak 98e29aa4a7 use frozen EMPTY_ARRAY for Variable filters when no filters present 2026-04-04 17:33:53 -07:00
Tobi LutkeandChris Pak a404cf10d0 fast-path variable_lookups: skip mutable string alloc when no dot/bracket follows 2026-04-04 17:33:53 -07:00
Tobi LutkeandChris Pak e73b41f3b6 fast-path String in render_obj_to_output, avoid Utils.to_s dispatch for common case 2026-04-04 17:33:53 -07:00
Tobi LutkeandChris Pak 9d9d094e20 short-circuit parse_number with first-byte check before regex 2026-04-04 17:33:53 -07:00
Tobi LutkeandChris Pak 4c96b5f682 avoid unnecessary strip allocation in Expression.parse, use byteslice for string literals 2026-04-04 17:33:53 -07:00
Tobi LutkeandChris Pak 0b19c0ce44 use equal? for frozen array comparison in Lexer, skip whitespace with \s+ 2026-04-04 17:33:53 -07:00
Tobi LutkeandChris Pak a43d970473 use getbyte instead of string indexing in whitespace_handler and create_variable 2026-04-04 17:33:53 -07:00
Tobi LutkeandChris Pak 3751f95da2 add auto/bench.sh: unit tests + liquid-spec + perf benchmark 2026-04-04 17:33:53 -07:00
Tobi LutkeandChris Pak 6ffd0e2339 replace VariableParser regex scan with manual byte parser in VariableLookup 2026-04-04 17:33:53 -07:00
Tobi LutkeandChris Pak dc7dda2253 replace FullToken regex with manual byte parsing in parse_for_document 2026-04-04 17:33:53 -07:00
Tobi LutkeandChris Pak 2479b26f5c add quick benchmark script for autoresearch 2026-04-04 17:33:53 -07:00
Kevin MenardandGitHub a9c85622dd Merge pull request #2066 from eregon/skip-slow-test
Skip slow test raising many exceptions on non-CRuby
2026-03-26 02:14:10 -04:00
Benoit Daloze 9f4d7e78b8 Skip slow test raising many exceptions on non-CRuby 2026-03-25 17:20:30 +01:00
Kevin MenardandGitHub fd68d076dd Merge pull request #2038 from eregon/truffleruby-ci
Add TruffleRuby in CI
2026-03-25 11:53:01 -04:00
Benoit Daloze 96aa47d13f Add TruffleRuby in CI 2026-03-25 16:46:47 +01:00
Benoit Daloze ad70c5c459 Improve no Symbol leak tests to be more reliable 2026-03-25 16:46:47 +01:00
Benoit Daloze d824de701c Adapt slice filter to work on non-64 bit platforms 2026-03-25 16:46:47 +01:00
Alok SwamyandGitHub dd37353cca Merge pull request #2062 from Shopify/update-setup-ruby
Update ruby/setup-ruby to v1.295.0 for ubuntu-24.04 support
2026-03-19 15:29:47 -04:00
32 changed files with 1901 additions and 387 deletions
+1
View File
@@ -22,6 +22,7 @@ jobs:
}
- { ruby: 4.0, allowed-failure: false, rubyopt: "--yjit" }
- { ruby: 4.0, allowed-failure: false, rubyopt: "--zjit" }
- { ruby: truffleruby, allowed-failure: false }
# Head can have failures due to being in development
- { ruby: head, allowed-failure: true }
+30
View File
@@ -0,0 +1,30 @@
# Autoresearch Ideas
## Dead Ends (tried and failed)
- **Tag name interning** (skip+byte dispatch): saves 878 allocs but verification loop overhead kills speed
- **String dedup (-@)** for filter names: no alloc savings, creates temp strings anyway
- **Split-based tokenizer**: 2.5x faster C-level split but can't handle {{ followed by %} nesting
- **Streaming tokenizer**: needs own StringScanner (+alloc), per-shift overhead worse than eager array
- **Merge simple_lookup? into initialize**: logic overhead offsets saved index call
- **Cursor for filter scanning**: cursor.reset overhead worse than inline byte loops
- **Direct strainer call**: YJIT already inlines context.invoke_single well
- **TruthyCondition subclass**: YJIT polymorphism at evaluate call site hurts more than 115 saved allocs
- **Index loop for filters**: YJIT optimizes each+destructure MUCH better than manual filter[0]/filter[1]
## Key Insights
- YJIT monomorphism > allocation reduction at this scale
- C-level StringScanner.scan/skip > Ruby-level byte loops (already applied)
- String#split is 2.5x faster than manual tokenization, but Liquid's grammar is too complex for regex
- 74% of total CPU time is GC — alloc reduction is the highest-leverage optimization
- But YJIT-deoptimization from polymorphism costs more than the GC savings
## Remaining Ideas
- **Tokenizer: use String#index + byteslice instead of StringScanner**: avoid the StringScanner overhead entirely for the simple case of finding {%/{{ delimiters
- **Pre-freeze all Condition operator lambdas**: reduce alloc in Condition initialization
- **Avoid `@blocks = []` in If with single-element optimization**: use `@block` ivar for single condition, only create array for elsif
- **Reduce ForloopDrop allocation**: reuse ForloopDrop objects across iterations or use a lighter-weight object
- **VariableLookup: single-segment optimization**: for "product.title" (1 lookup), use an ivar instead of 1-element Array
+109
View File
@@ -0,0 +1,109 @@
# Autoresearch: Liquid Parse+Render Performance
## Objective
Optimize the Shopify Liquid template engine's parse and render performance.
The workload is the ThemeRunner benchmark which parses and renders real Shopify
theme templates (dropify, ripen, tribble, vogue) with realistic data from
`performance/shopify/database.rb`. We measure parse time, render time, and
object allocations. The optimization target is combined parse+render time (µs).
## How to Run
Run `./auto/autoresearch.sh` — it runs unit tests, liquid-spec conformance,
then the performance benchmark, outputting metrics in parseable format.
## Metrics
- **Primary (optimization target)**: `combined_µs` (µs, lower is better) — sum of parse + render time
- **Secondary (tradeoff monitoring)**:
- `parse_µs` — time to parse all theme templates (Liquid::Template#parse)
- `render_µs` — time to render all pre-compiled templates
- `allocations` — total object allocations for one parse+render cycle
Parse dominates (~70-75% of combined). Allocations correlate with GC pressure.
## Files in Scope
- `lib/liquid/*.rb` — core Liquid library (parser, lexer, context, expression, etc.)
- `lib/liquid/tags/*.rb` — tag implementations (for, if, assign, etc.)
- `performance/bench_quick.rb` — benchmark script
## Off Limits
- `test/` — tests must continue to pass unchanged
- `performance/tests/` — benchmark templates, do not modify
- `performance/shopify/` — benchmark data/filters, do not modify
## Constraints
- All unit tests must pass (`bundle exec rake base_test`)
- liquid-spec failures must not increase beyond 2 (pre-existing UTF-8 edge cases)
- No new gem dependencies
- Semantic correctness must be preserved — templates must render identical output
- **Security**: Liquid runs untrusted user code. See Strategic Direction for details.
## Strategic Direction
The long-term goal is to converge toward a **single-pass, forward-only parsing
architecture** using one shared StringScanner instance. The current system has
multiple redundant passes: Tokenizer → BlockBody → Lexer → Parser → Expression
→ VariableLookup, each re-scanning portions of the source. A unified scanner
approach would:
1. **One StringScanner** flows through the entire parse — no intermediate token
arrays, no re-lexing filter chains, no string reconstruction in Parser#expression.
2. **Emit a lightweight IL or normalized AST** during the single forward pass,
decoupling strictness checking from the hot parse path. The LiquidIL project
(`~/src/tries/2026-01-05-liquid-il`) demonstrated this: a recursive-descent
parser emitting IL directly achieved significant speedups.
3. **Minimal backtracking** — the scanner advances forward, byte-checking as it
goes. liquid-c (`~/src/tries/2026-01-16-Shopify-liquid-c`) showed that a
C-level cursor-based tokenizer eliminates most allocation overhead.
Current fast-path optimizations (byte-level tag/variable/for/if parsing) are
steps toward this goal. Each one replaces a regex+MatchData pattern with
forward-only byte scanning. The remaining Lexer→Parser path for filter args
is the next target for elimination.
**Security note**: Liquid executes untrusted user templates. All parsing must
use explicit byte-range checks. Never use eval, send on user input, dynamic
method dispatch, const_get, or any pattern that lets template authors escape
the sandbox.
## Baseline
- **Commit**: 4ea835a (original, before any optimizations)
- **combined_µs**: 7,374
- **parse_µs**: 5,928
- **render_µs**: 1,446
- **allocations**: 62,620
## Progress Log
- 3329b09: Replace FullToken regex with manual byte parsing → combined 7,262 (-1.5%)
- 97e6893: Replace VariableParser regex with manual byte scanner → combined 6,945 (-5.8%), allocs 58,009
- 2b78e4b: getbyte instead of string indexing in whitespace_handler/create_variable → allocs 51,477
- d291e63: Lexer equal? for frozen arrays, \s+ whitespace skip → combined ~6,331
- d79b9fa: Avoid strip alloc in Expression.parse, byteslice for strings → allocs 49,151
- fa41224: Short-circuit parse_number with first-byte check → allocs 48,240
- c1113ad: Fast-path String in render_obj_to_output → combined ~6,071
- 25f9224: Fast-path simple variable parsing (skip Lexer/Parser) → combined ~5,860, allocs 45,202
- 3939d74: Replace SIMPLE_VARIABLE regex with byte scanner → combined ~5,717, allocs 42,763
- fe7a2f5: Fast-path simple if conditions → combined ~5,444, allocs 41,490
- cfa0dfe: Replace For tag Syntax regex with manual byte parser → combined ~4,974, allocs 39,847
- 8a92a4e: Unified fast-path Variable: parse name directly, only lex filter chain → combined ~5,060, allocs 40,520
- 58d2514: parse_tag_token returns [tag_name, markup, newlines] → combined ~4,815, allocs 37,355
- db43492: Hoist write score check out of render loop → render ~1,345
- 17daac9: Extend fast-path to quoted string literal variables → all 1,197 variables fast-pathed
- 9fd7cec: Split filter parsing: no-arg filters scanned directly, Lexer only for args → combined ~4,595, allocs 35,159
- e5933fc: Avoid array alloc in parse_tag_token via class ivars → allocs 34,281
- 2e207e6: Replace WhitespaceOrNothing regex with byte-level blank_string? → combined ~4,800
- 526af22: invoke_single fast path for no-arg filter invocation → allocs 32,621
- 76ae8f1: find_variable top-scope fast path → combined ~4,740
- 4cda1a5: slice_collection: skip copy for full Array → allocs 32,004
- 79840b1: Replace SIMPLE_CONDITION regex with manual byte parser → combined ~4,663, allocs 31,465
- 69430e9: Replace INTEGER_REGEX/FLOAT_REGEX with byte-level parse_number → allocs 31,129
- 405e3dc: Frozen EMPTY_ARRAY/EMPTY_HASH for Context @filters/@disabled_tags → allocs 31,009
- b90d7f0: Avoid unnecessary array wrapping for Context environments → allocs 30,709
- 3799d4c: Lazy seen={} hash in Utils.to_s/inspect → allocs 30,169
- 0b07487: Fast-path VariableLookup: skip scan_variable for simple identifiers → allocs 29,711
- 9de1527: Introduce Cursor class for centralized byte-level scanning
- dd4a100: Remove dead parse_tag_token/SIMPLE_CONDITION (now in Cursor)
- cdc3438: For tag: migrate lax_parse to Cursor with zero-alloc scanning → allocs 29,620
## Current Best
- **combined_µs**: ~3,400 (-54% from original 7,374 baseline)
- **parse_µs**: ~2,300
- **render_µs**: ~1,100
- **allocations**: 24,882 (-60% from original 62,620 baseline)
+48
View File
@@ -0,0 +1,48 @@
#!/usr/bin/env bash
# Autoresearch benchmark runner for Liquid performance optimization
# Runs: unit tests → performance benchmark (3 runs, takes best)
# Outputs METRIC lines for the agent to parse
# Exit code 0 = all good, non-zero = broken
set -euo pipefail
cd "$(dirname "$0")/.."
# ── Step 1: Unit tests (fast gate) ──────────────────────────────────
echo "=== Unit Tests ==="
TEST_OUT=$(bundle exec rake base_test 2>&1)
TEST_RESULT=$(echo "$TEST_OUT" | tail -1)
if echo "$TEST_OUT" | grep -q 'failures\|errors' && ! echo "$TEST_RESULT" | grep -q '0 failures, 0 errors'; then
echo "$TEST_OUT" | grep -E 'Failure|Error|failures|errors' | head -20
echo "FATAL: unit tests failed"
exit 1
fi
echo "$TEST_RESULT"
# ── Step 2: Performance benchmark (3 runs, take best) ──────────────
echo ""
echo "=== Performance Benchmark (3 runs) ==="
BEST_COMBINED=999999
BEST_PARSE=0
BEST_RENDER=0
BEST_ALLOC=0
for i in 1 2 3; do
OUT=$(bundle exec ruby performance/bench_quick.rb 2>&1)
P=$(echo "$OUT" | grep '^parse_us=' | cut -d= -f2)
R=$(echo "$OUT" | grep '^render_us=' | cut -d= -f2)
C=$(echo "$OUT" | grep '^combined_us=' | cut -d= -f2)
A=$(echo "$OUT" | grep '^allocations=' | cut -d= -f2)
echo " run $i: combined=${C}µs (parse=${P} render=${R}) allocs=${A}"
if [ "$C" -lt "$BEST_COMBINED" ]; then
BEST_COMBINED=$C
BEST_PARSE=$P
BEST_RENDER=$R
BEST_ALLOC=$A
fi
done
echo ""
echo "METRIC combined_us=$BEST_COMBINED"
echo "METRIC parse_us=$BEST_PARSE"
echo "METRIC render_us=$BEST_RENDER"
echo "METRIC allocations=$BEST_ALLOC"
Executable
+40
View File
@@ -0,0 +1,40 @@
#!/usr/bin/env bash
# Auto-research benchmark script for Liquid
# Runs: unit tests → liquid-spec → performance benchmark
# Outputs machine-readable metrics on success
# Exit code 0 = all good, non-zero = broken
set -euo pipefail
cd "$(dirname "$0")/.."
# ── Step 1: Unit tests (fast gate) ──────────────────────────────────
echo "=== Unit Tests ==="
if ! bundle exec rake base_test 2>&1; then
echo "FATAL: unit tests failed"
exit 1
fi
# ── Step 2: liquid-spec (correctness gate) ──────────────────────────
echo ""
echo "=== Liquid Spec ==="
SPEC_OUTPUT=$(bundle exec liquid-spec run spec/ruby_liquid.rb 2>&1 || true)
echo "$SPEC_OUTPUT" | tail -3
# Extract failure count from "Total: N passed, N failed, N errors" line
# Allow known pre-existing failures (≤2)
TOTAL_LINE=$(echo "$SPEC_OUTPUT" | grep "^Total:" || echo "Total: 0 passed, 0 failed, 0 errors")
FAILURES=$(echo "$TOTAL_LINE" | sed -n 's/.*\([0-9][0-9]*\) failed.*/\1/p')
ERRORS=$(echo "$TOTAL_LINE" | sed -n 's/.*\([0-9][0-9]*\) error.*/\1/p')
FAILURES=${FAILURES:-0}
ERRORS=${ERRORS:-0}
TOTAL_BAD=$((FAILURES + ERRORS))
if [ "$TOTAL_BAD" -gt 2 ]; then
echo "FATAL: liquid-spec has $FAILURES failures and $ERRORS errors (threshold: 2)"
exit 1
fi
# ── Step 3: Performance benchmark ──────────────────────────────────
echo ""
echo "=== Performance Benchmark ==="
bundle exec ruby performance/bench_quick.rb 2>&1
+30
View File
@@ -0,0 +1,30 @@
{"type":"config","name":"Liquid parse+render performance (tenderlove-inspired)","metricName":"combined_µs","metricUnit":"µs","bestDirection":"lower"}
{"run":1,"commit":"c09e722","metric":3818,"metrics":{"parse_µs":2722,"render_µs":1096,"allocations":24881},"status":"keep","description":"Baseline: 3,818µs combined, 24,881 allocs","timestamp":1773348490227}
{"run":2,"commit":"c09e722","metric":4063,"metrics":{"parse_µs":2901,"render_µs":1162,"allocations":24003},"status":"discard","description":"Tag name interning via skip+byte dispatch: saves 878 allocs but verification loop slower than scan","timestamp":1773348738557,"segment":0}
{"run":3,"commit":"c09e722","metric":3881,"metrics":{"parse_µs":2720,"render_µs":1161,"allocations":24881},"status":"discard","description":"String dedup (-@) for filter names: no alloc savings, no speed benefit","timestamp":1773348781481,"segment":0}
{"run":4,"commit":"c09e722","metric":3970,"metrics":{"parse_µs":2829,"render_µs":1141,"allocations":24881},"status":"discard","description":"Streaming tokenizer: needs own StringScanner (+1 alloc), per-shift overhead worse than saved array","timestamp":1773348883093,"segment":0}
{"run":5,"commit":"c09e722","metric":0,"metrics":{"parse_µs":0,"render_µs":0,"allocations":0},"status":"crash","description":"REVERTED: split-based tokenizer — regex can't handle unclosed tags inside raw blocks","timestamp":1773349089230,"segment":0}
{"run":6,"commit":"c09e722","metric":0,"metrics":{"parse_µs":0,"render_µs":0,"allocations":0},"status":"crash","description":"REVERTED: split regex tokenizer v2 — can't handle {{ followed by %} (variable-becomes-tag nesting)","timestamp":1773349248313,"segment":0}
{"run":7,"commit":"c09e722","metric":3861,"metrics":{"parse_µs":2744,"render_µs":1117,"allocations":24881},"status":"discard","description":"Merge simple_lookup? dot position into initialize — logic overhead offsets saved index call","timestamp":1773349376707,"segment":0}
{"run":8,"commit":"c09e722","metric":4048,"metrics":{"parse_µs":2929,"render_µs":1119,"allocations":24881},"status":"discard","description":"Use Cursor regex for filter name scanning — cursor.reset + method dispatch overhead worse than inline bytes","timestamp":1773349447172,"segment":0}
{"run":9,"commit":"c09e722","metric":3872,"metrics":{"parse_µs":2744,"render_µs":1128,"allocations":24881},"status":"discard","description":"Direct strainer call in Variable#render — YJIT already inlines context.invoke_single well","timestamp":1773349497593,"segment":0}
{"run":10,"commit":"c09e722","metric":3839,"metrics":{"parse_µs":2732,"render_µs":1107,"allocations":24879},"status":"discard","description":"Array#[] fast path for slice_collection with limit/offset — only 2 alloc savings, not meaningful","timestamp":1773349555348,"segment":0}
{"run":11,"commit":"c09e722","metric":3889,"metrics":{"parse_µs":2770,"render_µs":1119,"allocations":24766},"status":"discard","description":"TruthyCondition for simple if checks: -115 allocs but YJIT polymorphism at evaluate call site hurts speed","timestamp":1773349649377,"segment":0}
{"run":12,"commit":"c09e722","metric":4150,"metrics":{"parse_µs":2769,"render_µs":1381,"allocations":24881},"status":"discard","description":"Index loop for filters: YJIT optimizes each+destructure better than manual indexing","timestamp":1773349699285,"segment":0}
{"run":13,"commit":"b7ae55f","metric":3556,"metrics":{"parse_µs":2388,"render_µs":1168,"allocations":24882},"status":"keep","description":"Replace StringScanner tokenizer with String#byteindex — 12% faster parse, no regex overhead for delimiter finding","timestamp":1773349875890,"segment":0}
{"run":14,"commit":"e25f2f1","metric":3464,"metrics":{"parse_µs":2335,"render_µs":1129,"allocations":24882},"status":"keep","description":"Confirmation run: byteindex tokenizer consistently 3,400-3,600µs","timestamp":1773349889465,"segment":0}
{"run":15,"commit":"b37fa98","metric":3490,"metrics":{"parse_µs":2331,"render_µs":1159,"allocations":24882},"status":"keep","description":"Clean up tokenizer: remove unused StringScanner setup and regex constants","timestamp":1773349928672,"segment":0}
{"run":16,"commit":"b37fa98","metric":3638,"metrics":{"parse_µs":2460,"render_µs":1178,"allocations":24882},"status":"discard","description":"Single-char byteindex for %} search: Ruby loop overhead worse for nearby targets","timestamp":1773349985509,"segment":0}
{"run":17,"commit":"b37fa98","metric":3553,"metrics":{"parse_µs":2431,"render_µs":1122,"allocations":25256},"status":"discard","description":"Regex simple_variable_markup: MatchData creates 374 extra allocs, offsetting speed gain","timestamp":1773350066627,"segment":0}
{"run":18,"commit":"b37fa98","metric":3629,"metrics":{"parse_µs":2455,"render_µs":1174,"allocations":25002},"status":"discard","description":"String.new(capacity: 4096) for output buffer: allocates more objects, not fewer","timestamp":1773350101852,"segment":0}
{"run":19,"commit":"f6baeae","metric":3350,"metrics":{"parse_µs":2212,"render_µs":1138,"allocations":24882},"status":"keep","description":"parse_tag_token without StringScanner: pure byte ops avoid reset(token) overhead, -12% combined","timestamp":1773350230252,"segment":0}
{"run":20,"commit":"f6baead","metric":0,"metrics":{"parse_µs":0,"render_µs":0,"allocations":0},"status":"crash","description":"REVERTED: regex ultra-fast path for Variable — name pattern too broad, matches invalid trailing dots","timestamp":1773350472859,"segment":0}
{"run":21,"commit":"ae9a2e2","metric":3314,"metrics":{"parse_µs":2203,"render_µs":1111,"allocations":24882},"status":"keep","description":"Clean confirmation run: 3,314µs (-55% from main), stable","timestamp":1773350544354,"segment":0}
{"run":22,"commit":"ae9a2e2","metric":3497,"metrics":{"parse_µs":2336,"render_µs":1161,"allocations":24882},"status":"discard","description":"Regex fast path for no-filter variables: include? + match? overhead exceeds byte scan savings","timestamp":1773350641375,"segment":0}
{"run":23,"commit":"ca327b0","metric":3445,"metrics":{"parse_µs":2284,"render_µs":1161,"allocations":24647},"status":"keep","description":"Condition#evaluate: skip loop block for simple conditions (no child_relation) — saves 235 allocs","timestamp":1773350691752,"segment":0}
{"run":24,"commit":"99454a9","metric":3489,"metrics":{"parse_µs":2353,"render_µs":1136,"allocations":24647},"status":"keep","description":"Replace simple_lookup? byte scan with match? regex — 8x faster per call, cleaner code","timestamp":1773350837721,"segment":0}
{"run":25,"commit":"99454a9","metric":3797,"metrics":{"parse_µs":2636,"render_µs":1161,"allocations":29627},"status":"discard","description":"Regex name extraction in try_fast_parse: MatchData creates 5K extra allocs, much worse","timestamp":1773351048938,"segment":0}
{"run":26,"commit":"db348e0","metric":3459,"metrics":{"parse_µs":2318,"render_µs":1141,"allocations":24647},"status":"keep","description":"Inline to_liquid_value in If render — avoids one method dispatch per condition evaluation","timestamp":1773351080001,"segment":0}
{"run":27,"commit":"b195d09","metric":3496,"metrics":{"parse_µs":2356,"render_µs":1140,"allocations":24530},"status":"keep","description":"Replace @blocks.each with while loop in If render — avoids block proc allocation per render","timestamp":1773351101134,"segment":0}
{"run":28,"commit":"b195d09","metric":3648,"metrics":{"parse_µs":2457,"render_µs":1191,"allocations":24530},"status":"discard","description":"While loop in For render: YJIT optimizes each well for hot loops with many iterations","timestamp":1773351142275,"segment":0}
{"run":29,"commit":"b195d09","metric":3966,"metrics":{"parse_µs":2641,"render_µs":1325,"allocations":24060},"status":"discard","description":"While loop for environment search: -470 allocs but YJIT deopt makes render 16% slower","timestamp":1773351193863,"segment":0}
+2
View File
@@ -52,6 +52,8 @@ end
require "liquid/version"
require "liquid/deprecations"
require "liquid/const"
require 'liquid/byte_tables'
require 'liquid/cursor'
require 'liquid/standardfilters'
require 'liquid/file_system'
require 'liquid/parser_switching'
+4 -1
View File
@@ -60,8 +60,11 @@ module Liquid
@tag_name
end
# Cache block delimiters per tag name to avoid repeated string allocation
BLOCK_DELIMITER_CACHE = Hash.new { |h, k| h[k] = "end#{k}".freeze }
def block_delimiter
@block_delimiter ||= "end#{block_name}"
@block_delimiter ||= BLOCK_DELIMITER_CACHE[block_name]
end
private
+106 -66
View File
@@ -1,7 +1,5 @@
# frozen_string_literal: true
require 'English'
module Liquid
class BlockBody
LiquidTagToken = /\A\s*(#{TagName})\s*(.*?)\z/o
@@ -38,7 +36,7 @@ module Liquid
private def parse_for_liquid_tag(tokenizer, parse_context)
while (token = tokenizer.shift)
unless token.empty? || token.match?(WhitespaceOrNothing)
unless token.empty? || BlockBody.blank_string?(token)
unless token =~ LiquidTagToken
# line isn't empty but didn't match tag syntax, yield and let the
# caller raise a syntax error
@@ -53,8 +51,7 @@ module Liquid
end
unless (tag = parse_context.environment.tag_for_name(tag_name))
# end parsing if we reach an unknown tag and let the caller decide
# determine how to proceed
# end parsing if we reach an unknown tag; let the caller determine how to proceed
return yield tag_name, markup
end
new_tag = tag.parse(tag_name, markup, tokenizer, parse_context)
@@ -124,48 +121,38 @@ module Liquid
end
end
def self.blank_string?(str)
str.match?(WhitespaceOrNothing)
end
private def parse_for_document(tokenizer, parse_context, &block)
while (token = tokenizer.shift)
next if token.empty?
case
when token.start_with?(TAGSTART)
whitespace_handler(token, parse_context)
unless token =~ FullToken
return handle_invalid_tag_token(token, parse_context, &block)
end
tag_name = Regexp.last_match(2)
markup = Regexp.last_match(4)
if parse_context.line_number
# newlines inside the tag should increase the line number,
# particularly important for multiline {% liquid %} tags
parse_context.line_number += Regexp.last_match(1).count("\n") + Regexp.last_match(3).count("\n")
first_byte = token.getbyte(0)
if first_byte == Cursor::LCURLY
second_byte = token.getbyte(1)
if second_byte == Cursor::PCT
# handle_tag_token returns:
# nil — tag parsed normally, continue (update line number)
# :next — 'liquid' inline tag; skip line number update
# :unknown — end tag or unknown tag; yield to caller and return
# :invalid — malformed tag token; delegate to handle_invalid_tag_token
result = handle_tag_token(token, parse_context, tokenizer)
next unless result # nil: normal
next if result == :next # :next: 'liquid'
return yield(@_unknown_tag_name, parse_context.cursor.tag_markup) if result == :unknown
return handle_invalid_tag_token(token, parse_context, &block) # :invalid
elsif second_byte == Cursor::LCURLY
whitespace_handler(token, parse_context)
@nodelist << create_variable(token, parse_context)
@blank = false
else
# Fallback: text token starting with '{'
append_text_token(token, parse_context)
end
if tag_name == 'liquid'
parse_liquid_tag(markup, parse_context)
next
end
unless (tag = parse_context.environment.tag_for_name(tag_name))
# end parsing if we reach an unknown tag and let the caller decide
# determine how to proceed
return yield tag_name, markup
end
new_tag = tag.parse(tag_name, markup, tokenizer, parse_context)
@blank &&= new_tag.blank?
@nodelist << new_tag
when token.start_with?(VARSTART)
whitespace_handler(token, parse_context)
@nodelist << create_variable(token, parse_context)
@blank = false
else
if parse_context.trim_whitespace
token.lstrip!
end
parse_context.trim_whitespace = false
@nodelist << token
@blank &&= token.match?(WhitespaceOrNothing)
append_text_token(token, parse_context)
end
parse_context.line_number = tokenizer.line_number
end
@@ -173,8 +160,54 @@ module Liquid
yield nil, nil
end
def whitespace_handler(token, parse_context)
if token[2] == WhitespaceControl
# Handles a {%...%} tag token. Does not receive the outer block — callers handle
# yield/block passing themselves, keeping the Proc off the hot path.
# Returns:
# nil — tag parsed, caller continues the loop
# :next — 'liquid' inline tag; caller skips line number update
# :unknown — unknown/end tag; @_unknown_tag_name holds the tag name;
# markup is in parse_context.cursor.tag_markup
# :invalid — malformed token; caller delegates to handle_invalid_tag_token
private def handle_tag_token(token, parse_context, tokenizer)
whitespace_handler(token, parse_context)
cursor = parse_context.cursor
tag_name = cursor.parse_tag_token(token)
return :invalid unless tag_name
markup = cursor.tag_markup
if parse_context.line_number
newlines = cursor.tag_newlines
parse_context.line_number += newlines if newlines > 0
end
if tag_name == 'liquid'
parse_liquid_tag(markup, parse_context)
return :next
end
tag = parse_context.environment.tag_for_name(tag_name)
unless tag
# end parsing if we reach an unknown tag; let the caller determine how to proceed
@_unknown_tag_name = tag_name
return :unknown
end
new_tag = tag.parse(tag_name, markup, tokenizer, parse_context)
@blank &&= new_tag.blank?
@nodelist << new_tag
nil
end
def append_text_token(token, parse_context)
token.lstrip! if parse_context.trim_whitespace
parse_context.trim_whitespace = false
@nodelist << token
@blank &&= BlockBody.blank_string?(token)
end
private :append_text_token
private def whitespace_handler(token, parse_context)
if token.getbyte(2) == Cursor::DASH
previous_token = @nodelist.last
if previous_token.is_a?(String)
first_byte = previous_token.getbyte(0)
@@ -184,7 +217,7 @@ module Liquid
end
end
end
parse_context.trim_whitespace = (token[-3] == WhitespaceControl)
parse_context.trim_whitespace = (token.getbyte(token.bytesize - 3) == Cursor::DASH)
end
def blank?
@@ -216,24 +249,35 @@ module Liquid
end
def render_to_output_buffer(context, output)
freeze unless frozen?
freeze
context.resource_limits.increment_render_score(@nodelist.length)
resource_limits = context.resource_limits
resource_limits.increment_render_score(@nodelist.length)
# Hot render loop — split on check_write so the common case (no resource
# limits) pays zero branch cost per node.
idx = 0
while (node = @nodelist[idx])
if node.instance_of?(String)
output << node
else
render_node(context, output, node)
# If we get an Interrupt that means the block must stop processing. An
# Interrupt is any command that stops block execution such as {% break %}
# or {% continue %}. These tags may also occur through Block or Include tags.
break if context.interrupt? # might have happened in a for-block
if resource_limits.render_length_limit || resource_limits.last_capture_length
while (node = @nodelist[idx])
if node.instance_of?(String)
output << node
else
render_node(context, output, node)
break if context.interrupt?
end
idx += 1
resource_limits.increment_write_score(output)
end
else
while (node = @nodelist[idx])
if node.instance_of?(String)
output << node
else
render_node(context, output, node)
break if context.interrupt?
end
idx += 1
end
idx += 1
context.resource_limits.increment_write_score(output)
end
output
@@ -241,19 +285,15 @@ module Liquid
private
# Indirection allows subclasses to intercept per-node rendering.
def render_node(context, output, node)
BlockBody.render_node(context, output, node)
end
def create_variable(token, parse_context)
if token.end_with?("}}")
i = 2
i = 3 if token[i] == "-"
parse_end = token.length - 3
parse_end -= 1 if token[parse_end] == "-"
markup_end = parse_end - i + 1
markup = markup_end <= 0 ? "" : token.slice(i, markup_end)
len = token.bytesize
if len >= 4 && token.getbyte(len - 1) == Cursor::RCURLY && token.getbyte(len - 2) == Cursor::RCURLY
markup = parse_context.cursor.parse_variable_token(token)
return Variable.new(markup, parse_context)
end
+40
View File
@@ -0,0 +1,40 @@
# frozen_string_literal: true
module Liquid
# Pre-computed 256-entry boolean lookup tables for byte classification.
# Built once at load time; used as TABLE[byte] — a single array index
# instead of 3-5 comparison operators per check.
#
# Performance: neutral to slightly faster vs. chained comparisons.
# Readability: replaces expressions like
# (b >= 97 && b <= 122) || (b >= 65 && b <= 90) || b == 95
# with the intent-revealing
# ByteTables::IDENT_START[b]
module ByteTables
# [a-zA-Z_] — valid first byte of an identifier
IDENT_START = Array.new(256, false).tap do |t|
(97..122).each { |b| t[b] = true } # a-z
(65..90).each { |b| t[b] = true } # A-Z
t[95] = true # _
end.freeze
# [a-zA-Z0-9_-] — valid continuation byte of an identifier
IDENT_CONT = Array.new(256, false).tap do |t|
(97..122).each { |b| t[b] = true } # a-z
(65..90).each { |b| t[b] = true } # A-Z
(48..57).each { |b| t[b] = true } # 0-9
t[95] = true # _
t[45] = true # -
end.freeze
# [0-9] — ASCII digit
DIGIT = Array.new(256, false).tap do |t|
(48..57).each { |b| t[b] = true }
end.freeze
# [ \t\n\v\f\r] — ASCII whitespace (mirrors Ruby's \s)
WHITESPACE = Array.new(256, false).tap do |t|
[32, 9, 10, 11, 12, 13].each { |b| t[b] = true } # space, tab, \n, \v, \f, \r
end.freeze
end
end
+7 -6
View File
@@ -65,20 +65,21 @@ module Liquid
end
def evaluate(context = deprecated_default_context)
condition = self
result = nil
loop do
result = interpret_condition(condition.left, condition.right, condition.operator, context)
result = interpret_condition(@left, @right, @operator, context)
# Fast path: no child conditions (most common)
return result unless @child_relation
condition = self
while condition.child_relation
case condition.child_relation
when :or
break if Liquid::Utils.to_liquid_value(result)
when :and
break unless Liquid::Utils.to_liquid_value(result)
else
break
end
condition = condition.child_condition
result = interpret_condition(condition.left, condition.right, condition.operator, context)
end
result
end
+66 -31
View File
@@ -24,10 +24,15 @@ module Liquid
def initialize(environments = {}, outer_scope = {}, registers = {}, rethrow_errors = false, resource_limits = nil, static_environments = {}, environment = Environment.default)
@environment = environment
@environments = [environments]
@environments.flatten!
@environments = environments.is_a?(Array) ? environments : [environments]
@static_environments = [static_environments].flatten(1).freeze
@static_environments = if static_environments.is_a?(Array)
static_environments.frozen? ? static_environments : static_environments.freeze
elsif static_environments.empty?
Const::EMPTY_ARRAY
else
[static_environments].freeze
end
@scopes = [outer_scope || {}]
@registers = registers.is_a?(Registers) ? registers : Registers.new(registers)
@errors = []
@@ -35,14 +40,13 @@ module Liquid
@strict_variables = false
@resource_limits = resource_limits || ResourceLimits.new(environment.default_resource_limits)
@base_scope_depth = 0
@interrupts = []
@filters = []
@interrupts = Const::EMPTY_ARRAY
@filters = Const::EMPTY_ARRAY
@global_filter = nil
@disabled_tags = {}
@disabled_tags = Const::EMPTY_HASH
# Instead of constructing new StringScanner objects for each Expression parse,
# we recycle the same one.
@string_scanner = StringScanner.new("")
# Lazy-init StringScanner — only needed if Context#[] is called during render
@string_scanner = nil
@registers.static[:cached_partials] ||= {}
@registers.static[:file_system] ||= environment.file_system
@@ -73,7 +77,7 @@ module Liquid
# Note that this does not register the filters with the main Template object. see <tt>Template.register_filter</tt>
# for that
def add_filters(filters)
filters = [filters].flatten.compact
filters = Array(filters).flatten.compact
@filters += filters
@strainer = nil
end
@@ -84,11 +88,12 @@ module Liquid
# are there any not handled interrupts?
def interrupt?
!@interrupts.empty?
!@interrupts.equal?(Const::EMPTY_ARRAY) && @interrupts.any?
end
# push an interrupt to the stack. this interrupt is considered not handled.
def push_interrupt(e)
@interrupts = [] if @interrupts.frozen?
@interrupts.push(e)
end
@@ -109,6 +114,20 @@ module Liquid
strainer.invoke(method, *args).to_liquid
end
# Arity-specialized filter delegation — generated to match StrainerTemplate's specializations.
# The pattern (avoid *args splat) is the same for each arity; generating makes it explicit.
{
invoke_single: ['input'],
invoke_two: ['input', 'arg1'],
}.each do |method_name, params|
all_params = (["method"] + params).join(", ")
module_eval(<<~RUBY, __FILE__, __LINE__ + 1)
def #{method_name}(#{all_params})
strainer.#{method_name}(#{all_params}).to_liquid
end
RUBY
end
# Push new local scope on the stack. use <tt>Context#stack</tt> instead
def push(new_scope = {})
@scopes.unshift(new_scope)
@@ -180,11 +199,11 @@ module Liquid
# Example:
# products == empty #=> products.empty?
def [](expression)
evaluate(Expression.parse(expression, @string_scanner))
evaluate(Expression.parse(expression, @string_scanner ||= StringScanner.new("")))
end
def key?(key)
find_variable(key, raise_on_not_found: false) != nil
!find_variable(key, raise_on_not_found: false).nil?
end
def evaluate(object)
@@ -193,22 +212,38 @@ module Liquid
# Fetches an object starting at the local scope and then moving up the hierachy
def find_variable(key, raise_on_not_found: true)
# This was changed from find() to find_index() because this is a very hot
# path and find_index() is optimized in MRI to reduce object allocation
index = @scopes.find_index { |s| s.key?(key) }
variable = if index
lookup_and_evaluate(@scopes[index], key, raise_on_not_found: raise_on_not_found)
# Fast path: check top scope first (most common in for loops)
scope = @scopes[0]
if scope.key?(key)
variable = lookup_and_evaluate(scope, key, raise_on_not_found: raise_on_not_found)
elsif @scopes.length == 1
# Only one scope and key not found — go straight to environments
variable = try_variable_find_in_environments(key, raise_on_not_found: raise_on_not_found)
else
try_variable_find_in_environments(key, raise_on_not_found: raise_on_not_found)
# Multiple scopes — search through all of them
scope = @scopes.find { |s| s.key?(key) }
variable = if scope
lookup_and_evaluate(scope, key, raise_on_not_found: raise_on_not_found)
else
try_variable_find_in_environments(key, raise_on_not_found: raise_on_not_found)
end
end
# update variable's context before invoking #to_liquid
# Fast path: primitive types don't need context= or to_liquid conversion
case variable
when String, Integer, Float, NilClass, TrueClass, FalseClass, Array, Hash, Time
return variable
end
variable.context = self if variable.respond_to?(:context=)
liquid_variable = variable.to_liquid
liquid_variable.context = self if variable != liquid_variable && liquid_variable.respond_to?(:context=)
if variable != liquid_variable
liquid_variable.context = self if liquid_variable.respond_to?(:context=)
end
liquid_variable
end
@@ -228,6 +263,7 @@ module Liquid
end
def with_disabled_tags(tag_names)
@disabled_tags = {} if @disabled_tags.frozen?
tag_names.each do |name|
@disabled_tags[name] = @disabled_tags.fetch(name, 0) + 1
end
@@ -251,17 +287,16 @@ module Liquid
attr_reader :base_scope_depth
def try_variable_find_in_environments(key, raise_on_not_found:)
@environments.each do |environment|
found = find_in_envs(@environments, key, raise_on_not_found: raise_on_not_found)
return found unless found.nil? && !(@strict_variables && raise_on_not_found)
find_in_envs(@static_environments, key, raise_on_not_found: raise_on_not_found)
end
def find_in_envs(envs, key, raise_on_not_found:)
envs.each do |environment|
found_variable = lookup_and_evaluate(environment, key, raise_on_not_found: raise_on_not_found)
if !found_variable.nil? || @strict_variables && raise_on_not_found
return found_variable
end
end
@static_environments.each do |environment|
found_variable = lookup_and_evaluate(environment, key, raise_on_not_found: raise_on_not_found)
if !found_variable.nil? || @strict_variables && raise_on_not_found
return found_variable
end
return found_variable if !found_variable.nil? || (@strict_variables && raise_on_not_found)
end
nil
end
+317
View File
@@ -0,0 +1,317 @@
# frozen_string_literal: true
require "strscan"
module Liquid
# Single-pass forward-only scanner for Liquid parsing.
# Wraps StringScanner with higher-level methods for common Liquid constructs.
# One Cursor per template parse — threaded through all parsing code.
class Cursor
# Byte constants
SPACE = 32
TAB = 9
NL = 10
CR = 13
FF = 12
DASH = 45 # '-'
DOT = 46 # '.'
COLON = 58 # ':'
PIPE = 124 # '|'
QUOTE_S = 39 # "'"
QUOTE_D = 34 # '"'
LBRACK = 91 # '['
RBRACK = 93 # ']'
LPAREN = 40 # '('
RPAREN = 41 # ')'
QMARK = 63 # '?'
HASH = 35 # '#'
USCORE = 95 # '_'
COMMA = 44
ZERO = 48
NINE = 57
PCT = 37 # '%'
LCURLY = 123 # '{'
RCURLY = 125 # '}'
attr_reader :ss
def initialize(source)
@source = source
@ss = StringScanner.new(source)
end
# ── Position ────────────────────────────────────────────────────
def pos = @ss.pos
def pos=(n)
@ss.pos = n
end
def eos? = @ss.eos?
def peek_byte = @ss.peek_byte
def scan_byte = @ss.scan_byte
# Reset scanner to a new string (for reuse on sub-markup)
def reset(source)
@source = source
@ss.string = source
end
# Extract a slice from the source (deferred allocation)
def slice(start, len)
@source.byteslice(start, len)
end
# ── Whitespace ──────────────────────────────────────────────────
# Skip spaces/tabs/newlines/cr
def skip_ws
while (b = @ss.peek_byte)
case b
when SPACE, TAB, CR, FF, NL then @ss.scan_byte
else break
end
end
end
# Check if remaining bytes are all whitespace (or EOS).
# exist?(/\S/) returns nil when no non-whitespace remains, without advancing position.
def rest_blank?
!@ss.exist?(/\S/)
end
# Regex for identifier: [a-zA-Z_][\w-]*\??
ID_REGEX = /[a-zA-Z_][\w-]*\??/
# ── Identifiers ─────────────────────────────────────────────────
# Skip an identifier without allocating a string. Returns length skipped, or 0.
def skip_id
@ss.skip(ID_REGEX) || 0
end
# Check if next id matches expected string, consume if so. No allocation.
def expect_id(expected)
start = @ss.pos
len = @ss.skip(ID_REGEX)
if len == expected.bytesize
# Compare bytes directly without allocating a string
i = 0
while i < len
unless @source.getbyte(start + i) == expected.getbyte(i)
@ss.pos = start
return false
end
i += 1
end
return true
end
@ss.pos = start if len
false
end
# Scan a single identifier: [a-zA-Z_][\w-]*\??
# Returns the string or nil if not at an identifier
def scan_id
@ss.scan(ID_REGEX)
end
# Scan a tag name: '#' or \w+
def scan_tag_name
if @ss.peek_byte == HASH
@ss.scan_byte
"#"
else
scan_id
end
end
# Regex for numbers: -?\d+(\.\d+)?
FLOAT_REGEX = /-?\d+\.\d+/
INT_REGEX = /-?\d+/
# ── Numbers ─────────────────────────────────────────────────────
# Try to scan an integer or float. Returns the number or nil.
def scan_number
if (s = @ss.scan(FLOAT_REGEX))
s.to_f
elsif (s = @ss.scan(INT_REGEX))
s.to_i
end
end
# Regex for quoted string content (without quotes)
SINGLE_QUOTED_CONTENT = /'([^']*)'/
DOUBLE_QUOTED_CONTENT = /"([^"]*)"/
# ── Strings ─────────────────────────────────────────────────────
# Scan a quoted string ('...' or "..."). Returns the content without quotes, or nil.
def scan_quoted_string
if @ss.scan(SINGLE_QUOTED_CONTENT) || @ss.scan(DOUBLE_QUOTED_CONTENT)
@ss[1]
end
end
# Regex for quoted strings (single or double quoted, including quotes)
QUOTED_STRING_RAW = /"[^"]*"|'[^']*'/
# Scan a quoted string including quotes. Returns the full "..." or '...' string, or nil.
def scan_quoted_string_raw
@ss.scan(QUOTED_STRING_RAW)
end
# Regex for dotted identifier: name(.name)*
DOTTED_ID_REGEX = /[a-zA-Z_][\w-]*\??(?:\.[a-zA-Z_][\w-]*\??)*/
# ── Expressions ─────────────────────────────────────────────────
# Scan a simple variable lookup: name(.name)* — no brackets, no filters
# Returns the string or nil
def scan_dotted_id
@ss.scan(DOTTED_ID_REGEX)
end
# Skip a fragment without allocating. Returns length skipped, or 0.
def skip_fragment
@ss.skip(QUOTED_STRING_RAW) || @ss.skip(UNQUOTED_FRAGMENT) || 0
end
# Regex for unquoted fragment: non-whitespace/comma/pipe sequence
UNQUOTED_FRAGMENT = /[^\s,|]+/
# Scan a "QuotedFragment" — a quoted string or non-whitespace/comma/pipe run
def scan_fragment
@ss.scan(QUOTED_STRING_RAW) || @ss.scan(UNQUOTED_FRAGMENT)
end
# ── Comparison operators ────────────────────────────────────────
# Identity map used for frozen string interning: StringScanner#scan returns a
# new unfrozen String on every call. Indexing into this hash returns the frozen
# literal stored here, avoiding a separate allocation and enabling faster
# equality checks downstream (frozen strings can be compared by identity).
COMPARISON_OPS = {
'==' => '==',
'!=' => '!=',
'<>' => '<>',
'<=' => '<=',
'>=' => '>=',
'<' => '<',
'>' => '>',
'contains' => 'contains',
}.freeze
# Scan a comparison operator. Returns frozen string or nil.
# Regex for comparison operators
COMPARISON_OP_REGEX = /==|!=|<>|<=|>=|<|>|contains(?!\w)/
def scan_comparison_op
if (op = @ss.scan(COMPARISON_OP_REGEX))
COMPARISON_OPS[op]
end
end
# ── Tag parsing helpers ─────────────────────────────────────────
# Results from last parse_tag_token call (avoids array allocation)
attr_reader :tag_markup, :tag_newlines
# Parse the interior of a tag token: "{%[-] tag_name markup [-]%}"
# Pure byte operations — avoids StringScanner reset overhead.
# Returns tag_name string or nil. Sets tag_markup and tag_newlines.
def parse_tag_token(token)
len = token.bytesize
pos = 2 # skip "{%"
pos += 1 if token.getbyte(pos) == DASH # skip '-'
nl = 0
# Skip whitespace, count newlines
while pos < len
b = token.getbyte(pos)
case b
when SPACE, TAB, CR, FF then pos += 1
when NL then pos += 1
nl += 1
else break
end
end
# Scan tag name: '#' or [a-zA-Z_][\w-]*
name_start = pos
b = token.getbyte(pos)
if b == HASH
pos += 1
elsif b && ByteTables::IDENT_START[b]
pos += 1
while pos < len
b = token.getbyte(pos)
break unless ByteTables::IDENT_CONT[b]
pos += 1
end
pos += 1 if pos < len && token.getbyte(pos) == QMARK
else
return
end
tag_name = token.byteslice(name_start, pos - name_start)
# Skip whitespace after tag name, count newlines
while pos < len
b = token.getbyte(pos)
case b
when SPACE, TAB, CR, FF then pos += 1
when NL then pos += 1
nl += 1
else break
end
end
# markup is everything up to optional '-' before '%}'
markup_end = len - 2
markup_end -= 1 if markup_end > pos && token.getbyte(markup_end - 1) == DASH
@tag_markup = pos >= markup_end ? "" : token.byteslice(pos, markup_end - pos)
@tag_newlines = nl
tag_name
end
# Parse variable token interior: extract markup from "{{[-] ... [-]}}"
def parse_variable_token(token)
len = token.bytesize
return if len < 4
i = 2
i = 3 if token.getbyte(i) == DASH
parse_end = len - 3
parse_end -= 1 if token.getbyte(parse_end) == DASH
markup_len = parse_end - i + 1
markup_len <= 0 ? "" : token.byteslice(i, markup_len)
end
# ── Simple condition parser ─────────────────────────────────────
# Results from last parse_simple_condition call
attr_reader :cond_left, :cond_op, :cond_right
# Parse "expr [op expr]" from current position to end.
# Returns true on success, nil on failure. Sets cond_left, cond_op, cond_right.
def parse_simple_condition
skip_ws
@cond_left = scan_fragment
return unless @cond_left
skip_ws
if eos?
@cond_op = nil
@cond_right = nil
return true
end
@cond_op = scan_comparison_op
return unless @cond_op
skip_ws
@cond_right = scan_fragment
return unless @cond_right
skip_ws
return unless eos? # trailing junk
true
end
end
end
+77 -50
View File
@@ -16,16 +16,9 @@ module Liquid
'-' => VariableLookup.parse("-", nil).freeze,
}.freeze
DOT = ".".ord
ZERO = "0".ord
NINE = "9".ord
DASH = "-".ord
# Use an atomic group (?>...) to avoid pathological backtracing from
# malicious input as described in https://github.com/Shopify/liquid/issues/1357
RANGES_REGEX = /\A\(\s*(?>(\S+)\s*\.\.)\s*(\S+)\s*\)\z/
INTEGER_REGEX = /\A(-?\d+)\z/
FLOAT_REGEX = /\A(-?\d+)\.\d+\z/
class << self
def safe_parse(parser, ss = StringScanner.new(""), cache = nil)
@@ -35,11 +28,17 @@ module Liquid
def parse(markup, ss = StringScanner.new(""), cache = nil)
return unless markup
markup = markup.strip # markup can be a frozen string
# Only strip if there's leading/trailing whitespace (avoids allocation)
first_byte = markup.getbyte(0)
if first_byte && ByteTables::WHITESPACE[first_byte]
markup = markup.strip
elsif first_byte
markup = markup.strip if ByteTables::WHITESPACE[markup.getbyte(markup.bytesize - 1)]
end
if (markup.start_with?('"') && markup.end_with?('"')) ||
(markup.start_with?("'") && markup.end_with?("'"))
return markup[1..-2]
return markup.byteslice(1, markup.bytesize - 2)
elsif LITERALS.key?(markup)
return LITERALS[markup]
end
@@ -71,57 +70,85 @@ module Liquid
end
end
def parse_number(markup, ss)
# check if the markup is simple integer or float
case markup
when INTEGER_REGEX
def parse_number(markup, _ss = nil)
len = markup.bytesize
return if len == 0
# Quick reject: first byte must be digit or dash
pos = 0
first = markup.getbyte(pos)
if first == Cursor::DASH
pos += 1
return if pos >= len
b = markup.getbyte(pos)
return unless ByteTables::DIGIT[b]
pos += 1
elsif ByteTables::DIGIT[first]
pos += 1
else
return
end
# Scan digits
while pos < len
b = markup.getbyte(pos)
break unless ByteTables::DIGIT[b]
pos += 1
end
# If we consumed everything, it's a simple integer
if pos == len
return Integer(markup, 10)
when FLOAT_REGEX
return markup.to_f
end
ss.string = markup
# the first byte must be a digit or a dash
byte = ss.scan_byte
# Check for dot (float)
if markup.getbyte(pos) == Cursor::DOT
dot_pos = pos
pos += 1
# Must have at least one digit after dot
digit_after_dot = pos
while pos < len
b = markup.getbyte(pos)
break unless ByteTables::DIGIT[b]
return false if byte != DASH && (byte < ZERO || byte > NINE)
pos += 1
end
if byte == DASH
peek_byte = ss.peek_byte
# if it starts with a dash, the next byte must be a digit
return false if peek_byte.nil? || !(peek_byte >= ZERO && peek_byte <= NINE)
end
# The markup could be a float with multiple dots
first_dot_pos = nil
num_end_pos = nil
while (byte = ss.scan_byte)
return false if byte != DOT && (byte < ZERO || byte > NINE)
# we found our number and now we are just scanning the rest of the string
next if num_end_pos
if byte == DOT
if first_dot_pos.nil?
first_dot_pos = ss.pos
else
# we found another dot, so we know that the number ends here
num_end_pos = ss.pos - 1
end
if pos > digit_after_dot && pos == len
# Simple float like "123.456"
return markup.to_f
elsif pos > digit_after_dot
# Float followed by more content: "1.2.3.4" — scan to find where the
# numeric portion ends (stop at next dot or non-digit).
return scan_float_with_trailing(markup, pos, len)
else
# dot at end: "123."
return markup.byteslice(0, dot_pos).to_f
end
end
num_end_pos = markup.length if ss.eos?
# Not a number (has non-digit, non-dot characters)
nil
end
if num_end_pos
# number ends with a number "123.123"
markup.byteslice(0, num_end_pos).to_f
else
# number ends with a dot "123."
markup.byteslice(0, first_dot_pos).to_f
private
# Scans forward from `pos` through digits, returning the float up to the
# next dot or the end of string. Returns nil when a non-digit, non-dot
# byte is found (not a valid number). Used by parse_number for inputs
# like "1.2.3.4" where the float literal ends at the second dot.
def scan_float_with_trailing(markup, pos, len)
while pos < len
b = markup.getbyte(pos)
return markup.byteslice(0, pos).to_f if b == Cursor::DOT
return unless ByteTables::DIGIT[b]
pos += 1
end
markup.byteslice(0, pos).to_f
end
end
end
+4 -3
View File
@@ -29,6 +29,7 @@ module Liquid
RUBY_WHITESPACE = [" ", "\t", "\r", "\n", "\f"].freeze
SINGLE_STRING_LITERAL = /'[^\']*'/
WHITESPACE_OR_NOTHING = /\s*/
WHITESPACE = /\s+/
SINGLE_COMPARISON_TOKENS = [].tap do |table|
table["<".ord] = COMPARISON_LESS_THAN
@@ -104,7 +105,7 @@ module Liquid
output = []
until ss.eos?
ss.skip(WHITESPACE_OR_NOTHING)
ss.skip(WHITESPACE)
break if ss.eos?
@@ -114,10 +115,10 @@ module Liquid
if (special = SPECIAL_TABLE[peeked])
ss.scan_byte
# Special case for ".."
if special == DOT && ss.peek_byte == DOT_ORD
if special.equal?(DOT) && ss.peek_byte == DOT_ORD
ss.scan_byte
output << DOTDOT
elsif special == DASH
elsif special.equal?(DASH)
# Special case for negative numbers
if (peeked_byte = ss.peek_byte) && NUMBER_TABLE[peeked_byte]
ss.pos -= 1
+3 -1
View File
@@ -3,7 +3,7 @@
module Liquid
class ParseContext
attr_accessor :locale, :line_number, :trim_whitespace, :depth
attr_reader :partial, :warnings, :error_mode, :environment
attr_reader :partial, :warnings, :error_mode, :environment, :expression_cache, :string_scanner, :cursor
def initialize(options = Const::EMPTY_HASH)
@environment = options.fetch(:environment, Environment.default)
@@ -24,6 +24,8 @@ module Liquid
{}
end
@cursor = Cursor.new("")
self.depth = 0
self.partial = false
end
+3
View File
@@ -83,6 +83,9 @@ module Liquid
end
def variable_lookups
# Fast path: no lookups at all (most common case for simple identifiers)
return "" unless look(:dot) || look(:open_square)
str = +""
loop do
if look(:open_square)
+6 -6
View File
@@ -6,15 +6,15 @@ module Liquid
def initialize(registers = {})
@static = registers.is_a?(Registers) ? registers.static : registers
@changes = {}
@changes = nil
end
def []=(key, value)
@changes[key] = value
(@changes ||= {})[key] = value
end
def [](key)
if @changes.key?(key)
if @changes&.key?(key)
@changes[key]
else
@static[key]
@@ -22,13 +22,13 @@ module Liquid
end
def delete(key)
@changes.delete(key)
@changes&.delete(key)
end
UNDEFINED = Object.new
def fetch(key, default = UNDEFINED, &block)
if @changes.key?(key)
if @changes&.key?(key)
@changes.fetch(key)
elsif default != UNDEFINED
if block_given?
@@ -42,7 +42,7 @@ module Liquid
end
def key?(key)
@changes.key?(key) || @static.key?(key)
@changes&.key?(key) || @static.key?(key)
end
end
+1
View File
@@ -9,6 +9,7 @@ module Liquid
:cumulative_assign_score_limit
attr_reader :render_score,
:assign_score,
:last_capture_length,
:cumulative_render_score,
:cumulative_assign_score
+80 -18
View File
@@ -8,10 +8,19 @@ module Liquid
MAX_I32 = (1 << 31) - 1
private_constant :MAX_I32
MIN_I64 = -(1 << 63)
MAX_I64 = (1 << 63) - 1
I64_RANGE = MIN_I64..MAX_I64
private_constant :MIN_I64, :MAX_I64, :I64_RANGE
supports_64bit_indices = begin
[][1 << 33, 1 << 33]
true
rescue RangeError
false
end
INDEX_RANGE = if supports_64bit_indices
(-(1 << 63))..((1 << 63) - 1)
else
(-(1 << 31))..((1 << 31) - 1)
end
private_constant :INDEX_RANGE
HTML_ESCAPE = {
'&' => '&amp;',
@@ -214,11 +223,11 @@ module Liquid
Utils.to_s(input).slice(offset, length) || ''
end
rescue RangeError
if I64_RANGE.cover?(length) && I64_RANGE.cover?(offset)
if INDEX_RANGE.cover?(length) && INDEX_RANGE.cover?(offset)
raise # unexpected error
end
offset = offset.clamp(I64_RANGE)
length = length.clamp(I64_RANGE)
offset = offset.clamp(INDEX_RANGE)
length = length.clamp(INDEX_RANGE)
retry
end
end
@@ -266,18 +275,71 @@ module Liquid
words = Utils.to_integer(words)
words = 1 if words <= 0
wordlist = begin
input.split(" ", words + 1)
rescue RangeError
# integer too big for String#split, but we can semantically assume no truncation is needed
return input if words + 1 > MAX_I32
raise # unexpected error
end
return input if wordlist.length <= words
return input if words + 1 > MAX_I32
wordlist.pop
truncate_string = Utils.to_s(truncate_string)
wordlist.join(" ").concat(truncate_string)
# Scan words tracking byte positions; build the normalized (single-space)
# result string only when truncation is actually needed.
len = input.bytesize
pos = 0
word_count = 0
# Flat array of [start, end, start, end, ...] for up to `words` words.
# Avoids allocating a result string in the common no-truncation case.
positions = []
# Skip leading whitespace
while pos < len
break unless ByteTables::WHITESPACE[input.getbyte(pos)]
pos += 1
end
while pos < len
word_start = pos
word_count += 1
# Scan to end of word
while pos < len
break if ByteTables::WHITESPACE[input.getbyte(pos)]
pos += 1
end
if word_count <= words
positions.push(word_start, pos) # [start, end, start, end, ...]
else
# Truncation confirmed — build normalized result from stored positions
result = +input.byteslice(positions[0], positions[1] - positions[0])
i = 2
while i < positions.length
result << " " << input.byteslice(positions[i], positions[i + 1] - positions[i])
i += 2
end
return result << Utils.to_s(truncate_string)
end
# Skip whitespace between words
while pos < len
break unless ByteTables::WHITESPACE[input.getbyte(pos)]
pos += 1
end
end
# Fewer words than requested — no truncation needed, return original unchanged.
return input if word_count < words
# Exactly `words` words. Ruby's split(" ", words+1) would produce a words+1-th
# empty element when input has trailing whitespace, triggering the truncation path.
# Match that behaviour: if the input ends with whitespace, normalize and append
# truncate_string even though no word was cut.
if len > 0 && ByteTables::WHITESPACE[input.getbyte(len - 1)]
result = +input.byteslice(positions[0], positions[1] - positions[0])
i = 2
while i < positions.length
result << " " << input.byteslice(positions[i], positions[i + 1] - positions[i])
i += 2
end
return result << Utils.to_s(truncate_string)
end
input
end
# @liquid_public_docs
+26
View File
@@ -58,5 +58,31 @@ module Liquid
rescue ::ArgumentError => e
raise Liquid::ArgumentError, e.message, e.backtrace
end
# Arity-specialized filter invocation.
# Avoids *args splat allocation for the common 0-arg and 1-arg cases.
# `invoke` (general case) still uses *args for 2+ extra arguments.
{
invoke_single: ['input'],
invoke_two: ['input', 'arg1'],
}.each do |method_name, params|
all_params = (["method"] + params).join(", ")
send_params = params.join(", ")
# __LINE__ + 1 is a parse-time constant; both generated methods will report
# the same file:line in backtraces. The method name in the trace distinguishes them.
module_eval(<<~RUBY, __FILE__, __LINE__ + 1)
def #{method_name}(#{all_params})
if self.class.invokable?(method)
send(method, #{send_params})
elsif @context.strict_filters
raise Liquid::UndefinedFilter, "undefined filter \#{method}"
else
input
end
rescue ::ArgumentError => e
raise Liquid::ArgumentError, e.message, e.backtrace
end
RUBY
end
end
end
+47 -29
View File
@@ -25,8 +25,6 @@ module Liquid
# @liquid_optional_param range [untyped] A custom numeric range to iterate over.
# @liquid_optional_param reversed [untyped] Iterate in reverse order.
class For < Block
Syntax = /\A(#{VariableSegment}+)\s+in\s+(#{QuotedFragment}+)\s*(reversed)?/o
attr_reader :collection_name, :variable_name, :limit, :from
def initialize(tag_name, markup, options)
@@ -72,18 +70,52 @@ module Liquid
protected
# Fast byte-level parser for "var in collection [reversed] [limit:N] [offset:N]"
def lax_parse(markup)
if markup =~ Syntax
@variable_name = Regexp.last_match(1)
collection_name = Regexp.last_match(2)
@reversed = !!Regexp.last_match(3)
@name = "#{@variable_name}-#{collection_name}"
@collection_name = parse_expression(collection_name)
markup.scan(TagAttributes) do |key, value|
set_attribute(key, value)
c = @parse_context.cursor
c.reset(markup)
c.skip_ws
# Parse variable name
var_start = c.pos
var_len = c.skip_id
raise SyntaxError, options[:locale].t("errors.syntax.for") if var_len == 0
@variable_name = c.slice(var_start, var_len)
# Expect "in"
c.skip_ws
raise SyntaxError, options[:locale].t("errors.syntax.for") unless c.expect_id("in")
c.skip_ws
# Parse collection name
col_start = c.pos
if c.peek_byte == Cursor::LPAREN
# Parenthesized range: (1..10)
depth = 1
c.scan_byte
while !c.eos? && depth > 0
b = c.scan_byte
depth += 1 if b == Cursor::LPAREN
depth -= 1 if b == Cursor::RPAREN
end
else
raise SyntaxError, options[:locale].t("errors.syntax.for")
c.skip_fragment
end
collection_name = c.slice(col_start, c.pos - col_start)
@name = "#{@variable_name}-#{collection_name}"
@collection_name = parse_expression(collection_name)
c.skip_ws
@reversed = c.expect_id("reversed")
c.skip_ws
# Parse limit:/offset: if present.
# Cursor doesn't handle key:value attributes — delegate to regex for limit:/offset:.
if !c.eos? && (rest = c.slice(c.pos, markup.bytesize - c.pos)).include?(':')
rest.scan(TagAttributes) do |key, value|
set_attribute(key, value)
end
end
end
@@ -111,9 +143,7 @@ module Liquid
private
def strict2_parse(markup)
strict_parse(markup)
end
alias_method :strict2_parse, :strict_parse
def collection_segment(context)
offsets = context.registers[:for] ||= {}
@@ -122,22 +152,14 @@ module Liquid
offsets[@name].to_i
else
from_value = context.evaluate(@from)
if from_value.nil?
0
else
Utils.to_integer(from_value)
end
from_value.nil? ? 0 : Utils.to_integer(from_value)
end
collection = context.evaluate(@collection_name)
collection = collection.to_a if collection.is_a?(Range)
limit_value = context.evaluate(@limit)
to = if limit_value.nil?
nil
else
Utils.to_integer(limit_value) + from
end
to = limit_value && (Utils.to_integer(limit_value) + from)
segment = Utils.slice_collection(collection, from, to)
segment.reverse! if @reversed
@@ -192,11 +214,7 @@ module Liquid
end
def render_else(context, output)
if @else_block
@else_block.render_to_output_buffer(context, output)
else
output
end
@else_block ? @else_block.render_to_output_buffer(context, output) : output
end
class ParseTreeVisitor < Liquid::ParseTreeVisitor
+28 -4
View File
@@ -51,14 +51,17 @@ module Liquid
end
def render_to_output_buffer(context, output)
@blocks.each do |block|
result = Liquid::Utils.to_liquid_value(
block.evaluate(context),
)
idx = 0
blocks = @blocks
while idx < blocks.length
block = blocks[idx]
result = block.evaluate(context)
result = result.to_liquid_value if result.respond_to?(:to_liquid_value)
if result
return block.attachment.render_to_output_buffer(context, output)
end
idx += 1
end
output
@@ -86,6 +89,27 @@ module Liquid
end
def lax_parse(markup)
# Fastest path: simple identifier truthiness like "product.available" or "forloop.first"
if (simple = Variable.simple_variable_markup(markup))
return Condition.new(parse_expression(simple))
end
# Fast path: simple condition without and/or — use Cursor.
# The include? pre-checks are both a correctness guard (parse_simple_condition
# only handles a single comparison) and a perf gate (avoids cursor allocation
# for the compound-condition case that will always fall through to lax_parse).
if !markup.include?(' and ') && !markup.include?(' or ')
cursor = @parse_context.cursor
cursor.reset(markup)
if cursor.parse_simple_condition
return Condition.new(
parse_expression(cursor.cond_left),
cursor.cond_op,
cursor.cond_right ? parse_expression(cursor.cond_right) : nil,
)
end
end
expressions = markup.scan(ExpressionsAndOperators)
raise SyntaxError, options[:locale].t("errors.syntax.if") unless expressions.pop =~ Syntax
+93 -102
View File
@@ -1,37 +1,23 @@
# frozen_string_literal: true
require "strscan"
module Liquid
class Tokenizer
attr_reader :line_number, :for_liquid_tag
TAG_END = /%\}/
TAG_OR_VARIABLE_START = /\{[\{\%]/
NEWLINE = /\n/
OPEN_CURLEY = "{".ord
CLOSE_CURLEY = "}".ord
PERCENTAGE = "%".ord
def initialize(
source:,
string_scanner:,
string_scanner: nil,
line_numbers: false,
line_number: nil,
for_liquid_tag: false
)
@line_number = line_number || (line_numbers ? 1 : nil)
@for_liquid_tag = for_liquid_tag
@source = source.to_s.to_str
@source = source.to_s
@offset = 0
@tokens = []
if @source
@ss = string_scanner
@ss.string = @source
tokenize
end
tokenize
end
def shift
@@ -54,108 +40,113 @@ module Liquid
if @for_liquid_tag
@tokens = @source.split("\n")
else
@tokens << shift_normal until @ss.eos?
tokenize_fast
end
@source = nil
@ss = nil
end
def shift_normal
token = next_token
# Fast tokenizer using String#byteindex instead of StringScanner regex.
# String#byteindex is ~40% faster for finding { delimiters.
def tokenize_fast
src = @source
unless src.valid_encoding?
raise SyntaxError, "Invalid byte sequence in #{src.encoding}"
end
return unless token
len = src.bytesize
pos = 0
token
end
while pos < len
# Find next { which could start a tag or variable
idx = src.byteindex('{', pos)
def next_token
# possible states: :text, :tag, :variable
byte_a = @ss.peek_byte
if byte_a == OPEN_CURLEY
@ss.scan_byte
byte_b = @ss.peek_byte
if byte_b == PERCENTAGE
@ss.scan_byte
return next_tag_token
elsif byte_b == OPEN_CURLEY
@ss.scan_byte
return next_variable_token
unless idx
# No more tags/variables — rest is text
@tokens << src.byteslice(pos, len - pos) if pos < len
break
end
@ss.pos -= 1
end
next_byte = idx + 1 < len ? src.getbyte(idx + 1) : nil
next_text_token
end
if next_byte == Cursor::PCT # {%
# Emit text before tag
@tokens << src.byteslice(pos, idx - pos) if idx > pos
def next_text_token
start = @ss.pos
unless @ss.skip_until(TAG_OR_VARIABLE_START)
token = @ss.rest
@ss.terminate
return token
end
pos = @ss.pos -= 2
@source.byteslice(start, pos - start)
rescue ::ArgumentError => e
if e.message == "invalid byte sequence in #{@ss.string.encoding}"
raise SyntaxError, "Invalid byte sequence in #{@ss.string.encoding}"
else
raise
end
end
def next_variable_token
start = @ss.pos - 2
byte_a = byte_b = @ss.scan_byte
while byte_b
byte_a = @ss.scan_byte while byte_a && byte_a != CLOSE_CURLEY && byte_a != OPEN_CURLEY
break unless byte_a
if @ss.eos?
return byte_a == CLOSE_CURLEY ? @source.byteslice(start, @ss.pos - start) : "{{"
end
byte_b = @ss.scan_byte
if byte_a == CLOSE_CURLEY
if byte_b == CLOSE_CURLEY
return @source.byteslice(start, @ss.pos - start)
elsif byte_b != CLOSE_CURLEY
@ss.pos -= 1
return @source.byteslice(start, @ss.pos - start)
# Find %} to close the tag
close = src.byteindex('%}', idx + 2)
if close
@tokens << src.byteslice(idx, close + 2 - idx)
pos = close + 2
else
# Emit malformed token to propagate a missing-terminator error in the parser
@tokens << "{%"
pos = idx + 2
end
elsif next_byte == Cursor::LCURLY # {{
# Emit text before variable, then scan for the closing }}.
@tokens << src.byteslice(pos, idx - pos) if idx > pos
pos = scan_variable_token(src, idx, len)
else
# Lone '{' — not the start of a tag or variable.
# Find the next '{{' or '{%' to know where this text token ends.
# Using two byteindex calls avoids a nested loop and is always O(n).
tag_start = src.byteindex('{%', idx + 1)
var_start = src.byteindex('{{', idx + 1)
next_token = [tag_start, var_start].compact.min
if next_token
@tokens << src.byteslice(pos, next_token - pos)
pos = next_token
else
@tokens << src.byteslice(pos, len - pos)
pos = len
end
elsif byte_a == OPEN_CURLEY && byte_b == PERCENTAGE
return next_tag_token_with_start(start)
end
byte_a = byte_b
end
"{{"
end
def next_tag_token
start = @ss.pos - 2
if (len = @ss.skip_until(TAG_END))
@source.byteslice(start, len + 2)
else
"{%"
end
end
def next_tag_token_with_start(start)
@ss.skip_until(TAG_END)
@source.byteslice(start, @ss.pos - start)
# Scans a {{ ... }} variable token starting at `idx` in `src`.
# Emits the token to @tokens and returns the new position after the token.
# Handles }}, single }, and embedded {% ... %} (nested tag inside variable).
private def scan_variable_token(src, idx, len)
# Byte-by-byte scan: find } or {, then inspect the next byte.
scan_pos = idx + 2
while scan_pos < len
b = src.getbyte(scan_pos)
if b == Cursor::RCURLY # }
if scan_pos + 1 >= len
# } at end of string — emit token up to here
@tokens << src.byteslice(idx, scan_pos + 1 - idx)
return scan_pos + 1
end
b2 = src.getbyte(scan_pos + 1)
if b2 == Cursor::RCURLY
# Found }} — close variable
@tokens << src.byteslice(idx, scan_pos + 2 - idx)
return scan_pos + 2
else
# } followed by non-} — emit token up to here (matches original: @ss.pos -= 1)
@tokens << src.byteslice(idx, scan_pos + 1 - idx)
return scan_pos + 1
end
elsif b == Cursor::LCURLY && scan_pos + 1 < len && src.getbyte(scan_pos + 1) == Cursor::PCT
# Found {% inside {{ — scan to %} and emit as one token
close = src.byteindex('%}', scan_pos + 2)
if close
@tokens << src.byteslice(idx, close + 2 - idx)
return close + 2
else
@tokens << src.byteslice(idx, len - idx)
return len
end
else
scan_pos += 1
end
end
# Reached end without finding }} — malformed
@tokens << "{{"
idx + 2
end
end
end
+19 -16
View File
@@ -8,6 +8,9 @@ module Liquid
def self.slice_collection(collection, from, to)
if (from != 0 || !to.nil?) && collection.respond_to?(:load_slice)
collection.load_slice(from, to)
elsif from == 0 && to.nil? && collection.is_a?(Array)
# Fast path: no offset/limit on an Array — return as-is (avoid copy)
collection
else
slice_collection_using_each(collection, from, to)
end
@@ -15,23 +18,17 @@ module Liquid
def self.slice_collection_using_each(collection, from, to)
segments = []
index = 0
# Maintains Ruby 1.8.7 String#each behaviour on 1.9
# String is Enumerable but #each is not defined; handle it as a single-element collection
if collection.is_a?(String)
return collection.empty? ? [] : [collection]
end
return [] unless collection.respond_to?(:each)
index = 0
collection.each do |item|
if to && to <= index
break
end
if from <= index
segments << item
end
break if to && to <= index
segments << item if from <= index
index += 1
end
@@ -93,8 +90,14 @@ module Liquid
obj
end
def self.to_s(obj, seen = {})
# Cached string representations for common small integers (0-999)
# Avoids repeated Integer#to_s allocations during rendering
SMALL_INT_STRINGS = Array.new(1000) { |i| i.to_s.freeze }.freeze
def self.to_s(obj, seen = nil)
case obj
when Integer
obj >= 0 && obj < 1000 ? SMALL_INT_STRINGS[obj] : obj.to_s
when BigDecimal
obj.to_s("F")
when Hash
@@ -102,30 +105,30 @@ module Liquid
# custom implementation. Otherwise we use Liquid's default
# implementation.
if obj.class.instance_method(:to_s) == HASH_TO_S_METHOD
hash_inspect(obj, seen)
hash_inspect(obj, seen || {})
else
obj.to_s
end
when Array
array_inspect(obj, seen)
array_inspect(obj, seen || {})
else
obj.to_s
end
end
def self.inspect(obj, seen = {})
def self.inspect(obj, seen = nil)
case obj
when Hash
# If the custom hash implementation overrides `#inspect`, use their
# custom implementation. Otherwise we use Liquid's default
# implementation.
if obj.class.instance_method(:inspect) == HASH_INSPECT_METHOD
hash_inspect(obj, seen)
hash_inspect(obj, seen || {})
else
obj.inspect
end
when Array
array_inspect(obj, seen)
array_inspect(obj, seen || {})
else
obj.inspect
end
+329 -15
View File
@@ -12,6 +12,26 @@ module Liquid
# {{ user | link }}
#
class Variable
# Checks if markup is a simple "name.lookup.chain" with no filters/brackets/quotes.
# Returns the trimmed markup string, or nil if not simple.
def self.simple_variable_markup(markup)
return if markup.empty?
return unless markup.match?(SIMPLE_VARIABLE_RE)
# Avoid allocation when there's no surrounding whitespace (the common case)
first = markup.getbyte(0)
last = markup.getbyte(markup.bytesize - 1)
needs_strip = first == Cursor::SPACE || first == Cursor::TAB || first == Cursor::NL || first == Cursor::CR ||
last == Cursor::SPACE || last == Cursor::TAB || last == Cursor::NL || last == Cursor::CR
needs_strip ? markup.strip : markup
end
# Cache for [filtername, EMPTY_ARRAY] tuples — avoids repeated array creation
NO_ARG_FILTER_CACHE = Hash.new { |h, k| h[k] = [k, Const::EMPTY_ARRAY].freeze }
# Regex for a simple variable lookup with optional surrounding whitespace.
# Shares the identifier grammar with VariableLookup::SIMPLE_LOOKUP_RE.
SIMPLE_VARIABLE_RE = /\A\s*[\w-]+\??(?:\.[\w-]+\??)*\s*\z/
FilterMarkupRegex = /#{FilterSeparator}\s*(.*)/om
FilterParser = /(?:\s+|#{QuotedFragment}|#{ArgumentSeparator})+/o
FilterArgsRegex = /(?:#{FilterArgumentSeparator}|#{ArgumentSeparator})\s*((?:\w+\s*\:\s*)?#{QuotedFragment})/o
@@ -30,7 +50,278 @@ module Liquid
@parse_context = parse_context
@line_number = parse_context.line_number
strict_parse_with_error_mode_fallback(markup)
# Fast path: try to parse without going through Lexer → Parser
# Skip for strict2/rigid modes which require different parsing
# Fast path only for lax/warn modes — strict modes need full error checking
error_mode = parse_context.error_mode
if error_mode == :strict2 || error_mode == :rigid || error_mode == :strict || !try_fast_parse(markup, parse_context)
strict_parse_with_error_mode_fallback(markup)
end
end
private def try_fast_parse(markup, parse_context)
pos = fast_scan_name(markup)
return false unless pos
# fast_resolve_name calls VariableLookup.parse_simple / Expression::LITERALS — the
# only sites that can raise SyntaxError on malformed input. The byte scanners return
# false instead of raising.
begin
fast_resolve_name(markup, parse_context)
rescue SyntaxError
return false
end
# End of markup — no filters
if pos >= markup.bytesize
@filters = Const::EMPTY_ARRAY
return true
end
# Must be followed by a pipe filter separator
return false unless markup.getbyte(pos) == Cursor::PIPE
fast_scan_filters(markup, pos, parse_context)
end
# Scan the variable name (quoted string or identifier chain) at the start of markup.
# Returns the position after the name + trailing whitespace, or false on failure.
# Sets @_fast_name_start and @_fast_name_end for fast_resolve_name.
private def fast_scan_name(markup)
len = markup.bytesize
return false if len == 0
# Skip leading whitespace
pos = 0
while pos < len
b = markup.getbyte(pos)
break unless b == Cursor::SPACE || b == Cursor::TAB || b == Cursor::NL || b == Cursor::CR
pos += 1
end
return false if pos >= len
b = markup.getbyte(pos)
if b == Cursor::QUOTE_S || b == Cursor::QUOTE_D
# Quoted string literal: scan to matching close quote
quote = b
@_fast_name_start = pos
pos += 1
pos += 1 while pos < len && markup.getbyte(pos) != quote
pos += 1 if pos < len # skip closing quote
@_fast_name_end = pos
elsif ByteTables::IDENT_START[b]
# Identifier chain: [a-zA-Z_][a-zA-Z0-9_-]*(.[a-zA-Z_][a-zA-Z0-9_-]*)*
@_fast_name_start = pos
pos += 1
while pos < len
b = markup.getbyte(pos)
if ByteTables::IDENT_CONT[b]
pos += 1
elsif b == Cursor::DOT
pos += 1
return false if pos >= len
b = markup.getbyte(pos)
return false unless ByteTables::IDENT_START[b]
pos += 1
else
break
end
end
@_fast_name_end = pos
else
return false
end
# Skip whitespace after name
while pos < len
b = markup.getbyte(pos)
break unless b == Cursor::SPACE || b == Cursor::TAB || b == Cursor::NL || b == Cursor::CR
pos += 1
end
pos
end
# Resolve the scanned name bytes to a Liquid expression object.
# Reads @_fast_name_start / @_fast_name_end set by fast_scan_name.
# Sets @name. May raise SyntaxError (rescued in try_fast_parse).
private def fast_resolve_name(markup, parse_context)
name_start = @_fast_name_start
name_end = @_fast_name_end
len = markup.bytesize
# Avoid byteslice when the name spans the whole markup (no surrounding whitespace/filters)
expr_markup = name_start == 0 && name_end == len ? markup : markup.byteslice(name_start, name_end - name_start)
cache = parse_context.expression_cache
ss = parse_context.string_scanner
first_byte = expr_markup.getbyte(0)
@name = if first_byte == Cursor::QUOTE_S || first_byte == Cursor::QUOTE_D
# String literal — strip enclosing quotes
expr_markup.byteslice(1, expr_markup.bytesize - 2)
elsif Expression::LITERALS.key?(expr_markup)
Expression::LITERALS[expr_markup]
elsif cache
cache[expr_markup] || (cache[expr_markup] = VariableLookup.parse_simple(expr_markup, ss, cache).freeze)
else
VariableLookup.parse_simple(expr_markup, ss || StringScanner.new(""), nil).freeze
end
end
# Scan the filter chain starting at `pos` (the first '|').
# Returns true on success (sets @filters), false to fall back to the Lexer.
# Rescues SyntaxError from Expression.parse inside fast_scan_filter_args.
private def fast_scan_filters(markup, pos, parse_context)
len = markup.bytesize
@filters = []
filter_pos = pos
while filter_pos < len && markup.getbyte(filter_pos) == Cursor::PIPE
filter_pos += 1
# Skip spaces after pipe (tabs/newlines handled in the between-filters skip below)
filter_pos += 1 while filter_pos < len && markup.getbyte(filter_pos) == Cursor::SPACE
# Scan filter name: must start with [a-zA-Z_]
fname_start = filter_pos
b = filter_pos < len ? markup.getbyte(filter_pos) : nil
break unless b && ByteTables::IDENT_START[b]
filter_pos += 1
while filter_pos < len
b = markup.getbyte(filter_pos)
break unless ByteTables::IDENT_CONT[b]
filter_pos += 1
end
filtername = markup.byteslice(fname_start, filter_pos - fname_start)
# Skip whitespace after filter name
filter_pos += 1 while filter_pos < len && markup.getbyte(filter_pos) == Cursor::SPACE
if filter_pos < len && markup.getbyte(filter_pos) == Cursor::COLON
# Has arguments — fast-scan positional args; fall to Lexer on keyword args
filter_pos += 1 # skip ':'
filter_pos += 1 while filter_pos < len && markup.getbyte(filter_pos) == Cursor::SPACE
result = fast_scan_filter_args(markup, filter_pos, parse_context)
return fall_to_lexer_filters(markup, pos, fname_start, len, parse_context) if result == :fall_to_lexer
filter_args, filter_pos = result
@filters << [filtername, filter_args]
else
# No-arg filter — reuse the cached [name, EMPTY_ARRAY] tuple
@filters << NO_ARG_FILTER_CACHE[filtername]
end
# Skip whitespace (including tabs and newlines) between filters
filter_pos += 1 while filter_pos < len && (
markup.getbyte(filter_pos) == Cursor::SPACE ||
markup.getbyte(filter_pos) == Cursor::TAB ||
markup.getbyte(filter_pos) == Cursor::NL ||
markup.getbyte(filter_pos) == Cursor::CR
)
end
# Trailing bytes that aren't a pipe mean something the fast path doesn't handle
return false if filter_pos < len
@filters = Const::EMPTY_ARRAY if @filters.empty?
true
rescue SyntaxError
# Expression.parse (called inside fast_scan_filter_args for identifier args) can
# raise SyntaxError on malformed input. Fall back to full Lexer parse.
@name = nil
@filters = nil
false
end
# Called when fast_scan_filter_args encounters keyword args or an unrecognised
# token. Hands the remaining filter chain (from the pipe before fname_start)
# to the full Lexer-based parser, merges results into @filters, and returns true.
private def fall_to_lexer_filters(markup, pos, fname_start, len, parse_context)
# Walk back from fname_start to find the pipe that opened this filter.
# Equivalent to: markup.rindex('|', fname_start), bounded by pos.
rest_start = fname_start
rest_start -= 1 while rest_start > pos && markup.getbyte(rest_start) != Cursor::PIPE
rest_markup = markup.byteslice(rest_start, len - rest_start)
p = parse_context.new_parser(rest_markup)
while p.consume?(:pipe)
fn = p.consume(:id)
fa = p.consume?(:colon) ? parse_filterargs(p) : Const::EMPTY_ARRAY
@filters << lax_parse_filter_expressions(fn, fa)
end
p.consume(:end_of_string)
@filters = Const::EMPTY_ARRAY if @filters.empty?
true
end
# Scan positional filter arguments starting at `filter_pos`.
# Returns [filter_args_array, new_filter_pos] on success, or :fall_to_lexer when
# keyword args or unrecognised tokens are encountered.
private def fast_scan_filter_args(markup, filter_pos, parse_context)
len = markup.bytesize
filter_args = []
loop do
arg_start = filter_pos
b = filter_pos < len ? markup.getbyte(filter_pos) : nil
if b == Cursor::QUOTE_S || b == Cursor::QUOTE_D
# Quoted string argument
quote = b
filter_pos += 1
filter_pos += 1 while filter_pos < len && markup.getbyte(filter_pos) != quote
filter_pos += 1 if filter_pos < len # skip closing quote
filter_args << markup.byteslice(arg_start + 1, filter_pos - arg_start - 2)
elsif b && (ByteTables::DIGIT[b] ||
(b == Cursor::DASH && filter_pos + 1 < len && ByteTables::DIGIT[markup.getbyte(filter_pos + 1)]))
# Numeric argument (integer or float, optionally negative)
filter_pos += 1 if b == Cursor::DASH
filter_pos += 1 while filter_pos < len && ByteTables::DIGIT[markup.getbyte(filter_pos)]
if filter_pos < len && markup.getbyte(filter_pos) == Cursor::DOT # float
filter_pos += 1
filter_pos += 1 while filter_pos < len && ByteTables::DIGIT[markup.getbyte(filter_pos)]
end
num_str = markup.byteslice(arg_start, filter_pos - arg_start)
filter_args << (num_str.include?('.') ? num_str.to_f : num_str.to_i)
elsif b && ByteTables::IDENT_START[b]
# Identifier argument — may be a variable lookup or keyword arg
id_start = filter_pos
filter_pos += 1
while filter_pos < len
b2 = markup.getbyte(filter_pos)
break unless ByteTables::IDENT_CONT[b2] || b2 == Cursor::DOT
filter_pos += 1
end
filter_pos += 1 if filter_pos < len && markup.getbyte(filter_pos) == Cursor::QMARK
# Peek past whitespace: if followed by ':', this is a keyword arg → fall to Lexer
kw_check = filter_pos
kw_check += 1 while kw_check < len && markup.getbyte(kw_check) == Cursor::SPACE
return :fall_to_lexer if kw_check < len && markup.getbyte(kw_check) == Cursor::COLON
id_markup = markup.byteslice(id_start, filter_pos - id_start)
filter_args << Expression.parse(id_markup, parse_context.string_scanner, parse_context.expression_cache)
else
return :fall_to_lexer
end
# Skip whitespace after argument
filter_pos += 1 while filter_pos < len && markup.getbyte(filter_pos) == Cursor::SPACE
# Comma: more arguments follow; anything else: done with this filter's args
if filter_pos < len && markup.getbyte(filter_pos) == Cursor::COMMA
filter_pos += 1
filter_pos += 1 while filter_pos < len && markup.getbyte(filter_pos) == Cursor::SPACE
else
break
end
end
[filter_args, filter_pos]
end
def raw
@@ -42,7 +333,7 @@ module Liquid
end
def lax_parse(markup)
@filters = []
@filters = Const::EMPTY_ARRAY
return unless markup =~ MarkupWithQuotedFragment
name_markup = Regexp.last_match(1)
@@ -54,19 +345,21 @@ module Liquid
next unless f =~ /\w+/
filtername = Regexp.last_match(0)
filterargs = f.scan(FilterArgsRegex).flatten
@filters = [] if @filters.frozen?
@filters << lax_parse_filter_expressions(filtername, filterargs)
end
end
end
def strict_parse(markup)
@filters = []
@filters = Const::EMPTY_ARRAY
p = @parse_context.new_parser(markup)
return if p.look(:end_of_string)
@name = parse_context.safe_parse_expression(p)
while p.consume?(:pipe)
@filters = [] if @filters.frozen?
filtername = p.consume(:id)
filterargs = p.consume?(:colon) ? parse_filterargs(p) : Const::EMPTY_ARRAY
@filters << lax_parse_filter_expressions(filtername, filterargs)
@@ -75,13 +368,16 @@ module Liquid
end
def strict2_parse(markup)
@filters = []
@filters = Const::EMPTY_ARRAY
p = @parse_context.new_parser(markup)
return if p.look(:end_of_string)
@name = parse_context.safe_parse_expression(p)
@filters << strict2_parse_filter_expressions(p) while p.consume?(:pipe)
while p.consume?(:pipe)
@filters = [] if @filters.frozen?
@filters << strict2_parse_filter_expressions(p)
end
p.consume(:end_of_string)
end
@@ -97,24 +393,37 @@ module Liquid
obj = context.evaluate(@name)
@filters.each do |filter_name, filter_args, filter_kwargs|
filter_args = evaluate_filter_expressions(context, filter_args, filter_kwargs)
obj = context.invoke(filter_name, obj, *filter_args)
if filter_args.empty? && !filter_kwargs
obj = context.invoke_single(filter_name, obj)
elsif !filter_kwargs && filter_args.length == 1
# Single positional arg — most common after no-arg
obj = context.invoke_two(filter_name, obj, context.evaluate(filter_args[0]))
else
filter_args = evaluate_filter_expressions(context, filter_args, filter_kwargs)
obj = context.invoke(filter_name, obj, *filter_args)
end
end
context.apply_global_filter(obj)
end
def render_to_output_buffer(context, output)
obj = render(context)
# Fast path: no filters and no global filter
obj = if @filters.empty? && context.global_filter.nil?
context.evaluate(@name)
else
render(context)
end
render_obj_to_output(obj, output)
output
end
def render_obj_to_output(obj, output)
case obj
when NilClass
if obj.instance_of?(String)
output << obj
elsif obj.nil?
# Do nothing
when Array
elsif obj.instance_of?(Array)
obj.each do |o|
render_obj_to_output(o, output)
end
@@ -128,7 +437,7 @@ module Liquid
end
def disabled_tags
[]
Const::EMPTY_ARRAY
end
private
@@ -137,7 +446,8 @@ module Liquid
filter_args = []
keyword_args = nil
unparsed_args.each do |a|
if (matches = a.match(JustTagAttributes))
# Fast check: keyword args must contain ':'
if a.include?(':') && (matches = a.match(JustTagAttributes))
keyword_args ||= {}
keyword_args[matches[1]] = parse_context.parse_expression(matches[2])
else
@@ -190,15 +500,19 @@ module Liquid
end
def evaluate_filter_expressions(context, filter_args, filter_kwargs)
parsed_args = filter_args.map { |expr| context.evaluate(expr) }
if filter_kwargs
parsed_args = filter_args.map { |expr| context.evaluate(expr) }
parsed_kwargs = {}
filter_kwargs.each do |key, expr|
parsed_kwargs[key] = context.evaluate(expr)
end
parsed_args << parsed_kwargs
parsed_args
elsif filter_args.empty?
Const::EMPTY_ARRAY
else
filter_args.map { |expr| context.evaluate(expr) }
end
parsed_args
end
class ParseTreeVisitor < Liquid::ParseTreeVisitor
+140 -18
View File
@@ -10,11 +10,108 @@ module Liquid
new(markup, string_scanner, cache)
end
def initialize(markup, string_scanner = StringScanner.new(""), cache = nil)
lookups = markup.scan(VariableParser)
# Fast parse that skips simple_lookup? check — caller guarantees simple identifier chain
def self.parse_simple(markup, string_scanner = nil, cache = nil)
new(markup, string_scanner, cache, true)
end
# Fast manual scanner replacing markup.scan(VariableParser)
# VariableParser = /\[(?>[^\[\]]+|\g<0>)*\]|[\w-]+\??/
# Splits "product.variants[0].title" into ["product", "variants", "[0]", "title"]
def self.scan_variable(markup)
result = []
pos = 0
len = markup.bytesize
while pos < len
byte = markup.getbyte(pos)
if byte == 91 # '['
# Scan balanced brackets
depth = 1
start = pos
pos += 1
while pos < len && depth > 0
b = markup.getbyte(pos)
depth += 1 if b == 91
depth -= 1 if b == 93
pos += 1
end
if depth == 0
result << markup.byteslice(start, pos - start)
else
# Unbalanced bracket - skip '[' and continue
pos = start + 1
end
elsif byte == 46 # '.'
pos += 1
elsif ByteTables::IDENT_CONT[byte] # [\w-]
start = pos
pos += 1
while pos < len
b = markup.getbyte(pos)
break unless ByteTables::IDENT_CONT[b]
pos += 1
end
# Check trailing '?'
if pos < len && markup.getbyte(pos) == 63
pos += 1
end
result << markup.byteslice(start, pos - start)
else
pos += 1
end
end
result
end
# Check if markup is a simple identifier chain: [\w-]+\??(.[\w-]+\??)*
# Uses C-level match? — 8x faster than Ruby byte scanning
SIMPLE_LOOKUP_RE = /\A[\w-]+\??(?:\.[\w-]+\??)*\z/
def self.simple_lookup?(markup)
markup.bytesize > 0 && markup.match?(SIMPLE_LOOKUP_RE)
end
def initialize(markup, string_scanner = StringScanner.new(""), cache = nil, simple = false)
# Fast path: simple identifier chain without brackets
if simple || self.class.simple_lookup?(markup)
dot_pos = markup.index('.')
if dot_pos.nil?
@name = markup
@lookups = Const::EMPTY_ARRAY
@command_flags = 0
return
end
@name = markup.byteslice(0, dot_pos)
# Build lookups array from remaining dot-separated segments
lookups = []
@command_flags = 0
pos = dot_pos + 1
len = markup.bytesize
while pos < len
seg_start = pos
while pos < len
b = markup.getbyte(pos)
break if b == 46 # '.'
pos += 1
end
seg = markup.byteslice(seg_start, pos - seg_start)
if COMMAND_METHODS.include?(seg)
@command_flags |= 1 << lookups.length
end
lookups << seg
pos += 1 # skip dot
end
@lookups = lookups
return
end
lookups = self.class.scan_variable(markup)
name = lookups.shift
if name&.start_with?('[') && name&.end_with?(']')
if name&.start_with?('[') && name.end_with?(']')
name = Expression.parse(
name[1..-2],
string_scanner,
@@ -26,9 +123,8 @@ module Liquid
@lookups = lookups
@command_flags = 0
@lookups.each_index do |i|
lookup = lookups[i]
if lookup&.start_with?('[') && lookup&.end_with?(']')
@lookups.each_with_index do |lookup, i|
if lookup&.start_with?('[') && lookup.end_with?(']')
lookups[i] = Expression.parse(
lookup[1..-2],
string_scanner,
@@ -49,26 +145,34 @@ module Liquid
object = context.find_variable(name)
@lookups.each_index do |i|
key = context.evaluate(@lookups[i])
lookup = @lookups[i]
key = lookup.instance_of?(String) ? lookup : context.evaluate(lookup)
# Cast "key" to its liquid value to enable it to act as a primitive value
key = Liquid::Utils.to_liquid_value(key)
# Fast path: strings and integers (most common key types) don't need conversion
unless key.instance_of?(String) || key.instance_of?(Integer)
key = Liquid::Utils.to_liquid_value(key)
end
# If object is a hash- or array-like object we look for the
# presence of the key and if its available we return it
if object.respond_to?(:[]) &&
((object.respond_to?(:key?) && object.key?(key)) ||
(object.respond_to?(:fetch) && key.is_a?(Integer)))
if accessible?(object, key)
# if its a proc we will replace the entry with the proc
res = context.lookup_and_evaluate(object, key)
object = res.to_liquid
object = context.lookup_and_evaluate(object, key)
# Skip to_liquid for common primitive types (they return self)
unless object.instance_of?(String) || object.instance_of?(Integer) || object.instance_of?(Float) ||
object.instance_of?(Array) || object.instance_of?(Hash) || object.nil?
object = liquidize(object, context)
end
# Some special cases. If the part wasn't in square brackets and
# no key with the same name was found we interpret following calls
# as commands and call them on the current object
elsif lookup_command?(i) && object.respond_to?(key)
object = object.send(key).to_liquid
object = object.send(key)
unless object.instance_of?(String) || object.instance_of?(Integer) || object.instance_of?(Array) || object.nil?
object = liquidize(object, context)
end
# Handle string first/last like ActiveSupport does (returns first/last character)
# ActiveSupport returns "" for empty strings, not nil
@@ -82,9 +186,6 @@ module Liquid
return nil unless context.strict_variables
raise Liquid::UndefinedVariable, "undefined variable #{key}"
end
# If we are dealing with a drop here we have to
object.context = context if object.respond_to?(:context=)
end
object
@@ -94,6 +195,27 @@ module Liquid
self.class == other.class && state == other.state
end
private
# Returns true if +object+ has +key+ accessible via [] lookup.
def accessible?(object, key)
if object.instance_of?(Hash)
object.key?(key)
else
object.respond_to?(:[]) &&
((object.respond_to?(:key?) && object.key?(key)) ||
(object.respond_to?(:fetch) && key.is_a?(Integer)))
end
end
# Calls to_liquid on +object+ and wires up the context reference if needed.
# Skipped for primitive types that return self from to_liquid.
def liquidize(object, context)
object = object.to_liquid
object.context = context if object.respond_to?(:context=)
object
end
protected
def state
+62
View File
@@ -0,0 +1,62 @@
# frozen_string_literal: true
# Quick benchmark for autoresearch: measures parse µs, render µs, and object allocations
# Outputs machine-readable metrics to stdout
require_relative 'theme_runner'
RubyVM::YJIT.enable if defined?(RubyVM::YJIT)
runner = ThemeRunner.new
# Warmup — enough iterations for YJIT to fully optimize hot paths
20.times { runner.compile }
20.times { runner.render }
GC.start
GC.compact if GC.respond_to?(:compact)
# Measure parse
parse_times = []
10.times do
GC.disable
t0 = Process.clock_gettime(Process::CLOCK_MONOTONIC)
runner.compile
t1 = Process.clock_gettime(Process::CLOCK_MONOTONIC)
GC.enable
GC.start
parse_times << (t1 - t0) * 1_000_000 # µs
end
# Measure render
render_times = []
10.times do
GC.disable
t0 = Process.clock_gettime(Process::CLOCK_MONOTONIC)
runner.render
t1 = Process.clock_gettime(Process::CLOCK_MONOTONIC)
GC.enable
GC.start
render_times << (t1 - t0) * 1_000_000 # µs
end
# Measure object allocations for one parse+render cycle
require 'objspace'
GC.start
GC.disable
before = ObjectSpace.count_objects.values_at(:TOTAL).first - ObjectSpace.count_objects.values_at(:FREE).first
runner.compile
runner.render
after = ObjectSpace.count_objects.values_at(:TOTAL).first - ObjectSpace.count_objects.values_at(:FREE).first
GC.enable
allocations = after - before
parse_us = parse_times.min.round(0)
render_us = render_times.min.round(0)
combined_us = parse_us + render_us
puts "RESULTS"
puts "parse_us=#{parse_us}"
puts "render_us=#{render_us}"
puts "combined_us=#{combined_us}"
puts "allocations=#{allocations}"
+28 -21
View File
@@ -44,32 +44,39 @@ class SecurityTest < Minitest::Test
end
def test_does_not_permanently_add_filters_to_symbol_table
current_symbols = Symbol.all_symbols
assert_no_new_symbols do
# MRI imprecisely marks objects found on the C stack, which can result
# in uninitialized memory being marked. This can even result in the test failing
# deterministically for a given compilation of ruby. Using a separate thread will
# keep these writes of the symbol pointer on a separate stack that will be garbage
# collected after Thread#join.
Thread.new do
test = %( {{ "some_string" | a_bad_filter }} )
Template.parse(test).render!
nil
end.join
# MRI imprecisely marks objects found on the C stack, which can result
# in uninitialized memory being marked. This can even result in the test failing
# deterministically for a given compilation of ruby. Using a separate thread will
# keep these writes of the symbol pointer on a separate stack that will be garbage
# collected after Thread#join.
Thread.new do
test = %( {{ "some_string" | a_bad_filter }} )
Template.parse(test).render!
nil
end.join
GC.start
assert_equal([], Symbol.all_symbols - current_symbols)
GC.start
end
end
def test_does_not_add_drop_methods_to_symbol_table
assert_no_new_symbols do
assigns = { 'drop' => Drop.new }
assert_equal("", Template.parse("{{ drop.custom_method_1 }}", assigns).render!)
assert_equal("", Template.parse("{{ drop.custom_method_2 }}", assigns).render!)
assert_equal("", Template.parse("{{ drop.custom_method_3 }}", assigns).render!)
end
end
def assert_no_new_symbols
# Run once to trigger any first-time initialization which might create some symbols,
# for example autoload or lazy method parsing might create symbols on first execution.
yield
# Ensure no new symbols for further runs, i.e. the code does not leak symbols
current_symbols = Symbol.all_symbols
assigns = { 'drop' => Drop.new }
assert_equal("", Template.parse("{{ drop.custom_method_1 }}", assigns).render!)
assert_equal("", Template.parse("{{ drop.custom_method_2 }}", assigns).render!)
assert_equal("", Template.parse("{{ drop.custom_method_3 }}", assigns).render!)
yield
assert_equal([], Symbol.all_symbols - current_symbols)
end
+2
View File
@@ -1181,6 +1181,8 @@ class StandardFiltersTest < Minitest::Test
end
def test_all_filters_never_raise_non_liquid_exception
skip("too slow on non-CRuby due to many exceptions") unless RUBY_ENGINE == 'ruby'
test_drop = TestDrop.new(value: "test")
test_drop.context = Context.new
test_enum = TestEnumerable.new
+27
View File
@@ -48,6 +48,33 @@ class TokenizerTest < Minitest::Test
assert_equal(["{%%}", "}"], tokenize('{%%}}'))
end
# Regression: lone '{' at or near end of string previously caused an infinite
# loop. The stray-{ else branch left `pos` unchanged when no further '{{' or
# '{%' existed, so the outer loop found the same '{' on every iteration.
def test_lone_brace_does_not_loop
assert_equal(["{"], tokenize('{'))
assert_equal(["a{"], tokenize('a{'))
assert_equal(["hello { world {"], tokenize('hello { world {'))
assert_equal(["{ world"], tokenize('{ world'))
assert_equal(["x{y"], tokenize('x{y'))
assert_equal(["{b{c"], tokenize('{b{c'))
end
def test_lone_brace_before_real_token
assert_equal(
["a { b ", "{% if x %}", "yes", "{% endif %}", " c"],
tokenize('a { b {% if x %}yes{% endif %} c'),
)
assert_equal(
["x { ", "{{ var }}", " y"],
tokenize('x { {{ var }} y'),
)
assert_equal(
["{ ", "{{ var }}"],
tokenize('{ {{ var }}'),
)
end
private
def new_tokenizer(source, parse_context: Liquid::ParseContext.new, start_line_number: nil)
+126
View File
@@ -0,0 +1,126 @@
# frozen_string_literal: true
require 'test_helper'
# Tests that the fast-path parser (try_fast_parse) produces the same result as the
# full Lexer → Parser pipeline for every input we expect it to handle.
#
# This protects against silent regressions where a change to try_fast_parse causes it
# to produce different output from the slow path (the existing test suite would still
# pass because the slow path catches it, but correctness would be silently lost).
class VariableFastParseTest < Minitest::Test
include Liquid
EQUIVALENCE_CASES = [
# Simple lookups
"product",
"product.title",
"product.variants.first.title",
# Quoted string literals
"'hello'",
'"hello"',
# Variables with no-arg filters
"product | upcase",
"product | upcase | downcase",
"product | strip | upcase | downcase",
# Variables with single-arg filters
"product | truncate: 50",
"product | plus: 1",
"product | plus: -3",
"product | round: 2",
"product | append: ' world'",
# Variables with multi-arg filters
"product | replace: 'a', 'b'",
"product | pluralize: 'item', 'items'",
"product | slice: 0, 5",
# Chained mixed filters
"product.title | truncate: 50",
"'hello' | append: ' world' | upcase",
"name | prepend: 'Dr. ' | append: ' PhD' | upcase",
# Numeric args
"count | plus: 1.5",
"price | minus: 0.99",
# No whitespace around pipe
"x|upcase",
"x|replace:'a','b'|upcase",
# Leading/trailing whitespace
" product ",
" product.title | upcase ",
].freeze
EQUIVALENCE_CASES.each_with_index do |markup, i|
define_method(:"test_fast_parse_equivalence_#{i.to_s.rjust(2, "0")}") do
lax_ctx = Liquid::ParseContext.new(error_mode: :lax)
strict_ctx = Liquid::ParseContext.new(error_mode: :strict)
lax_var = Liquid::Variable.new(markup, lax_ctx)
strict_var = Liquid::Variable.new(markup, strict_ctx)
assert_equal strict_var.name,
lax_var.name,
"Name mismatch for #{markup.inspect}: " \
"lax=#{lax_var.name.inspect} strict=#{strict_var.name.inspect}"
assert_equal strict_var.filters.length,
lax_var.filters.length,
"Filter count mismatch for #{markup.inspect}: " \
"lax=#{lax_var.filters.inspect} strict=#{strict_var.filters.inspect}"
strict_var.filters.each_with_index do |(s_name, *), i|
l_name = lax_var.filters[i][0]
assert_equal s_name,
l_name,
"Filter name mismatch at index #{i} for #{markup.inspect}"
end
end
end
# Verify the fast path is actually taken for simple variables (i.e. filters is the
# shared frozen EMPTY_ARRAY, not a newly allocated array).
def test_fast_path_taken_for_simple_variable
ctx = Liquid::ParseContext.new(error_mode: :lax)
var = Liquid::Variable.new("product.title", ctx)
assert_same(
Liquid::Const::EMPTY_ARRAY,
var.filters,
"Expected fast path (frozen EMPTY_ARRAY) for simple variable",
)
end
def test_fast_path_taken_for_no_arg_filter
ctx = Liquid::ParseContext.new(error_mode: :lax)
var = Liquid::Variable.new("product | upcase", ctx)
assert_equal(1, var.filters.length)
assert_equal("upcase", var.filters[0][0])
# The no-arg filter tuple should come from NO_ARG_FILTER_CACHE (frozen)
assert_predicate(var.filters[0], :frozen?)
end
def test_fast_path_taken_for_single_arg_filter
ctx = Liquid::ParseContext.new(error_mode: :lax)
var = Liquid::Variable.new("product | truncate: 50", ctx)
assert_equal(1, var.filters.length)
assert_equal("truncate", var.filters[0][0])
assert_equal([50], var.filters[0][1])
end
# Keyword args must fall through to the Lexer — verify the result is still correct.
def test_keyword_arg_falls_to_lexer_and_parses_correctly
ctx = Liquid::ParseContext.new(error_mode: :lax)
var = Liquid::Variable.new("img | img_tag: class: 'hero'", ctx)
assert_equal(1, var.filters.length)
assert_equal("img_tag", var.filters[0][0])
end
# Numeric filter arguments: integers and floats
def test_numeric_filter_args
ctx = Liquid::ParseContext.new(error_mode: :lax)
int_var = Liquid::Variable.new("price | plus: 3", ctx)
assert_equal([3], int_var.filters[0][1])
neg_var = Liquid::Variable.new("price | minus: -1", ctx)
assert_equal([-1], neg_var.filters[0][1])
float_var = Liquid::Variable.new("price | round: 2.5", ctx)
assert_equal([2.5], float_var.filters[0][1])
end
end