Fixes infinite loop in tokenizer on trailing stray '{'

The stray-{ else branch in tokenize_fast had a nested while loop that could
exit via its condition (when next_open >= len) without advancing pos.
The outer while pos < len loop would then find the same { again forever.

Reproduced by: Liquid::Template.parse('a{') -- hangs indefinitely.

Replaces the nested scan loop with two String#byteindex calls to find the
next '{%' and '{{' directly, then takes the minimum. Always O(n), eliminates
the nested loop entirely, and impossible to leave pos stranded.

Adds regression tests covering the three inputs that previously hung
plus adjacent stray-brace cases.
This commit is contained in:
Chris Pak
2026-04-04 21:25:59 -07:00
parent bc60deb671
commit d9c42fd2eb
2 changed files with 39 additions and 18 deletions
+12 -18
View File
@@ -146,24 +146,18 @@ module Liquid
pos = idx + 2
end
else
# { followed by something else — it's text
# Keep scanning from after this {
# Find next { that could be {% or {{
next_open = idx + 1
while next_open < len
ni = src.byteindex('{', next_open)
unless ni
@tokens << src.byteslice(pos, len - pos)
pos = len
break
end
nb = ni + 1 < len ? src.getbyte(ni + 1) : nil
if nb == PERCENTAGE || nb == OPEN_CURLEY
@tokens << src.byteslice(pos, ni - pos)
pos = ni
break
end
next_open = ni + 1
# Lone '{' — not the start of a tag or variable.
# Find the next '{{' or '{%' to know where this text token ends.
# Using two byteindex calls avoids a nested loop and is always O(n).
tag_start = src.byteindex('{%', idx + 1)
var_start = src.byteindex('{{', idx + 1)
next_token = [tag_start, var_start].compact.min
if next_token
@tokens << src.byteslice(pos, next_token - pos)
pos = next_token
else
@tokens << src.byteslice(pos, len - pos)
pos = len
end
end
end
+27
View File
@@ -48,6 +48,33 @@ class TokenizerTest < Minitest::Test
assert_equal(["{%%}", "}"], tokenize('{%%}}'))
end
# Regression: lone '{' at or near end of string previously caused an infinite
# loop. The stray-{ else branch left `pos` unchanged when no further '{{' or
# '{%' existed, so the outer loop found the same '{' on every iteration.
def test_lone_brace_does_not_loop
assert_equal(["{"], tokenize('{'))
assert_equal(["a{"], tokenize('a{'))
assert_equal(["hello { world {"], tokenize('hello { world {'))
assert_equal(["{ world"], tokenize('{ world'))
assert_equal(["x{y"], tokenize('x{y'))
assert_equal(["{b{c"], tokenize('{b{c'))
end
def test_lone_brace_before_real_token
assert_equal(
["a { b ", "{% if x %}", "yes", "{% endif %}", " c"],
tokenize('a { b {% if x %}yes{% endif %} c'),
)
assert_equal(
["x { ", "{{ var }}", " y"],
tokenize('x { {{ var }} y'),
)
assert_equal(
["{ ", "{{ var }}"],
tokenize('{ {{ var }}'),
)
end
private
def new_tokenizer(source, parse_context: Liquid::ParseContext.new, start_line_number: nil)