Fixes infinite loop in tokenizer on trailing stray '{'

The stray-{ else branch in tokenize_fast had a nested while loop that could
exit via its condition (when next_open >= len) without advancing pos.
The outer while pos < len loop would then find the same { again forever.

Reproduced by: Liquid::Template.parse('a{') -- hangs indefinitely.

Replaces the nested scan loop with two String#byteindex calls to find the
next '{%' and '{{' directly, then takes the minimum. Always O(n), eliminates
the nested loop entirely, and impossible to leave pos stranded.

Adds regression tests covering the three inputs that previously hung
plus adjacent stray-brace cases.
This commit is contained in:
Chris Pak
2026-04-04 21:25:59 -07:00
parent bc60deb671
commit d9c42fd2eb
2 changed files with 39 additions and 18 deletions
+27
View File
@@ -48,6 +48,33 @@ class TokenizerTest < Minitest::Test
assert_equal(["{%%}", "}"], tokenize('{%%}}'))
end
# Regression: lone '{' at or near end of string previously caused an infinite
# loop. The stray-{ else branch left `pos` unchanged when no further '{{' or
# '{%' existed, so the outer loop found the same '{' on every iteration.
def test_lone_brace_does_not_loop
assert_equal(["{"], tokenize('{'))
assert_equal(["a{"], tokenize('a{'))
assert_equal(["hello { world {"], tokenize('hello { world {'))
assert_equal(["{ world"], tokenize('{ world'))
assert_equal(["x{y"], tokenize('x{y'))
assert_equal(["{b{c"], tokenize('{b{c'))
end
def test_lone_brace_before_real_token
assert_equal(
["a { b ", "{% if x %}", "yes", "{% endif %}", " c"],
tokenize('a { b {% if x %}yes{% endif %} c'),
)
assert_equal(
["x { ", "{{ var }}", " y"],
tokenize('x { {{ var }} y'),
)
assert_equal(
["{ ", "{{ var }}"],
tokenize('{ {{ var }}'),
)
end
private
def new_tokenizer(source, parse_context: Liquid::ParseContext.new, start_line_number: nil)