Does this have a marked performance improvement? I believe we only call this method once per lex.

Does this have a marked performance improvement?

Not at all. Just reductions in array allocations.

I believe we only call this method once per lex.

I checked this by running against Jekyll's docs source-code. The Lexer.assert_utf8! was called a total of 5196 times

Wait, hang on. Why is this method being called so often?

Also, we don't really need an array, so if the allocation is really where the problem is (and not that this method is being called upwards of 5000 times for some reason) I'd prefer to just use ||.

Wait, hang on. Why is this method being called so often?

@jneen Because this method is part of two frequently called methods (based on how Jekyll and kramdown is using Rouge):

I'd prefer to just use ||

Indeed, my issue here is to eliminate as much unnecessary allocations (and retained objects) as possible. (Irrespective of whether they have a significant impact on the performance (speed)). To that end using just strings and an operator call is all the more welcome.

Can you tell me more about that pattern of calling? I want to make sure we're optimizing for the right stuff.

Can you tell me more about that pattern of calling?

@jneen Jekyll has a custom Liquid tag that can be used to render code-blocks with syntax-highlighting in a Jekyll site's templates and themes.
This is how the tag uses Rouge:

      def render_rouge(code)
        require "rouge"
        formatter = ::Rouge::Formatters::HTMLLegacy.new(
          :line_numbers => @highlight_options[:linenos],
          :wrap         => false,
          :css_class    => "highlight",
          :gutter_class => "gutter",
          :code_class   => "code"
        )
        lexer = ::Rouge::Lexer.find_fancy(@lang, code) || Rouge::Lexers::PlainText
        formatter.format(lexer.lex(code))
      end

This method will be called *for every use of {% highlight %} some code {% endhighlight %} in a site configured to use Rouge as the site's syntax-highlighter.
The line of interest here is the very last expression:

formatter.format(lexer.lex(code))

A similar code exists in kramdown that is used to render fenced code-blocks in Markdown:

    def self.call(converter, text, lang, type, call_opts)
      opts = options(converter, type)
      call_opts[:default_lang] = opts[:default_lang]
      return nil unless lang || opts[:default_lang] || opts[:guess_lang]

      lexer = ::Rouge::Lexer.find_fancy(lang || opts[:default_lang], text)
      return nil if opts[:disable] || !lexer || (lexer.tag == "plaintext" && !opts[:guess_lang])

      opts[:css_class] ||= 'highlight' # For backward compatibility when using Rouge 2.0
      formatter = formatter_class(opts).new(opts)
      formatter.format(lexer.lex(text))
    end

Makes sense, thank you!