Ruby: pm_token_buffer_t Struct Reference

Ruby 4.0.5p0 (2026-05-20 revision 64336ffd0ee9e1f4c05891695a3d7b49cb709721)

Data Fields
pm_token_buffer_t Struct Reference

When we're lexing certain types (strings, symbols, lists, etc.) we have string content associated with the tokens. More...

## Data Fields
pm_buffer_t | buffer
| The buffer that we're using to keep track of the string content.
const uint8_t * | cursor
| The cursor into the source string that points to how far we have currently copied into the buffer.

## Detailed Description

When we're lexing certain types (strings, symbols, lists, etc.) we have string content associated with the tokens.

For example:

    "foo"

In this case, the string content is foo. Since there is no escaping, there's no need to track additional information and the token can be returned as normal. However, if we have escape sequences:

    "foo\n"

then the bytes in the string are "f", "o", "o", "\", "n", but we want to provide our consumers with the string content "f", "o", "o", "
". In these cases, when we find the first escape sequence, we initialize a pm_buffer_t to keep track of the string content. Then in the parser, it will automatically attach the string content to the node that it belongs to.

Definition at line 9465 of file prism.c.

## Field Documentation

## ◆ buffer

pm_buffer_t pm_token_buffer_t::buffer

The buffer that we're using to keep track of the string content.

It will only be initialized if we receive an escape sequence.

Definition at line 9470 of file prism.c.

## ◆ cursor

const uint8_t* pm_token_buffer_t::cursor

The cursor into the source string that points to how far we have currently copied into the buffer.

Definition at line 9476 of file prism.c.

The documentation for this struct was generated from the following file:

- prism/prism.c