Tukukkk

I am trying to parse a very simple language with PEGTL. I think I have found the problem, but don't understand why; white spaces are not ignored. I understand it must be possible to not ignore white space so that indentation-aware languages can also be parsed. But I couldn't find a mechanism to "eat" white spaces by default. Given:

struct kw_enum : tao::pegtl::string<'e', 'n', 'u', 'm'> { };

struct enum_decl : tao::pegtl::seq<kw_enum, tao::pegtl::identifier, tao::pegtl::one<';'>> { };

the following cannot be parsed:

enum thing;

If I add pegtl::space between each token explicitly, then it works. But it would be major burden to do it in the entire grammar.

How can white spaces be ignored/eaten/skipped, like in C, without specifying those explicitly?

I don't think there is a shortcut, you have to specify the grammar in a way that makes it unambiguous where and how many whitespaces you allow.

The best way to do it I think is to add a convenience rule template that allows matching a list (tao::pegtl::seq) of rules separated by any allowed separater (generally whitespace plus comments).

struct comment : tao::pegtl::disable< /* whatever your comment syntax is */ > {};

struct separator : tao::pegtl::sor< tao::pegtl::ascii::space, comment > {}; // either/or

struct seps : tao::pegtl::star< separator > {}; // Any separators, whitespace or comments



// Template to generate rule

// tao::pegtl::seq<Rule0, Separator, Rule1, Separator, Rule2, ... , Separator, RuleN>

template <typename Separator, typename... Rules>

struct interleaved;



template <typename Separator, typename Rule0, typename... RulesRest>

struct interleaved<Separator, Rule0, RulesRest...>

  : tao::pegtl::seq<Rule0, Separator, interleaved<Separator, RulesRest...>> {};



template <typename Separator, typename Rule0>

struct interleaved<Separator, Rule0>

  : Rule0 {};



// Note: interleaved<Separator /*, no Rule! */> intentionally not defined.



struct enum_decl : interleaved<seps, kw_enum, tao::pegtl::identifier, tao::pegtl::one<';'> {};

// Expands to:

seq<kw_enum, seps, interleaved<seps, identifier, one<';'>>> ==

seq<kw_enum, seps, seq<identifier, seps, interleaved<seps, one<';'>>>> ==

seq<kw_enum, seps, seq<identifier, seps, one<';'>> ==

seq<kw_enum, seps, identifier, seps, one<';'>>

Basically, doing something like the above, you only need to replace tao::pegtl::seq<R...> with interleaved<seps, R...>, but you could even make a separate alias for that:

template<typename... Rules>

using sseq = interleaved<seps, Rules...>;



// Now you only have to replace tao::pegtl::seq with sseq

struct enum_decl : sseq<kw_enum, tao::pegtl::identifier, tao::pegtl::one<';'>> {};

This strategy is mostly needed not because of whitespace-significant languages but because you don't have an earlier tokenizer step in the parsing that would separate the enum from the thing. Another strategy would be to implement that first, and do the later-stage parsing based on a stream of tokens instead of a stream of characters, but that is a larger rewrite and has its own drawbacks.

Note: I haven't compiled the code, but the point here should be the strategy. Please leave a comment if there is a compile error (I hope I haven't messed up the variadic templates) or if something is unclear.

Note 2: Also you probably don't want to replace every seq with sseq. For example, if you have the logical and (&&) defined as seq<one<'&'>, one<'&'>>, that's something you probably don't want to change.