Tukukkk

I am writing a parser with pyparsing for a particular file structure. It's mainly a list of element definitions with the syntax: ElementName: ElementType, ParameterList. The ParameterList is optional and has the structure of a pp.delimitedList. An example excerpt of this file follows:

"L_000333": DRIF,L=0.0607

BPM15FL2EXTR: MONI

Q8FL2EXTR: QUAD, &

            L=0.3286,K1=-0.7494081,&

               DY="vert_offset_1"

"L_000334": DRIF,L=0.359694

D8FL2EXTR: CSRCSBEN,L=0.2200112619037261,       &

       ANGLE=-0.015708,E1=-0.007854, E2=-0.007854,  &

       SG_HALFWIDTH = 1, DY = "vert_offset_1",  &

       N_KICKS = "num_CSR_kicks", BINS = "bin_num", &

       ISR = "with_ISR", CSR = "with_CSR",      &

           OUTPUT_INTERVAL = "output_interval",     &

       INTEGRATION_ORDER = "order_integration"

"L_000335": DRIF,L=0.134994

V9FL2EXTR: VKICK,L=0.1

...

To match the ElementType I want to use a list of given options, because there's a particular type called LINE, which does not match the same structure definition as the ones given in the example just above this lines. For instance, the type LINE follows the syntax:

FL2EXTR: LINE=(STARTFLEXTR,L_000297,FSHUTTER, ... etc... )

and it can be found in between of the elements given above. But by now, I'll forget about how to parse this particular type of element, because the problems come before :-/.

To achieve the proper parsing of a given list of elements, I defined the following building blocks:

import pyparsing as pp



# allowed delimiters:

lte_delim_0 = pp.Literal(',')

lte_delim_1 = pp.Literal('&')

lte_delim_2 = pp.Literal(',') + lte_delim_1

lte_delim = pp.Or([lte_delim_0, lte_delim_1, lte_delim_2])



# ElementName:

elementName = pp.Or([pp.Word(pp.alphanums + '_'), pp.dblQuotedString]) + 

              pp.Literal(':').suppress()



# ElementType:

elementType = pp.Or([pp.CaselessLiteral('DRIF'),

                     pp.CaselessLiteral('ALPH'),

                     pp.CaselessLiteral('BGGEXP'),

                     pp.CaselessLiteral('BMAPXY'),

                     pp.CaselessLiteral('BMXYZ'),



                     ... looong list of possible ElementType ...



                     pp.CaselessLiteral('ZTRANSVERSE')])



# ParameterName and ParameterValue for the ParameterList:

parameterName = pp.Word(pp.alphanums + '_')

parameterValue = pp.Or([pp.Word(pp.alphanums + '_'),

                        pp.pyparsing_common.number,

                        pp.dblQuotedString])



# basic assignment definition:

assignmentExpr = pp.Group(parameterName('ParameterName') + 

                          pp.Literal('=').suppress() +  

                          parameterValue('ParameterValue'))



parameterList = pp.Dict(pp.delimitedList(assignmentExpr, 

                        delim = lte_delim, combine = False))



# element definition

elementDefinition = pp.Group(elementName('ElementName') + 

                             elementType('ElementType') + 

                             pp.Optional(lte_delim.suppress() + 

                                         parameterList('ParameterList')))

Then I just create a parser for an element list as:

elementList = pp.OneOrMore(ElementDefinition)

However, when I parse the example list given above, the parser stops parsing when it encounters the element CSRCSBEN (even if it is defined in the elementType list of possible pp.CaselessLiterals!!!).

If instead of defining the elementType parser with the pp.Or([ ... list of pp.CaselessLiterals... ]) as I did above, I define the elementType simply as:

elementType = pp.Word(pp.alphanums + '_')

then I get all the element definitions in the list parsed right. But then, I cannot achieve the particular ElementType 'LINE' from being parsed differently...

Can anybody help me with that? Why is the parser stopping at the element CSRCSBEN even if it is given as a valid ElementType?

Thanks a lot in advance for your help!

Cheers,
Pau

Thanks for sharing an interesting problem. Sorry, I don't have a "good" solution for you. (Maybe if you spent a little time boiling it down to a simpler "repeat-by", a simpler grammar that exhibits the same effect?)

I will offer this suggestion. Consider pre-processing so that (A.) continuation lines with '&' become single long lines, and more importantly (B.) each line gets a type prefix, based on presence of a string like 'FL2EXTR' or 'CSRCSBEN'. Call them {TYPE1, TYPE2} or whatever is convenient, and prepend that token to the line.

Now you're faced with a pair of simpler problems, writing a pair of simpler grammars which have fewer details to cope with. The initial type token should make it easier to get just the 1st or 2nd grammar productions to trigger as appropriate.