Should comments be treated as whitespaces?

[igordejanovic]

textX/Arpeggio treat comments in a special way. More specifically, noskipws rule modifier will disable whitespace skipping but will leave comments skipping in place which may lead to some gotchas.

If comments should be treated like whitespaces than noskipws would mean don't skip whitespaces and comments. This would be a backward incompatible change. This would also mean that comments should be handled manually in noskipws rules.

This issue is created as a poll to gather feedback from the community. Please vote with emoji thumbs up/down.

[Aluriak]

Whitespaces and comments are different things. Why handling them as the same object ? Is it not simpler (user-side) to have options for both ? noskipws would retain his actual meaning, and commentaswhitespace would allow for both behaviors.

Note that in most languages, comments are handled per rule. This is an example in C, where two comments have different side-effects:

int main() {
    int a/*break the identifier rule*/b = /*no side effect*/ 2;
    printf("%i", ab);
}

I do not know if it is possible at all, but letting user provides a set of rule names indicating which rules may (not) be broken by a comment seems to fix a lot of problems too.
Maybe this rule may be simplified to: any raw data/regex rule can't be broken by comments.

[igordejanovic]

Note that in most languages, comments are handled per rule.

Can you elaborate more on this please. Not sure that I follow.

If I understand the example with C you are implying that comment between a and b breaks the identifier. That happens also if you put whitespace at that position. So, is there any difference in handling. I agree that comments and whitespaces are different in a sense that comments provide some useful information to the reader (and Arpeggio does collect comments) but is their handling any different? If you replace all comments in source code with whitespaces, would it make any difference?

Maybe this rule may be simplified to: any raw data/regex rule can't be broken by comments.

This is the default behaviour. Regex can't be broken by a comment or whitespace. It is in a sense atomic.

[dkrikun]

Please do not treat comments as whitespace:

1. Sometimes whitespace is significant (e.g. python) but comments aren't
2. Sometimes comments are significant and need be preserved -- aka docstrings in python, but whitespace isn't

[fzipi]

@igordejanovic In fact, one of the things that I miss the most is not having access to the comments, for example, when transpiling things. We just lose the comments.

I would like to have a flag to conserve them.

[igordejanovic]

@fzipi Totally agree. Actually, that is something I had been planning for quite some time. It requires some work in the Arpeggio. I thought that the issue for this is already registered in the tracker but it seems that I was wrong. Please feel free to register the feature request.

[fzipi]

@igordejanovic Super, will do! Thanks for your support :)

[fzipi]

Created textX/Arpeggio#56.