PMD Comments Removal

[Malmahrouqi3]

Description

This is subsequent to (#882) with the mere addition of filtering out inline comments and commented lines. For the PMD details, check out the original issue (#646)

To show off the difference:
Filter-off: pmd-old branch https://github.com/Malmahrouqi3/MFC-mo2/actions/runs/15693861383/job/44214858621
Filter-on: pmd-new branch https://github.com/Malmahrouqi3/MFC-mo2/actions/runs/15694477828/job/44216673771

[sbryngelson]

remove

[Malmahrouqi3]

@sbryngelson the number of violations is identical (63) whether the filter is on or off.

[sbryngelson]

strange

[sbryngelson]

idk about the number of violations, but the violations are different. This looks like an improvement to me.

[sbryngelson]

also remove line breaks: https://chatgpt.com/s/t_6850a2d138c0819187272dc044cfb208

sed -E '
  :a                      # ① label for looping
  /&$/ {                  # ② if the *current* line ends with “&”
        N                 #    read the *next* line into the pattern space
        s/&[[:space:]]*\n[[:space:]]*&/ /   # ③ delete the trailing “&”,
                                            #    the following newline, the
                                            #    leading “&” on the next line,
                                            #    and join them with one space
        ba                # ④ repeat until the next line no longer starts
                          #    with “&” (handles runs of many “&” lines)
  }
  s/&//g                  # ⑤ finally strip any stray “&” left in the file
'  input.txt  >  output.txt

(i haven't checked to see if this works myself). PS this will mess up the line numbers but whatever.

[sbryngelson]

this might break for openacc line continuations which have something like

!$acc <......> &
!$acc <......>

idk what it will do to code that looks like this.

[Malmahrouqi3]

It deems them as comments as far as I know

[sbryngelson]

supposedly this will work for the acc statements but the first command might break acc w/ line continuation. i would run the acc one first, then the one above

sed -E '
  :a
  /[[:space:]]*&[[:space:]]*$/ { N; s/[[:space:]]*&[[:space:]]*\n[[:space:]]*/ /; ba }
  s/^[[:space:]]*!\$acc[[:space:]]*/!$acc /
  :b
  s/[[:space:]]+!\$acc[[:space:]]*/ /g
  tb
'  source.f90  >  clean.f90

[sbryngelson]

It deems them as comments as far as I know

it has no notion of comments, it just looks at text. i'm trying to make the text as simple as possible, and having line continuations in different places is not a meaningful code difference so we should remove them before CPD

[Malmahrouqi3]

There were some syntax errors and I got rid off em. Also I lowered the number of tokens=20 kinda semi-helpful cuz at 40 we only see big blocks mostly that are barely affected by comments. Further, I added these flags to avoid dummy errors/violations. --no-fail-on-violation \--no-fail-on-error so the test will fail iff pmd.yml itself is messed up - syntax wise.

Current implementation

                  sed -E '
                    :a
                    /&$/ {
                          N
                          s/&[[:space:]]*\n[[:space:]]*&/ /
                          ba
                    }
                    s/&//g
                  ' "$file" | \
                  sed -E \
                      -e '/^\s*!/s/.*//' \
                      -e '/^[cC*dD]/s/.*//' \
                      -e '/^[ \t]*[cC*dD]/s/.*//' \
                      -e 's/([^"'\'']*("[^"]*"[^"'\'']*|'\''[^'\'']*'\''[^"'\'']*)*[^"'\'']*)[!].*$/\1/' \
                      > "$TMP_FILE"

[Malmahrouqi3]

(i haven't checked to see if this works myself). PS this will mess up the line numbers but whatever.

You can just copy and paste the redundant lines and search them up. It ain't gonna be a wild hunting job.

[sbryngelson]

something isn't right with concatenating the line continuations because this is a repeated pattern in pmd:

                                q_sf(j, k, l) = q_sf(j, k, l) 
                                                + q_prim_vf0(mom_idx%beg)%sf(j, k, l)*fd_coeff_x(r, j)* 
                                                q_prim_vf0(mom_idx%beg)%sf(r + j, k, l) 
                                                + q_prim_vf0(mom_idx%beg + 1)%sf(j, k, l)*fd_coeff_y(r, k)* 
                                                q_prim_vf0(mom_idx%beg)%sf(j, r + k, l)

if they were concatted then this would be on one line

[Malmahrouqi3]

I guess it should now presumably detect more duplicate lines if they are exactly the same operation but split out into lines differently.

q_sf(j, k, l) = q_sf(j, k, l)+q_prim_vf0(mom_idx%beg)%sf(j, k, l)*fd_coeff_x(r, j)*q_prim_vf0(mom_idx%beg)%sf(r+j, k, l)+q_prim_vf0(mom_idx%beg+1)%sf(j, k, l)*fd_coeff_y(r, k)*q_prim_vf0(mom_idx%beg)%sf(j, r+k, l)+q_prim_vf0(mom_idx%end)%sf(j, k, l)*fd_coeff_z(r, l)*q_prim_vf0(mom_idx%beg)%sf(j, k, r+l)/y_cc(k)

[sbryngelson]

yes that's what i was thinking. you could even strip spaces (maybe?)

[sbryngelson]

this is actually a very valuable tool!

[Malmahrouqi3]

I left behind = .or. .and. and few subtle things.
Other than that, all spaces should be taken off around math/comparison operators, inside indexing parentheses and brackets.

[sbryngelson]

I left behind = .or. .and. and few subtle things. Other than that, all spaces should be taken off around math/comparison operators, inside indexing parentheses and brackets.

yes agreed. so you already did this or not yet?

[Malmahrouqi3]

yup, you can check out the last commit PMD check.

[Malmahrouqi3]

Filter Full Implementation

                  sed -E '
                    # First handle & continuation style (modern Fortran)
                    :ampersand_loop
                    /&[[:space:]]*$/ {
                      N
                      s/&[[:space:]]*\n[[:space:]]*(&)?/ /g
                      tampersand_loop
                    }

                    # Handle fixed-form continuation (column 6 indicator)
                    :fixed_form_loop
                    /^[[:space:]]{0,5}[^[:space:]!&]/ {
                      N
                      s/\n[[:space:]]{5}[^[:space:]]/ /g
                      tfixed_form_loop
                    }

                    # Remove any remaining continuation markers
                    s/&//g

                    # Normalize spacing - replace multiple spaces with single space
                    s/[[:space:]]{2,}/ /g

                    # Remove spaces around mathematical operators
                    s/[[:space:]]*\*[[:space:]]*/*/g
                    s/[[:space:]]*\+[[:space:]]*/+/g
                    s/[[:space:]]*-[[:space:]]*/-/g
                    s/[[:space:]]*\/[[:space:]]*/\//g
                    s/[[:space:]]*\*\*[[:space:]]*/\*\*/g

                    # Remove spaces in common Fortran constructs (array indexing, function calls)
                    s/\([[:space:]]*([^,)[:space:]]+)[[:space:]]*,/(\1,/g      # First argument
                    s/,[[:space:]]*([^,)[:space:]]+)[[:space:]]*,/,\1,/g       # Middle arguments
                    s/,[[:space:]]*([^,)[:space:]]+)[[:space:]]*\)/,\1)/g      # Last argument
                    s/\([[:space:]]*([^,)[:space:]]+)[[:space:]]*\)/(\1)/g     # Single argument

                    # Remove spaces around brackets and parentheses
                    s/\[[[:space:]]*/</g
                    s/\[[[:space:]]*/>/g
                    s/\[[[:space:]]*/</g
                    s/[[:space:]]*\]/]/g
                    s/\([[:space:]]*/(/g
                    s/[[:space:]]*\)/)/g

                    # Remove spaces around comparison operators
                    s/[[:space:]]*<=[[:space:]]*/</g
                    s/[[:space:]]*>=[[:space:]]*/>/g
                    s/[[:space:]]*<[[:space:]]*/</g
                    s/[[:space:]]*>[[:space:]]*/>/g
                    s/[[:space:]]*==[[:space:]]*/==/g

                    # Remove full-line comments
                    /^\s*!/d
                    /^[cC*dD]/d
                    /^[ \t]*[cC*dD]/d

                    # Remove end-of-line comments, preserving quoted strings
                    s/([^"'\''\\]*("[^"]*")?('\''[^'\'']*'\''?)?[^"'\''\\]*)[!].*$/\1/
                  ' "$file" > "$TMP_FILE"