fix: Support auto scan mode with Spark 4.0.0

[andygrove]

Which issue does this PR close?

Closes #1967

Rationale for this change

Provide better performance with Spark 4.0.0 by supporting auto scan mode.

What changes are included in this PR?

• Remove auto fallback for Spark 4
• Fall back to native_comet for schemas containing collated string types
• Update Spark 4.0.0 diff to match changes made in other versions related to skipping tests for native scans

How are these changes tested?

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 20.00000% with 12 lines in your changes missing coverage. Please review.

Project coverage is 58.22%. Comparing base (f09f8af) to head (fd4cdb6).
Report is 315 commits behind head on main.

Files with missing lines	Patch %	Lines
...n/scala/org/apache/comet/rules/CometScanRule.scala	22.22%	3 Missing and 4 partials ⚠️
.../scala/org/apache/spark/sql/comet/util/Utils.scala	16.66%	5 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #1975      +/-   ##
============================================
+ Coverage     56.12%   58.22%   +2.09%     
- Complexity      976     1155     +179     
============================================
  Files           119      133      +14     
  Lines         11743    13037    +1294     
  Branches       2251     2421     +170     
============================================
+ Hits           6591     7591    +1000     
- Misses         4012     4209     +197     
- Partials       1140     1237      +97     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[andygrove]

hive-1 failure to investigate:

2025-07-09T20:18:09.4999985Z [info]   Cause: org.apache.comet.CometNativeException: External error: Arrow: Parquet argument error: Parquet error: encountered non UTF-8 data
2025-07-09T20:18:09.5000678Z [info]   at org.apache.comet.parquet.Native.readNextRecordBatch(Native Method)
2025-07-09T20:18:09.5001239Z [info]   at org.apache.comet.parquet.NativeBatchReader.loadNextBatch(NativeBatchReader.java:812)