Skip to content

Conversation

@jpountz
Copy link
Contributor

@jpountz jpountz commented Jan 30, 2025

The recent optimization from #14164 interfered in a bad way with a prior optimization.

The recent optimization from apache#14164 interfered in a bad way with a prior
optimization.
@jpountz jpountz added this to the 10.2.0 milestone Jan 30, 2025
@jpountz
Copy link
Contributor Author

jpountz commented Jan 30, 2025

Benchmarks suggest that the additional overhead in nextDoc() is fine:

                            TaskQPS baseline      StdDevQPS my_modified_version      StdDev                Pct diff p-value
                      OrHighRare      517.98      (4.5%)      506.55      (6.5%)   -2.2% ( -12% -    9%) 0.211
                DismaxOrHighHigh      163.10      (2.2%)      160.59      (3.4%)   -1.5% (  -6% -    4%) 0.090
             FilteredOrStopWords       71.58      (6.1%)       70.50      (6.1%)   -1.5% ( -12% -   11%) 0.433
                       OrHighMed      292.53      (2.3%)      288.61      (2.4%)   -1.3% (  -5% -    3%) 0.066
              FilteredOrHighHigh       97.76      (6.8%)       96.47      (6.0%)   -1.3% ( -13% -   12%) 0.514
                     OrStopWords       44.42      (3.4%)       43.84      (3.7%)   -1.3% (  -8% -    5%) 0.240
                      OrHighHigh       67.46      (2.2%)       66.59      (1.9%)   -1.3% (  -5% -    2%) 0.050
                        Wildcard      131.06      (3.0%)      129.44      (4.2%)   -1.2% (  -8% -    6%) 0.280
             And2Terms2StopWords      267.40      (3.7%)      264.15      (3.2%)   -1.2% (  -7% -    5%) 0.264
                    AndStopWords       40.26      (3.9%)       39.78      (3.8%)   -1.2% (  -8% -    6%) 0.322
                        PKLookup      374.88      (3.7%)      370.62      (3.9%)   -1.1% (  -8% -    6%) 0.342
                       And3Terms      236.13      (3.1%)      233.61      (3.3%)   -1.1% (  -7% -    5%) 0.287
                 DismaxOrHighMed      249.56      (2.0%)      247.08      (2.6%)   -1.0% (  -5% -    3%) 0.178
                          OrMany       25.96      (5.5%)       25.71      (4.0%)   -1.0% (  -9% -    9%) 0.529
                         Prefix3      235.88      (3.1%)      233.63      (4.0%)   -1.0% (  -7% -    6%) 0.395
              Or2Terms2StopWords      253.41      (3.5%)      251.17      (3.7%)   -0.9% (  -7% -    6%) 0.439
                  FilteredOrMany       20.15      (9.8%)       19.98      (9.3%)   -0.8% ( -18% -   20%) 0.779
                      AndHighMed      176.43      (2.6%)      174.96      (2.8%)   -0.8% (  -6% -    4%) 0.328
                     AndHighHigh       57.64      (3.0%)       57.16      (3.5%)   -0.8% (  -7% -    5%) 0.423
                FilteredOr3Terms      234.70      (6.7%)      233.02      (5.9%)   -0.7% ( -12% -   12%) 0.719
      FilteredOr2Terms2StopWords      237.64      (5.6%)      235.96      (5.3%)   -0.7% ( -11% -   10%) 0.683
                      TermDTSort      606.25      (4.7%)      602.01      (3.8%)   -0.7% (  -8% -    8%) 0.605
              CombinedOrHighHigh       29.13      (2.7%)       28.93      (2.7%)   -0.7% (  -5% -    4%) 0.432
               FilteredOrHighMed      236.89      (6.5%)      235.59      (5.9%)   -0.5% ( -12% -   12%) 0.781
               CombinedOrHighMed      116.23      (2.4%)      115.60      (2.7%)   -0.5% (  -5% -    4%) 0.503
                      DismaxTerm      823.43      (4.1%)      819.10      (5.2%)   -0.5% (  -9% -    9%) 0.721
                        Or3Terms      220.05      (3.9%)      219.03      (3.9%)   -0.5% (  -7% -    7%) 0.706
                 CountAndHighMed      425.17      (3.3%)      423.34      (2.8%)   -0.4% (  -6% -    5%) 0.656
                  TermBGroup1M1P       53.55      (2.7%)       53.35      (2.8%)   -0.4% (  -5% -    5%) 0.673
     FilteredAnd2Terms2StopWords      308.51      (2.3%)      307.49      (2.6%)   -0.3% (  -5% -    4%) 0.669
              FilteredAndHighMed      184.14      (2.4%)      183.64      (2.7%)   -0.3% (  -5% -    4%) 0.735
                AndMedOrHighHigh       91.28      (3.5%)       91.04      (3.1%)   -0.3% (  -6% -    6%) 0.803
                    CombinedTerm       50.34      (2.4%)       50.27      (3.1%)   -0.2% (  -5% -    5%) 0.863
             FilteredAndHighHigh       98.97      (4.6%)       98.83      (4.4%)   -0.1% (  -8% -    9%) 0.918
                     CountPhrase        7.07      (2.9%)        7.06      (2.7%)   -0.1% (  -5% -    5%) 0.878
                    TermGroup10K       30.48      (2.4%)       30.44      (1.8%)   -0.1% (  -4% -    4%) 0.851
               TermDayOfYearSort     3408.11      (1.9%)     3403.87      (1.5%)   -0.1% (  -3% -    3%) 0.821
                         Respell       76.33      (2.3%)       76.24      (2.4%)   -0.1% (  -4% -    4%) 0.875
                  CountOrHighMed      522.17      (3.3%)      521.60      (2.9%)   -0.1% (  -6% -    6%) 0.912
                     TermGroup1M       32.16      (2.1%)       32.14      (1.7%)   -0.1% (  -3% -    3%) 0.917
         CountFilteredOrHighHigh      221.09      (3.1%)      220.96      (2.7%)   -0.1% (  -5% -    5%) 0.949
          CountFilteredOrHighMed      241.72      (2.3%)      241.63      (1.8%)   -0.0% (  -4% -    4%) 0.953
                    TermBGroup1M       43.00      (2.6%)       42.98      (2.5%)   -0.0% (  -5% -    5%) 0.968
             CountFilteredOrMany       40.87      (4.9%)       40.86      (4.2%)   -0.0% (  -8% -    9%) 0.988
             CombinedAndHighHigh       24.90      (3.4%)       24.90      (3.1%)   -0.0% (  -6% -    6%) 0.988
                       CountTerm    15466.46      (3.6%)    15465.67      (4.6%)   -0.0% (  -7% -    8%) 0.997
                    TermGroup100       33.90      (2.4%)       33.91      (2.9%)    0.0% (  -5% -    5%) 0.993
              CombinedAndHighMed       96.43      (3.3%)       96.44      (2.9%)    0.0% (  -6% -    6%) 0.992
                 CountOrHighHigh      465.82      (6.8%)      466.21      (5.3%)    0.1% ( -11% -   13%) 0.965
                CountAndHighHigh      509.45      (6.6%)      509.90      (5.7%)    0.1% ( -11% -   13%) 0.964
                          Fuzzy1      111.93      (2.9%)      112.05      (2.6%)    0.1% (  -5% -    5%) 0.907
                          Fuzzy2      105.56      (2.6%)      105.69      (2.5%)    0.1% (  -4% -    5%) 0.887
               FilteredAnd3Terms      270.45      (2.4%)      270.82      (3.6%)    0.1% (  -5% -    6%) 0.888
                   TermTitleSort      236.14      (2.2%)      236.48      (2.2%)    0.1% (  -4% -    4%) 0.836
                   TermMonthSort     3512.78      (2.0%)     3518.37      (2.5%)    0.2% (  -4% -    4%) 0.820
            FilteredAndStopWords       81.77      (4.4%)       81.90      (4.4%)    0.2% (  -8% -    9%) 0.908
                          IntNRQ      222.55     (12.7%)      223.07      (9.2%)    0.2% ( -19% -   25%) 0.946
                    FilteredTerm      269.47      (2.4%)      270.24      (2.2%)    0.3% (  -4% -    4%) 0.693
                  FilteredPhrase       55.62      (2.7%)       55.89      (2.8%)    0.5% (  -4% -    6%) 0.575
                 AndHighOrMedMed       71.48      (2.5%)       71.83      (2.5%)    0.5% (  -4% -    5%) 0.533
             CountFilteredPhrase       50.57      (3.3%)       50.82      (2.9%)    0.5% (  -5% -    6%) 0.618
                          Phrase       23.97      (3.5%)       24.13      (3.4%)    0.6% (  -6% -    7%) 0.561
                            Term      786.42      (4.5%)      799.69      (3.5%)    1.7% (  -6% -   10%) 0.187

@jpountz jpountz requested a review from gf2121 January 30, 2025 18:38
Copy link
Contributor

@gf2121 gf2121 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @jpountz !

I intended to make the opt works only if needsImpacts == false while i forgot it when i refactor needsDocsOnly to needsFreq. Sorry!

@jpountz jpountz merged commit b429c43 into apache:main Feb 1, 2025
5 checks passed
@jpountz jpountz deleted the fix/next_doc_refill branch February 1, 2025 16:41
jpountz added a commit that referenced this pull request Feb 1, 2025
The recent optimization from #14164 interfered in a bad way with a prior
optimization.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants