Performance and Benchmarking Concerns

[alexevanczuk]

After releasing a handful of performance features, I wanted to make sure I shared other future performance improvements as well as documented some of the things I did to make performance work easier.

future performance/refactor improvements

• We can generate an artifact during the Dockerfile build that is then ingested quickly during production boot.
• We could process files in parallel, which saves a couple of seconds off of whole codebase bin/codeownership validate
• We could separate the dev and prod portions of the gem, making changes less risky and opening up options for the (currently internal only) rust-based alternative (https://github.com/rubyatscale/codeowners-rs)
• Could potentially separate things out into two types of ownership: file and folder ownership. For example file annotations are file based, and everything else is a derivative of folder based. This could allow for simpler UX (e.g. owned_folders instead of the cryptic owned_globs). It'd also allow us to optimize how different types of mappers worked. In this world, the cache would become a file map and folder map cache
• We could potentially use git ls-files, which is faster than glowing, although we would still need to glob "excluded" files (e.g. in ZP https://github.com/Gusto/zenpayroll/pull/180263)

benchmarking script

I use this script to compare how a handful of operation types behave

# typed: true

require 'sorbet-runtime'
require 'pry'

class InlineGemfile
  extend T::Sig
  require 'bundler/inline'

  sig { void }
  def self.bundle
    send(:gemfile) do # send is for sorbet to not complain
      gem 'gusto_teams', path: 'components/gusto_teams'
      gem 'code_ownership', path: '../code_ownership'
    end
  end
end

class Benchmarker
  require 'benchmark'
  extend T::Sig

  sig { params(benchmarks: T::Array[Numeric]).returns(Numeric) }
  def self.average(benchmarks)
    (benchmarks.sum / benchmarks.count.to_f).round(2)
  end

  sig { params(message: String, block: T.untyped).returns(T::Array[Numeric]) }
  def self.with_benchmarks(message:, &block)
    puts "Now benchmarking: #{message}"
    benchmarks = []
    6.times do
      benchmarks << Benchmark.measure do
        block.call
      end
    end

    ret = benchmarks.last(5).map{|bm| bm.real.round(2) }
    puts "Results: #{ret.inspect}, average #{average(ret)}\n\n"
    ret
  end
end

class AccuracyVerifier
  extend T::Sig

  VERSION_CURRENT = 'tmp/old_code_ownership'
  VERSION_NEW = 'tmp/new_code_ownership'

  OUTPUT_FILE = VERSION_NEW

  def self.write_file_map(file_map)
    puts "Writing to #{OUTPUT_FILE}) for accuracy testing"
    Pathname.new(OUTPUT_FILE).write(YAML.dump(file_map))
  end
end

InlineGemfile.bundle

row = []
file_map = {}
benchmarks = Benchmarker.with_benchmarks(message: "CodeOwnership.for_file(...) for all files") do
  CodeOwnership.bust_caches!
  CodeOwnership.const_get(:Private).tracked_files.each do |file|
    file_map[file] = CodeOwnership.for_file(file)&.name 
  end
end

AccuracyVerifier.write_file_map(file_map)
row << Benchmarker.average(benchmarks)

benchmarks = Benchmarker.with_benchmarks(message: "bin/codeownership validate") do
  `bin/codeownership validate`
end

row << Benchmarker.average(benchmarks)

benchmarks = Benchmarker.with_benchmarks(message: "bin/codeownership validate --diff") do
  `bin/codeownership validate --diff`
end

row << Benchmarker.average(benchmarks)

files = [
  # File annotation ownership
  'packs/some_pack/path/to/file',
  # Pack-based ownership
  'packs/some_pack/path/to/file',
  # owned_globs ownership
  'packs/some_pack/path/to/file'
]
CodeOwnership.for_file('')
benchmarks = Benchmarker.with_benchmarks(message: "First-execution of CodeOwnership.for_file(...) for 3 files") do
  files.each do |file|
    CodeOwnership.bust_caches!
    file_map[file] = CodeOwnership.for_file(file)&.name 
  end
end

row << Benchmarker.average(benchmarks)

benchmarks = Benchmarker.with_benchmarks(message: "bin/codeownership for_team \"Product Infrastructure Backend\"") do
  `bin/codeownership for_team "Product Infrastructure Backend"`
end

row << Benchmarker.average(benchmarks)

puts "Copy and paste this into https://docs.google.com/spreadsheets/d/1srEan_sWOaqLCuRTBRHDgXpmUWey61O-rDmOXPixq8c/edit#gid=0"
puts row.join("\t")