Add a new delegate to allow API tracing

[daniel-raffler]

This is a preliminary draft for adding API tracing to JavaSMT with the help of a new delegate. The idea is to record all API calls and generate a new Java program from them. By running this program the exact sequence of calls can then be recreated. The main application here is debugging, where the traces allow us to create easy to reproduce examples for solver errors. This is especially useful when the error occurs as part of a larger program where it can be hard to pin down the exact sequence of JavaSMT calls that are needed to trigger the bug.

We use a new delegate to implement this feature. Setting solver.trace to true will enable tracing, and the output will be stored in a file called trace*.java

TODO

• Finish the implementation. Currently we only have (parts of) the ArrayFormulaManager, IntegerFormulaManager, BooleanFormulaManager, UFManager and ProverEnvironment
• Write the trace to a file while it's being created. We'll need this to debug segfaults as the trace is otherwise lost done
• Consider adding an option to skip duplicate calls. (The trace is currently way too long)
• Write a simple delta-debugger to shrink the trace down even further

[daniel-raffler]

Write a simple delta-debugger to shrink the trace down even further

I started working on the delta-debugger today and wrote a python script to reduce the size of the traces. So far it does little more that some dead-code elimination, but that's already enough to bring down the size of the trace by a factor of ten. I believe that another factor of two should be possible with some aggressive optimization.

The issue now is that I don't quite know where to put such a script in JavaSMT. We could handle this as a separate project, or maybe include it in the JavaSMT source tree, similar to the Example projects. However, neither really seems quite ideal.

@baierd, @kfriedberger: What is your opinion?

Here is the file in question:

#!/usr/bin/env python3
import re
import sys
from collections import defaultdict
from pathlib import Path


# Read a trace file
def readTrace(path):
    with open(path) as file:
        return [line.rstrip() for line in file]


# Build a map with line numbers for all variable definitions
def getLinesForDefinitions(trace):
    lineNumber = 1
    lineDefs = dict()
    for line in trace:
        if line.find('=') >= 0:
            leftSide = line[0:(line.find('=') - 1)]
            name = re.match('var (.*)', leftSide)
            lineDefs[name.group(1)] = lineNumber
        lineNumber = lineNumber + 1
    return lineDefs


# Build a dependency graph for the definitions
# Maps from variables to the places where they are used
def buildDependencies(lineDefs, trace):
    lineNumber = 1
    deps = defaultdict(list)
    for line in trace:
        expr = line[(line.find('=') + 2):] if line.find('=') >= 0 else line
        object = expr[0:expr.find('.')]
        if object[0].islower():
            deps[lineDefs[object]].append(lineNumber)
        # FIXME Parse the expression to get the variables
        for m in re.finditer('(config|logger|notifier|var[0-9]+)', expr):
            deps[lineDefs[m.group()]].append(lineNumber)
        lineNumber += 1
    return deps


# Collect all top-level statements
# Top-level statements are:
#  *.addConstraint(*)
#  *.isUnsat()
#  *.getModel()
#  *.asList()
# FIXME Finish this list
def usedTopLevel(lineDefs, trace):
    tl = set()
    for line in trace:
        m = re.fullmatch(
            'var (var[0-9]+) = (var[0-9]+).(isUnsat\\(\\)|getModel\\(\\)|asList\\(\\)|addConstraint\\((var[0-9]+)\\));',
            line)
        if m != None:
            tl.add(lineDefs[m.group(1)])
    return tl


# Calculate the closure of all used definitions, starting with the top-level statements
def usedClosure(tl, deps):
    cl = set()
    st = set(tl)
    while cl.union(st) != cl:
        cl = cl.union(st)
        st = set()
        for (key, val) in deps.items():
            if set(val).intersection(cl) != set():
                st.add(key)
    return cl


# Keep only statements and definitions that are used
def filterUnused(used, trace):
    lineNumber = 1
    reduced = []
    for line in trace:
        if line.find('=') == -1 or lineNumber in used:
            reduced.append(line)
        lineNumber += 1
    return reduced


# Remove all definitions that are not used (recursively)
def removeDeadCode(trace):
    lineDefs = getLinesForDefinitions(trace)
    deps = buildDependencies(lineDefs, trace)
    tl = usedTopLevel(lineDefs, trace)
    cl = usedClosure(tl, deps)
    return filterUnused(cl, trace)


# We'll use multiple passes to reduce the size of the trace:
# 1. Read the trace
# 2. Remove unused code
# 3. Remove unnecessary toplevel commands
# 4. Loop: Remove aliasing (by duplicating the definitions)
# 5.    Loop: Reduce terms
# 6. Remove unused prover environments
if __name__ == '__main__':
    arg = sys.argv
    if not len(sys.argv) == 2:
        print('Expecting a path to a trace file as argument')
        exit(-1)

    path = Path(sys.argv[1])
    if not (path.is_file()):
        print(f'Could not find file "{path}"')
        exit(-1)

    # TODO Implement steps 3-6
    # TODO Check that the reduced trace still crashes

    trace = readTrace(path)
    for line in removeDeadCode(trace):
        print(line)

The idea is to run JavaSMT with solver.trace=true to collect a trace of the crash, and then use the script to reduce the trace. Since we're "crashing" (posssibly with a segfault) there doesn't seem to be a good way to do this in one go