Support intermediate insights and controls for synthesizer training

[srinify]

Problem Description

SDV users want to be able to interact more with the synthesizer training process. Additional controls include being able to:

• track and analyze model metrics they care about
• export intermediate model state at different intervals during training
• stop model training early if desirable metrics are achieved

These can help users compare and identify the best models and reduce model training time.

Additional context

Related issues have been combined into this one:

• ModelCheckpoint Callback for Best Model Preservation #2397
• EarlyStoppingCallback #2396
• Tracking metrics while training #460