Skip to content

Add benchmarks #491

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 24 commits into from
May 21, 2025
Merged

Add benchmarks #491

merged 24 commits into from
May 21, 2025

Conversation

Hrovatin
Copy link
Collaborator

@Hrovatin Hrovatin commented Feb 19, 2025

EDIT:

  • Added 2 chemical data TL benchmarks and 3 synthetic ones.
  • Lines:
    • For chemical ones and Hartmann: TL model with 0-10% of data (e.g. 0) and one with non-TL model using only task data (nonTL).
    • For Michaelewicz and Easom: TL model with single source data size(TL), TL with no source data (TL-noSource), and non-TL model with only target data (non-TL).

OLD:
Started by adding direct arylation TL campaign with Temp as task - adapted from the paper.

Will continue adding more. If someone could already check if this goes in the right direction that would be great so that I do not repeat the mistakes for the others.

Things to discuss:

  • Where to read data from?
  • How should TL benchmarks be set up? E.g. should we always include comparison to no TL and different source data proportions? - We didnt do this before on some datasets as we mainly focused on comparing different TL methods.

@Hrovatin Hrovatin changed the title Add direct arylation benchmark for TL with temperature as a task Add benchmarks Feb 19, 2025
@Scienfitz
Copy link
Collaborator

@Hrovatin
if there are discussion items before we start the review, can you open one thread per item so we can collect thoughts? if needed we can also have meetings

@Hrovatin Hrovatin force-pushed the feature/benchmark_cases branch from 0ea9be1 to 87d9deb Compare March 3, 2025 12:42
Copy link
Collaborator

@AdrianSosic AdrianSosic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First batch of comments, only regarding one benchmark file, but probably apply to the others as well

@AVHopp AVHopp force-pushed the feature/benchmark_cases branch 4 times, most recently from e29bc15 to 31fa3c0 Compare March 12, 2025 16:15
@AVHopp AVHopp force-pushed the feature/benchmark_cases branch 5 times, most recently from 0c76238 to a5c47bc Compare March 18, 2025 11:16
Copy link
Collaborator

@AdrianSosic AdrianSosic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @AVHopp: went throught the aryl halides folder. Here some comment first, before I go trhough the other folders

@AVHopp AVHopp force-pushed the feature/benchmark_cases branch from c99d45f to 482114c Compare March 19, 2025 15:52
@AVHopp AVHopp force-pushed the feature/benchmark_cases branch from 477db15 to 82449ec Compare May 20, 2025 12:48
@AVHopp AVHopp force-pushed the feature/benchmark_cases branch 2 times, most recently from 0279a29 to 1866ce1 Compare May 21, 2025 11:04
@AVHopp AVHopp force-pushed the feature/benchmark_cases branch from 1866ce1 to 2624df2 Compare May 21, 2025 11:04
@AVHopp AVHopp merged commit c91f42a into main May 21, 2025
18 of 26 checks passed
@AVHopp AVHopp deleted the feature/benchmark_cases branch May 21, 2025 12:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants