Skip to content

Document the difference between tracing time and execution time #8858

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
tengyifei opened this issue Mar 19, 2025 · 4 comments · May be fixed by #9133
Open

Document the difference between tracing time and execution time #8858

tengyifei opened this issue Mar 19, 2025 · 4 comments · May be fixed by #9133
Assignees
@sdasgup3
Labels
documentation enhancement New feature or request good first issue Good for newcomers

Comments

@tengyifei
Copy link
Collaborator

📚 Documentation

If we write a loop like

start = time.time()
for step in range(num_steps):
  run_model()
  xm.mark_step()
end = time.time()

Then end - start will only measure the tracing time. We'll need to do torch_xla.sync(wait=True) to block on device execution to measure the execution time.

We should document this in some "common FAQs/sharp edges" maybe
@ysiraichi ysiraichi added the enhancement New feature or request label Mar 20, 2025
@tengyifei tengyifei added the good first issue Good for newcomers label Apr 22, 2025
@sdasgup3 sdasgup3 linked a pull request May 9, 2025 that will close this issue
Open
@sdasgup3
Copy link
Collaborator

sdasgup3 commented May 19, 2025 •
edited
Loading

@tengyifei As mark_step is synonymous with torxh_xla.sync (

xla/torch_xla/core/xla_model.py

Line 1056 in 01b5408

def mark_step(wait: bool = False, reset_scope: bool = True):
), should this issue be deprecated?

@tengyifei
Copy link
Collaborator Author

@sdasgup3 no. we can replace mark_step with sync and the same ideas still persist:

start = time.time()
for step in range(num_steps):
  run_model()
  torch_xla.sync()
end = time.time()

@sdasgup3
Copy link
Collaborator

@tengyifei I have added a section ## Case Study: Correctly Profiling Loops with torch_xla.sync in #9133 for the same. Please take a look.

@tengyifei
Copy link
Collaborator Author

That looks great!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sdasgup3 sdasgup3

Labels
documentation enhancement New feature or request good first issue Good for newcomers
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Document the difference between tracing time and execution time sdasgup3/xla
3 participants
@tengyifei @ysiraichi @sdasgup3