Added dataset download files to write the dataset from the HF api to … #37
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…disk. Also added bash script to create tar files from the IR files on disk.
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
llvm_ir_dataset_utils/compile_time_analysis_tools/datatset_download/write_data_files.py
Outdated
Show resolved
Hide resolved
llvm_ir_dataset_utils/compile_time_analysis_tools/datatset_download/write_data_files.py
Outdated
Show resolved
Hide resolved
llvm_ir_dataset_utils/compile_time_analysis_tools/datatset_download/write_data_files.py
Outdated
Show resolved
Hide resolved
llvm_ir_dataset_utils/compile_time_analysis_tools/datatset_download/write_data_files.py
Outdated
Show resolved
Hide resolved
llvm_ir_dataset_utils/compile_time_analysis_tools/datatset_download/write_data_files.py
Outdated
Show resolved
Hide resolved
llvm_ir_dataset_utils/compile_time_analysis_tools/datatset_download/create_tar.sh
Outdated
Show resolved
Hide resolved
llvm_ir_dataset_utils/compile_time_analysis_tools/datatset_download/write_data_files.py
Outdated
Show resolved
Hide resolved
llvm_ir_dataset_utils/compile_time_analysis_tools/datatset_download/write_data_files.py
Outdated
Show resolved
Hide resolved
…de argparse functionality, elimination of global variables, and script execution layout.
llvm_ir_dataset_utils/compile_time_analysis_tools/datatset_download/write_data_files.py
Outdated
Show resolved
Hide resolved
llvm_ir_dataset_utils/compile_time_analysis_tools/datatset_download/write_data_files.py
Outdated
Show resolved
Hide resolved
llvm_ir_dataset_utils/compile_time_analysis_tools/datatset_download/write_data_files.py
Outdated
Show resolved
Hide resolved
llvm_ir_dataset_utils/compile_time_analysis_tools/dataset_download/write_data_files.py
Outdated
Show resolved
Hide resolved
llvm_ir_dataset_utils/compile_time_analysis_tools/dataset_download/write_data_files.py
Outdated
Show resolved
Hide resolved
llvm_ir_dataset_utils/compile_time_analysis_tools/dataset_download/write_data_files.py
Outdated
Show resolved
Hide resolved
| def get_args(): | ||
| """Function to return the provided storage argument for the script. | ||
|
|
||
| Returns: argparse.Namespace |
There was a problem hiding this comment.
Nit: Can we have a type annotation for this?
| tarinfo = tarfile.TarInfo(name=f'bc_files/file{x[0]+1+start_index}.bc') | ||
| file_obj = BytesIO(x[1]) | ||
| tarinfo.size = file_obj.getbuffer().nbytes | ||
| tarinfo.mtime = time() |
There was a problem hiding this comment.
Why do we need to set the time here?
There was a problem hiding this comment.
The tar file will otherwise be created with an irregular modification time (e.g. the year 1969) and the tar will complain that the modification time is irregular.
| tarinfo.mtime = time() | ||
| tar.addfile(tarinfo, fileobj=file_obj) | ||
|
|
||
| with parallel.parallel_backend('spark'): |
There was a problem hiding this comment.
Can you add a comment on the performance benefits of using the parallel backend?
| end_index = file_indices[i]["end_index"] | ||
| dir_name = f'{storage}/{file_indices[i]["language"]}' | ||
| makedirs(dir_name, exist_ok=True) | ||
| thread = threading.Thread( |
There was a problem hiding this comment.
It would probably be more natural to use a ThreadPoolExecutor here, submit jobs, and get back futures?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added dataset download files to write the dataset from the HF api to disk. Also added bash script to create tar files from the IR files on disk.