Add basic Git options for data interpreter and target file name extraction utilities

[stellaHSR]

Features

1. Add git reset and checkout options, which are helpful in checking files related to specific issues.

2. Add git clone options to make it easier to clone repositories.

3. Use EnvManager for local repository management.

4. Add a basic parser to extract files from code text based on the input from swe-bench.

Feature Docs

Influence

Result
Here is the runtime result (batch) for the default SCIKIT_LEARN_IDS.

Other

[shenchucheng]

Does this function only aim to support parsing Python code? Because of the regex with a '.py' suffix.

[stellaHSR]

Yes, currently, it is only available for Python scripts.

[shenchucheng]

LGTM

[geekan]

Provide a cp method instead of clone

[stellaHSR]

add

[seehi]

/review

[better629]

/review

[qodo-merge-pro]

PR-Agent was enabled for this repository. To continue using it, please link your git user with your CodiumAI identity here.

PR Review 🔍

⏱️ Estimated effort to review [1-5]	4, due to the complexity and breadth of the changes involving multiple files and functionalities such as Git operations, environment management, and text parsing. The PR also integrates these functionalities, which requires careful consideration of integration points and potential side effects.
🧪 Relevant tests	No
⚡ Possible issues	Possible Bug: The reset_task_env method in repo_utils.py uses a broad exception handling strategy which might suppress important errors that should be addressed or logged differently.
	Performance Concern: The method clone_repo in repo_utils.py could be inefficient if cloning large repositories frequently without checking if the repository already exists at the specified path.
🔒 Security concerns	No

Code feedback:

---

relevant file	data/inference/make_datasets/repo_utils.py
suggestion	Consider checking if the repository already exists before cloning in clone_repo. This can prevent unnecessary network usage and speed up the process if the repository is already present. [important]
relevant line	git.Repo.clone_from(repo_url, path)

---

relevant file	data/inference/make_datasets/repo_utils.py
suggestion	Replace the broad exception handling in reset_task_env with more specific exceptions. This will help in identifying and resolving potential issues more effectively. [important]
relevant line	@handle_exception(exception_type=Exception, default_return=False)

---

relevant file	data/inference/make_datasets/parse_utils.py
suggestion	Optimize the regex pattern in extract_scripts_from_codetext by compiling it once and reusing it, which can improve performance, especially if this function is called multiple times. [medium]
relevant line	matches = re.findall(r"\[start of ([^\]]+\.py)\]", codetext)

---

relevant file	data/inference/make_datasets/repo_utils.py
suggestion	In clone_repo, validate the token more robustly to handle cases where it might be an empty string, which could lead to authentication issues. [important]
relevant line	if not token:

---

[qodo-merge-pro]

PR Review 🔍

⏱️ Estimated effort to review [1-5]	4, because the PR introduces multiple new features and changes across several files, involving complex operations such as Git commands and file parsing. The complexity of the changes and the potential for bugs in critical operations like repository cloning, environment resetting, and file extraction from text make this a more challenging PR to review thoroughly.
🧪 Relevant tests	No
⚡ Possible issues	Possible Bug: The reset_task_env method in repo_utils.py attempts to handle ignored files using a Git command that is commented as needing platform-specific adjustments. This could lead to issues on different operating systems if not properly handled.
	Error Handling: The clone_repo method in repo_utils.py raises a generic ValueError if the GitHub token is not found, which might not provide enough information for debugging in cases where the token is actually provided but incorrect.
🔒 Security concerns	No

Code feedback:

---

relevant file	data/inference/make_datasets/repo_utils.py
suggestion	Consider using a more specific exception for token errors in the clone_repo method. This can help in debugging and handling specific token-related issues more effectively. [important]
relevant line	raise ValueError("GitHub token is required for cloning repositories.")

---

relevant file	data/inference/make_datasets/repo_utils.py
suggestion	Implement platform-specific handling for the reset_task_env method where ignored files are processed. This could involve checking the operating system and adjusting the command accordingly. [important]
relevant line	# fixme: need detect platform and change this cmd

---

relevant file	data/inference/make_datasets/parse_utils.py
suggestion	Optimize the regex pattern in extract_scripts_from_codetext to ensure it handles cases where script names might include unusual characters or patterns that could break the current simple regex. [medium]
relevant line	matches = re.findall(r"\[start of ([^\]]+\.py)\]", codetext)

---

relevant file	data/inference/run_api.py
suggestion	Refactor the openai_inference function to handle repository existence checks before changing the working directory, which can prevent errors if the directory does not exist. [important]
relevant line	os.chdir(repo_path)

---