Improve File Operations, Filtering Logic, and Code Robustness

[TusharNaugain]

Issue Description:
The current implementation of the script has several areas for improvement, including:

File Operations: Manual file operations lack clarity and scalability.
Filtering Logic: The logic is not optimized, leading to potential performance issues when handling large datasets.
Error Handling: Missing data or invalid entries are not handled gracefully, increasing the risk of runtime errors.
Code Readability: The code uses non-descriptive variable names and lacks sufficient comments or documentation.
Scalability: Key operations are not modular, making the script harder to extend or maintain for larger projects.
Steps to Resolve:

Replace manual file operations with pathlib for cleaner and more intuitive path management.
Optimize filtering logic using str.contains() for better performance and runtime simplicity.
Use .dropna() to handle missing or invalid data more elegantly.
Refactor code to include self-explanatory variable names and add comments/docstrings to enhance readability.
Modularize key operations to improve scalability and maintainability.
Impact:
These changes will make the code:

Easier to maintain and extend.
Faster and more efficient for large datasets.
More robust against errors caused by missing or invalid data.
Request for Review:
Please validate the changes made in the linked PR to ensure that all modifications align with the project's requirements and standards. Feedback is highly appreciated to make further improvements if necessary.

[dikshant182004]

i was going through this issue and found that it is completed by u in a pr ,so do i have to modify that or create my own pr

[TusharNaugain]

i was going through this issue and found that it is completed by u in a pr ,so do i have to modify that or create my own p
No, there’s no need to create a new PR as I’ve already resolved this issue in a previous PR. If there’s anything specific that still needs attention, let me know!"

[AJ0070]

Is this issue solved?

[pawanupadhyaay]

Is this issue solved?

yes, they all are resolved!