Async file close and S3 upload (Good First Issue)

[github-actions]

construct full file path

---

olake/writers/parquet/parquet.go

Lines 233 to 238 in 1ed362e

	// TODO: Async file close and S3 upload (Good First Issue)
	// construct full file path
	filePath := filepath.Join(p.config.Path, basePath, fileMetadata.fileName)
	// Remove empty files
	if fileMetadata.recordCount == 0 {

---

This issue was generated by todo-issue based on a TODO comment in 1ed362e. It's been assigned to @hash-data because they committed the code.

[mrmagicpotato007]

@hash-data i can take this if its still unassigned

[hash-data]

@mrmagicpotato007 sure you can take it.

[mrmagicpotato007]

@hash-data my thought process is to use errgroup on array of parquetFiles. Am I on the right track?

func processPartitionedFiles(p *Processor) error {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()

g, ctx := errgroup.WithContext(ctx)

for basePath, parquetFiles := range p.partitionedFiles {
	for _, fileMetadata := range parquetFiles {
		basePath, fileMetadata := basePath, fileMetadata // Capture loop variables

		g.Go(func() error {
			return processFile(ctx, basePath, fileMetadata, p)
		})
	}
}

return g.Wait() // Wait for all tasks to complete, returns first error if any

}

[mrmagicpotato007]

@hash-data pls let me know your thoughts on this.

[hash-data]

@mrmagicpotato007 we have concurrent function in utils that can be utilized here.

[mrmagicpotato007]

yes will use Concurrent func from utils .

[mrmagicpotato007]

pr : #101

@hash-data can you review?

[hash-data]

will review it @mrmagicpotato007