Skip to content

Commit bc6df8b

Browse files
guyernestclaude
andcommitted
fix(batch-processor): Fix result aggregation and add integration docs
Fixes critical data structure handling bugs in result_aggregator.py that prevented proper CSV generation from batch processing results. ## Bug Fixes ### 1. Incorrect Data Structure Assumption **Problem**: Code expected `{row: {...}, structured_result: {...}}` structure, but actual Map state output from Lambda has: `{Payload: {...}, ExecutedVersion: "$LATEST", ...}` **Fix**: Updated `process_result_item()` to correctly extract from `item['Payload']` **Location**: lambda/tools/batch_processor/result_aggregator.py:149-177 ### 2. Wrong Status Checking **Problem**: Status was checked on raw result item instead of extracted payload, causing incorrect success/failure counts **Fix**: Process all items first, then count based on extracted results - Lines 48-61: Count after extraction, not during - Check `_status` on processed results, not raw items ### 3. Overly Complex Column Ordering **Problem**: Multiple redundant passes through results with complex logic **Fix**: Simplified to single pass with clearer separation - Lines 64-79: Regular columns + sorted metadata columns - Removed unnecessary `seen_columns` tracking in multiple passes ## Improvements ### Code Simplification - Removed nested structure handling that wasn't needed - Cleaner error handling with descriptive messages - More maintainable column ordering logic ### Better Logging - Added warnings for unexpected formats - Included truncated raw data in error cases for debugging ## Integration Documentation **New File**: docs/BATCH_PROCESSOR_BROADBAND_INTEGRATION.md Comprehensive guide covering: - ✅ Compatibility verification with broadband agent - ✅ Complete processing flow diagrams - ✅ Configuration examples - ✅ Output format specifications - ✅ Error handling details - ✅ Performance characteristics - ✅ Step-by-step testing guide - ✅ Troubleshooting section ## Expected Output With the fixes, CSV output now correctly shows: ```csv address,postcode,availability,cabinet,downstream_mbps,exchange,screenshot_url,service_type,success,upstream_mbps,_status,_timestamp 1 Church view London,DN12 1RH,False,22,80,CONISBROUGH,s3://...,VDSL,True,68.4,SUCCESS,2025-10-18T18:06:23.491908 40 Withers Place Fordham,CB7 5LG,False,Cabinet 4,80,FORDHAM CAMBS,s3://...,VDSL,True,20,SUCCESS,2025-10-18T18:09:42.157336 ``` ## Testing Verified with: - Sample broadband availability data - Multiple row processing - Success and failure cases - Column ordering preservation ## Breaking Changes None - maintains backward compatibility with existing batch configs 🤖 Generated with Claude Code https://claude.com/claude-code Co-Authored-By: Claude <[email protected]>
1 parent ff0a3af commit bc6df8b

File tree

2 files changed

+641
-77
lines changed

2 files changed

+641
-77
lines changed

0 commit comments

Comments
 (0)