-
Notifications
You must be signed in to change notification settings - Fork 8
Commit bc6df8b
fix(batch-processor): Fix result aggregation and add integration docs
Fixes critical data structure handling bugs in result_aggregator.py that
prevented proper CSV generation from batch processing results.
## Bug Fixes
### 1. Incorrect Data Structure Assumption
**Problem**: Code expected `{row: {...}, structured_result: {...}}` structure,
but actual Map state output from Lambda has: `{Payload: {...}, ExecutedVersion: "$LATEST", ...}`
**Fix**: Updated `process_result_item()` to correctly extract from `item['Payload']`
**Location**: lambda/tools/batch_processor/result_aggregator.py:149-177
### 2. Wrong Status Checking
**Problem**: Status was checked on raw result item instead of extracted payload,
causing incorrect success/failure counts
**Fix**: Process all items first, then count based on extracted results
- Lines 48-61: Count after extraction, not during
- Check `_status` on processed results, not raw items
### 3. Overly Complex Column Ordering
**Problem**: Multiple redundant passes through results with complex logic
**Fix**: Simplified to single pass with clearer separation
- Lines 64-79: Regular columns + sorted metadata columns
- Removed unnecessary `seen_columns` tracking in multiple passes
## Improvements
### Code Simplification
- Removed nested structure handling that wasn't needed
- Cleaner error handling with descriptive messages
- More maintainable column ordering logic
### Better Logging
- Added warnings for unexpected formats
- Included truncated raw data in error cases for debugging
## Integration Documentation
**New File**: docs/BATCH_PROCESSOR_BROADBAND_INTEGRATION.md
Comprehensive guide covering:
- ✅ Compatibility verification with broadband agent
- ✅ Complete processing flow diagrams
- ✅ Configuration examples
- ✅ Output format specifications
- ✅ Error handling details
- ✅ Performance characteristics
- ✅ Step-by-step testing guide
- ✅ Troubleshooting section
## Expected Output
With the fixes, CSV output now correctly shows:
```csv
address,postcode,availability,cabinet,downstream_mbps,exchange,screenshot_url,service_type,success,upstream_mbps,_status,_timestamp
1 Church view London,DN12 1RH,False,22,80,CONISBROUGH,s3://...,VDSL,True,68.4,SUCCESS,2025-10-18T18:06:23.491908
40 Withers Place Fordham,CB7 5LG,False,Cabinet 4,80,FORDHAM CAMBS,s3://...,VDSL,True,20,SUCCESS,2025-10-18T18:09:42.157336
```
## Testing
Verified with:
- Sample broadband availability data
- Multiple row processing
- Success and failure cases
- Column ordering preservation
## Breaking Changes
None - maintains backward compatibility with existing batch configs
🤖 Generated with Claude Code
https://claude.com/claude-code
Co-Authored-By: Claude <[email protected]>1 parent ff0a3af commit bc6df8bCopy full SHA for bc6df8b
File tree
Expand file treeCollapse file tree
2 files changed
+641
-77
lines changedOpen diff view settings
Filter options
- docs
- lambda/tools/batch_processor
Expand file treeCollapse file tree
2 files changed
+641
-77
lines changedOpen diff view settings
0 commit comments