Skip to content

Commit b89aa89

Browse files
sriram-mvsriram-mv
committed
feat: Python UV workflow
Core Features: * Intelligent handling of pyproject.toml with automatic uv.lock detection * Multi-format support: pyproject.toml, requirements.txt, and requirements-*.txt variants * Lock file optimization: Automatic uv.lock usage for reproducible builds when available * Platform targeting: Lambda-compatible builds for x86_64 and ARM64 architectures * Python version control: Precise Python version targeting via UV's --python flags UV Functionality: * `uv sync --python X.Y` for lock-based builds (pyproject.toml + uv.lock) * `uv lock && uv export` workflow for pyproject.toml without lock files * `uv pip install --python-version X.Y --python-platform` for requirements.txt * Proper virtual environment handling and site-packages extraction Documentation: * Complete DESIGN.md with architecture overview and usage patterns * Accurate API documentation reflecting constructor + method call pattern * Smart dispatch logic clearly explained with examples * Updated to match actual implementation (no outdated references) Compatibility: * Compatible with existing Lambda Builders interface * Supports standard build parameters (source_dir, artifacts_dir, scratch_dir) * Runtime parameter properly extracted and used for Python version targeting * Architecture parameter mapped to UV platform specifications
1 parent 024f96d commit b89aa89

File tree

21 files changed

+2141
-2
lines changed

21 files changed

+2141
-2
lines changed

‎.github/workflows/build.yml

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -254,7 +254,7 @@ jobs:
254254
- run: pytest -vv tests/integration/workflows/custom_make
255255

256256
python-integration:
257-
name: ${{ matrix.os }} / ${{ matrix.python }} / python
257+
name: ${{ matrix.os }} / ${{ matrix.python }} / python (pip + uv)
258258
if: github.repository_owner == 'aws'
259259
runs-on: ${{ matrix.os }}
260260
strategy:
@@ -279,8 +279,15 @@ jobs:
279279
python -m pip install --upgrade pip
280280
pip install --upgrade setuptools
281281
if: ${{ matrix.os }} == 'ubuntu-latest' && ${{ matrix.python }} == '3.12'
282+
# Install UV for python_uv workflow tests
283+
- name: Install UV
284+
uses: astral-sh/setup-uv@v4
285+
with:
286+
enable-cache: true
282287
- run: make init
288+
# Test both python_pip and python_uv workflows
283289
- run: pytest -vv tests/integration/workflows/python_pip
290+
- run: pytest -vv tests/integration/workflows/python_uv
284291

285292
ruby-integration:
286293
name: ${{ matrix.os }} / ${{ matrix.python }} / ruby

‎Makefile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ lint:
2121
ruff check aws_lambda_builders
2222

2323
lint-fix:
24-
ruff aws_lambda_builders --fix
24+
ruff check aws_lambda_builders --fix
2525

2626
# Command to run everytime you make changes to verify everything works
2727
dev: lint test

‎aws_lambda_builders/workflows/__init__.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,5 +10,6 @@
1010
import aws_lambda_builders.workflows.nodejs_npm
1111
import aws_lambda_builders.workflows.nodejs_npm_esbuild
1212
import aws_lambda_builders.workflows.python_pip
13+
import aws_lambda_builders.workflows.python_uv
1314
import aws_lambda_builders.workflows.ruby_bundler
1415
import aws_lambda_builders.workflows.rust_cargo

‎aws_lambda_builders/workflows/python_uv/DESIGN.md

Lines changed: 355 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,355 @@
1+
## Python - UV Lambda Builder
2+
3+
### Scope
4+
5+
This package provides a Python dependency builder that uses UV (An extremely fast Python package installer and resolver, written in Rust) as an alternative to the traditional pip-based workflow. The scope for this builder is to take an existing directory containing customer code, and dependency specification files (such as `pyproject.toml` or `requirements.txt`) and use UV to build and include the dependencies in the customer code bundle in a way that makes them importable in AWS Lambda.
6+
7+
UV offers several advantages over pip:
8+
- **Performance**: UV is significantly faster than pip for dependency resolution and installation
9+
- **Better dependency resolution**: More reliable and consistent dependency resolution
10+
- **Lock file support**: Native support for lock files for reproducible builds
11+
- **Modern Python packaging**: Built-in support for modern Python packaging standards (PEP 517/518)
12+
- **Virtual environment management**: Integrated virtual environment handling
13+
14+
### Challenges
15+
16+
Similar to the Python PIP workflow, Python packaging for AWS Lambda presents unique challenges:
17+
18+
1. **Platform compatibility**: Python packages often contain platform-specific code or compiled extensions that must be compatible with the AWS Lambda runtime environment (Amazon Linux 2)
19+
20+
2. **Architecture compatibility**: Packages must be compatible with the target Lambda architecture (x86_64 or arm64)
21+
22+
3. **Dependency resolution complexity**: Complex dependency trees with potential conflicts need to be resolved consistently
23+
24+
4. **Binary dependencies**: Some packages require compilation of C extensions or have binary dependencies that must be built for the target platform
25+
26+
5. **Package size optimization**: Lambda has deployment package size limits, requiring efficient dependency packaging
27+
28+
UV addresses many of these challenges through:
29+
- Better dependency resolution algorithms
30+
- Improved handling of platform-specific wheels
31+
- More efficient caching mechanisms
32+
- Better support for lock files ensuring reproducible builds
33+
34+
### Interface
35+
36+
The top level interface is presented by the `PythonUvDependencyBuilder` class. There will be one public method `build_dependencies`, which takes the provided arguments and builds python dependencies using UV under the hood.
37+
38+
```python
39+
def build_dependencies(artifacts_dir_path,
40+
scratch_dir_path,
41+
manifest_path,
42+
architecture=None,
43+
config=None,
44+
):
45+
"""Builds a python project's dependencies into an artifact directory using UV.
46+
47+
Note: The runtime parameter is passed to the PythonUvDependencyBuilder constructor,
48+
not to this method.
49+
50+
:type artifacts_dir_path: str
51+
:param artifacts_dir_path: Directory to write dependencies into.
52+
53+
:type scratch_dir_path: str
54+
:param scratch_dir_path: Temporary directory for build operations and intermediate files.
55+
56+
:type manifest_path: str
57+
:param manifest_path: Path to a dependency manifest file. Supported manifests:
58+
- pyproject.toml (preferred for modern Python projects)
59+
- requirements.txt (traditional pip format)
60+
- requirements-*.txt (environment-specific: dev, test, prod, etc.)
61+
62+
Note: uv.lock is NOT a valid manifest - it's a lock file that automatically
63+
enhances pyproject.toml builds when present in the same directory.
64+
65+
:type runtime: str
66+
:param runtime: Python version to build dependencies for. This can
67+
be python3.8, python3.9, python3.10, python3.11, python3.12, or python3.13.
68+
These are currently the only supported values.
69+
Note: This parameter is passed to the PythonUvDependencyBuilder constructor.
70+
71+
:type config: :class:`lambda_builders.actions.python_uv.utils.UvConfig`
72+
:param config: Optional config object for customizing UV behavior,
73+
including cache settings, index URLs, and build options.
74+
75+
:type architecture: str
76+
:param architecture: Target architecture for Lambda compatibility (x86_64 or arm64).
77+
Defaults to x86_64 if not specified.
78+
"""
79+
```
80+
81+
### Usage Pattern
82+
83+
The `PythonUvDependencyBuilder` follows a constructor + method call pattern:
84+
85+
```python
86+
# 1. Create builder with runtime
87+
builder = PythonUvDependencyBuilder(
88+
osutils=osutils,
89+
runtime="python3.9", # Runtime specified here
90+
uv_runner=uv_runner
91+
)
92+
93+
# 2. Call build_dependencies method
94+
builder.build_dependencies(
95+
artifacts_dir_path="/path/to/artifacts",
96+
scratch_dir_path="/path/to/scratch",
97+
manifest_path="/path/to/pyproject.toml",
98+
architecture="x86_64",
99+
config=uv_config
100+
)
101+
```
102+
103+
### Implementation
104+
105+
The general algorithm for preparing a python package using UV for use on AWS Lambda follows a streamlined approach that leverages UV's advanced capabilities:
106+
107+
#### Step 1: Smart manifest detection and dispatch
108+
109+
The workflow uses a smart dispatch system that recognizes actual manifest files:
110+
111+
**Supported Manifests:**
112+
- `pyproject.toml` - Modern Python project manifest (preferred)
113+
- `requirements.txt` - Traditional pip requirements file
114+
- `requirements-*.txt` - Environment-specific variants (dev, prod, test, etc.)
115+
116+
**Smart Lock File Detection:**
117+
- When `pyproject.toml` is the manifest, automatically checks for `uv.lock` in the same directory
118+
- If `uv.lock` exists alongside `pyproject.toml`, uses lock-based build for precise dependencies
119+
- If no `uv.lock`, uses standard pyproject.toml build with UV's lock and export workflow
120+
121+
**Important:** `uv.lock` is NOT a standalone manifest - it's a lock file that enhances `pyproject.toml` builds when present.
122+
123+
#### Step 2: Build dependencies based on manifest type
124+
125+
**For pyproject.toml with uv.lock present:**
126+
- Use `uv sync` to install exact dependencies from lock file
127+
- Provides reproducible builds with locked dependency versions
128+
129+
**For pyproject.toml without uv.lock:**
130+
- Use `uv lock` to create temporary lock file with resolved dependencies
131+
- Use `uv export` to convert lock file to requirements.txt format
132+
- Install dependencies using the exported requirements
133+
134+
**For requirements.txt files:**
135+
- Use `uv pip install` directly with Lambda-compatible settings
136+
137+
#### Step 3: Configure Lambda-compatible installation
138+
139+
UV is configured with Lambda-specific settings:
140+
- Target platform: `linux` (Amazon Linux 2)
141+
- Target architecture: `x86_64` or `aarch64`
142+
- Python version matching Lambda runtime
143+
- Prefer wheels over source distributions for faster builds
144+
145+
#### Step 4: Install to target directory
146+
147+
Install resolved dependencies to the Lambda deployment package:
148+
- Extract packages to artifacts directory
149+
- Maintain proper Python package structure
150+
- Ensure all packages are importable from Lambda function
151+
152+
This streamlined approach leverages UV's built-in capabilities rather than manually implementing dependency resolution, compilation handling, and optimization steps that UV already performs efficiently.
153+
154+
### UV-Specific Features
155+
156+
This workflow leverages several UV-specific features that provide advantages over the traditional pip workflow:
157+
158+
#### Lock File Support
159+
- **Reproducible builds**: `uv.lock` files ensure identical dependency versions across builds
160+
- **Faster subsequent builds**: Lock files eliminate dependency resolution time
161+
- **Conflict detection**: Early detection of dependency conflicts during resolution
162+
163+
#### Advanced Dependency Resolution
164+
- **Better conflict resolution**: UV's resolver handles complex dependency graphs more reliably
165+
- **Version range optimization**: More intelligent selection of compatible versions
166+
- **Platform-aware resolution**: Better handling of platform-specific dependencies
167+
168+
#### Performance Optimizations
169+
- **Parallel downloads**: Multiple packages downloaded simultaneously
170+
- **Efficient caching**: Smart caching reduces redundant downloads and builds
171+
- **Fast installs**: Rust-based implementation provides significant speed improvements
172+
173+
#### Modern Python Standards
174+
- **PEP 517/518 support**: Native support for modern Python packaging standards
175+
- **pyproject.toml first**: Preferred support for modern project configuration
176+
- **Build isolation**: Proper build environment isolation for reliable builds
177+
178+
### Error Handling and Diagnostics
179+
180+
The UV workflow provides enhanced error handling:
181+
182+
1. **Dependency resolution errors**: Clear reporting of version conflicts and resolution failures
183+
2. **Platform compatibility warnings**: Explicit warnings about potential platform issues
184+
3. **Build failures**: Detailed error messages for compilation and build failures
185+
4. **Lock file conflicts**: Detection and reporting of lock file inconsistencies
186+
5. **Performance metrics**: Optional reporting of build times and cache efficiency
187+
188+
### Configuration Options
189+
190+
The workflow supports various configuration options through the config parameter:
191+
192+
```python
193+
config = {
194+
"index_url": "https://pypi.org/simple/", # Custom package index
195+
"extra_index_urls": [], # Additional package indexes
196+
"cache_dir": "/tmp/uv-cache", # Custom cache directory
197+
"no_cache": False, # Disable caching
198+
"prerelease": "disallow", # Handle pre-release versions
199+
"resolution": "highest", # Resolution strategy
200+
"compile_bytecode": True, # Compile .pyc files
201+
"exclude_newer": None, # Exclude packages newer than date
202+
"generate_hashes": False, # Generate package hashes
203+
}
204+
```
205+
206+
### Compatibility with Existing Workflows
207+
208+
The UV workflow is designed to be a drop-in replacement for the pip workflow:
209+
- Supports the same manifest formats (requirements.txt, pyproject.toml)
210+
- Uses native UV commands for pyproject.toml (lock/export workflow)
211+
- Maintains the same output structure and package layout
212+
- Compatible with existing Lambda deployment processes
213+
- Provides migration path from pip-based builds
214+
- Follows established requirements file naming conventions
215+
216+
### Architecture Components
217+
218+
The UV workflow consists of several key components that mirror the PIP workflow structure:
219+
220+
#### Core Classes
221+
222+
1. **PythonUvWorkflow**: Main workflow class that orchestrates the build process
223+
2. **PythonUvBuildAction**: Action class that handles dependency resolution
224+
3. **UvRunner**: Wrapper around UV command execution
225+
4. **SubprocessUv**: Low-level UV subprocess interface
226+
6. **PythonUvDependencyBuilder**: High-level dependency builder orchestrator
227+
228+
#### File Structure
229+
```
230+
python_uv/
231+
├── __init__.py
232+
├── DESIGN.md
233+
├── workflow.py # Main workflow implementation
234+
├── actions.py # Build actions
235+
├── packager.py # Core packaging logic
236+
├── utils.py # Utility functions
237+
└── exceptions.py # UV-specific exceptions
238+
```
239+
240+
#### Capability Definition
241+
```python
242+
CAPABILITY = Capability(
243+
language="python",
244+
dependency_manager="uv",
245+
application_framework=None
246+
)
247+
```
248+
249+
#### Smart Manifest Detection and Dispatch
250+
251+
The workflow uses intelligent manifest detection:
252+
253+
**Supported Manifests (in order of preference):**
254+
1. `pyproject.toml` - Modern Python project manifest (preferred)
255+
2. `requirements.txt` - Standard pip format
256+
3. `requirements-*.txt` - Environment-specific variants (dev, test, prod, etc.)
257+
258+
**Smart Lock File Enhancement:**
259+
- When `pyproject.toml` is used, automatically detects `uv.lock` in the same directory
260+
- If `uv.lock` exists, uses lock-based build for reproducible dependencies
261+
- If no `uv.lock`, uses standard pyproject.toml workflow with UV's lock and export
262+
263+
**Important:** `uv.lock` is NOT a standalone manifest - attempting to use it as one will result in an "Unsupported manifest file" error.
264+
265+
#### Requirements File Naming Conventions
266+
The workflow follows Python ecosystem standards for requirements files:
267+
- `requirements.txt` - Standard format (primary)
268+
- `requirements-dev.txt` - Development dependencies
269+
- `requirements-test.txt` - Test dependencies
270+
- `requirements-prod.txt` - Production dependencies
271+
- `requirements-staging.txt` - Staging dependencies
272+
273+
Note: `requirements.in` (pip-tools format) is not supported to keep the implementation simple and focused.
274+
275+
#### UV Binary Requirements
276+
- UV must be available on the system PATH
277+
- Minimum UV version: 0.1.0 (to be determined based on feature requirements)
278+
- Fallback: Attempt to install UV using pip if not found (optional behavior)
279+
280+
#### Error Handling Strategy
281+
- **MissingUvError**: UV binary not found on PATH
282+
- **UvInstallationError**: UV installation/setup failures
283+
- **UvResolutionError**: Dependency resolution failures
284+
- **UvBuildError**: Package build failures
285+
- **LockFileError**: Lock file parsing or validation errors
286+
287+
#### Platform Compatibility Matrix
288+
| Python Version | x86_64 | arm64 | Status |
289+
|---------------|--------|-------|---------|
290+
| python3.8 | ✓ | ✓ | Supported |
291+
| python3.9 | ✓ | ✓ | Supported |
292+
| python3.10 | ✓ | ✓ | Supported |
293+
| python3.11 | ✓ | ✓ | Supported |
294+
| python3.12 | ✓ | ✓ | Supported |
295+
| python3.13 | ✓ | ✓ | Supported |
296+
297+
#### Integration with Lambda Builders
298+
- Registers with the workflow registry automatically
299+
- Follows the same build lifecycle as other workflows
300+
- Compatible with existing SAM CLI integration
301+
- Supports all standard build options (scratch_dir, dependencies_dir, etc.)
302+
303+
### Implementation Phases
304+
305+
#### Phase 1: Core Infrastructure
306+
1. Basic workflow and action classes
307+
2. UV binary detection and validation
308+
3. Simple requirements.txt support
309+
4. Basic error handling
310+
311+
#### Phase 2: Advanced Features
312+
1. pyproject.toml support
313+
2. Lock file handling
314+
3. Advanced configuration options
315+
4. Performance optimizations
316+
317+
#### Phase 3: Production Readiness
318+
1. Comprehensive testing
319+
2. Error message improvements
320+
3. Documentation and examples
321+
4. Performance benchmarking
322+
323+
### Testing Strategy
324+
325+
#### Unit Tests
326+
- UV binary detection and validation
327+
- Manifest file parsing and detection
328+
- Dependency resolution logic
329+
- Error handling scenarios
330+
- Platform compatibility checks
331+
332+
#### Integration Tests
333+
- End-to-end build scenarios
334+
- Different manifest file formats
335+
- Lock file generation and usage
336+
- Multi-architecture builds
337+
- Performance comparisons with pip
338+
339+
#### Compatibility Tests
340+
- Migration from pip to UV workflows
341+
- Existing SAM CLI integration
342+
- Various Python project structures
343+
- Different dependency complexity levels
344+
345+
### Future Enhancements
346+
347+
Potential future improvements to the UV workflow:
348+
- **Dependency vulnerability scanning**: Integration with security scanning tools
349+
- **Package size optimization**: Advanced techniques for reducing package size
350+
- **Multi-platform builds**: Support for building packages for multiple architectures simultaneously
351+
- **Custom build hooks**: Support for custom build steps and transformations
352+
- **Integration with other tools**: Better integration with other Python development tools
353+
- **UV auto-installation**: Automatic UV installation if not present on system
354+
- **Build caching**: Advanced caching strategies for faster subsequent builds
355+
- **Dependency analysis**: Detailed dependency tree analysis and reporting

0 commit comments

Comments
 (0)