Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -13,3 +13,10 @@ API_PORT=8000

# Scraping Limits
MAX_LISTINGS_PER_SESSION=25

# Optional Proxy Configuration
# If using rotating proxies (e.g., Webshare), uncomment and provide the proxy URL.
# Example: http://username:password@proxyhost:port
# PROXY_SERVER=
# PROXY_USERNAME=
# PROXY_PASSWORD=
25 changes: 25 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,3 +20,28 @@ The scraper can also be run standalone:
```bash
python main.py scrape_test
```

## Environment Variables

Set the following variables in a `.env` file or your deployment environment:

| Variable | Description | Default |
| --- | --- | --- |
| `DATABASE_URL` | Database connection URL | `sqlite+aiosqlite:///./vehicle_data.db` |
| `HEADLESS` | Run the browser in headless mode | `true` |
| `BROWSER_TIMEOUT` | Playwright launch timeout (ms) | `60000` |
| `PAGE_DELAY` | Base delay after page loads (ms) | `5000` |
| `MIN_DELAY_BETWEEN_ACTIONS` | Delay between scraping actions (s) | `2.5` |
| `API_HOST` | Host for the FastAPI server | `127.0.0.1` |
| `API_PORT` | Port for the FastAPI server | `8000` |
| `MAX_LISTINGS_PER_SESSION` | Maximum listings fetched per scrape | `25` |
| `PROXY_SERVER` | *(Optional)* Proxy URL for Playwright | - |
| `PROXY_USERNAME` | *(Optional)* Proxy username | - |
| `PROXY_PASSWORD` | *(Optional)* Proxy password | - |

### Pagination

The `/api/v1/vehicles/` endpoint accepts `skip` and `limit` query parameters to paginate results.
Example: `/api/v1/vehicles/?skip=25&limit=25`.


123 changes: 0 additions & 123 deletions app.py

This file was deleted.

Empty file removed app/__init__.py
Empty file.
50 changes: 0 additions & 50 deletions app/crud.py

This file was deleted.

25 changes: 0 additions & 25 deletions app/database.py

This file was deleted.

Loading