Skip to content

Commit 244ec60

Browse files
VinciGit00VinciGit00
committed
feat: add examples
1 parent 3cc0761 commit 244ec60

File tree

10 files changed

+2342
-0
lines changed

10 files changed

+2342
-0
lines changed

‎README.md‎

Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -183,6 +183,76 @@ CompletableFuture<CompletedSmartscraper> completedSmartscraper = client.smartscr
183183

184184
The asynchronous client supports the same options as the synchronous one, except most methods return `CompletableFuture`s.
185185

186+
## Examples
187+
188+
The `scrapegraphai-java-example` module contains comprehensive examples demonstrating all SDK features:
189+
190+
### Running Examples
191+
192+
Run the basic example:
193+
```bash
194+
./gradlew :scrapegraphai-java-example:run
195+
```
196+
197+
Run specific examples:
198+
```bash
199+
# SmartScraper examples
200+
./gradlew :scrapegraphai-java-example:run -Pexample=Smartscraper
201+
202+
# SearchScraper examples
203+
./gradlew :scrapegraphai-java-example:run -Pexample=Searchscraper
204+
205+
# Crawl examples
206+
./gradlew :scrapegraphai-java-example:run -Pexample=Crawl
207+
208+
# Markdownify examples
209+
./gradlew :scrapegraphai-java-example:run -Pexample=Markdownify
210+
211+
# Async examples
212+
./gradlew :scrapegraphai-java-example:run -Pexample=Async
213+
214+
# Schema Generation examples
215+
./gradlew :scrapegraphai-java-example:run -Pexample=SchemaGeneration
216+
217+
# Utility examples (validation, health, credits)
218+
./gradlew :scrapegraphai-java-example:run -Pexample=Utility
219+
```
220+
221+
### Example Categories
222+
223+
| Example | Description | Features Demonstrated |
224+
|---------|-------------|----------------------|
225+
| **Main** | Basic SmartScraper usage | Simple scraping, error handling |
226+
| **SmartscraperExample** | Comprehensive scraping scenarios | Custom schemas, pagination, JavaScript rendering, headers/cookies |
227+
| **SearchscraperExample** | Web search + scraping | Multi-source data aggregation, product comparison, news aggregation |
228+
| **CrawlExample** | Website crawling | Site exploration, path filtering, progress monitoring |
229+
| **MarkdownifyExample** | HTML to Markdown conversion | Content formatting, documentation generation |
230+
| **AsyncExample** | Asynchronous operations | Non-blocking requests, parallel processing, error handling |
231+
| **SchemaGenerationExample** | Automatic schema creation | Schema analysis, structured data extraction |
232+
| **UtilityExample** | Service utilities | API validation, health checks, credit monitoring, feedback |
233+
234+
### Quick Start Example
235+
236+
```java
237+
import com.scrapegraphai.api.client.ScrapegraphaiClient;
238+
import com.scrapegraphai.api.client.okhttp.ScrapegraphaiOkHttpClient;
239+
import com.scrapegraphai.api.models.smartscraper.CompletedSmartscraper;
240+
import com.scrapegraphai.api.models.smartscraper.SmartscraperCreateParams;
241+
242+
// Initialize client (reads from SCRAPEGRAPHAI_API_KEY environment variable)
243+
ScrapegraphaiClient client = ScrapegraphaiOkHttpClient.fromEnv();
244+
245+
// Create scraping request
246+
SmartscraperCreateParams params = SmartscraperCreateParams.builder()
247+
.userPrompt("Extract the main heading and description")
248+
.websiteUrl("https://example.com")
249+
.build();
250+
251+
// Execute scraping
252+
CompletedSmartscraper result = client.smartscraper().create(params);
253+
System.out.println("Extracted data: " + result.result());
254+
```
255+
186256
## Raw responses
187257

188258
The SDK defines methods that deserialize responses into instances of Java classes. However, these methods don't provide access to the response headers, status code, or the raw response body.

‎scrapegraphai-java-example/README.md‎

Lines changed: 186 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,186 @@
1+
# ScrapeGraphAI Java SDK Examples
2+
3+
This module contains comprehensive examples demonstrating all features of the ScrapeGraphAI Java SDK.
4+
5+
## Prerequisites
6+
7+
1. **API Key**: Set your ScrapeGraphAI API key as an environment variable:
8+
```bash
9+
export SCRAPEGRAPHAI_API_KEY="your_api_key_here"
10+
```
11+
12+
Or set it as a system property:
13+
```bash
14+
-Dscrapegraphai.apiKey=your_api_key_here
15+
```
16+
17+
2. **Java 8+**: This SDK requires Java 8 or later.
18+
19+
## Running Examples
20+
21+
### Basic Example
22+
Run the main example (basic SmartScraper usage):
23+
```bash
24+
./gradlew :scrapegraphai-java-example:run
25+
```
26+
27+
### Specific Examples
28+
Run any specific example using the `-Pexample` parameter:
29+
30+
```bash
31+
# Comprehensive SmartScraper examples
32+
./gradlew :scrapegraphai-java-example:run -Pexample=Smartscraper
33+
34+
# Web search and scraping examples
35+
./gradlew :scrapegraphai-java-example:run -Pexample=Searchscraper
36+
37+
# Website crawling examples
38+
./gradlew :scrapegraphai-java-example:run -Pexample=Crawl
39+
40+
# HTML to Markdown conversion examples
41+
./gradlew :scrapegraphai-java-example:run -Pexample=Markdownify
42+
43+
# Asynchronous operation examples
44+
./gradlew :scrapegraphai-java-example:run -Pexample=Async
45+
46+
# Schema generation examples
47+
./gradlew :scrapegraphai-java-example:run -Pexample=SchemaGeneration
48+
49+
# Utility service examples (validation, health, credits)
50+
./gradlew :scrapegraphai-java-example:run -Pexample=Utility
51+
```
52+
53+
## Example Overview
54+
55+
| Example File | Description | Key Features |
56+
|-------------|-------------|--------------|
57+
| **Main.java** | Basic SmartScraper usage | Simple scraping, error handling, getting started |
58+
| **SmartscraperExample.java** | Comprehensive scraping scenarios | Custom schemas, pagination, JavaScript rendering, headers/cookies, result retrieval |
59+
| **SearchscraperExample.java** | Web search + data extraction | Multi-source aggregation, product comparison, news collection, academic research |
60+
| **CrawlExample.java** | Website crawling operations | Site exploration, path filtering, depth control, progress monitoring |
61+
| **MarkdownifyExample.java** | HTML to Markdown conversion | Content formatting, documentation generation, custom options |
62+
| **AsyncExample.java** | Asynchronous operations | Non-blocking requests, parallel processing, chaining operations, error handling |
63+
| **SchemaGenerationExample.java** | Automatic schema creation | Schema analysis, structured data extraction, schema-driven scraping |
64+
| **UtilityExample.java** | Service utilities | API validation, health checks, credit monitoring, feedback submission |
65+
66+
## Example Details
67+
68+
### 🕷️ SmartScraper Examples
69+
- Basic product information extraction
70+
- Structured data with custom JSON schemas
71+
- Dynamic content requiring JavaScript rendering
72+
- Pagination handling
73+
- Custom headers and cookies for authenticated scraping
74+
- Retrieving results by request ID
75+
76+
### 🔍 SearchScraper Examples
77+
- General web search and data aggregation
78+
- Product comparison across multiple e-commerce sites
79+
- News aggregation from various sources
80+
- Academic research paper collection
81+
- Search progress monitoring
82+
83+
### 🕸️ Crawl Examples
84+
- Basic website crawling
85+
- E-commerce catalog exploration
86+
- Blog and news site crawling
87+
- Documentation site mapping
88+
- Crawl progress monitoring and result retrieval
89+
90+
### 📝 Markdownify Examples
91+
- Basic web page to Markdown conversion
92+
- Blog post formatting
93+
- Documentation page conversion
94+
- Direct HTML content processing
95+
- Conversion progress monitoring
96+
97+
### ⚡ Async Examples
98+
- Basic asynchronous operations
99+
- Parallel processing of multiple URLs
100+
- Chaining async operations
101+
- Mixed service operations
102+
- Comprehensive error handling
103+
104+
### 🏗️ Schema Generation Examples
105+
- E-commerce page schema generation
106+
- Blog content schema creation
107+
- Product listing schemas
108+
- Using generated schemas with SmartScraper
109+
- Schema analysis and monitoring
110+
111+
### 🛠️ Utility Examples
112+
- API key validation and user information
113+
- Service health checks
114+
- Credit balance monitoring
115+
- Feedback submission
116+
- Comprehensive service status checks
117+
118+
## Common Use Cases
119+
120+
### E-commerce Data Extraction
121+
```java
122+
// Use SmartscraperExample for product details
123+
// Use SearchscraperExample for price comparison
124+
// Use CrawlExample for catalog exploration
125+
```
126+
127+
### Content Management
128+
```java
129+
// Use MarkdownifyExample for content conversion
130+
// Use CrawlExample for site mapping
131+
// Use SchemaGenerationExample for consistent data structure
132+
```
133+
134+
### Research and Analysis
135+
```java
136+
// Use SearchscraperExample for information gathering
137+
// Use SmartscraperExample for specific data extraction
138+
// Use AsyncExample for high-volume processing
139+
```
140+
141+
### Monitoring and Maintenance
142+
```java
143+
// Use UtilityExample for service health checks
144+
// Use AsyncExample for performance optimization
145+
// Use CrawlExample for site monitoring
146+
```
147+
148+
## Error Handling
149+
150+
All examples include comprehensive error handling patterns:
151+
- Network connectivity issues
152+
- API key validation problems
153+
- Rate limiting and quota management
154+
- Service availability checks
155+
- Data parsing and validation errors
156+
157+
## Best Practices Demonstrated
158+
159+
1. **Resource Management**: Proper client initialization and cleanup
160+
2. **Error Handling**: Graceful degradation and user-friendly error messages
161+
3. **Async Operations**: Efficient parallel processing and non-blocking operations
162+
4. **Configuration**: Environment-based configuration management
163+
5. **Monitoring**: Service health and usage monitoring
164+
6. **Documentation**: Clear code comments and usage examples
165+
166+
## Getting Help
167+
168+
- Check the main [README.md](../README.md) for general SDK documentation
169+
- Review the [API documentation](https://docs.scrapegraphai.com) for detailed endpoint information
170+
- Run the UtilityExample to verify your setup and check service status
171+
- Use the feedback functionality in UtilityExample to report issues or suggestions
172+
173+
## Contributing
174+
175+
If you have ideas for additional examples or improvements to existing ones, please:
176+
1. Fork the repository
177+
2. Create a new example following the existing patterns
178+
3. Add appropriate documentation
179+
4. Submit a pull request
180+
181+
Examples should be:
182+
- Self-contained and runnable
183+
- Well-documented with clear explanations
184+
- Include error handling
185+
- Demonstrate best practices
186+
- Cover real-world use cases

0 commit comments

Comments
 (0)