EFCoreDapper.Bulk

A high-performance bulk operations library that provides efficient Insert, Update, Delete, and Upsert operations for both Entity Framework Core and Dapper. Built for scenarios requiring high-volume data processing with optimal performance.

Features

🚀 High Performance: Bulk operations dramatically reduce database round-trips
🔄 Dual Compatibility: Works seamlessly with both Entity Framework Core and Dapper
⚙️ Flexible Configuration: Customizable batch sizes, transaction control, and retry policies
🔒 Type Safety: Strongly-typed operations with attribute-based configuration
🛡️ Robust Error Handling: Built-in retry mechanisms and transaction support
📊 Batch Processing: Configurable batch sizes to optimize memory and performance

Quickstart

Installation

Install the EFCoreDapper.Bulk package from NuGet:

dotnet add package EFCoreDapper.Bulk

Entity Framework Core Example

1. Install Required Packages

dotnet add package EFCoreDapper.Bulk
dotnet add package Microsoft.EntityFrameworkCore.SqlServer
dotnet add package Microsoft.Extensions.Hosting

2. Configure Services in Dependency Injection

using EFCoreDapper.Bulk;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

public class Program
{
    public static async Task Main(string[] args)
    {
        var host = Host.CreateDefaultBuilder(args)
            .ConfigureServices((hostContext, services) =>
            {
                // Register DbContext
                services.AddDbContext<SampleDbContext>(options =>
                    options.UseSqlServer("Server=(localdb)\\mssqllocaldb;Database=EFCoreDapper.Bulk.Sample;Trusted_Connection=True;MultipleActiveResultSets=true"));

                // Register Bulk Services with configuration
                services.AddBulkServices(options =>
                {
                    options.BatchSize = 5000; // Process 5000 records per batch
                    options.UseTransaction = true; // Wrap operations in transactions
                    options.CommandTimeoutSeconds = 300; // 5-minute timeout for large operations
                });
            })
            .Build();

        using (var scope = host.Services.CreateScope())
        {
            var services = scope.ServiceProvider;
            var dbContext = services.GetRequiredService<SampleDbContext>();
            var bulkService = services.GetRequiredService<IBulkService>();

            // Your bulk operations here
        }
    }
}

3. Define Your Entity

using EFCoreDapper.Bulk;

public class SampleEntity
{
    [BulkKey] // Marks this property as the primary key
    public int Id { get; set; }
    
    public string Name { get; set; }
    
    [BulkColumn("EmailAddress")] // Custom column mapping
    public string Email { get; set; }
    
    [BulkIgnore] // Exclude from bulk operations
    public string InternalData { get; set; }
    
    public DateTime CreatedAt { get; set; }
}

4. Perform Bulk Operations

// Bulk Insert
var items = new List<SampleEntity>();
for (int i = 0; i < 100000; i++)
{
    items.Add(new SampleEntity { 
        Name = $"Name {i}", 
        Email = $"user{i}@example.com",
        CreatedAt = DateTime.UtcNow 
    });
}

await bulkService.InsertAsync(items);

// Bulk Update
var itemsToUpdate = items.Take(1000).ToList();
foreach (var item in itemsToUpdate)
{
    item.Name = $"Updated {item.Name}";
}
await bulkService.UpdateAsync(itemsToUpdate);

// Bulk Delete
var keysToDelete = items.Skip(5000).Take(1000).Select(x => (object)x.Id);
await bulkService.DeleteAsync<SampleEntity>(keysToDelete);

// Bulk Upsert (Insert or Update)
var mixedItems = GetMixedItems(); // Some new, some existing
await bulkService.UpsertAsync(mixedItems, item => item.Id);

Dapper Example

1. Install Required Packages

dotnet add package EFCoreDapper.Bulk
dotnet add package Dapper
dotnet add package Microsoft.Data.SqlClient

2. Direct Usage with Dapper

using Dapper;
using EFCoreDapper.Bulk;
using Microsoft.Data.SqlClient;

public class Program
{
    public static async Task Main(string[] args)
    {
        var connectionString = "Server=(localdb)\\mssqllocaldb;Database=EFCoreDapper.Bulk.Sample;Trusted_Connection=True;MultipleActiveResultSets=true";

        using (var connection = new SqlConnection(connectionString))
        {
            await connection.OpenAsync();

            // Create bulk service instance
            var service = new DapperBulkService(connection);

            // Prepare data
            var items = new List<SampleEntity>();
            for (int i = 0; i < 100000; i++)
            {
                items.Add(new SampleEntity { 
                    Name = $"Name {i}", 
                    Email = $"user{i}@example.com" 
                });
            }

            // Perform bulk insert with custom options
            await service.InsertAsync(items, new BulkOptions { 
                TableName = "SampleEntities",
                BatchSize = 2000,
                UseTransaction = true
            });
        }
    }
}

// Entity class for Dapper usage
public class SampleEntity
{
    public int Id { get; set; }
    public string Name { get; set; }
    public string Email { get; set; }
}

Configuration

BulkOptions Class

Customize bulk operations with the BulkOptions class:

public class BulkOptions
{
    /// <summary>
    /// Number of records to process per batch (default: 1000)
    /// Higher values = more memory, lower values = more round-trips
    /// </summary>
    public int BatchSize { get; set; } = 1000;
    
    /// <summary>
    /// Use temporary tables for merge operations (default: true)
    /// Improves performance for large datasets
    /// </summary>
    public bool UseTempTable { get; set; } = true;
    
    /// <summary>
    /// Wrap operations in a transaction (default: true)
    /// Ensures atomicity of bulk operations
    /// </summary>
    public bool UseTransaction { get; set; } = true;
    
    /// <summary>
    /// Command timeout in seconds (default: 120)
    /// </summary>
    public int CommandTimeoutSeconds { get; set; } = 120;
    
    /// <summary>
    /// Retry policy for transient errors
    /// </summary>
    public RetryPolicy? Retry { get; set; }
    
    /// <summary>
    /// Table name (required for Dapper, auto-detected for EF Core)
    /// </summary>
    public string? TableName { get; set; }
    
    /// <summary>
    /// Primary key column name (required for Update/Delete operations)
    /// </summary>
    public string? PrimaryKey { get; set; }
}

RetryPolicy Configuration

Handle transient database errors with retry policies:

var options = new BulkOptions
{
    Retry = new RetryPolicy
    {
        MaxRetryCount = 3,
        RetryInterval = TimeSpan.FromSeconds(2),
        ExponentialBackoff = true
    }
};

await bulkService.InsertAsync(items, options);

API Reference

IBulkService Interface

public interface IBulkService
{
    /// <summary>
    /// Bulk insert a collection of entities
    /// </summary>
    Task InsertAsync<T>(IEnumerable<T> items, BulkOptions? options = null, CancellationToken ct = default);
    
    /// <summary>
    /// Bulk update a collection of entities
    /// Requires primary key to be specified via attributes or BulkOptions
    /// </summary>
    Task UpdateAsync<T>(IEnumerable<T> items, BulkOptions? options = null, CancellationToken ct = default);
    
    /// <summary>
    /// Bulk delete entities by their primary keys
    /// </summary>
    Task DeleteAsync<T>(IEnumerable<object> keys, BulkOptions? options = null, CancellationToken ct = default);
    
    /// <summary>
    /// Bulk upsert (insert or update) a collection of entities
    /// </summary>
    Task UpsertAsync<T>(IEnumerable<T> items, Func<T, object> keySelector, BulkOptions? options = null, CancellationToken ct = default);
}

Custom Attributes

`[BulkKey]` Attribute

Marks a property as the primary key for bulk operations:

public class Product
{
    [BulkKey]
    public int ProductId { get; set; }
    public string Name { get; set; }
}

`[BulkColumn("columnName")]` Attribute

Specifies custom column mapping:

public class User
{
    [BulkKey]
    public int Id { get; set; }
    
    [BulkColumn("FullName")]
    public string Name { get; set; }
    
    [BulkColumn("EmailAddress")]
    public string Email { get; set; }
}

`[BulkIgnore]` Attribute

Excludes properties from bulk operations:

public class Order
{
    [BulkKey]
    public int OrderId { get; set; }
    public decimal Amount { get; set; }
    
    [BulkIgnore]
    public string AuditData { get; set; } // Won't be included in bulk operations
}

Performance Considerations

Batch Size Optimization

Small datasets (< 10,000 records): Use default batch size (1000)
Medium datasets (10,000 - 100,000 records): Increase to 2000-5000
Large datasets (> 100,000 records): Test with 5000-10000, monitor memory usage

Memory Management

Bulk operations load data into memory - monitor your application's memory usage
For extremely large datasets, consider processing in smaller chunks
Use streaming or pagination for datasets that don't fit in memory

Transaction Strategy

UseTransaction = true: Recommended for atomic operations
UseTransaction = false: Can improve performance for non-critical data
Always test with your specific workload and data size

Error Handling and Retry

Built-in Retry Mechanism

var retryPolicy = new RetryPolicy
{
    MaxRetryCount = 5,
    RetryInterval = TimeSpan.FromSeconds(1),
    ExponentialBackoff = true // Doubles interval after each retry
};

var options = new BulkOptions { Retry = retryPolicy };

Exception Handling

try
{
    await bulkService.InsertAsync(largeDataset, options);
}
catch (SqlException ex) when (ex.Number == 1205) // Deadlock
{
    // Handle deadlock specifically
    await Task.Delay(1000);
    await bulkService.InsertAsync(largeDataset, options); // Retry once
}
catch (Exception ex)
{
    // Log and handle other exceptions
    logger.LogError(ex, "Bulk operation failed");
}

Best Practices

1. Connection Management

EF Core: Let DI container manage connections
Dapper: Ensure proper using statements or connection pooling

2. Data Validation

Validate data before bulk operations to avoid partial failures
Consider using database constraints for data integrity

3. Monitoring

Monitor performance with logging and metrics
Set appropriate timeouts based on expected data volume

4. Testing

Test with representative data volumes
Include integration tests with real database
Monitor memory usage during performance tests

Migration Scenarios

From EF Core SaveChanges

// Before: Slow for large datasets
foreach (var item in items)
{
    context.Entities.Add(item);
}
await context.SaveChangesAsync(); // Multiple round-trips

// After: High performance
await bulkService.InsertAsync(items); // Single round-trip

From Dapper Individual Inserts

// Before: Individual insert statements
foreach (var item in items)
{
    await connection.ExecuteAsync("INSERT INTO Table (...) VALUES (...)", item);
}

// After: Bulk insert
var bulkService = new DapperBulkService(connection);
await bulkService.InsertAsync(items, new BulkOptions { TableName = "Table" });

Supported Databases

SQL Server (Primary support)
Other databases with SQL Server compatibility mode
Future versions may include additional database providers

Contributing

We welcome contributions! Please see our Contributing Guide for details.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Support

Issues: GitHub Issues
Documentation: Wiki
Questions: Discussions

Release Notes

See CHANGELOG.md for version history and changes.# EFCoreDapper.Bulk

High-performance bulk operations (Insert, Update, Delete, Upsert) for Entity Framework Core and Dapper.

Quickstart

Install the EFCoreDapper.Bulk package from NuGet:

dotnet add package EFCoreDapper.Bulk

EF Core Example

Install packages

dotnet add package EFCoreDapper.Bulk dotnet add package Microsoft.EntityFrameworkCore.SqlServer dotnet add package Microsoft.Extensions.Hosting ```

Configure services

using EFCoreDapper.Bulk;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

public class Program
{
    public static async Task Main(string[] args)
    {
        var host = Host.CreateDefaultBuilder(args)
            .ConfigureServices((hostContext, services) =>
            {
                services.AddDbContext<SampleDbContext>(options =>
                    options.UseSqlServer("Server=(localdb)\\\\mssqllocaldb;Database=EFCoreDapper.Bulk.Sample;Trusted_Connection=True;MultipleActiveResultSets=true"));

                services.AddBulkServices(options =>
                {
                    options.BatchSize = 5000;
                });
            })
            .Build();

        using (var scope = host.Services.CreateScope())
        {
            var services = scope.ServiceProvider;
            var dbContext = services.GetRequiredService<SampleDbContext>();
            var bulkService = services.GetRequiredService<IBulkService>();

            // ...
        }
    }
}

Use the service

var items = new List<SampleEntity>();
for (int i = 0; i < 100000; i++)
{
    items.Add(new SampleEntity { Name = $"Name {i}" });
}

await bulkService.InsertAsync(items);

Dapper Example

Install packages

dotnet add package EFCoreDapper.Bulk dotnet add package Dapper dotnet add package Microsoft.Data.SqlClient ```

Use the service

using Dapper;
using EFCoreDapper.Bulk;
using Microsoft.Data.SqlClient;
using System.Collections.Generic;
using System.Threading.Tasks;

public class Program
{
    public static async Task Main(string[] args)
    {
        var connectionString = "Server=(localdb)\\\\mssqllocaldb;Database=EFCoreDapper.Bulk.Sample;Trusted_Connection=True;MultipleActiveResultSets=true";

        using (var connection = new SqlConnection(connectionString))
        {
            await connection.OpenAsync();

            var service = new DapperBulkService(connection);

            var items = new List<SampleEntity>();
            for (int i = 0; i < 100000; i++)
            {
                items.Add(new SampleEntity { Name = $"Name {i}" });
            }

            await service.InsertAsync(items, new BulkOptions { TableName = "SampleEntities" });
        }
    }
}

Configuration

You can configure the bulk operations by passing a BulkOptions object to the methods.

public class BulkOptions
{
    public int BatchSize { get; set; } = 1000;
    public bool UseTempTable { get; set; } = true;
    public bool UseTransaction { get; set; } = true;
    public int CommandTimeoutSeconds { get; set; } = 120;
    public RetryPolicy? Retry { get; set; }
    public string? TableName { get; set; }
    public string? PrimaryKey { get; set; }
}

API Reference

`IBulkService`

public interface IBulkService
{
    Task InsertAsync<T>(IEnumerable<T> items, BulkOptions? options = null, CancellationToken ct = default);
    Task UpdateAsync<T>(IEnumerable<T> items, BulkOptions? options = null, CancellationToken ct = default);
    Task DeleteAsync<T>(IEnumerable<object> keys, BulkOptions? options = null, CancellationToken ct = default);
    Task UpsertAsync<T>(IEnumerable<T> items, Func<T, object> keySelector, BulkOptions? options = null, CancellationToken ct = default);
}

Attributes

[BulkKey]: Specifies the primary key of the entity.
[BulkColumn("name")]: Specifies the column name of the property.
[BulkIgnore]: Ignores the property during bulk operations.

Name		Name	Last commit message	Last commit date
Latest commit History 2 Commits
samples		samples
src/EFCoreDapper.Bulk		src/EFCoreDapper.Bulk
tests		tests
.gitattributes		.gitattributes
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
CONTRIBUTING.md		CONTRIBUTING.md
EFCoreDapper.Bulk.sln		EFCoreDapper.Bulk.sln
GEMINI.md		GEMINI.md
LICENSE.md		LICENSE.md
README.md		README.md

Folders and files

Latest commit

History

Repository files navigation

EFCoreDapper.Bulk

Features

Quickstart

Installation

Entity Framework Core Example

1. Install Required Packages

2. Configure Services in Dependency Injection

3. Define Your Entity

4. Perform Bulk Operations

Dapper Example

1. Install Required Packages

2. Direct Usage with Dapper

Configuration

BulkOptions Class

RetryPolicy Configuration

API Reference

IBulkService Interface

Custom Attributes

[BulkKey] Attribute

[BulkColumn("columnName")] Attribute

[BulkIgnore] Attribute

Performance Considerations

Batch Size Optimization

Memory Management

Transaction Strategy

Error Handling and Retry

Built-in Retry Mechanism

Exception Handling

Best Practices

1. Connection Management

2. Data Validation

3. Monitoring

4. Testing

Migration Scenarios

From EF Core SaveChanges

From Dapper Individual Inserts

Supported Databases

Contributing

License

Support

Release Notes

Quickstart

EF Core Example

Dapper Example

Configuration

API Reference

IBulkService

Attributes

About

Resources

License

Contributing

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

`[BulkKey]` Attribute

`[BulkColumn("columnName")]` Attribute

`[BulkIgnore]` Attribute

`IBulkService`

Packages