Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions .github/workflows/dotnet.yml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,19 @@ jobs:

runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v3
- name: Setup .NET
uses: actions/setup-dotnet@v3
with:
dotnet-version: 8.0.x
- name: Build
run: ./build.sh
shell: bash
build-mac:

runs-on: macos-latest

steps:
- uses: actions/checkout@v3
- name: Setup .NET
Expand Down
215 changes: 192 additions & 23 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,38 +1,146 @@
## Build Status
# NuGet.CatalogReader

| Github |
| --- |
| [![.NET test](https://github.com/emgarten/NuGet.CatalogReader/actions/workflows/dotnet.yml/badge.svg)](https://github.com/emgarten/NuGet.CatalogReader/actions/workflows/dotnet.yml) |
Read NuGet v3 feed catalogs, mirror packages to disk, and query package metadata.

# What are these tools?
| | NuGet.CatalogReader | NuGetMirror |
| --- | --- | --- |
| **Package** | [![NuGet](https://img.shields.io/nuget/v/NuGet.CatalogReader.svg)](https://www.nuget.org/packages/NuGet.CatalogReader/) | [![NuGet](https://img.shields.io/nuget/v/NuGetMirror.svg)](https://www.nuget.org/packages/NuGetMirror/) |
| **Downloads** | [![NuGet Downloads](https://img.shields.io/nuget/dt/NuGet.CatalogReader.svg)](https://www.nuget.org/packages/NuGet.CatalogReader/) | [![NuGet Downloads](https://img.shields.io/nuget/dt/NuGetMirror.svg)](https://www.nuget.org/packages/NuGetMirror/) |

NuGetMirror.exe is a command line tool to mirror nuget.org to disk, or any NuGet v3 feed. It supports filtering package ids and wildcards to narrow down the set of mirrored packages.
[![.NET test](https://github.com/emgarten/NuGet.CatalogReader/actions/workflows/dotnet.yml/badge.svg)](https://github.com/emgarten/NuGet.CatalogReader/actions/workflows/dotnet.yml)

NuGet.CatalogReader is a library for reading package ids, versions, and the change history of a NuGet v3 feeds or nuget.org.
## Table of contents

## Getting NuGetMirror
- [Overview](#overview)
- [Prerequisites](#prerequisites)
- [Installation](#installation)
- [NuGetMirror CLI tool](#nugetmirror-cli-tool)
- [Mirror packages (nupkgs)](#mirror-packages-nupkgs)
- [List packages (list)](#list-packages-list)
- [Auth support](#auth-support)
- [NuGet.CatalogReader library](#nugetcatalogreader-library)
- [Reading catalog entries](#reading-catalog-entries)
- [Time range filtering](#time-range-filtering)
- [Reading feeds without a catalog](#reading-feeds-without-a-catalog)
- [Download modes](#download-modes)
- [API overview](#api-overview)
- [Building from source](#building-from-source)
- [Contributing](#contributing)
- [License](#license)

### Install dotnet global tool
1. `dotnet tool install -g nugetmirror`
1. `nugetmirror` should now be on your *PATH*
## Overview

# Quick start
This repository contains two packages:

## Using NuGetMirror
- **NuGet.CatalogReader** — A library for reading package ids, versions, and the change history of NuGet v3 feeds or nuget.org.
- **NuGetMirror** — A command line tool to mirror nuget.org (or any NuGet v3 feed) to disk. Supports filtering by package id and wildcards.

Mirror all packages to a folder on disk.
## Prerequisites

``NuGetMirror nupkgs https://api.nuget.org/v3/index.json -o d:\tmp``
- .NET 8.0, 9.0, or 10.0 SDK

NuGetMirror can be used to continually sync the latest packages. Runs store to the last commit time to disk, future runs will resume from this point and only get new or updated packages.
## Installation

### NuGet.CatalogReader library

```
dotnet add package NuGet.CatalogReader
```

### NuGetMirror CLI tool

```
dotnet tool install -g nugetmirror
```

After installation, `nugetmirror` will be available on your PATH.

## NuGetMirror CLI tool

### Mirror packages (nupkgs)

Mirror all packages to a folder on disk:

```
nugetmirror nupkgs https://api.nuget.org/v3/index.json -o /tmp/packages
```

NuGetMirror stores the last commit time to disk so that future runs resume from that point and only download new or updated packages.

#### Options

| Option | Description |
| --- | --- |
| `-o\|--output` | Output directory for nupkgs (required) |
| `--folder-format` | Output folder format: `v2` or `v3` (default: `v3`) |
| `-i\|--include-id` | Include only matching package ids (supports wildcards, can be repeated) |
| `-e\|--exclude-id` | Exclude matching package ids (supports wildcards, can be repeated) |
| `--latest-only` | Mirror only the latest version of each package |
| `--stable-only` | Exclude pre-release packages |
| `--start` | Beginning of commit time range (exclusive) |
| `--end` | End of commit time range (inclusive) |
| `--max-threads` | Maximum concurrent downloads (default: 8) |
| `--delay` | Delay in minutes before downloading the latest packages (default: 10) |
| `--additional-output` | Additional output directory for load balancing across drives |
| `--ignore-errors` | Continue on download errors |

#### Examples

Mirror only packages matching a wildcard:

```
nugetmirror nupkgs https://api.nuget.org/v3/index.json -o /tmp/packages -i "Newtonsoft.*"
```

Mirror stable-only, latest versions:

```
nugetmirror nupkgs https://api.nuget.org/v3/index.json -o /tmp/packages --latest-only --stable-only
```

### List packages (list)

List all packages in a feed:

```
nugetmirror list https://api.nuget.org/v3/index.json
```

#### Options

| Option | Description |
| --- | --- |
| `-s\|--start` | Beginning of commit time range (exclusive) |
| `-e\|--end` | End of commit time range (inclusive) |
| `-v\|--verbose` | Write additional network call information |

### Auth support

NuGetMirror can use credentials from a *nuget.config* file. Pass the name of the source instead of the index.json URI and ensure that the config is in the working directory or one of the common nuget.config locations.
NuGetMirror can use credentials from a `nuget.config` file. Pass the source name instead of the index.json URI and ensure that the config is in the working directory or one of the [common nuget.config locations](https://learn.microsoft.com/nuget/consume-packages/configuring-nuget-behavior#config-file-locations-and-uses).

```
nugetmirror nupkgs MyPrivateFeed -o /tmp/packages
```

With a `nuget.config` like:

```xml
<packageSources>
<add key="MyPrivateFeed" value="https://myfeed.example.com/v3/index.json" />
</packageSources>
<packageSourceCredentials>
<MyPrivateFeed>
<add key="Username" value="user" />
<add key="ClearTextPassword" value="token" />
</MyPrivateFeed>
</packageSourceCredentials>
```

## NuGet.CatalogReader library

## Using NuGet.CatalogReader
### Reading catalog entries

Discover all packages in a feed using ``GetFlattenedEntriesAsync``. To see the complete history including edits use ``GetEntriesAsync``.
Discover all packages in a feed using `GetFlattenedEntriesAsync`. To see the complete history including edits, use `GetEntriesAsync`.

```csharp
var feed = new Uri("https://api.nuget.org/v3/index.json");
Expand All @@ -42,12 +150,32 @@ using (var catalog = new CatalogReader(feed))
foreach (var entry in await catalog.GetFlattenedEntriesAsync())
{
Console.WriteLine($"[{entry.CommitTimeStamp}] {entry.Id} {entry.Version}");
await entry.DownloadNupkgAsync(@"d:\output");
}
}
```

NuGet v3 feeds that do not have a catalog can be read using `FeedReader`.
### Time range filtering

Retrieve entries within a specific time range:

```csharp
var feed = new Uri("https://api.nuget.org/v3/index.json");

using (var catalog = new CatalogReader(feed))
{
var start = DateTimeOffset.UtcNow.AddHours(-1);
var end = DateTimeOffset.UtcNow;

foreach (var entry in await catalog.GetEntriesAsync(start, end, CancellationToken.None))
{
Console.WriteLine($"[{entry.CommitTimeStamp}] {entry.Id} {entry.Version}");
}
}
```

### Reading feeds without a catalog

NuGet v3 feeds that do not have a catalog can be read using `FeedReader`:

```csharp
var feed = new Uri("https://api.nuget.org/v3/index.json");
Expand All @@ -57,14 +185,55 @@ using (var feedReader = new FeedReader(feed))
foreach (var entry in await feedReader.GetPackagesById("NuGet.Versioning"))
{
Console.WriteLine($"{entry.Id} {entry.Version}");
await entry.DownloadNupkgAsync(@"d:\output");
await entry.DownloadNupkgAsync("/tmp/output");
}
}
```

### Download modes

When downloading packages, specify how to handle existing files with `DownloadMode`:

| Mode | Behavior |
| --- | --- |
| `FailIfExists` | Throw if the file already exists |
| `SkipIfExists` | Skip the download if the file already exists |
| `OverwriteIfNewer` | Overwrite only if the new package is newer |
| `Force` | Always overwrite |

```csharp
await entry.DownloadNupkgAsync("/tmp/output", DownloadMode.SkipIfExists, CancellationToken.None);
```

### API overview

| Type | Description |
| --- | --- |
| `CatalogReader` | Reads catalog entries from a NuGet v3 feed |
| `FeedReader` | Reads packages from a NuGet v3 feed without requiring a catalog |
| `CatalogEntry` | A catalog page entry with commit metadata (`CommitTimeStamp`, `IsAddOrUpdate`, `IsDelete`) |
| `PackageEntry` | A package with download and metadata methods (`DownloadNupkgAsync`, `GetNuspecAsync`, `IsListedAsync`) |
| `CatalogPageEntry` | Represents a page in the catalog index |
| `DownloadMode` | Controls file overwrite behavior when downloading |

## Building from source

Clone the repository and run the build script for your platform:

```bash
# macOS / Linux
./build.sh

# Windows
./build.ps1
```

The build script will install the required .NET SDKs locally, restore packages, build, pack, and run tests.

## Contributing

We welcome contributions. If you are interested in contributing you can report an issue or open a pull request to propose a change.

### License
## License

[MIT License](https://raw.githubusercontent.com/emgarten/NuGet.CatalogReader/main/LICENSE)
3 changes: 3 additions & 0 deletions ReleaseNotes.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
# Release Notes

## 4.0.0
* Add net10.0 support, remove net6.0

## 3.3.2
* Fix for download nuspec helpers [PR](https://github.com/emgarten/NuGet.CatalogReader/pull/33)

Expand Down
44 changes: 16 additions & 28 deletions build/common/build.shared.proj
Original file line number Diff line number Diff line change
Expand Up @@ -75,12 +75,13 @@
</PropertyGroup>

<!-- Run tests as a batch -->
<!-- Note: TestProjectPaths must be escaped because it contains semicolons which would otherwise be interpreted as property separators -->
<MSBuild
Projects="$(MSBuildThisFileFullPath)"
Targets="RunTestsOnProjects"
Properties="$(CommonMSBuildProperties);
TestResultsFileName=$(TestResultsFileName);
TestProjectPaths=$(TestProjectPaths)" />
TestProjectPaths=$([MSBuild]::Escape($(TestProjectPaths)))" />
</Target>

<!--
Expand Down Expand Up @@ -119,8 +120,9 @@
<Target Name="RunTestsOnProjects">
<Message Text="Running $(TestResultsFileName)" Importance="high" />

<!-- Unescape the TestProjectPaths property to restore semicolons as item separators -->
<ItemGroup>
<TestProjectToSearch Include="$(TestProjectPaths)" />
<TestProjectToSearch Include="$([MSBuild]::Unescape($(TestProjectPaths)))" />
</ItemGroup>

<MSBuild
Expand All @@ -140,38 +142,24 @@
<MakeDir Directories="$(TestResultsDirectory)" />

<PropertyGroup>
<!-- Sort assemblies -->
<DesktopInputTestAssemblies>@(TestAssemblyPath->WithMetadataValue("IsDesktop", "true"))</DesktopInputTestAssemblies>
<DesktopInputTestAssembliesSpaced>$(DesktopInputTestAssemblies.Replace(';', ' '))</DesktopInputTestAssembliesSpaced>
<CoreInputTestAssemblies>@(TestAssemblyPath->WithMetadataValue("IsCore", "true"))</CoreInputTestAssemblies>
<CoreInputTestAssembliesSpaced>$(CoreInputTestAssemblies.Replace(';', ' '))</CoreInputTestAssembliesSpaced>

<!-- Build exe commands -->
<TestResultsHtml Condition=" '$(TestResultsFileName)' != '' ">$(TestResultsDirectory)$(TestResultsFileName).html</TestResultsHtml>
<VSTestCommand>$(DotnetExePath) vstest $(CoreInputTestAssembliesSpaced)</VSTestCommand>
<DesktopTestCommand>$(XunitConsoleExePath) $(DesktopInputTestAssembliesSpaced)</DesktopTestCommand>
<DesktopTestCommand Condition=" '$(TestResultsHtml)' != '' ">$(DesktopTestCommand) -html $(TestResultsHtml)</DesktopTestCommand>
</PropertyGroup>
<!-- All test assemblies run via dotnet vstest -->
<TestInputAssemblies>@(TestAssemblyPath)</TestInputAssemblies>
<TestInputAssembliesSpaced>$(TestInputAssemblies.Replace(';', ' '))</TestInputAssembliesSpaced>

<!-- Desktop -->
<Exec Command="$(DesktopTestCommand)"
ContinueOnError="true"
Condition=" '$(DesktopInputTestAssemblies)' != '' AND '$(SkipDesktopTests)' != 'true' ">
<Output TaskParameter="ExitCode" PropertyName="DesktopTestErrorCode"/>
</Exec>
<!-- Build vstest command -->
<VSTestCommand>$(DotnetExePath) vstest $(TestInputAssembliesSpaced)</VSTestCommand>
</PropertyGroup>

<!-- VSTest/NETCore -->
<!-- Run all tests via VSTest -->
<Exec Command="$(VSTestCommand)"
ContinueOnError="true"
Condition=" '$(CoreInputTestAssemblies)' != '' AND '$(SkipCoreTests)' != 'true' ">
Condition=" '$(TestInputAssemblies)' != '' ">
<Output TaskParameter="ExitCode" PropertyName="VSTestErrorCode"/>
</Exec>

<Error Text="NETFramework $(TestResultsFileName) tests failed! Results: $(TestResultsHtml)" Condition=" '$(DesktopTestErrorCode)' != '0' AND '$(DesktopTestErrorCode)' != '' " />
<Error Text="NETCoreApp $(TestResultsFileName) tests failed!" Condition=" '$(VSTestErrorCode)' != '0' AND '$(VSTestErrorCode)' != '' " />
<Error Text="$(TestResultsFileName) tests failed!" Condition=" '$(VSTestErrorCode)' != '0' AND '$(VSTestErrorCode)' != '' " />

<Message Text="NETFramework $(TestResultsFileName) tests passed!" Condition=" '$(DesktopTestErrorCode)' == '0' " />
<Message Text="NETCoreApp $(TestResultsFileName) tests passed!" Condition=" '$(VSTestErrorCode)' == '0' " />
<Message Text="$(TestResultsFileName) tests passed!" Condition=" '$(VSTestErrorCode)' == '0' " />
</Target>

<!--
Expand All @@ -186,12 +174,12 @@
<CICommonPackagesConfig>$(CIRootDirectory)common\packages.common.config</CICommonPackagesConfig>
<CIPackagesConfig>$(CIRootDirectory)common\packages.config</CIPackagesConfig>
</PropertyGroup>

<Exec Command="$(NuGetExePath) restore $(CommonPackagesConfig) -PackagesDirectory $(PackagesConfigDirectory)" Condition="Exists($(CommonPackagesConfig))" />
<Exec Command="$(NuGetExePath) restore $(RepoPackagesConfig) -PackagesDirectory $(PackagesConfigDirectory)" Condition="Exists($(RepoPackagesConfig))" />
<Exec Command="$(NuGetExePath) restore $(CICommonPackagesConfig) -PackagesDirectory $(PackagesConfigDirectory)" Condition="Exists($(CICommonPackagesConfig))" />
<Exec Command="$(NuGetExePath) restore $(CIPackagesConfig) -PackagesDirectory $(PackagesConfigDirectory)" Condition="Exists($(CIPackagesConfig))" />
</Target>

<Import Project="$([MSBuild]::GetDirectoryNameOfFileAbove($(MSBuildThisFileDirectory), 'README.md'))\build\common\gitversion.targets" />
</Project>
</Project>
4 changes: 2 additions & 2 deletions build/common/common.ps1
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,9 @@ Function Install-DotnetCLI {

Invoke-WebRequest https://dot.net/v1/dotnet-install.ps1 -OutFile $installDotnet

& $installDotnet -Channel 6.0 -i $CLIRoot
& $installDotnet -Channel 8.0 -i $CLIRoot
& $installDotnet -Channel 9.0 -i $CLIRoot
& $installDotnet -Channel 10.0 -i $CLIRoot

if (-not (Test-Path $DotnetExe)) {
Write-Log "Missing $DotnetExe"
Expand Down Expand Up @@ -73,7 +73,7 @@ Function Install-NuGetExe {
$nugetDir = Split-Path $nugetExe
New-Item -ItemType Directory -Force -Path $nugetDir

Invoke-WebRequest https://dist.nuget.org/win-x86-commandline/v6.12.1/nuget.exe -OutFile $nugetExe
Invoke-WebRequest https://dist.nuget.org/win-x86-commandline/v7.0.1/nuget.exe -OutFile $nugetExe
}
}

Expand Down
Loading