SDK 1.0 by mo4islona · Pull Request #37 · subsquid-labs/pipes-sdk

mo4islona · 2026-01-19T18:06:59Z

No description provided.

…mers

…ion and clarity

… sources

… improved context management

…mproved clarity and transformation capabilities

… for improved clarity

… handling and improved state management

…d error handling - Introduced flexible pipe selection in the Sidebar with progress and ETA. - Added error state handling with alerts for disconnected pipes. - Updated `displayEstimatedTime` for customizable label prefixes. - Refactored and standardized API types for stats and pipes. - Enhanced `Pipeline` component with dynamic state awareness. - Improved style adjustments for consistent rendering.

… method names for consistency

…d dashboard components

…est compatibility Export pipeline metrics (last_block, progress_ratio, eta_seconds, blocks_per_second, bytes_downloaded_total, pipeline_running) from progress-tracker. Remove obsolete metrics.test.ts and adapt solana-instruction-decoder tests to new outputs-based API.

Add batch and reorg metrics to portal-source (sqd_reorgs_total, sqd_batch_size_blocks, sqd_batch_size_bytes). Remove duplicate HistogramConfiguration interface in metrics-server.

Test all Prometheus metrics: sqd_current_block, sqd_last_block, sqd_progress_ratio, sqd_eta_seconds, sqd_blocks_per_second, sqd_bytes_downloaded_total, sqd_pipeline_running, sqd_reorgs_total, sqd_batch_size_blocks, sqd_batch_size_bytes. Co-Authored-By: Claude Opus 4.6 <[email protected]>

…d dashboard components

* feat: implement server-side metrics fetching and enhance component structure * feat: refactor imports and enhance fullscreen toggle functionality in transformation exemplar * feat: adjust height properties in profiler and transformation exemplar components * feat: add loading panel component and integrate it into profiler and transformation exemplar * feat: enhance profiler layout and add gradient stroke to sidebar circle * feat: add Select, Separator, and Switch components with Radix UI integration * feat: extend ApiPipe type to include dataset details for improved metrics handling * feat: implement server-side metrics fetching with enhanced transformation history and UI controls

…lity

add rpc calls examples

* impr(subsquid-pipes): thrown error if range.from > range.to * refactor(subsquid-pipes): improve block range handling * refactor(subsquid-pipes): use BlockRangeConfigurationError for block range validation * chore(subsquid-pipes): update release notes * fix(subsquid-pipes): use case-insensitive match for portal timestamp error Co-Authored-By: Claude Opus 4.6 <[email protected]> --------- Co-authored-by: Claude Opus 4.6 <[email protected]>

* refactor!: naming and ergonomics overhaul BREAKING CHANGE: Comprehensive rename of the public API surface. Types: - ResultOf<T> → OutputOf<T> - Added SingleOutput / MultiOutput type aliases - BatchCtx → BatchContext with internals nesting (dataset, head, state, meta, query under ctx.internals) - TransformerFn removed from public exports - RunConfig → PipeContext - FactoryOptions → ContractFactoryOptions - Added EventFilter<T> union type Functions: - evmPortalSource() → evmPortalStream() - factory() → contractFactory() - factorySqliteDatabase() → contractFactoryStore() - chunk() → batchForInsert() - readAll() made public (removed @internal) Options fields: - id is now required in all portal sources - parameter → childAddressField in factory options - contracts → contractFactory in evmDecoder options - stream → handler in runner StreamConfig New features: - Declarative preindex: set preindex: true in contractFactory options instead of calling imperative preindex() method - pipe()/pipeTo() misuse guards with diagnostic error messages Co-Authored-By: Claude Opus 4.6 <[email protected]> * test: add missing acceptance criteria tests for naming refactor Add 11 tests covering Phase 2 (pipe/pipeTo guards), Phase 4 (EventFilter type validation and declarative preindex behavior) acceptance criteria that were not covered by the initial refactor. Co-Authored-By: Claude Opus 4.6 <[email protected]> * style: apply biome fixes across docs and packages Co-Authored-By: Claude Opus 4.6 <[email protected]> --------- Co-authored-by: Claude Opus 4.6 <[email protected]> Co-authored-by: Eugene Formanenko <[email protected]>

Co-Authored-By: Claude Opus 4.6 <[email protected]>

…ith single maxBytes (#55) Remove the minBytes option and use maxBytes as both the flush threshold and backpressure limit. Rename internal futures for clarity (flushSignal, consumedSignal, hasDataSignal). Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>

…packaging scripts - Remove deprecated `metrics.ts` for cleanup. - Upgrade `@biomejs/biome` to `v2.4.8`. - Adjust package scripts and rename `bin.js` to `cli.js`. - Add end-to-end test script for package validation. - Update package metadata and versioning for `@subsquid/pipes` and `@subsquid/pipes-ui`. - Refine UI for connection status and version display.

Copilot

Pull request overview

This PR is a broad “SDK 1.0” update that introduces a new declarative “outputs/streams” model for portal sources and query-aware transformers, expands testing utilities, and migrates the Pipe UI to Next.js while upgrading/reshaping several core APIs.

Changes:

Introduces *PortalStream sources with declarative outputs and new QueryAwareTransformer-based query builders (evmQuery, solanaQuery, hyperliquidFillsQuery).
Refactors core transformer/query/pipeline context structures (BatchContext, ctx.stream.*) and adds new testing helpers + EVM mock block utilities.
Migrates pipe-ui from React Router/Vite to Next.js standalone packaging and adds new runtime/dev-runner + optional OpenTelemetry profiler hooks.

Reviewed changes

Copilot reviewed 185 out of 188 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
rfc/01.STREAMS.md	Adds RFC describing the new declarative outputs/streams/query model.
rfc/00.CHANGES.md	Summarizes SDK API pain points and references the streams proposal.
packages/subsquid-pipes/vitest.config.ts	Minor formatting/blank-line change.
packages/subsquid-pipes/tsconfig.json	Adds `$context` path alias for runtime context module.
packages/subsquid-pipes/src/tests/test-block-stream.ts	Removes old test helper (moved to new testing module).
packages/subsquid-pipes/src/tests/index.ts	Removes old tests barrel exports (replaced by `src/testing`).
packages/subsquid-pipes/src/testing/test-portal.ts	Enhances mock portal (close method, richer metadata, REST API mock, `readAll` typing).
packages/subsquid-pipes/src/testing/test-metrics-server.ts	Adds mock metrics server for tests.
packages/subsquid-pipes/src/testing/test-block-stream.ts	Adds internal block decoder built on `evmQuery`.
packages/subsquid-pipes/src/testing/index.ts	New testing barrel exports.
packages/subsquid-pipes/src/testing/evm/mock-block.ts	Adds EVM portal-format block builder + event encoding helpers.
packages/subsquid-pipes/src/testing/evm/index.ts	Exports EVM testing utilities.
packages/subsquid-pipes/src/testing/evm/evm-portal-mock-stream.ts	Adds convenience mock portal stream for EVM blocks.
packages/subsquid-pipes/src/targets/memory/memory-target.ts	Updates target context access to `ctx.stream.*`.
packages/subsquid-pipes/src/targets/drizzle/node-postgres/utils.ts	Renames `chunk()` to `batchForInsert()` and deprecates the old name.
packages/subsquid-pipes/src/targets/drizzle/node-postgres/postgres-state.ts	Updates to `BatchContext` and new `ctx.stream.*` structure.
packages/subsquid-pipes/src/targets/drizzle/node-postgres/drizzle-target.ts	Updates context usage to `ctx.stream.*` and profiler naming.
packages/subsquid-pipes/src/targets/clickhouse/clickhouse-state.ts	Updates to `BatchContext` and new `ctx.stream.*` structure.
packages/subsquid-pipes/src/solana/solana-rpc-latency-watcher.ts	Reworks watcher to become a query-aware stream (`solanaQuery().build().pipe(...)`).
packages/subsquid-pipes/src/solana/solana-query-builder.ts	Adds `build()` returning `QueryAwareTransformer` and updates typing helpers.
packages/subsquid-pipes/src/solana/solana-query-builder.test.ts	Updates tests to use `mockPortalRestApi`.
packages/subsquid-pipes/src/solana/solana-portal-source.ts	Introduces `solanaPortalStream(id, outputs, ...)` and merges outputs.
packages/subsquid-pipes/src/solana/solana-portal-source.test.ts	Updates tests for `solanaPortalStream` and new data shape.
packages/subsquid-pipes/src/solana/solana-instruction-decoder.ts	Rebuilds decoder as query-aware output via `solanaQuery().build().pipe(...)`.
packages/subsquid-pipes/src/solana/solana-instruction-decoder.test.ts	Updates test setup to new testing helpers and stream API.
packages/subsquid-pipes/src/runtime/node/runner.ts	Adds a dev runner to run multiple pipes concurrently with retries + shared metrics.
packages/subsquid-pipes/src/runtime/node/index.ts	Exports runtime node modules.
packages/subsquid-pipes/src/runtime/node/context.ts	Implements runtime context via AsyncLocalStorage.
packages/subsquid-pipes/src/runtime/node/browser_context.ts	Adds browser stub for runtime context.
packages/subsquid-pipes/src/portal-client/query/index.ts	Removes `createQuery`, tweaks exports, adds FIXME note.
packages/subsquid-pipes/src/portal-client/query/evm.ts	Changes transaction nonce type to `bigint` and validator to accept larger inputs.
packages/subsquid-pipes/src/portal-client/query/evm.test.ts	Adds nonce casting/validation tests.
packages/subsquid-pipes/src/portal-client/client.ts	Updates metadata API typing, adds timestamp resolver, refactors stream buffering.
packages/subsquid-pipes/src/opentelemetry/opentelemetry-profiler.ts	Adds OTEL span exporter hooks implementing `SpanHooks`.
packages/subsquid-pipes/src/opentelemetry/index.ts	Exports OTEL profiler helper.
packages/subsquid-pipes/src/monitoring/rpc-latency/rpc-latency-watcher.ts	Refactors watcher into pure transformer (query handled by query builders).
packages/subsquid-pipes/src/monitoring/rpc-latency/index.ts	Exports ws-client from monitoring module.
packages/subsquid-pipes/src/hyperliquid/hyperliquid-fills-query-builder.ts	Adds `build()` returning `QueryAwareTransformer` and updates typing.
packages/subsquid-pipes/src/hyperliquid/hyperliquid-fills-query-builder.test.ts	Updates tests to use `mockPortalRestApi`.
packages/subsquid-pipes/src/hyperliquid/hyperliquid-fills-portal-source.ts	Introduces `hyperliquidFillsPortalStream(id, outputs, ...)`.
packages/subsquid-pipes/src/hyperliquid/hyperliquid-fills-portal-source.test.ts	Updates for new stream API and new data shape.
packages/subsquid-pipes/src/evm/factory.ts	Renames/reshapes factory API (`contractFactory`, `childAddressField`) and updates preindex flow.
packages/subsquid-pipes/src/evm/factory-adapters/sqlite.ts	Renames adapter helper to `contractFactoryStore` + improves generics.
packages/subsquid-pipes/src/evm/evm-rpc-latency-watcher.ts	Refactors watcher into query-aware pipeline (`evmQuery().build().pipe(...)`).
packages/subsquid-pipes/src/evm/evm-query-builder.ts	Adds `build()` returning `QueryAwareTransformer` and exports `EvmPortalData`.
packages/subsquid-pipes/src/evm/evm-query-builder.test.ts	Expands tests for timestamp ranges and range validation.
packages/subsquid-pipes/src/evm/evm-portal-source.ts	Introduces `evmPortalStream(id, outputs, ...)` and merges outputs.
packages/subsquid-pipes/src/evm/evm-portal-source.test.ts	Updates tests for new stream API and new data shape.
packages/subsquid-pipes/src/evm/abi/define-abi.bench.ts	Adds benchmarks comparing event decoding implementations.
packages/subsquid-pipes/src/evm/abi/common.ts	Exposes `defineAbi` types and helper.
packages/subsquid-pipes/src/core/types.ts	Adds `OutputOf` helper and `Subset` type utility.
packages/subsquid-pipes/src/core/transformer.ts	Major refactor: removes query hook from transformers, adds `QueryAwareTransformer`, new profiling naming.
packages/subsquid-pipes/src/core/target.ts	Removes unused `Ctx` import and minor formatting.
packages/subsquid-pipes/src/core/query-builder.ts	Adds timestamp-aware ranges + `resolveTimestamp` support and introduces query-aware build contract.
packages/subsquid-pipes/src/core/portal-range.ts	Adds Date/date-string parsing and validation rules.
packages/subsquid-pipes/src/core/output.ts	Adds `Outputs` model and `mergeOutputs()` transformer.
packages/subsquid-pipes/src/core/metrics-server.ts	Updates metrics server interface (`registerPipe`, `batchProcessed`) and histogram label API.
packages/subsquid-pipes/src/core/logger.ts	Adds per-pipe `id` base field + prettier formatting options.
packages/subsquid-pipes/src/core/index.ts	Exports new modules (`errors`, `output`, `SpanHooks`).
packages/subsquid-pipes/src/core/errors.ts	Adds structured SDK error types with doc links and error codes.
packages/subsquid-pipes/src/core/composite-transformer.ts	Removes old composite transformer (replaced by outputs merge).
packages/subsquid-pipes/src/core/aggregator.ts	Updates profiler naming and `ctx.stream.*` access.
packages/subsquid-pipes/scripts/fix-imports.ts	Adds rewrite rule for `$context` alias.
packages/subsquid-pipes/package.json	Bumps version, adds new exports (`runtime/node`, `opentelemetry`, `testing`) and new deps/peers.
packages/subsquid-pipes/README.md	Updates docs to `evmPortalStream` and profiler naming.
packages/pipe-ui/vite.config.ts	Removes Vite config (migration away from Vite).
packages/pipe-ui/tsconfig.json	Switches TypeScript config to Next.js conventions.
packages/pipe-ui/scripts/package.js	Reworks packaging to Next standalone output.
packages/pipe-ui/scripts/e2e-test.sh	Adds e2e packaging test script for the UI tarball.
packages/pipe-ui/scripts/bin.js	Replaces express static server with executing Next standalone server.
packages/pipe-ui/react-router.config.ts	Removes React Router config (migration complete).
packages/pipe-ui/postcss.config.mjs	Adds Tailwind PostCSS config for Next.
packages/pipe-ui/package.json	Renames package and switches scripts/deps to Next.js stack.
packages/pipe-ui/next.config.ts	Adds Next standalone output config.
packages/pipe-ui/components.json	Enables RSC in shadcn config.
packages/pipe-ui/app/routes/home.tsx	Removes React Router route component.
packages/pipe-ui/app/routes.ts	Removes React Router routes config.
packages/pipe-ui/app/root.tsx	Removes React Router root wrapper.
packages/pipe-ui/app/page.tsx	Adds Next app router page entry.
packages/pipe-ui/app/lib/config.ts	Adds YAML-driven server config loader.
packages/pipe-ui/app/layout.tsx	Adds Next root layout and font preconnects.
packages/pipe-ui/app/hooks/use-servers.ts	Adds hook to fetch configured metrics servers.
packages/pipe-ui/app/hooks/use-server-context.ts	Adds context for selected metrics server index.
packages/pipe-ui/app/hooks/use-portal.ts	Switches portal status fetching to Next API proxy route.
packages/pipe-ui/app/dashboard/server-selector.tsx	Adds server selector UI for multiple metrics servers.
packages/pipe-ui/app/dashboard/query-exemplar.tsx	Refactors query panel to Next client component and adds cURL view.
packages/pipe-ui/app/dashboard/profiler.tsx	Refactors profiler panel to new hooks + loading state.
packages/pipe-ui/app/dashboard/pipeline-disconnected.tsx	Replaces raw import with embedded example and uses static docs URL.
packages/pipe-ui/app/dashboard/panel-loading.tsx	Adds reusable loading panel component.
packages/pipe-ui/app/dashboard/formatters.ts	Updates ETA formatter to use pipe status + pipe object.
packages/pipe-ui/app/dashboard/dashboard.tsx	Refactors dashboard to use server selection and per-pipe selection.
packages/pipe-ui/app/dashboard/dashboard-skeleton.tsx	Adds skeleton loading UI.
packages/pipe-ui/app/dashboard/code.example	Removes external example file (now inlined).
packages/pipe-ui/app/components/ui/toggle.tsx	Marks as Next client component.
packages/pipe-ui/app/components/ui/tabs.tsx	Marks as Next client component and updates imports/styles.
packages/pipe-ui/app/components/ui/switch.tsx	Marks as Next client component + formatting update.
packages/pipe-ui/app/components/ui/separator.tsx	Adds separator component for Next client usage.
packages/pipe-ui/app/components/ui/select.tsx	Adds select component for Next client usage.
packages/pipe-ui/app/components/ui/copy-button.tsx	Marks as Next client component.
packages/pipe-ui/app/components/ui/code.tsx	Marks as Next client component.
packages/pipe-ui/app/components/ui/button.tsx	Marks as Next client component.
packages/pipe-ui/app/components/theme-provider.tsx	Makes theme provider SSR-safe for Next.
packages/pipe-ui/app/components/providers.tsx	Adds Next client Providers wrapper for React Query + Theme.
packages/pipe-ui/app/app.css	Updates global styles, fonts, and skeleton/content animations.
packages/pipe-ui/app/api/servers/route.ts	Adds API route for configured server list.
packages/pipe-ui/app/api/portal/route.ts	Adds API route proxying portal status endpoint.
packages/pipe-ui/app/api/metrics/[...path]/route.ts	Adds API route proxying metrics server endpoints by server index.
packages/pipe-ui/app/api/metrics.ts	Removes axios-based metrics hooks (replaced by Next routes + hooks).
packages/pipe-ui/app/api/client.ts	Removes axios client helper.
packages/pipe-ui/.gitignore	Updates ignores for Next build outputs.
packages/clickhouse-ui/src/app/page.tsx	Adds redirect page in clickhouse-ui Next app.
package.json	Updates Biome version.
docs/vitest.config.ts	Adds vitest config for docs examples.
docs/package.json	Adds docs testing deps and OTEL/viem/date-fns for examples.
docs/examples/solana/abi/tokenProgram/types.ts	Reformats generated code to project style.
docs/examples/solana/abi/tokenProgram/instructions.ts	Minor import ordering/formatting.
docs/examples/solana/abi/raydium-cpmm/types.ts	Minor import ordering/formatting.
docs/examples/solana/abi/raydium-cpmm/instructions.ts	Minor import ordering/formatting.
docs/examples/solana/abi/raydium-clmm/types.ts	Minor import ordering/formatting.
docs/examples/solana/abi/raydium-clmm/instructions.ts	Minor import ordering/formatting.
docs/examples/solana/abi/raydium-amm/types.ts	Minor import ordering/formatting.
docs/examples/solana/abi/raydium-amm/instructions.ts	Minor import ordering/formatting.
docs/examples/solana/abi/orca_whirlpool/types.ts	Minor import ordering/formatting.
docs/examples/solana/abi/orca_whirlpool/instructions.ts	Minor import ordering/formatting.
docs/examples/solana/abi/meteora-dlmm/types.ts	Minor import ordering/formatting.
docs/examples/solana/abi/meteora-dlmm/instructions.ts	Minor import ordering/formatting.
docs/examples/solana/abi/meteora-damm/types.ts	Minor import ordering/formatting.
docs/examples/solana/abi/meteora-damm/instructions.ts	Minor import ordering/formatting.
docs/examples/solana/abi/abi.support.ts	Fixes import order/formatting.
docs/examples/solana/07.indexing-latency.example.ts	Updates example for `solanaPortalStream` + new outputs model.
docs/examples/solana/02.swaps.example.ts	Updates example for `solanaPortalStream` + outputs map.
docs/examples/hyperliquid/01.user-fills.example.ts	Updates example for `hyperliquidFillsPortalStream` + new data shape.
docs/examples/evm/decoders/uniswap-v3.ts	Updates for `contractFactory` + profiler naming.
docs/examples/evm/decoders/erc20-transfers.ts	Updates profiler naming (`name`) and pipe chain usage.
docs/examples/evm/99.factory-pre-index.example.ts	Removes test-only example for experimental preindex.
docs/examples/evm/13.jaeger-tracing.example.ts	Adds example showing OTEL tracing integration.
docs/examples/evm/12.runner.example.ts	Adds example demonstrating dev runner usage.
docs/examples/evm/11.simple.decoder.example.ts	Adds example showing `OutputOf` typing + multi-output usage.
docs/examples/evm/10.factory-event-filtering.example.ts	Updates example to `contractFactory` + `evmPortalStream`.
docs/examples/evm/09.filtering-by-event-params.example.ts	Updates example to `evmPortalStream` + outputs.
docs/examples/evm/08.drizzle.example.ts	Updates to `evmPortalStream` and `batchForInsert`.
docs/examples/evm/07.indexing-latency.example.ts	Updates example to `evmPortalStream` + outputs.
docs/examples/evm/06.custom-metrics.example.ts	Updates example to `evmPortalStream` + outputs.
docs/examples/evm/05.custom-logs-transport.example.ts	Updates example to `evmPortalStream` + outputs.
docs/examples/evm/04.clickhouse.example.ts	Updates example to `evmPortalStream` + outputs.
docs/examples/evm/03.factory.example.ts	Updates example to `contractFactory` + `contractFactoryStore`.
docs/examples/evm/02.combining-pipes.example.ts	Updates example to new outputs map and `contractFactoryStore`.
docs/examples/evm/01.evm-decoder.example.ts	Updates example to `evmPortalStream`, adds metrics server usage.
biome.json	Updates schema version and adds restricted import rule for runtime context.
README.md	Updates root README usage to `evmPortalStream` and profiler naming.
.claude/skills/typescript-vitest	Adds skill link file.
.claude/skills/typescript-pino-logger	Adds skill link file.
.claude/skills/typescript-code-style	Adds skill link file.
.claude/skills/typescript-biome	Adds skill link file.
.claude/skills/typescript-advanced-types	Adds skill link file.
.agents/skills/typescript-pino-logger/SKILL.md	Adds internal skill documentation.
.agents/skills/typescript-biome/SKILL.md	Adds internal skill documentation.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2026-03-23T14:22:43Z

+    <ServerContext value={{ serverIndex, setServerIndex }}>
+      <div className="flex flex-col items-center pt-16 pb-4 gap-10 content-fade-in">


ServerContext is a context object returned by createContext, not a React component. In React, you must render ServerContext.Provider to provide a value; <ServerContext value={...}> will not work and should fail type-check/build. Change this to <ServerContext.Provider value={{ serverIndex, setServerIndex }}> (and update the closing tag accordingly).

Copilot · 2026-03-23T14:22:44Z

+  return undefined
+}
+
+async function runWithContext(ctx: any, fn: () => Promise<void>) {


The browser stub for the runtime context does not export runWithContext, but the Node implementation exports it and consumers will expect the same named exports via conditional exports. Export runWithContext from this module (matching the signature) so the ./runtime/node/context export surface is consistent in browser builds.

Suggested change

async function runWithContext(ctx: any, fn: () => Promise<void>) {

export async function runWithContext(ctx: any, fn: () => Promise<void>) {

Copilot · 2026-03-23T14:22:44Z

+              const worker = new Worker(new URL('worker.ts', import.meta.url).href, {
+                env: {
+                  ...process.env,
+                  PORT: '3333',
+                },
+              })
+
+              await new Promise<void>((resolve, reject) =>
+                worker.addEventListener('close', (event) => {
+                  if (event.code !== 0) {
+                    reject(new Error(`Worker stopped with exit code ${event.code}`))
+                  }
+
+                  resolve()
+                }),
+              )


This worker-thread branch appears to use the Node worker_threads API, but Worker is not imported and the event wiring is incorrect for worker_threads (it uses EventEmitter-style events like 'exit'/'error', not addEventListener('close', ...)). Import Worker from node:worker_threads and listen for 'exit' (and 'error') to reliably detect worker termination.

Copilot · 2026-03-23T14:22:44Z

+export async function GET(request: NextRequest) {
+  const host = request.nextUrl.searchParams.get('host')
+
+  if (!host) {
+    return NextResponse.json({ error: 'Missing host parameter' }, { status: 400 })
+  }
+
+  const targetUrl = `${host}/status`
+
+  try {


This endpoint proxies an arbitrary host provided by the client and performs a server-side fetch to it, which is a classic SSRF vector (can reach internal network resources, metadata services, etc.). Restrict host to an allowlist (e.g., only values from config.metrics_server_url or a dedicated allowlist), validate scheme/hostname with new URL(host), and reject non-HTTP(S) or private/IP-literal targets.

Suggested change

export async function GET(request: NextRequest) {

const host = request.nextUrl.searchParams.get('host')

if (!host) {

return NextResponse.json({ error: 'Missing host parameter' }, { status: 400 })

}

const targetUrl = `${host}/status`

try {

// Allowlist of backend origins that this endpoint is permitted to call.

// Populate via environment variables to avoid hard-coding secrets or deployment-specific hosts.

const ALLOWED_HOSTS = new Set<string>(

[

process.env.METRICS_SERVER_URL,

process.env.PORTAL_BACKEND_URL,

].filter((value): value is string => typeof value === 'string' && value.length > 0),

)

function isIpAddress(hostname: string): boolean {

// Simple IPv4 check

const ipv4Match = hostname.match(/^(\d{1,3}\.){3}\d{1,3}$/)

if (ipv4Match) {

const parts = hostname.split('.').map(Number)

if (parts.every((part) => part >= 0 && part <= 255)) {

return true

}

}

// Simple IPv6 check (compressed or full). We reject all IP literals, so a loose check is sufficient.

if (hostname.includes(':')) {

return true

}

return false

}

function getValidatedTargetUrl(hostParam: string | null): string | null {

if (!hostParam) {

return null

}

let parsed: URL

try {

parsed = new URL(hostParam)

} catch {

return null

}

if (parsed.protocol !== 'http:' && parsed.protocol !== 'https:') {

return null

}

if (isIpAddress(parsed.hostname)) {

return null

}

if (ALLOWED_HOSTS.size > 0) {

const isAllowed = Array.from(ALLOWED_HOSTS).some((allowed) => {

try {

const allowedUrl = new URL(allowed)

return allowedUrl.origin === parsed.origin

} catch {

return false

}

})

if (!isAllowed) {

return null

}

}

const target = new URL('/status', parsed)

return target.toString()

}

export async function GET(request: NextRequest) {

const hostParam = request.nextUrl.searchParams.get('host')

const targetUrl = getValidatedTargetUrl(hostParam)

if (!targetUrl) {

return NextResponse.json({ error: 'Invalid host parameter' }, { status: 400 })

}

try {

Copilot · 2026-03-23T14:22:45Z

+function toDate(value: string): Date {
+  // Treat date-only strings as UTC (e.g., "2024-01-01" → "2024-01-01T00:00:00Z")
+  const date = new Date(value.includes('T') ? value : `${value}T00:00:00Z`)
+  if (Number.isNaN(date.getTime())) {
+    throw new Error(`Invalid date value: "${value}"`)
+  }
+  return date
+}


User-facing range parsing errors should use the SDK's structured error type (BlockRangeConfigurationError) so they carry the correct error code/doc link and consistent messaging. Replace the generic Error thrown here with BlockRangeConfigurationError (and consider making the message actionable, e.g. expected formats).

Copilot · 2026-03-23T14:22:45Z

+
+## Proposed solution: Declarative Streams
+
+Introduce a new `streams` field in `evmPortalSource` configuration.


The RFC text says the config field is streams (line 64), but the examples and the rest of the doc use outputs. This is confusing for readers trying to map the proposal to the API. Align terminology throughout (either rename the field in examples or update the prose). Also fix the typo transofrmation → transformation.

Copilot · 2026-03-23T14:22:45Z

+        "noRestrictedImports": {
+          "level": "error",
+          "options": {
+            "patterns": [
+              {
+                "group": ["~/runtime/context/*", "../**/runtime/context", "../**/runtime/context/*"],
+                "message": "Use $runtime/context instead. This module is platform-specific and is loaded via conditional exports to avoid pulling the Node-only async_hooks dependency into browser builds."
+              }
+            ]
+          }
+        }


The restricted-import rule instructs developers to use $runtime/context, but the PR introduces a $context tsconfig alias and the import-fixer maps '$context' to @subsquid/pipes/runtime/node/context. The guidance message should match the actual supported alias to avoid churn and confusion (update the message and/or the alias naming to be consistent).

Copilot · 2026-03-23T14:22:46Z

+        const res = {} as any
+        for (const key in output) {
+          res[key] = await output[key].run(data, ctx)


mergeOutputs() runs each output transformer sequentially, which can significantly slow down pipelines when outputs are independent (e.g., multiple decoders over the same batch). Consider running these in parallel with Promise.all (and then assembling the result object). If ordering/side-effects matter, document that constraint explicitly and keep sequential execution; otherwise parallelizing is usually the better default.

Suggested change

const res = {} as any

for (const key in output) {

res[key] = await output[key].run(data, ctx)

const entries = await Promise.all(

Object.entries(output).map(async ([key, transformer]) => {

const value = await transformer.run(data, ctx)

return [key, value] as const

}),

)

const res = {} as any

for (const [key, value] of entries) {

res[key] = value

Copilot · 2026-03-23T14:22:46Z

  if (range.from === 'latest') {
-    return { from: 'latest', to: range.to ? parseBlock(range.to) : undefined }
+    if (range.to instanceof Date || (typeof range.to === 'string' && isDateString(range.to))) {
+      throw new BlockRangeConfigurationError(
+        "Cannot use a Date for 'to' when 'from' is 'latest'. The portal cannot resolve a timestamp to a block number for blocks that have not been produced yet. Use a block number instead.",
+      )
+    }
+    const to = range.to ? parseBlock(range.to) : undefined
+    return { from: 'latest', to }
  }


New range parsing/validation behavior is introduced here (notably: rejecting from: 'latest' with a Date/date-string to). There are tests for latest and timestamp resolution in the query-builder layer, but this specific validation path in parsePortalRange() should also be covered with a focused unit test to prevent regressions.

) * refactor: extract StreamBuffer and splitLines from portal client into separate modules Add comprehensive tests for StreamBuffer (29 tests) covering buffering, backpressure, flush strategies, close/fail lifecycle, and async iteration. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * fix: guard idle timeout on empty buffer and enable streaming TextDecoder - StreamBuffer: only arm idle timeout when buffer is non-empty, preventing stale timers after backpressured flushes - LineSplitter: use TextDecoder streaming mode to handle multi-byte chars split across chunk boundaries Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>

…multi-decoder isolation test (#59) - Pass `config` (serializable isolation level) to the fork transaction in drizzle-target, matching the write and trigger setup paths - Add test verifying two evmDecoders with different contracts correctly isolate events in a shared portal stream Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>

Cover mergeRangeRequests, applyRangeBound, rangeIntersection, and rangeDifference with 35 tests. Remove the TODO comment requesting tests. Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>

* chore: remove dead code and add pr-quality-gate skill - Remove unused aggregator.ts, factory-adapters/errors.ts, runtime context module - Remove runWithContext wrapper from runner (unused after context.ts removal) - Fix indentation in clickhouse/fs.ts - Add pr-quality-gate skill for automated test coverage checks on PRs - Add CLAUDE.md with project conventions Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> * feat(pipe-ui): profiler flame chart + data flow visualization Profiler tab: - Heat-colored tree view (violet→fuchsia→amber→red by time %) - Colored left borders and visible background bars - Self-time toggle via shadcn Switch Data samples tab — redesigned as vertical flow pipeline: - Shape badges (Array · 5556 items, Object · 1 collections) - Change indicators (↓ passthrough, ↓ reshape Array[282]→Object, ↓ transform) - Sibling shape chaining for correct change detection - Field pills with nested array counts (transactions · 11) - New fields highlighted green (+ traces, + stateDiffs) - Side-by-side diff view for reshape stages (INPUT → OUTPUT) - Core stages labeled via API with blue dots and reduced opacity - Batch blocks badge on root card, timing with percentage - Tab and fullscreen state persisted via URL params - Child stages indented for hierarchy Backend: - Add elapsed, dataSize, labels to TransformationResult API - Add bytesSize to batch info in exemplar endpoint - Add 'core' label to batch, fetch data, apply transformers, metrics processing spans - Expose labels on Profiler interface Code component: - Custom syntax theme (purple keys, green strings, blue numbers) Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>

getInstructionD2 and getInstructionD4 used wrong slice offsets for 0x-prefixed hex strings, extracting 3/6 bytes instead of 2/4, causing silent zero matches for programs using 2-byte or 4-byte discriminators. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

…o 1.0.0-alpha.5 Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

…a tsup Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

…mp alpha.3 - Added `block: { number, hash }` to Solana DecodedInstruction, deprecated `blockNumber` - version.ts reads package.json at runtime via createRequire (no more tsup inlining) - Removed worker_threads logic from runner.ts - Bumped to 1.0.0-alpha.3 - Updated release notes Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

) saveCursor was running SELECT * FROM sync FINAL on every batch, which forces a full CollapsingMergeTree merge scan and was taking ~7 seconds per batch. This mirrors the Postgres pattern (postgres-state.ts) which only cleans up every 25 saves. Adds a #saves counter. Cleanup now runs on the first save and then every min(25, maxRows) saves - the min() fallback keeps tiny-maxRows test cases converging. Bumps @subsquid/pipes to 1.0.0-alpha.4. Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>

Shared hooks and URL sync: - Add useUrlParam hook and migrate server selector, pipeline tab and exemplar fullscreen to URL query params so they survive reloads. Wrap <Dashboard /> in <Suspense> (Next 15 useSearchParams requirement). - Add useLocalStorage hook for persisting UI toggles across sessions. - Clamp an out-of-range ?server= index to 0 once servers load so a stale URL can't cause /api/metrics/* to 400. Profiler: - Re-integrate FlameChart alongside the tree view, gated by a segmented Tree / Flame view toggle in the profiler header. - Rename "Self time" → "Exclude children" (and the matching calcStats option) for clarity. Persist the toggle. - Segmented Total / Avg time mode so the cumulative-vs-per-batch distinction is explicit. Tree nodes, flame cells and tooltips render `avg. 0.06ms` in avg mode. Header reads `Σ over N batches` or `avg over N batches`. - Hide Exclude-children and Time controls behind a Settings gear popover with click-outside / Escape to close and inline help text explaining each option. - Fall back to the first pipe on the active server when the stored selection isn't there so switching servers no longer shows a false "pipe is offline" panel. Visual improvements (ported from feat/polymarket): - code.tsx: richer syntax highlight theme and roomier padding. - pipeline-disconnected.tsx: tightened getting-started layout with an inline docs link. - Optional `labels?: string[]` on profiler nodes, rendered as chips in the tree view and flame chart tooltip. Housekeeping: - Bump react/react-dom to 19.2.5 (+ matching @types). - Bump @subsquid/pipes-ui to 1.0.0-alpha.6. Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>

- ProfilerOptions.labels lets spans be tagged (e.g. core, db) for UI filtering - core spans (start/stop/fork roots, transformers, target_rollback) carry the core label - clickhouse/drizzle targets wrap each batch in a labeled span and emit granular insert cursor / cleanup cursors / cleanup snapshots sub-spans - bump @subsquid/pipes to 1.0.0-alpha.5 Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>

alpha.4/alpha.5 silently shipped a stale dist from the alpha.3 epoch because nothing forced a rebuild on publish and pnpm build was broken on a fresh checkout. As a result, PR #62 (clickhouse debounce cleanup) and the clickhouse half of PR #64 (profiler labels/granular spans) never reached npm even though gitHead pointed at them. - add prepublishOnly → pnpm build so npm/pnpm publish can never ship whatever happens to be sitting in dist/ - declare @babel/parser as an explicit root devDependency; recast's typescript parser requires it and pnpm strict isolation hid the transitive copy from fix-imports.ts, making pnpm build fail - fix type errors in clickhouse-target.test.ts introduced by PR #62 (MockResponse discriminated union needed an explicit annotation, and the spyOn this-type clashed with a direct ClickhouseStore cast); the .d.ts build has been failing on main since that PR - bump @subsquid/pipes to 1.0.0-alpha.6 so the next publish actually carries the missing clickhouse changes Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

- Expose `started` on the `Profiler` interface (was only on `Span`) - Serialize each span's `startOffset` (ms relative to root) in the `/profiler` and `/exemplars/transformation` payloads - pipe-ui: carry `startOffset` through `calcStats` aggregation and rewrite `flattenByDepth` to position children by real offset instead of a running duration accumulator - Legacy fallback kept for older SDK payloads with no offsets - Bump @subsquid/pipes and @subsquid/pipes-ui to 1.0.0-alpha.7 Root cause: `Span.started` was captured but never serialized, so the UI was forced to fabricate child positions from `(child.totalTime / siblingTotal)`. This hid parent self-time and misaligned nested children. Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>

mo4islona force-pushed the design/sdk1 branch 4 times, most recently from 2a413ca to ec7324a Compare January 21, 2026 09:04

mo4islona added 4 commits January 24, 2026 15:47

feat: introduce declarative streams API for evmPortalSource in Pipes SDK

f7e78d8

feat: update evmPortalSource to use streams for decoders and transfor…

9a3f9d2

…mers

docs: update 01.STREAMS.md with latest changes and clarifications

464870a

feat: implement runner PoC and context management for stream processing

82eeb60

mo4islona force-pushed the design/sdk1 branch from d7e7094 to 82eeb60 Compare January 24, 2026 07:50

mo4islona added 2 commits January 24, 2026 15:54

fix: broken tests

cc3eb4f

feat: enhance decoder and transformer types for improved type extract…

9975ac8

…ion and clarity

mo4islona force-pushed the design/sdk1 branch from f0abb84 to 2d34de4 Compare January 27, 2026 04:10

feat: refactor queries

8fe514a

mo4islona force-pushed the design/sdk1 branch from 2d34de4 to 8fe514a Compare January 27, 2026 04:28

mo4islona and others added 6 commits January 27, 2026 10:25

feat: update progress tracking types and logger integration in portal…

07e44dd

… sources

feat: enhance logger and portal source with optional ID parameter for…

63754eb

… improved context management

feat: update EVM query structure and enhance output definitions for i…

0cc7b74

…mproved clarity and transformation capabilities

fix(pipes): factory event params type (#40)

cf20e24

feat: refactor Solana instruction decoder and update output structure…

ae48643

… for improved clarity

feat: enhance metrics and pipeline components with pipe-specific data…

0f56584

… handling and improved state management

mo4islona force-pushed the design/sdk1 branch from 622b0ca to 91259ce Compare February 15, 2026 22:32

mo4islona force-pushed the design/sdk1 branch from 91259ce to bf6e83a Compare February 15, 2026 22:34

mo4islona and others added 7 commits February 16, 2026 12:43

feat: add registerPipe method to MetricsServer and update transformer…

06a9619

… method names for consistency

feat: enhance pipe management with status tracking and history handling

4c92d83

feat: improve stats handling and pipe status management in metrics an…

06bfce7

…d dashboard components

feat: add portal-source metrics and fix duplicate HistogramConfiguration

e944c47

Add batch and reorg metrics to portal-source (sqd_reorgs_total, sqd_batch_size_blocks, sqd_batch_size_bytes). Remove duplicate HistogramConfiguration interface in metrics-server.

feat: improve stats handling and pipe status management in metrics an…

dbb6a26

…d dashboard components

mo4islona added 2 commits February 27, 2026 18:20

feat(ui): adjust code display height and improve portal status visibi…

bf2e50c

…lity

mo4islona force-pushed the design/sdk1 branch from 7435cd4 to bf2e50c Compare February 27, 2026 21:03

shestakoven and others added 11 commits March 2, 2026 09:45

add rpc calls examples

a7d3635

feat(metrics): implement server status and metrics retrieval endpoints

cb4922d

lint

ee30d9e

Merge pull request #49 from subsquid-labs/rpc-call-examples

81c69a1

add rpc calls examples

chore(examples): remove deprecated EVM factory pre-indexing example

b2bf7d7

feat(ui): move connection status and version into server select

10bfbde

Co-Authored-By: Claude Opus 4.6 <[email protected]>

Change transaction.nonce to bigint (#56)

a331ce8

mo4islona marked this pull request as ready for review March 23, 2026 14:19

Copilot AI review requested due to automatic review settings March 23, 2026 14:19

Copilot AI reviewed Mar 23, 2026

View reviewed changes

mo4islona and others added 13 commits March 27, 2026 15:56

test: add unit tests for query-builder range utilities (#60)

790522e

Cover mergeRangeRequests, applyRangeBound, rangeIntersection, and rangeDifference with 35 tests. Remove the TODO comment requesting tests. Co-authored-by: Claude Opus 4.6 (1M context) <[email protected]>

chore: bump @subsquid/pipes to 1.0.0-alpha.2 and @subsquid/pipes-ui t…

fc0470e

…o 1.0.0-alpha.5 Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

fix: read version from package.json at runtime instead of inlining vi…

0f60238

…a tsup Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SDK 1.0#37

SDK 1.0#37
mo4islona wants to merge 95 commits intomainfrom
design/sdk1

mo4islona commented Jan 19, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Mar 23, 2026

Uh oh!

Copilot AI Mar 23, 2026

Uh oh!

Copilot AI Mar 23, 2026

Uh oh!

Copilot AI Mar 23, 2026

Uh oh!

Copilot AI Mar 23, 2026

Uh oh!

Copilot AI Mar 23, 2026

Uh oh!

Copilot AI Mar 23, 2026

Uh oh!

Copilot AI Mar 23, 2026

Uh oh!

Copilot AI Mar 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		<ServerContext value={{ serverIndex, setServerIndex }}>
		<div className="flex flex-col items-center pt-16 pb-4 gap-10 content-fade-in">

	async function runWithContext(ctx: any, fn: () => Promise<void>) {
	export async function runWithContext(ctx: any, fn: () => Promise<void>) {

-export async function GET(request: NextRequest) {
-  const host = request.nextUrl.searchParams.get('host')
-  if (!host) {
-    return NextResponse.json({ error: 'Missing host parameter' }, { status: 400 })
-  }
-  const targetUrl = `${host}/status`
-  try {
+// Allowlist of backend origins that this endpoint is permitted to call.
+// Populate via environment variables to avoid hard-coding secrets or deployment-specific hosts.
+const ALLOWED_HOSTS = new Set<string>(
+  [
+    process.env.METRICS_SERVER_URL,
+    process.env.PORTAL_BACKEND_URL,
+  ].filter((value): value is string => typeof value === 'string' && value.length > 0),
+)
+function isIpAddress(hostname: string): boolean {
+  // Simple IPv4 check
+  const ipv4Match = hostname.match(/^(\d{1,3}\.){3}\d{1,3}$/)
+  if (ipv4Match) {
+    const parts = hostname.split('.').map(Number)
+    if (parts.every((part) => part >= 0 && part <= 255)) {
+      return true
+    }
+  }
+  // Simple IPv6 check (compressed or full). We reject all IP literals, so a loose check is sufficient.
+  if (hostname.includes(':')) {
+    return true
+  }
+  return false
+}
+function getValidatedTargetUrl(hostParam: string | null): string | null {
+  if (!hostParam) {
+    return null
+  }
+  let parsed: URL
+  try {
+    parsed = new URL(hostParam)
+  } catch {
+    return null
+  }
+  if (parsed.protocol !== 'http:' && parsed.protocol !== 'https:') {
+    return null
+  }
+  if (isIpAddress(parsed.hostname)) {
+    return null
+  }
+  if (ALLOWED_HOSTS.size > 0) {
+    const isAllowed = Array.from(ALLOWED_HOSTS).some((allowed) => {
+      try {
+        const allowedUrl = new URL(allowed)
+        return allowedUrl.origin === parsed.origin
+      } catch {
+        return false
+      }
+    })
+    if (!isAllowed) {
+      return null
+    }
+  }
+  const target = new URL('/status', parsed)
+  return target.toString()
+}
+export async function GET(request: NextRequest) {
+  const hostParam = request.nextUrl.searchParams.get('host')
+  const targetUrl = getValidatedTargetUrl(hostParam)
+  if (!targetUrl) {
+    return NextResponse.json({ error: 'Invalid host parameter' }, { status: 400 })
+  }
+  try {


		## Proposed solution: Declarative Streams

		Introduce a new `streams` field in `evmPortalSource` configuration.

-        const res = {} as any
-        for (const key in output) {
-          res[key] = await output[key].run(data, ctx)
+        const entries = await Promise.all(
+          Object.entries(output).map(async ([key, transformer]) => {
+            const value = await transformer.run(data, ctx)
+            return [key, value] as const
+          }),
+        )
+        const res = {} as any
+        for (const [key, value] of entries) {
+          res[key] = value

Conversation

mo4islona commented Jan 19, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Mar 23, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants