Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,9 @@
- **自然语言查数**:基于 LLM + ReAct 工具调用,把自然语言转为 SQL 并在目标库执行,全程流式输出。
- **Python 数据分析**:对查询结果自动执行统计分析(相关性、回归、分布检验),补齐 SQL 在复杂统计计算上的短板。
- **多模型可切换**:内置 OpenAI / Ollama / 通义 DashScope / Anthropic 等提供商,换底座模型不影响已沉淀的业务知识。
- **多数据源(JDBC 抽象)**:数据读取与执行完全基于 JDBC 标准 API,因此支持**任意 JDBC 兼容数据库**——MySQL / PostgreSQL / Oracle 已验证,ClickHouse / SQL Server / 达梦 / OceanBase / SQLite 等只需引入对应驱动即可接入。
- **多数据源(JDBC 抽象)**:数据读取与执行完全基于 JDBC 标准 API,因此支持**任意 JDBC 兼容数据库**——内置类型已覆盖 MySQL / PostgreSQL / Oracle(已验证)与 ClickHouse / SQL Server / 达梦 / OceanBase / SQLite,其余 JDBC 兼容库扩展枚举即可接入。

> **默认仅内置 MySQL 驱动**。使用其他数据库前,需先在 `data-agent-backend/pom.xml` 中添加对应 JDBC 驱动依赖并重新构建后端,否则新增数据源时会提示「未找到数据库驱动」。类型列表与 Maven 坐标见 [docs/configuration.md](docs/configuration.md#4-查询数据源)。
- **语义层(无向量召回)**:以"域(Domain)"组织表,由 LLM 在工具调用时**推理出业务问题所属域、主动选表**,而非向量相似度召回——更精准、更稳定,也无需维护任何 embedding 索引。维度包括逻辑表 / 逻辑列 / 表关系 / 指标口径的业务映射。
- **会话式分析**:SSE 流式回答,会话历史可追溯、可调试。
- **报表生成**:内置报表工具与配套前端报表视图。
Expand All @@ -38,7 +40,7 @@ Data Agent 反其道而行——**不引入任何向量检索**。语义层把
- `SchemaReader` 完全基于 JDBC 标准 `DatabaseMetaData` 读取表 / 列 / 主键,不绑定任何数据库方言;
- `SqlExecutor` 只使用 `Connection` / `PreparedStatement` / `ResultSet` 执行查询,并叠加 SELECT 校验、自动 `LIMIT` 等安全护栏。

整条"读取表结构 → 执行查询"的路径都跑在 JDBC 标准 API 上,因此只要目标库**提供 JDBC 驱动**,Data Agent 就能接入。支持范围不局限于 MySQL / PostgreSQL / Oracle——ClickHouseSQL Server、达梦、OceanBaseSQLite 等任意 JDBC 兼容数据库在理论上都可直接支持,引入对应驱动即可
整条"读取表结构 → 执行查询"的路径都跑在 JDBC 标准 API 上,因此只要目标库**提供 JDBC 驱动**,Data Agent 就能接入。目前已内置 MySQL / PostgreSQL / Oracle / ClickHouse / SQL Server / 达梦 / OceanBase / SQLite 八种类型(前端数据源下拉与后端枚举同步支持),其余任意 JDBC 兼容数据库扩展 `DataSourceType` 枚举即可接入。注意:**默认发布包仅内置 MySQL 驱动**,接入其他数据库前请先按 [docs/configuration.md](docs/configuration.md#4-查询数据源) 在 `data-agent-backend/pom.xml` 中引入对应驱动

## 🏗️ 架构速览

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,13 +25,39 @@
@Getter
@AllArgsConstructor
public enum DataSourceType {
MYSQL("mysql", "com.mysql.cj.jdbc.Driver", "jdbc:mysql://"),
POSTGRESQL("postgresql", "org.postgresql.Driver", "jdbc:postgresql://"),
ORACLE("oracle", "oracle.jdbc.OracleDriver", "jdbc:oracle:thin:@");
MYSQL("mysql", "com.mysql.cj.jdbc.Driver", "jdbc:mysql://", "com.mysql:mysql-connector-j"),
POSTGRESQL(
"postgresql",
"org.postgresql.Driver",
"jdbc:postgresql://",
"org.postgresql:postgresql"),
ORACLE(
"oracle",
"oracle.jdbc.OracleDriver",
"jdbc:oracle:thin:@",
"com.oracle.database.jdbc:ojdbc11"),
CLICKHOUSE(
"clickhouse",
"com.clickhouse.jdbc.ClickHouseDriver",
"jdbc:clickhouse://",
"com.clickhouse:clickhouse-jdbc"),
SQLSERVER(
"sqlserver",
"com.microsoft.sqlserver.jdbc.SQLServerDriver",
"jdbc:sqlserver://",
"com.microsoft.sqlserver:mssql-jdbc"),
DAMENG("dameng", "dm.jdbc.driver.DmDriver", "jdbc:dm://", "com.dameng:DmJdbcDriver18"),
OCEANBASE(
"oceanbase",
"com.oceanbase.jdbc.Driver",
"jdbc:oceanbase://",
"com.oceanbase:oceanbase-client"),
SQLITE("sqlite", "org.sqlite.JDBC", "jdbc:sqlite:", "org.xerial:sqlite-jdbc");

private final String code;
private final String driverClassName;
private final String urlPrefix;
private final String mavenCoordinates;

public static Optional<DataSourceType> fromCode(String code) {
if (code == null) {
Expand All @@ -44,9 +70,15 @@ public static Optional<DataSourceType> fromCode(String code) {

public String buildJdbcUrl(String host, int port, String databaseName) {
return switch (this) {
case MYSQL -> String.format("%s%s:%d/%s", urlPrefix, host, port, databaseName);
case POSTGRESQL -> String.format("%s%s:%d/%s", urlPrefix, host, port, databaseName);
case MYSQL, POSTGRESQL, CLICKHOUSE, DAMENG, OCEANBASE ->
String.format("%s%s:%d/%s", urlPrefix, host, port, databaseName);
case ORACLE -> String.format("%s%s:%d:%s", urlPrefix, host, port, databaseName);
case SQLSERVER ->
String.format("%s%s:%d;databaseName=%s", urlPrefix, host, port, databaseName);
case SQLITE ->
throw new IllegalArgumentException(
"SQLite datasource does not support host/port-based URLs; please"
+ " provide a connectionUrl like jdbc:sqlite:/path/to/db");
};
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -58,15 +58,51 @@ private HikariDataSource createDataSource(Datasource datasource) {

String jdbcUrl = resolveJdbcUrl(datasource, type);

HikariConfig config = getHikariConfig(datasource, jdbcUrl, type);

log.info(
"Creating datasource pool for [{}] type={} url={}",
datasource.getName(),
type.getCode(),
jdbcUrl);
// 注意:HikariConfig.setDriverClassName() 会同步加载驱动类,缺驱动时在此即抛异常,
// 因此 config 构建与池创建必须同在一个 try 内,才能被转译为可操作的缺驱动提示。
try {
HikariConfig config = getHikariConfig(datasource, jdbcUrl, type);

log.info(
"Creating datasource pool for [{}] type={} url={}",
datasource.getName(),
type.getCode(),
jdbcUrl);

return new HikariDataSource(config);
} catch (RuntimeException e) {
throw translateInitializationFailure(type, e);
} catch (Error e) {
// 驱动依赖缺失时 JVM 抛的是 NoClassDefFoundError(Error 而非 Exception),
// 且 Error 会绕过 ToolExceptionMapper 的 catch(Exception),必须在此一并转译。
throw translateInitializationFailure(type, e);
}
}

return new HikariDataSource(config);
/**
* 区分池初始化失败的两类原因:驱动未打包(给出 pom.xml 修复指引)与连接失败(原样抛出,
* 交由全局异常映射处理)。
*/
private RuntimeException translateInitializationFailure(
DataSourceType type, Throwable exception) {
for (Throwable current = exception; current != null; current = current.getCause()) {
if (current instanceof ClassNotFoundException
|| current instanceof NoClassDefFoundError) {
return BusinessException.of(
ErrorCode.JDBC_DRIVER_NOT_FOUND,
String.format(
"未找到数据库驱动 %s(%s 类型)。后端默认仅内置 MySQL 驱动,"
+ "请在 data-agent-backend/pom.xml 中添加依赖 %s 后重新构建并启动后端。",
type.getDriverClassName(),
type.getCode(),
type.getMavenCoordinates()),
exception);
}
}
if (exception instanceof RuntimeException runtimeException) {
throw runtimeException;
}
throw new RuntimeException("Unexpected datasource initialization failure", exception);
}

private static HikariConfig getHikariConfig(
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,13 @@ public enum ErrorCode {
HttpStatus.BAD_REQUEST,
"Datasource type is not supported."),

/** 后端未打包对应数据库的 JDBC 驱动,需在 data-agent-backend/pom.xml 引入后重新构建。 */
JDBC_DRIVER_NOT_FOUND(
"JDBC_DRIVER_NOT_FOUND",
HttpStatus.BAD_REQUEST,
"The JDBC driver for this database is not bundled in the backend. "
+ "Please add it to data-agent-backend/pom.xml and rebuild."),

/** 逻辑关系类型非法或不受支持。 */
INVALID_RELATION_TYPE(
"INVALID_RELATION_TYPE", HttpStatus.BAD_REQUEST, "Relation type is invalid."),
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
/*
* Copyright (C) 2026 github.com/MaloneTalk
*
* This program is free software: you can redistribute it and/or modify
* it under the terms of the GNU Affero General Public License as
* published by the Free Software Foundation, either version 3 of the
* License, or any later version.
*
* This program is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
* GNU Affero General Public License for more details.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <https://www.gnu.org/licenses/>.
* limitations under the License.
*/
package io.github.malonetalk.agent.datasource;

import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;

import io.github.malonetalk.common.ErrorCode;
import io.github.malonetalk.entity.Datasource;
import io.github.malonetalk.exception.BusinessException;
import org.junit.jupiter.api.Test;

/** 池初始化失败的转译逻辑:缺驱动必须给出可操作的 pom.xml 指引,而不是裸 INTERNAL_ERROR。 */
class DynamicDataSourceManagerTest {

private final DynamicDataSourceManager manager = new DynamicDataSourceManager();

@Test
void missingDriverYieldsActionableBusinessException() {
Datasource ds = new Datasource();
ds.setId(1);
ds.setName("pg-without-driver");
ds.setType("postgresql");
ds.setHost("localhost");
ds.setPort(5432);
ds.setDatabaseName("db");

BusinessException ex =
assertThrows(BusinessException.class, () -> manager.getOrCreateDataSource(ds));

assertEquals(ErrorCode.JDBC_DRIVER_NOT_FOUND, ex.getErrorCode());
assertTrue(ex.getMessage().contains("org.postgresql:postgresql"));
assertTrue(ex.getMessage().contains("pom.xml"));
}
}
67 changes: 54 additions & 13 deletions data-agent-frontend/src/views/data-source/DataSource.vue
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
-->

<script setup lang="ts">
import { ref, reactive, onMounted } from 'vue';
import { ref, reactive, onMounted, computed } from 'vue';
import type { FormInstance, FormRules } from 'element-plus';
import { ElMessage, ElMessageBox } from 'element-plus';
import { useDatasource } from '@/composables/useDatasource';
Expand Down Expand Up @@ -57,21 +57,40 @@
fetchList();
});

const rules: FormRules = {
name: [{ required: true, message: '请输入数据源名称', trigger: 'blur' }],
type: [{ required: true, message: '请选择数据源类型', trigger: 'change' }],
host: [{ required: true, message: '请输入主机地址', trigger: 'blur' }],
port: [{ required: true, message: '请输入端口', trigger: 'blur' }],
databaseName: [{ required: true, message: '请输入数据库名', trigger: 'blur' }],
username: [{ required: true, message: '请输入用户名', trigger: 'blur' }],
};

// value 与后端 DataSourceType.code 一致
const dataSourceTypes = [
{ value: 'MySQL', label: 'MySQL' },
{ value: 'PostgreSQL', label: 'PostgreSQL' },
{ value: 'Oracle', label: 'Oracle' },
{ value: 'mysql', label: 'MySQL' },
{ value: 'postgresql', label: 'PostgreSQL' },
{ value: 'oracle', label: 'Oracle' },
{ value: 'clickhouse', label: 'ClickHouse' },
{ value: 'sqlserver', label: 'SQL Server' },
{ value: 'dameng', label: '达梦 DM' },
{ value: 'oceanbase', label: 'OceanBase' },
{ value: 'sqlite', label: 'SQLite' },
];

const isSqlite = computed(() => form.type === 'sqlite');

const rules = computed<FormRules>(() => ({
name: [{ required: true, message: '请输入数据源名称', trigger: 'blur' }],
type: [{ required: true, message: '请选择数据源类型', trigger: 'change' }],
host: isSqlite.value ? [] : [{ required: true, message: '请输入主机地址', trigger: 'blur' }],
port: isSqlite.value ? [] : [{ required: true, message: '请输入端口', trigger: 'blur' }],
databaseName: isSqlite.value
? []
: [{ required: true, message: '请输入数据库名', trigger: 'blur' }],
username: isSqlite.value ? [] : [{ required: true, message: '请输入用户名', trigger: 'blur' }],
connectionUrl: isSqlite.value
? [
{
required: true,
message: '请输入连接URL(如 jdbc:sqlite:/path/to/db)',
trigger: 'blur',
},
]
: [],
}));

const resetForm = () => {
Object.assign(form, {
id: undefined,
Expand Down Expand Up @@ -257,6 +276,15 @@
/>
</el-select>
</el-form-item>
<el-alert
v-if="!isEdit"
type="info"
:closable="false"
show-icon
title="后端默认仅内置 MySQL 驱动"
description="使用 PostgreSQL / Oracle 等其他数据库前,请先在 data-agent-backend/pom.xml 中添加对应 JDBC 驱动依赖并重新构建后端,否则连接时会提示「未找到数据库驱动」。"
class="driver-tip"
/>
<el-form-item label="主机地址" prop="host" :error="fieldErrors.host">
<el-input v-model="form.host" placeholder="请输入主机地址" />
</el-form-item>
Expand All @@ -272,6 +300,15 @@
<el-form-item label="密码" prop="password" :error="fieldErrors.password">
<el-input v-model="form.password" type="password" placeholder="请输入密码" show-password />
</el-form-item>
<el-alert
v-if="isSqlite"
type="info"
:closable="false"
show-icon
title="SQLite 为本地文件数据库"
description="无需填写主机 / 端口 / 数据库名,请在「连接URL」中直接填写形如 jdbc:sqlite:/path/to/db 的文件路径。"
class="driver-tip"
/>
<el-form-item label="连接URL" prop="connectionUrl" :error="fieldErrors.connectionUrl">
<el-input v-model="form.connectionUrl" placeholder="请输入连接URL(可选)" />
</el-form-item>
Expand Down Expand Up @@ -314,6 +351,10 @@
font-size: 14px;
}

.driver-tip {
margin-bottom: 18px;
}

.expand-content {
padding: 12px 48px;
display: grid;
Expand Down
33 changes: 27 additions & 6 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,13 +54,34 @@ export IO_GITHUB_MALONETALK_MODEL_API_KEY="sk-你的密钥"

## 4. 查询数据源

被查询的业务库在「数据源管理」界面(或对应 API)中配置,后端按类型动态建立连接。当前支持:
「数据源管理」中配置的是 **Agent 连接并执行 SQL 数据分析的目标数据库**(你的业务库),与后端自身使用的元数据库(MySQL `data_agent`,见上文第 1 节)是两回事:元数据库由环境变量 `DB_URL` 指定,目标数据源在页面或 API 中配置。当前支持:

| 类型 `type` | 驱动 | JDBC 前缀 | 默认端口 | Maven 坐标 |
| --- | --- | --- | --- | --- |
| `mysql` | `com.mysql.cj.jdbc.Driver` | `jdbc:mysql://` | 3306 | `com.mysql:mysql-connector-j`(已内置) |
| `postgresql` | `org.postgresql.Driver` | `jdbc:postgresql://` | 5432 | `org.postgresql:postgresql` |
| `oracle` | `oracle.jdbc.OracleDriver` | `jdbc:oracle:thin:@` | 1521 | `com.oracle.database.jdbc:ojdbc11` |
| `clickhouse` | `com.clickhouse.jdbc.ClickHouseDriver` | `jdbc:clickhouse://` | 8123 | `com.clickhouse:clickhouse-jdbc` |
| `sqlserver` | `com.microsoft.sqlserver.jdbc.SQLServerDriver` | `jdbc:sqlserver://` | 1433 | `com.microsoft.sqlserver:mssql-jdbc` |
| `dameng` | `dm.jdbc.driver.DmDriver` | `jdbc:dm://` | 5236 | `com.dameng:DmJdbcDriver18`(不在中央仓库,需从达梦官网获取后本地安装) |
| `oceanbase` | `com.oceanbase.jdbc.Driver` | `jdbc:oceanbase://` | 2881 | `com.oceanbase:oceanbase-client` |
| `sqlite` | `org.sqlite.JDBC` | `jdbc:sqlite:` | 无(文件路径) | `org.xerial:sqlite-jdbc` |

> **⚠️ 驱动依赖**:后端**默认仅内置 MySQL 驱动**。使用上表其他数据库前,需先在 `data-agent-backend/pom.xml` 中添加对应驱动依赖并重新构建、启动后端,否则新增/连接数据源会报「未找到数据库驱动」。其中**达梦驱动不在 Maven 中央仓库**,需从达梦官网下载 jar 后执行 `mvn install:install-file` 安装到本地仓库。

以 PostgreSQL 为例,在 `data-agent-backend/pom.xml` 的 `<dependencies>` 中加入:

```xml
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<scope>runtime</scope>
</dependency>
```

| 类型 `type` | 驱动 | JDBC 前缀 |
| --- | --- | --- |
| `mysql` | `com.mysql.cj.jdbc.Driver` | `jdbc:mysql://` |
| `postgresql` | `org.postgresql.Driver` | `jdbc:postgresql://` |
| `oracle` | `oracle.jdbc.OracleDriver` | `jdbc:oracle:thin:@` |
> **SQLite 特殊说明**:SQLite 是本地文件数据库,没有主机/端口概念。新增时无需填写主机/端口/数据库名,直接在「连接URL」中填写形如 `jdbc:sqlite:/path/to/db` 的文件路径即可。

> 上述列表之外的其他 JDBC 兼容数据库,可通过扩展 `DataSourceType` 枚举接入(见 [contributing.md](contributing.md))。

> 连接信息(host/port/database/username/password)保存在元数据库中,属敏感信息,请妥善管理元数据库访问权限。

Expand Down
Loading
Loading