feat: support disable thinking when VLLM model (#616)

* feat: add deep think switch button

Signed-off-by: Bob Du <[email protected]>

* feat: api model support VLLM

Signed-off-by: Bob Du <[email protected]>

* feat: api support disable thinking when VLLM model

Signed-off-by: Bob Du <[email protected]>

* docs: update readme

Signed-off-by: Bob Du <[email protected]>

---------

Signed-off-by: Bob Du <[email protected]>

Author: BobDu

README.en.md

@@ -34,6 +34,8 @@ Some unique features have been added:
 
 [✓] Web Search functionality (Real-time web search based on Tavily API)
 
+[✓] VLLM API model support & Optional disable deep thinking mode
+
 > [!CAUTION]
 > This project is only published on GitHub, based on the MIT license, free and for open source learning usage. And there will be no any form of account selling, paid service, discussion group, discussion group and other behaviors. Beware of being deceived.
 
@@ -125,6 +127,10 @@ For all parameter variables, check [here](#docker-parameter-example) or see:
 
 [✓] Interface themes
 
+[✓] VLLM API model support
+
+[✓] Deep thinking mode switch
+
 [✗] More...
 
 ## Prerequisites
@@ -318,6 +324,63 @@ PS: You can also run `pnpm start` directly on the server without packaging.
 pnpm build
 ```
 
+## VLLM API Deep Thinking Mode Control
+
+> [!TIP]
+> Deep thinking mode control is only available when the backend is configured to use VLLM API, allowing users to choose whether to enable the model's deep thinking functionality.
+
+### Features
+
+- **VLLM API Exclusive Feature**: Only available when the backend uses VLLM API
+- **Per-conversation Control**: Each conversation can independently enable or disable deep thinking mode
+- **Real-time Switching**: Deep thinking mode can be switched at any time during conversation
+- **Performance Optimization**: Disabling deep thinking can improve response speed and reduce computational costs
+
+### Prerequisites
+
+**The following conditions must be met to use this feature:**
+
+1. **Backend Configuration**: Backend must be configured to use VLLM API interface
+2. **Model Support**: The model used must support deep thinking functionality
+3. **API Compatibility**: VLLM API version needs to support thinking mode control parameters
+
+### Usage
+
+#### 1. Enable/Disable Deep Thinking Mode
+
+1. **Enter Conversation Interface**: In a conversation session that supports VLLM API
+2. **Find Control Switch**: Locate the "Deep Thinking" toggle button in the conversation interface
+3. **Switch Mode**: 
+   - Enable: Model will perform deep thinking, providing more detailed and in-depth responses
+   - Disable: Model will respond directly, faster but potentially more concise
+
+#### 2. Usage Scenarios
+
+**Recommended to enable deep thinking when:**
+- Complex problems require in-depth analysis
+- Logical reasoning and multi-step thinking are needed
+- High-quality responses are required
+- Time is not sensitive
+
+**Recommended to disable deep thinking when:**
+- Simple questions need quick answers
+- Fast response is required
+- Need to reduce computational costs
+- Batch processing simple tasks
+
+#### 3. Technical Implementation
+
+- **API Parameter**: Controlled through VLLM API's `disable_thinking` parameter
+- **State Persistence**: Each conversation session independently saves the deep thinking switch state
+- **Real-time Effect**: Takes effect immediately for the next message after switching
+
+### Notes
+
+- **VLLM API Only**: This feature is only available when the backend uses VLLM API, other APIs (such as OpenAI API) do not support this feature
+- **Model Dependency**: Not all models support deep thinking mode, please confirm that your model supports this feature
+- **Response Differences**: Disabling deep thinking may affect the detail and quality of responses
+- **Cost Considerations**: Enabling deep thinking typically increases computational costs and response time
+
 ## Frequently Asked Questions
 
 Q: Why does Git always report an error when committing?

README.md

@@ -34,6 +34,8 @@
 
 [✓] Web Search 网络搜索功能 (基于 Tavily API 实现实时网络搜索)
 
+[✓] VLLM API 模型支持 & 可选关闭深度思考模式
+
 
 > [!CAUTION]
 > 声明：此项目只发布于 Github，基于 MIT 协议，免费且作为开源学习使用。并且不会有任何形式的卖号、付费服务、讨论群、讨论组等行为。谨防受骗。
@@ -128,6 +130,10 @@
 
 [✓] 界面主题
 
+[✓] VLLM API 模型支持
+
+[✓] 深度思考模式开关
+
 [✗] More...
 
 ## 前置要求
@@ -454,6 +460,63 @@ Current time: {current_time}
 - 每个会话可以独立控制是否使用搜索功能
 
 
+## VLLM API 深度思考模式控制
+
+> [!TIP]
+> 深度思考模式控制功能仅在后端配置为 VLLM API 时可用，可以让用户选择是否启用模型的深度思考功能。
+
+### 功能特性
+
+- **VLLM API 专属功能**: 仅在后端使用 VLLM API 时可用
+- **按对话控制**: 每个对话可以独立开启或关闭深度思考模式
+- **实时切换**: 在对话过程中可以随时切换深度思考模式
+- **性能优化**: 关闭深度思考可以提高响应速度，降低计算成本
+
+### 使用前提
+
+**必须满足以下条件才能使用此功能：**
+
+1. **后端配置**: 后端必须配置为使用 VLLM API 接口
+2. **模型支持**: 使用的模型必须支持深度思考功能
+3. **API 兼容**: VLLM API 版本需要支持思考模式控制参数
+
+### 使用方式
+
+#### 1. 启用/关闭深度思考模式
+
+1. **进入对话界面**: 在支持 VLLM API 的对话会话中
+2. **找到控制开关**: 在对话界面中找到"深度思考"开关按钮
+3. **切换模式**: 
+   - 开启：模型将进行深度思考，提供更详细和深入的回答
+   - 关闭：模型将直接回答，响应更快但可能较为简洁
+
+#### 2. 使用场景
+
+**建议开启深度思考的情况：**
+- 复杂问题需要深入分析
+- 需要逻辑推理和多步骤思考
+- 对回答质量要求较高的场景
+- 时间不敏感的情况
+
+**建议关闭深度思考的情况：**
+- 简单问题快速回答
+- 需要快速响应的场景
+- 降低计算成本的需求
+- 批量处理简单任务
+
+#### 3. 技术实现
+
+- **API 参数**: 通过 VLLM API 的 `disable_thinking` 参数控制
+- **状态保存**: 每个对话会话独立保存深度思考开关状态
+- **实时生效**: 切换后立即对下一条消息生效
+
+### 注意事项
+
+- **仅限 VLLM API**: 此功能仅在后端使用 VLLM API 时可用，其他 API（如 OpenAI API）不支持此功能
+- **模型依赖**: 不是所有模型都支持深度思考模式，请确认您使用的模型支持此功能
+- **响应差异**: 关闭深度思考可能会影响回答的详细程度和质量
+- **成本考虑**: 开启深度思考通常会增加计算成本和响应时间
+
 ## 常见问题
 Q: 为什么 `Git` 提交总是报错？
 

service/src/chatgpt/index.ts

@@ -1,5 +1,4 @@
-import type { AuditConfig, KeyConfig, UserInfo } from '../storage/model'
-import type { ModelConfig } from '../types'
+import type { AuditConfig, Config, KeyConfig, UserInfo } from '../storage/model'
 import type { TextAuditService } from '../utils/textAudit'
 import type { ChatMessage, RequestOptions } from './types'
 import { tavily } from '@tavily/core'
@@ -102,10 +101,18 @@ async function chatReplyProcess(options: RequestOptions) {
     const searchConfig = globalConfig.searchConfig
     if (searchConfig.enabled && searchConfig?.options?.apiKey && searchEnabled) {
       messages[0].content = renderSystemMessage(searchConfig.systemMessageGetSearchQuery, dayjs().format('YYYY-MM-DD HH:mm:ss'))
-      const completion = await openai.chat.completions.create({
+
+      const getSearchQueryChatCompletionCreateBody: OpenAI.ChatCompletionCreateParamsNonStreaming = {
         model,
         messages,
-      })
+      }
+      if (key.keyModel === 'VLLM') {
+        // @ts-expect-error vLLM supports a set of parameters that are not part of the OpenAI API.
+        getSearchQueryChatCompletionCreateBody.chat_template_kwargs = {
+          enable_thinking: false,
+        }
+      }
+      const completion = await openai.chat.completions.create(getSearchQueryChatCompletionCreateBody)
       let searchQuery: string = completion.choices[0].message.content
       const match = searchQuery.match(/<search_query>([\s\S]*)<\/search_query>/i)
       if (match)
@@ -144,7 +151,7 @@ search result: <search_result>${searchResult}</search_result>`,
       messages[0].content = systemMessage
 
     // Create the chat completion with streaming
-    const stream = await openai.chat.completions.create({
+    const chatCompletionCreateBody: OpenAI.ChatCompletionCreateParamsStreaming = {
       model,
       messages,
       temperature: temperature ?? undefined,
@@ -153,9 +160,19 @@ search result: <search_result>${searchResult}</search_result>`,
       stream_options: {
         include_usage: true,
       },
-    }, {
-      signal: abort.signal,
-    })
+    }
+    if (key.keyModel === 'VLLM') {
+      // @ts-expect-error vLLM supports a set of parameters that are not part of the OpenAI API.
+      chatCompletionCreateBody.chat_template_kwargs = {
+        enable_thinking: options.room.thinkEnabled,
+      }
+    }
+    const stream = await openai.chat.completions.create(
+      chatCompletionCreateBody,
+      {
+        signal: abort.signal,
+      },
+    )
 
     // Process the stream
     let responseReasoning = ''
@@ -253,8 +270,8 @@ async function containsSensitiveWords(audit: AuditConfig, text: string): Promise
 }
 
 async function chatConfig() {
-  const config = await getOriginConfig() as ModelConfig
-  return sendResponse<ModelConfig>({
+  const config = await getOriginConfig()
+  return sendResponse<Config>({
     type: 'Success',
     data: config,
   })

service/src/index.ts

@@ -146,7 +146,6 @@ router.post('/session', async (req, res) => {
     const hasAuth = config.siteConfig.loginEnabled || config.siteConfig.authProxyEnabled
     const authProxyEnabled = config.siteConfig.authProxyEnabled
     const allowRegister = config.siteConfig.registerEnabled
-    config.apiModel = 'ChatGPTAPI'
     const userId = await getUserId(req)
     const chatModels: {
       label: string
@@ -173,7 +172,6 @@ router.post('/session', async (req, res) => {
           data: {
             auth: hasAuth,
             allowRegister,
-            model: config.apiModel,
             title: config.siteConfig.siteTitle,
             chatModels,
             allChatModels: chatModelOptions,
@@ -227,7 +225,6 @@ router.post('/session', async (req, res) => {
           auth: hasAuth,
           authProxyEnabled,
           allowRegister,
-          model: config.apiModel,
           title: config.siteConfig.siteTitle,
           chatModels,
           allChatModels: chatModelOptions,
@@ -246,7 +243,6 @@ router.post('/session', async (req, res) => {
         auth: hasAuth,
         authProxyEnabled,
         allowRegister,
-        model: config.apiModel,
         title: config.siteConfig.siteTitle,
         chatModels: chatModelOptions,
         allChatModels: chatModelOptions,
@@ -659,11 +655,10 @@ router.post('/verifyadmin', authLimiter, async (req, res) => {
 
 router.post('/setting-base', rootAuth, async (req, res) => {
   try {
-    const { apiKey, apiModel, apiBaseUrl, accessToken, timeoutMs, reverseProxy, socksProxy, socksAuth, httpsProxy } = req.body as Config
+    const { apiKey, apiBaseUrl, accessToken, timeoutMs, reverseProxy, socksProxy, socksAuth, httpsProxy } = req.body as Config
 
     const thisConfig = await getOriginConfig()
     thisConfig.apiKey = apiKey
-    thisConfig.apiModel = apiModel
     thisConfig.apiBaseUrl = apiBaseUrl
     thisConfig.accessToken = accessToken
     thisConfig.reverseProxy = reverseProxy

service/src/routes/room.ts

@@ -10,6 +10,7 @@ import {
   updateRoomChatModel,
   updateRoomPrompt,
   updateRoomSearchEnabled,
+  updateRoomThinkEnabled,
   updateRoomUsingContext,
 } from '../storage/mongo'
 
@@ -29,6 +30,7 @@ router.get('/chatrooms', auth, async (req, res) => {
         usingContext: r.usingContext === undefined ? true : r.usingContext,
         chatModel: r.chatModel,
         searchEnabled: !!r.searchEnabled,
+        thinkEnabled: !!r.thinkEnabled,
       })
     })
     res.send({ status: 'Success', message: null, data: result })
@@ -153,6 +155,22 @@ router.post('/room-search-enabled', auth, async (req, res) => {
   }
 })
 
+router.post('/room-think-enabled', auth, async (req, res) => {
+  try {
+    const userId = req.headers.userId as string
+    const { thinkEnabled, roomId } = req.body as { thinkEnabled: boolean, roomId: number }
+    const success = await updateRoomThinkEnabled(userId, roomId, thinkEnabled)
+    if (success)
+      res.send({ status: 'Success', message: 'Saved successfully', data: null })
+    else
+      res.send({ status: 'Fail', message: 'Saved Failed', data: null })
+  }
+  catch (error) {
+    console.error(error)
+    res.send({ status: 'Fail', message: 'Update error', data: null })
+  }
+})
+
 router.post('/room-context', auth, async (req, res) => {
   try {
     const userId = req.headers.userId as string

service/src/storage/config.ts

@@ -27,7 +27,7 @@ export async function getCacheConfig(): Promise<Config> {
 export async function getOriginConfig() {
   let config = await getConfig()
   if (config == null) {
-    config = new Config(new ObjectId(), !Number.isNaN(+process.env.TIMEOUT_MS) ? +process.env.TIMEOUT_MS : 600 * 1000, process.env.OPENAI_API_KEY, process.env.OPENAI_API_DISABLE_DEBUG === 'true', process.env.OPENAI_ACCESS_TOKEN, process.env.OPENAI_API_BASE_URL, 'ChatGPTAPI', process.env.API_REVERSE_PROXY, (process.env.SOCKS_PROXY_HOST && process.env.SOCKS_PROXY_PORT)
+    config = new Config(new ObjectId(), !Number.isNaN(+process.env.TIMEOUT_MS) ? +process.env.TIMEOUT_MS : 600 * 1000, process.env.OPENAI_API_KEY, process.env.OPENAI_API_DISABLE_DEBUG === 'true', process.env.OPENAI_ACCESS_TOKEN, process.env.OPENAI_API_BASE_URL, process.env.API_REVERSE_PROXY, (process.env.SOCKS_PROXY_HOST && process.env.SOCKS_PROXY_PORT)
       ? (`${process.env.SOCKS_PROXY_HOST}:${process.env.SOCKS_PROXY_PORT}`)
       : '', (process.env.SOCKS_PROXY_USERNAME && process.env.SOCKS_PROXY_PASSWORD)
       ? (`${process.env.SOCKS_PROXY_USERNAME}:${process.env.SOCKS_PROXY_PASSWORD}`)
@@ -149,9 +149,7 @@ export async function getApiKeys() {
   const result = await getKeys()
   const config = await getCacheConfig()
   if (result.keys.length <= 0) {
-    if (config.apiModel === 'ChatGPTAPI')
-      result.keys.push(await upsertKey(new KeyConfig(config.apiKey, 'ChatGPTAPI', [], [], '')))
-
+    result.keys.push(await upsertKey(new KeyConfig(config.apiKey, 'ChatGPTAPI', [], [], '')))
     result.total++
   }
   result.keys.forEach((key) => {

service/src/storage/model.ts

@@ -83,14 +83,16 @@ export class ChatRoom {
   status: Status = Status.Normal
   chatModel: string
   searchEnabled: boolean
-  constructor(userId: string, title: string, roomId: number, chatModel: string, searchEnabled: boolean) {
+  thinkEnabled: boolean
+  constructor(userId: string, title: string, roomId: number, chatModel: string, searchEnabled: boolean, thinkEnabled: boolean) {
     this.userId = userId
     this.title = title
     this.prompt = undefined
     this.roomId = roomId
     this.usingContext = true
     this.chatModel = chatModel
     this.searchEnabled = searchEnabled
+    this.thinkEnabled = thinkEnabled
   }
 }
 
@@ -197,7 +199,6 @@ export class Config {
     public apiDisableDebug?: boolean,
     public accessToken?: string,
     public apiBaseUrl?: string,
-    public apiModel?: APIMODEL,
     public reverseProxy?: string,
     public socksProxy?: string,
     public socksAuth?: string,
@@ -304,4 +305,4 @@ export class UserPrompt {
   }
 }
 
-export type APIMODEL = 'ChatGPTAPI'
+export type APIMODEL = 'ChatGPTAPI' | 'VLLM'