Skip to content

C
canifa_note_extension
Commit fbd1e348 authored by Vũ Hoàng Anh's avatar Vũ Hoàng Anh
Browse files
Options

Update chatbot UI and memo embeddings

parent 32666a41
Expand all Hide whitespace changes
Inline Side-by-side
Showing with 2410 additions and 6065 deletions
backend/agent/prompt.py
View file @ fbd1e348
"""
CiCi Fashion Consultant - System Prompt
Tư vấn thời trang CANIFA chuyên nghiệp
Version 3.0 - Dynamic from File
"""
......@@ -9,29 +8,35 @@ from datetime import datetime
PROMPT_FILE_PATH = os.path.join(os.path.dirname(__file__), "system_prompt.txt")
def _ensure_json_instruction(prompt_text: str) -> str:
if "json" in prompt_text.lower():
return prompt_text
return f"{prompt_text}\n\nReturn JSON (json) object with keys: ai_response, product_ids."
def get_system_prompt() -> str:
"""
System prompt cho CiCi Fashion Agent
Đọc từ file system_prompt.txt để có thể update dynamic.
System prompt for CiCi Fashion Agent.
Returns:
str: System prompt với ngày hiện tại
str: System prompt with the current date.
"""
now = datetime.now()
date_str = now.strftime("%d/%m/%Y")
date_str = datetime.now().strftime("%d/%m/%Y")
try:
if os.path.exists(PROMPT_FILE_PATH):
with open(PROMPT_FILE_PATH, "r", encoding="utf-8") as f:
prompt_template = f.read()
return prompt_template.replace("{date_str}", date_str)
except Exception as e:
print(f"Error reading system prompt file: {e}")
# Fallback default prompt if file error
return f"""# VAI TRÒ
Bạn là CiCi - Chuyên viên tư vấn thời trang CANIFA.
Hôm nay: {date_str}
KHÔNG BAO GIỜ BỊA ĐẶT. TRẢ LỜI NGẮN GỌN.
"""
\ No newline at end of file
with open(PROMPT_FILE_PATH, "r", encoding="utf-8") as handle:
prompt_template = handle.read()
rendered = prompt_template.replace("{date_str}", date_str)
return _ensure_json_instruction(rendered)
except Exception as exc:
print(f"Error reading system prompt file: {exc}")
fallback = f"""# ROLE
You are CiCi, a CANIFA fashion assistant.
Today: {date_str}
Never fabricate. Keep responses concise.
"""
return _ensure_json_instruction(fallback)
backend/agent/system_prompt.txt
View file @ fbd1e348
# VAI TRÒ
Bạn là CiCi - Chuyên viên tư vấn thời trang CANIFA.
- Nhiệt tình, thân thiện, chuyên nghiệp
- CANIFA BÁN QUẦN ÁO: áo, quần, váy, đầm, phụ kiện thời trang
- Hôm nay: {date_str}
---
# QUY TẮC TRUNG THỰC - BẮT BUỘC
KHÔNG BAO GIỜ BỊA ĐẶT - CHỈ NÓI THEO DỮ LIỆU
**ĐÚNG:**
- Tool trả về áo thun → Giới thiệu áo thun
- Tool trả về 0 sản phẩm → Nói "Shop chưa có sản phẩm này"
- Tool trả về quần nỉ mà khách hỏi bikini → Nói "Shop chưa có bikini"
**CẤM:**
- Tool trả về quần nỉ → Gọi là "đồ bơi"
- Tool trả về 0 kết quả → Nói "shop có sản phẩm X"
- Tự bịa mã sản phẩm, giá tiền, chính sách
Không có trong data = Không nói = Không tư vấn láo
---
# NGÔN NGỮ & XƯNG HÔ
- Mặc định: Xưng "mình" - gọi "bạn"
- Khi khách xưng anh/chị: Xưng "em" - gọi "anh/chị"
- Khách nói tiếng Việt → Trả lời tiếng Việt
- Khách nói tiếng Anh → Trả lời tiếng Anh
- Ngắn gọn, đi thẳng vào vấn đề
---
# KHI NÀO GỌI TOOL
**Gọi data_retrieval_tool khi:**
- Khách tìm sản phẩm: "Tìm áo...", "Có màu gì..."
- Khách hỏi sản phẩm cụ thể: "Mã 8TS24W001 có không?"
- Tư vấn phong cách: "Mặc gì đi cưới?", "Đồ công sở?"
**⚠️ QUY TẮC SINH QUERY (BẮT BUỘC):**
- **Query chỉ chứa MÔ TẢ SẢN PHẨM** (tên, chất liệu, màu, phong cách).
- **TUYỆT ĐỐI KHÔNG đưa giá tiền vào chuỗi `query`**.
- Giá tiền phải đưa vào tham số riêng: `price_min`, `price_max`.
Ví dụ ĐÚNG:
- Query: "Áo thun nam cotton thoáng mát basic"
- Price_max: 300000
Ví dụ SAI (Cấm):
- Query: "Áo thun nam giá dưới 300k" (SAI vì có giá trong query)
**Gọi canifa_knowledge_search khi:**
- Hỏi chính sách: freeship, đổi trả, bảo hành
- Hỏi thương hiệu: Canifa là gì, lịch sử
- Tìm cửa hàng: địa chỉ, giờ mở cửa
**Không gọi tool khi:**
- Chào hỏi đơn giản: "Hi", "Hello"
- Hỏi lại về sản phẩm vừa show
---
# XỬ LÝ KẾT QUẢ TỪ TOOL
## Sau khi gọi tool, kiểm tra kết quả:
**Trường hợp 1: CÓ sản phẩm phù hợp (đúng loại, đúng yêu cầu)**
- DỪNG LẠI, giới thiệu sản phẩm
- KHÔNG GỌI TOOL LẦN 2
**Trường hợp 2: CÓ kết quả NHƯNG SAI LOẠI**
Ví dụ: Khách hỏi bikini, tool trả về quần nỉ
→ Trả lời thẳng:
"Dạ shop chưa có bikini ạ. Shop chuyên về quần áo thời trang (áo, quần, váy). Bạn có muốn tìm sản phẩm nào khác không?"
CẤM TUYỆT ĐỐI:
- Giới thiệu quần nỉ như thể nó là bikini
- Nói "shop có đồ bơi này bạn tham khảo" khi thực tế là áo/quần thường
**Trường hợp 3: KHÔNG CÓ kết quả (count = 0)**
- Thử lại 1 LẦN với filter rộng hơn
- Nếu vẫn không có:
"Dạ shop chưa có sản phẩm [X] ạ. Bạn có thể tham khảo [loại gần nhất] hoặc ghé shop sau nhé!"
---
# FORMAT ĐẦU RA
Trả về JSON (KHÔNG có markdown backticks):
```json
{{
"ai_response": "Câu trả lời ngắn gọn, mô tả bằng [SKU]",
"product_ids": [
{{
"sku": "8TS24W001",
"name": "Áo thun nam basic",
"price": 200000,
"sale_price": 160000,
"url": "https://canifa.com/...",
"thumbnail_image_url": "https://..."
}}
]
}}
```
**Quy tắc ai_response:**
- Mô tả ngắn gọn, nhắc sản phẩm bằng [SKU]
- Nói qua giá, chất liệu, điểm nổi bật
- KHÔNG tạo bảng markdown
- KHÔNG đưa link, ảnh (frontend tự render)
---
# VÍ DỤ
## Example 1: Chào hỏi
Input: "Chào shop"
Output:
```json
{{
"ai_response": "Chào bạn! Mình là CiCi, tư vấn thời trang CANIFA. Mình có thể giúp gì cho bạn?",
"product_ids": []
}}
```
## Example 2: Tìm sản phẩm CÓ
Input: "Tìm áo thun nam dưới 300k"
Tool trả về: 2 sản phẩm áo thun phù hợp
Output:
```json
{{
"ai_response": "Shop có 2 mẫu áo thun nam giá dưới 300k:
- [8TS24W009]: Áo thun cotton basic, giá 250k đang sale 200k
- [6TN24W012]: Áo thun trơn thoải mái, giá 280k
Bạn kéo xuống xem ảnh nhé!",
"product_ids": [
{{"sku": "8TS24W009", "name": "Áo thun cotton basic", "price": 250000, "sale_price": 200000, "url": "...", "thumbnail_image_url": "..."}},
{{"sku": "6TN24W012", "name": "Áo thun trơn", "price": 280000, "sale_price": null, "url": "...", "thumbnail_image_url": "..."}}
]
}}
```
## Example 3: Khách hỏi KHÔNG CÓ trong kho
Input: "Shop có bikini không?"
Tool trả về: 0 sản phẩm
Output:
```json
{{
"ai_response": "Dạ shop chưa có bikini ạ. CANIFA chuyên về quần áo thời trang như áo, quần, váy, đầm. Bạn có muốn tìm mẫu nào khác không?",
"product_ids": []
}}
```
## Example 4: Tool trả về SAI LOẠI
Input: "Cho tôi xem đồ bơi"
Tool trả về: Quần nỉ, áo nỉ (SAI HOÀN TOÀN so với đồ bơi)
Output:
```json
{{
"ai_response": "Dạ shop chưa có đồ bơi ạ. Shop chuyên bán quần áo thời trang (áo, quần, váy, áo khoác). Bạn có muốn tìm loại sản phẩm nào khác không?",
"product_ids": []
}}
```
TUYỆT ĐỐI KHÔNG giới thiệu sản phẩm sai loại
## Example 5: Khách xưng anh/chị
Input: "Chào em, anh muốn tìm áo sơ mi"
Output:
```json
{{
"ai_response": "Chào anh ạ! Em là CiCi. Anh đang tìm áo sơ mi dài tay hay ngắn tay ạ? Để em tư vấn mẫu phù hợp nhất cho anh nhé!",
"product_ids": []
}}
```
---
# TÓM TẮT
1. CANIFA bán quần áo (áo, quần, váy, đầm, phụ kiện)
2. Không có trong data = Không nói
3. Kiểm tra kỹ tên sản phẩm trước khi giới thiệu
4. Nếu sai loại → Nói thẳng "shop chưa có X"
5. Không bịa giá, mã sản phẩm, chính sách
6. Có kết quả phù hợp = DỪNG, không gọi tool lần 2
7. Trả lời ngắn gọn, dựa 100% vào dữ liệu tool trả về
---
Luôn thành thật, khéo léo, và chuyên nghiệp.
\ No newline at end of file
# ROLE
You are CiCi, a CANIFA fashion assistant.
Today: {date_str}
# STYLE
- Friendly, concise, and professional.
- If the user writes in Vietnamese, respond in Vietnamese. If the user writes in English, respond in English.
- Never fabricate. If data is missing, say the shop does not have it.
# TOOL USAGE
- Use data_retrieval_tool for product search or product detail questions.
- If the tool returns 0 products, say the shop does not have that item.
- Do not call tools for simple greetings.
# OUTPUT (json)
Return a JSON object only (json).
Keys:
- ai_response: string
- product_ids: array of objects with sku, name, price, sale_price, url, thumbnail_image_url
No markdown and no extra keys.
backend/api/memos/__init__.py
View file @ fbd1e348
......@@ -3,11 +3,12 @@ from fastapi import APIRouter
from .instance_routes import router as instance_router
from .auth_routes import router as auth_router
from .user_routes import router as user_router
from .memo_routes import router as memo_router
from .attachment_routes import router as attachment_router
from .shortcut_routes import router as shortcut_router
from .activity_routes import router as activity_router
from .idp_routes import router as idp_router
from .memo_routes import router as memo_router
from .attachment_routes import router as attachment_router
from .shortcut_routes import router as shortcut_router
from .activity_routes import router as activity_router
from .idp_routes import router as idp_router
from .embedding_routes import router as embedding_router
router = APIRouter(prefix="/api/v1")
......@@ -18,8 +19,9 @@ router.include_router(auth_router, tags=["auth"])
router.include_router(user_router, tags=["users"])
router.include_router(memo_router, tags=["memos"])
router.include_router(attachment_router, tags=["attachments"])
router.include_router(shortcut_router, tags=["shortcuts"])
router.include_router(activity_router, tags=["activities"])
router.include_router(idp_router, tags=["idp"])
router.include_router(shortcut_router, tags=["shortcuts"])
router.include_router(activity_router, tags=["activities"])
router.include_router(idp_router, tags=["idp"])
router.include_router(embedding_router, tags=["memo-embeddings"])
backend/api/memos/embedding_routes.py 0 → 100644
View file @ fbd1e348
from typing import List
from fastapi import APIRouter, Body, Depends, HTTPException
from memos_core.schemas import MemoEmbeddingCreate, MemoEmbeddingResponse
from memos_core.services import get_memo_embedding_service
router = APIRouter(prefix="/memo-embeddings", tags=["memo-embeddings"])
@router.post("", response_model=MemoEmbeddingResponse)
async def create_embedding(
payload: MemoEmbeddingCreate = Body(...),
svc=Depends(get_memo_embedding_service),
):
try:
return await svc.create_embedding(payload)
except Exception as exc: # pragma: no cover
raise HTTPException(status_code=400, detail=str(exc)) from exc
@router.post("/search", response_model=List[MemoEmbeddingResponse])
async def search_embeddings(
payload: dict = Body(...),
svc=Depends(get_memo_embedding_service),
):
"""
Payload:
{
"queryEmbedding": [float],
"topK": 5,
"dateKey": "YYYY-MM-DD" (optional),
"memoId": 1 (optional)
}
"""
try:
query_embedding = payload.get("queryEmbedding") or payload.get("embedding")
top_k = int(payload.get("topK", 5))
date_key = payload.get("dateKey")
memo_id = payload.get("memoId")
return await svc.search_embeddings(query_embedding, top_k=top_k, date_key=date_key, memo_id=memo_id)
except Exception as exc: # pragma: no cover
raise HTTPException(status_code=400, detail=str(exc)) from exc
backend/common/conversation_manager.py
View file @ fbd1e348
import asyncio
import json
import logging
import asyncio
import os
from datetime import datetime, date
from typing import Any
import aiosqlite
import psycopg
from psycopg_pool import AsyncConnectionPool
from config import CHECKPOINT_POSTGRES_URL
from config import CHECKPOINT_POSTGRES_URL, SQLITE_DB_PATH, USE_SQLITE_HISTORY
logger = logging.getLogger(__name__)
......@@ -235,14 +237,185 @@ class ConversationManager:
await self._pool.close()
class SQLiteConversationManager:
"""SQLite-based chat history storage for local/dev testing."""
def __init__(self, db_path: str = SQLITE_DB_PATH, table_name: str = "langgraph_chat_histories"):
self.db_path = db_path
self.table_name = table_name
db_dir = os.path.dirname(self.db_path)
if db_dir:
os.makedirs(db_dir, exist_ok=True)
async def initialize_table(self):
"""Create table and index if not exists."""
try:
async with aiosqlite.connect(self.db_path) as db:
await db.execute(
f"""
CREATE TABLE IF NOT EXISTS {self.table_name} (
id INTEGER PRIMARY KEY AUTOINCREMENT,
identity_key TEXT NOT NULL,
message TEXT NOT NULL,
is_human INTEGER NOT NULL,
timestamp TEXT DEFAULT (DATETIME('now'))
)
"""
)
await db.execute(
f"""
CREATE INDEX IF NOT EXISTS idx_{self.table_name}_identity_timestamp
ON {self.table_name} (identity_key, timestamp)
"""
)
await db.commit()
logger.info(f"SQLite table {self.table_name} initialized at {self.db_path}")
except Exception as e:
logger.error(f"Error initializing SQLite table: {e}", exc_info=True)
raise
async def save_conversation_turn(self, identity_key: str, human_message: str, ai_message: str):
"""Save both human and AI messages in a single transaction."""
timestamp = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
try:
async with aiosqlite.connect(self.db_path) as db:
await db.execute(
f"""INSERT INTO {self.table_name} (identity_key, message, is_human, timestamp)
VALUES (?, ?, ?, ?), (?, ?, ?, ?)""",
(
identity_key,
human_message,
1,
timestamp,
identity_key,
ai_message,
0,
timestamp,
),
)
await db.commit()
logger.debug(f"Saved conversation turn for {identity_key} (SQLite)")
except Exception as e:
logger.error(f"Failed to save conversation for {identity_key} (SQLite): {e}", exc_info=True)
raise
async def get_chat_history(
self, identity_key: str, limit: int | None = None, before_id: int | None = None
) -> list[dict[str, Any]]:
"""Retrieve chat history with optional cursor pagination."""
try:
query = f"""
SELECT id, message, is_human, timestamp
FROM {self.table_name}
WHERE identity_key = ?
"""
params: list[Any] = [identity_key]
if before_id:
query += " AND id < ?"
params.append(before_id)
query += " ORDER BY id DESC"
if limit:
query += " LIMIT ?"
params.append(limit)
async with aiosqlite.connect(self.db_path) as db:
db.row_factory = aiosqlite.Row
async with db.execute(query, tuple(params)) as cursor:
rows = await cursor.fetchall()
history: list[dict[str, Any]] = []
for row in rows:
message_content = row["message"]
is_human = bool(row["is_human"])
entry = {
"is_human": is_human,
"timestamp": row["timestamp"],
"id": row["id"],
}
if is_human:
entry["message"] = message_content
else:
try:
parsed = json.loads(message_content)
entry["message"] = parsed.get("ai_response", message_content)
entry["product_ids"] = parsed.get("product_ids", [])
except (json.JSONDecodeError, TypeError):
entry["message"] = message_content
entry["product_ids"] = []
history.append(entry)
return history
except Exception as e:
logger.error(f"Error retrieving chat history (SQLite): {e}", exc_info=True)
return []
async def clear_history(self, identity_key: str):
"""Clear all chat history for an identity."""
try:
async with aiosqlite.connect(self.db_path) as db:
await db.execute(f"DELETE FROM {self.table_name} WHERE identity_key = ?", (identity_key,))
await db.commit()
logger.info(f"Cleared chat history for {identity_key} (SQLite)")
except Exception as e:
logger.error(f"Error clearing chat history for {identity_key} (SQLite): {e}", exc_info=True)
async def get_user_count(self) -> int:
"""Get total number of unique identities."""
try:
async with aiosqlite.connect(self.db_path) as db:
async with db.execute(
f"SELECT COUNT(DISTINCT identity_key) FROM {self.table_name}"
) as cursor:
row = await cursor.fetchone()
return row[0] if row else 0
except Exception as e:
logger.error(f"Error getting user count (SQLite): {e}", exc_info=True)
return 0
async def get_message_count_today(self, identity_key: str) -> int:
"""Count user messages today (for potential rate limiting)."""
try:
async with aiosqlite.connect(self.db_path) as db:
async with db.execute(
f"""
SELECT COUNT(*) FROM {self.table_name}
WHERE identity_key = ?
AND is_human = 1
AND DATE(timestamp) = DATE('now', 'localtime')
""",
(identity_key,),
) as cursor:
row = await cursor.fetchone()
return row[0] if row else 0
except Exception as e:
logger.error(f"Error counting messages for {identity_key} (SQLite): {e}", exc_info=True)
return 0
async def close(self):
# No persistent connection to close when using aiosqlite per-call
return
# --- Singleton ---
_instance: ConversationManager | None = None
_instance: ConversationManager | SQLiteConversationManager | None = None
async def get_conversation_manager() -> ConversationManager:
"""Get or create async ConversationManager singleton"""
async def get_conversation_manager() -> ConversationManager | SQLiteConversationManager:
"""Get or create async ConversationManager singleton (Postgres or SQLite based on config)."""
global _instance
if _instance is None:
_instance = ConversationManager()
use_sqlite = USE_SQLITE_HISTORY or not CHECKPOINT_POSTGRES_URL
if use_sqlite:
logger.warning("Using SQLite for chat history (USE_SQLITE_HISTORY=true or no Postgres URL).")
_instance = SQLiteConversationManager()
else:
_instance = ConversationManager()
await _instance.initialize_table()
return _instance
backend/common/middleware.py
View file @ fbd1e348
......@@ -9,6 +9,8 @@ from collections.abc import Callable
from typing import TYPE_CHECKING
from fastapi import HTTPException, Request, status
from config import DISABLE_AUTH
from starlette.middleware.base import BaseHTTPMiddleware
if TYPE_CHECKING:
......@@ -62,6 +64,10 @@ class CanifaAuthMiddleware(BaseHTTPMiddleware):
path = request.url.path
method = request.method
# Temporary bypass: skip auth/rate-limit when DISABLE_AUTH=true
if DISABLE_AUTH:
return await call_next(request)
# ✅ Allow OPTIONS requests (CORS preflight)
if method == "OPTIONS":
return await call_next(request)
......
backend/config.py
View file @ fbd1e348
......@@ -55,6 +55,10 @@ __all__ = [
"USE_MONGO_CONVERSATION",
"RATE_LIMIT_GUEST",
"RATE_LIMIT_USER",
"USE_SQLITE_HISTORY",
"SQLITE_DB_PATH",
"DISABLE_AUTH",
"MEMO_DB_PATH",
]
# ====================== SUPABASE CONFIGURATION ======================
......@@ -140,4 +144,20 @@ OTEL_EXPORTER_JAEGER_AGENT_SPLIT_OVERSIZED_BATCHES = os.getenv("OTEL_EXPORTER_JA
RATE_LIMIT_GUEST: int = int(os.getenv("RATE_LIMIT_GUEST", "10"))
RATE_LIMIT_USER: int = int(os.getenv("RATE_LIMIT_USER", "100"))
# ====================== LOCAL DEV / TESTING TOGGLES ======================
# Turn off auth + rate limit middleware when true (for local testing)
DISABLE_AUTH: bool = os.getenv("DISABLE_AUTH", "true").lower() == "true"
# Use SQLite for chat history instead of Postgres when true
USE_SQLITE_HISTORY: bool = os.getenv("USE_SQLITE_HISTORY", "true").lower() == "true"
SQLITE_DB_PATH: str = os.getenv(
"SQLITE_DB_PATH",
os.path.join(os.path.dirname(__file__), "data", "chat_history.db"),
)
# Memo DB for note backend
MEMO_DB_PATH: str = os.getenv(
"MEMO_DB_PATH",
os.path.join(os.path.dirname(__file__), "db", "memos.db"),
)
backend/datadb/add_context_to_sizes.py deleted 100644 → 0
View file @ 32666a41
"""
Script tự động thêm context (tên bảng + subsection) vào tất cả size entries trong tonghop.txt
Ví dụ: "Size 92 (2Y):" -> "Size 92 (2Y) - BẢNG SIZE CHUNG CHO UNISEX - TRẺ EM (Dải size lẻ):"
"""
import re
def add_context_to_sizes(input_file, output_file):
with open(input_file, encoding="utf-8") as f:
lines = f.readlines()
result = []
current_table = None # Tên bảng hiện tại
current_subsection = None # Subsection hiện tại (Dải size lẻ, chẵn...)
for line in lines:
stripped = line.strip()
# Phát hiện header bảng (bắt đầu bằng BẢNG hoặc QUẦN)
if stripped.startswith("BẢNG SIZE") or stripped.startswith("QUẦN"):
current_table = stripped
current_subsection = None # Reset subsection khi sang bảng mới
result.append(line)
continue
# Phát hiện subsection (Dải size lẻ, Dải size chẵn)
if "Dải size" in stripped or stripped.startswith("Dải"):
current_subsection = stripped.rstrip(":")
result.append(line)
continue
# Phát hiện dòng Size (bắt mọi pattern: Size XS:, Size 92 (2Y):, Size 26 (XS):)
size_match = re.match(r"^(Size\s+[A-Z0-9]+(?:\s*\([^)]+\))?):(.*)$", stripped)
if size_match and current_table:
size_part = size_match.group(1) # "Size 92 (2Y)" hoặc "Size XS"
rest = size_match.group(2) # Phần còn lại sau dấu :
# Xây dựng context
context_parts = [current_table]
if current_subsection:
context_parts.append(f"({current_subsection})")
context = " - ".join(context_parts)
# Tạo dòng mới với context
new_line = f"{size_part} - {context}:{rest}\n"
result.append(new_line)
continue
# Giữ nguyên các dòng khác
result.append(line)
# Ghi file output
with open(output_file, "w", encoding="utf-8") as f:
f.writelines(result)
print("✅ Đã thêm context vào tất cả size entries!")
print(f"📝 File output: {output_file}")
if __name__ == "__main__":
input_path = r"d:\cnf\chatbot_canifa\backend\datadb\tonghop.txt"
output_path = r"d:\cnf\chatbot_canifa\backend\datadb\tonghop_with_context.txt"
add_context_to_sizes(input_path, output_path)
print("\n🔍 Preview 10 dòng đầu của file mới:")
with open(output_path, encoding="utf-8") as f:
for i, line in enumerate(f):
if i >= 1160 and i < 1170: # Vùng có size entries
print(line.rstrip())
backend/datadb/ingest_knowledge.py deleted 100644 → 0
View file @ 32666a41
import os
import json
import pymysql
from openai import OpenAI
import time
# ==========================================
# 🔐 HARD KEY CONFIGURATION (As requested)
# ==========================================
OPENAI_API_KEY = "sk-proj-srJ3l3B5q1CzRezXAnaewbbRfuWzIjYHbcAdggzsa4MmtXEHaIwS1OTkMgLpMDikgh"
SR_HOST = "172.16.2.100"
SR_PORT = 9030
SR_USER = "anhvh"
SR_PASS = "v0WYGeyLRCckXotT"
SR_DB = "shared_source"
# Parameter
CHUNK_SIZE = 500
CHUNK_OVERLAP = 50
EMBEDDING_MODEL = "text-embedding-3-small" # 1536 dimensions
client = OpenAI(api_key=OPENAI_API_KEY)
def get_embedding(text):
"""Lấy vector 1536 chiều từ OpenAI"""
try:
text = text.replace("\n", " ")
return client.embeddings.create(input=[text], model=EMBEDDING_MODEL).data[0].embedding
except Exception as e:
print(f"❌ Lỗi Embedding: {e}")
return None
def connect_starrocks():
return pymysql.connect(
host=SR_HOST,
port=SR_PORT,
user=SR_USER,
password=SR_PASS,
database=SR_DB,
charset='utf8mb4',
cursorclass=pymysql.cursors.DictCursor
)
def chunk_text(text, size=CHUNK_SIZE, overlap=CHUNK_OVERLAP):
"""Chia nhỏ văn bản với overlap"""
chunks = []
start = 0
while start < len(text):
end = start + size
chunks.append(text[start:end])
start += size - overlap
return chunks
def ingest():
input_file = r"d:\cnf\chatbot_canifa\backend\datadb\tonghop.txt"
if not os.path.exists(input_file):
print(f"❌ Không tìm thấy file: {input_file}")
return
print(f"📖 Đang đọc file {input_file}...")
with open(input_file, "r", encoding="utf-8") as f:
full_content = f.read()
# Tách dữ liệu theo từng FILE giả định trong tonghop.txt
sections = full_content.split("================================================================================")
db = connect_starrocks()
cursor = db.cursor()
total_chunks = 0
record_id = int(time.time()) # Làm ID cơ bản
for section in sections:
if not section.strip(): continue
# Lấy tiêu đề file nếu có
lines = section.strip().split("\n")
title = "Canifa Knowledge"
if "FILE:" in lines[0]:
title = lines[0].replace("FILE:", "").strip()
content = "\n".join(lines[1:])
else:
content = section
print(f"🚀 Đang xử lý section: {title}")
chunks = chunk_text(content)
for i, chunk in enumerate(chunks):
if len(chunk.strip()) < 20: continue # Bỏ qua đoạn quá ngắn
vector = get_embedding(chunk)
if not vector: continue
metadata = {
"title": title,
"chunk_idx": i,
"source": "tonghop.txt",
"timestamp": time.time()
}
sql = "INSERT INTO shared_source.canifa_knowledge (id, content, metadata, embedding) VALUES (%s, %s, %s, %s)"
try:
cursor.execute(sql, (record_id, chunk, json.dumps(metadata, ensure_ascii=False), str(vector)))
record_id += 1
total_chunks += 1
if total_chunks % 10 == 0:
db.commit()
print(f"✅ Đã nạp {total_chunks} chunks...")
except Exception as e:
print(f"❌ Lỗi SQL: {e}")
db.commit()
db.close()
print(f"🎊 HOÀN THÀNH! Tổng cộng đã nạp {total_chunks} vào StarRocks.")
if __name__ == "__main__":
ingest()
backend/datadb/magento_product_dimension_with_text_embedding_202512221052.json deleted 100644 → 0
View file @ 32666a41
This source diff could not be displayed because it is too large. You can view the blob instead.
backend/datadb/tonghop.txt deleted 100644 → 0
View file @ 32666a41
This diff is collapsed.
backend/datadb/tonghop_with_context.txt deleted 100644 → 0
View file @ 32666a41
This diff is collapsed.
backend/memos_core/db.py 0 → 100644
View file @ fbd1e348
import os
from typing import Any
import aiosqlite
from config import MEMO_DB_PATH
DB_PATH = MEMO_DB_PATH
async def init_memo_db():
"""Ensure memo database and tables exist."""
os.makedirs(os.path.dirname(DB_PATH), exist_ok=True)
async with aiosqlite.connect(DB_PATH) as db:
await db.execute(
"""
CREATE TABLE IF NOT EXISTS memos (
id INTEGER PRIMARY KEY AUTOINCREMENT,
content TEXT NOT NULL,
visibility TEXT NOT NULL DEFAULT 'PRIVATE',
tags_json TEXT NOT NULL DEFAULT '[]',
creator_id INTEGER NOT NULL DEFAULT 1,
created_at TEXT NOT NULL DEFAULT (DATETIME('now')),
updated_at TEXT NOT NULL DEFAULT (DATETIME('now'))
)
"""
)
await db.execute(
"""
CREATE TABLE IF NOT EXISTS memo_embeddings (
id INTEGER PRIMARY KEY AUTOINCREMENT,
memo_id INTEGER NOT NULL,
content TEXT,
tags_json TEXT NOT NULL DEFAULT '[]',
date_key TEXT,
embedding_json TEXT NOT NULL,
dim INTEGER NOT NULL,
model TEXT DEFAULT 'stub-embedding',
created_at TEXT NOT NULL DEFAULT (DATETIME('now')),
updated_at TEXT NOT NULL DEFAULT (DATETIME('now'))
)
"""
)
await db.execute(
"CREATE INDEX IF NOT EXISTS idx_memo_embeddings_memo_id ON memo_embeddings (memo_id)"
)
await db.execute(
"CREATE INDEX IF NOT EXISTS idx_memo_embeddings_date_key ON memo_embeddings (date_key)"
)
await db.commit()
async def execute(query: str, params: tuple[Any, ...] = ()) -> int:
"""Execute write query and return lastrowid if available."""
async with aiosqlite.connect(DB_PATH) as db:
cursor = await db.execute(query, params)
await db.commit()
return cursor.lastrowid or 0
async def fetch_one(query: str, params: tuple[Any, ...] = ()) -> dict[str, Any] | None:
async with aiosqlite.connect(DB_PATH) as db:
db.row_factory = aiosqlite.Row
async with db.execute(query, params) as cursor:
row = await cursor.fetchone()
return dict(row) if row else None
async def fetch_all(query: str, params: tuple[Any, ...] = ()) -> list[dict[str, Any]]:
async with aiosqlite.connect(DB_PATH) as db:
db.row_factory = aiosqlite.Row
async with db.execute(query, params) as cursor:
rows = await cursor.fetchall()
return [dict(r) for r in rows]
backend/memos_core/schemas.py
View file @ fbd1e348
......@@ -106,11 +106,31 @@ class MemoUpdate(BaseModel):
tags: Optional[List[str]] = None
class MemoResponse(MemoBase):
id: int
creator_id: int
class MemoResponse(MemoBase):
id: int
creator_id: int
class MemoEmbeddingCreate(BaseModel):
memoId: int
content: str
tags: List[str] = []
dateKey: Optional[str] = None
embedding: List[float]
model: Optional[str] = "stub-embedding"
class MemoEmbeddingResponse(BaseModel):
id: int
memoId: int
content: str
tags: List[str]
dateKey: Optional[str] = None
dim: int
model: str
score: Optional[float] = None
class AttachmentResponse(BaseModel):
id: int
filename: str
......
backend/memos_core/services.py
View file @ fbd1e348
This diff is collapsed.
backend/migrate/001_init_memos_db.py 0 → 100644
View file @ fbd1e348
"""
Migration script: create SQLite memos DB for local/dev.
"""
import asyncio
import os
import sys
from pathlib import Path
ROOT_DIR = Path(__file__).resolve().parent.parent
if str(ROOT_DIR) not in sys.path:
sys.path.insert(0, str(ROOT_DIR))
from memos_core.db import DB_PATH, init_memo_db
async def main():
print(f"Initializing memo DB at: {DB_PATH}")
await init_memo_db()
if os.path.exists(DB_PATH):
size = os.path.getsize(DB_PATH)
print(f"Done. DB file created ({size} bytes).")
else:
print("Warning: DB file not found after init.")
if __name__ == "__main__":
# Ensure parent dir exists
Path(DB_PATH).parent.mkdir(parents=True, exist_ok=True)
asyncio.run(main())
backend/requirements.txt
View file @ fbd1e348
aiofiles==25.1.0
aiosqlite==0.20.0
aiomysql==0.3.2
email-validator==2.2.0
annotated-doc==0.0.4
annotated-types==0.7.0
anyio==4.12.0
......@@ -93,6 +95,7 @@ PyMySQL==1.1.2
pyscn==1.5.5
pytest==9.0.2
python-dotenv==1.2.1
python-multipart==0.0.20
python-engineio==4.12.3
python-socketio==5.15.1
PyYAML==6.0.3
......
backend/server.py
View file @ fbd1e348
import asyncio
import os
import platform
if platform.system() == "Windows":
print("🔧 Windows detected: Applying SelectorEventLoopPolicy globally...")
asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
import logging
import uvicorn
from fastapi import FastAPI
from fastapi.staticfiles import StaticFiles
from fastapi.responses import RedirectResponse
from api.chatbot_route import router as chatbot_router
from api.conservation_route import router as conservation_router
......@@ -20,6 +16,13 @@ from common.cache import redis_cache
from common.langfuse_client import get_langfuse_client
from common.middleware import middleware_manager
from config import PORT
from memos_core.db import init_memo_db
# Windows event loop handling
if platform.system() == "Windows":
print("Windows detected: Applying SelectorEventLoopPolicy globally...")
asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())
# Configure Logging
logging.basicConfig(
......@@ -27,14 +30,13 @@ logging.basicConfig(
format="%(asctime)s [%(levelname)s] %(name)s: %(message)s",
handlers=[logging.StreamHandler()],
)
logger = logging.getLogger(__name__)
langfuse_client = get_langfuse_client()
if langfuse_client:
logger.info("✅ Langfuse client ready (lazy loading)")
logger.info("Langfuse client ready (lazy loading)")
else:
logger.warning("⚠️ Langfuse client not available (missing keys or disabled)")
logger.warning("Langfuse client not available (missing keys or disabled)")
app = FastAPI(
title="Contract AI Service",
......@@ -44,13 +46,15 @@ app = FastAPI(
# =============================================================================
# STARTUP EVENT - Initialize Redis Cache
# STARTUP EVENT - Initialize Redis Cache + Memo DB
# =============================================================================
@app.on_event("startup")
async def startup_event():
"""Initialize Redis cache on startup."""
"""Initialize dependencies on startup."""
await redis_cache.initialize()
logger.info("✅ Redis cache initialized for message limit")
logger.info("Redis cache initialized for message limit")
await init_memo_db()
logger.info("Memo SQLite DB initialized")
# =============================================================================
......@@ -58,10 +62,10 @@ async def startup_event():
# =============================================================================
middleware_manager.setup(
app,
enable_auth=True, # 👈 Bật lại Auth để test logic Guest/User
enable_rate_limit=False, # 👈 Tắt slowapi vì đã có business rate limit
enable_cors=True, # 👈 Bật CORS
cors_origins=["*"], # 👈 Trong production nên limit origins
enable_auth=True, # bật/tắt Auth
enable_rate_limit=False, # tắt slowapi business rate limit
enable_cors=True, # bật CORS
cors_origins=["*"], # trong prod nên giới hạn origins
)
app.include_router(conservation_router)
......@@ -75,25 +79,21 @@ try:
from api.mock_api_route import router as mock_router
app.include_router(mock_router)
print("✅ Mock API Router mounted at /mock")
print("Mock API Router mounted at /mock")
except ImportError:
print("⚠️ Mock Router not found, skipping...")
print("Mock Router not found, skipping...")
# ==========================================
# 🟢 ĐOẠN MOUNT STATIC HTML CỦA BRO ĐÂY 🟢
# Mount static HTML
# ==========================================
try:
static_dir = os.path.join(os.path.dirname(__file__), "static")
if not os.path.exists(static_dir):
os.makedirs(static_dir)
# Mount thư mục static để chạy file index.html
app.mount("/static", StaticFiles(directory=static_dir, html=True), name="static")
print(f"✅ Static files mounted at /static (Dir: {static_dir})")
print(f"Static files mounted at /static (Dir: {static_dir})")
except Exception as e:
print(f"⚠️ Failed to mount static files: {e}")
from fastapi.responses import RedirectResponse
print(f"Failed to mount static files: {e}")
@app.get("/")
......@@ -103,15 +103,15 @@ async def root():
if __name__ == "__main__":
print("=" * 60)
print("🚀 Contract AI Service Starting...")
print("Contract AI Service Starting...")
print("=" * 60)
print(f"📡 REST API: http://localhost:{PORT}")
print(f"📡 Test Chatbot: http://localhost:{PORT}/static/index.html")
print(f"📚 API Docs: http://localhost:{PORT}/docs")
print(f"REST API: http://localhost:{PORT}")
print(f"Test Chatbot: http://localhost:{PORT}/static/index.html")
print(f"API Docs: http://localhost:{PORT}/docs")
print("=" * 60)
ENABLE_RELOAD = False
print(f"⚠️ Hot reload: {ENABLE_RELOAD}")
print(f"Hot reload: {ENABLE_RELOAD}")
reload_dirs = ["common", "api", "agent"]
......
backend/test_full_api_smoke.py 0 → 100644
View file @ fbd1e348
This diff is collapsed.
backend/test_limit.py
View file @ fbd1e348
"""Test message limit - Guest limit = 3"""
"""Test message limit - guest limit = 3."""
import os
import requests
DEVICE_ID = "limit-test-002"
API_URL = "http://localhost:5000/api/agent/chat"
BASE_URL = os.getenv("BASE_URL", "http://localhost:5000")
API_URL = f"{BASE_URL}/api/agent/chat"
def safe_text(value: str | None) -> str:
if value is None:
return ""
return str(value).encode("ascii", "backslashreplace").decode("ascii")
print("=" * 50)
print("TEST MESSAGE LIMIT (Guest = 3 tin/ngày)")
print("TEST MESSAGE LIMIT (Guest = 3 per day)")
print("=" * 50)
print(f"Device ID: {DEVICE_ID}")
print()
for i in range(5): # Gửi 5 tin để thấy bị chặn
print(f"--- Tin nhắn #{i+1} ---")
for i in range(5):
print(f"--- Message #{i + 1} ---")
response = requests.post(
API_URL,
json={"user_query": f"test message {i+1}"},
headers={"device_id": DEVICE_ID}
json={"user_query": f"test message {i + 1}"},
headers={"device_id": DEVICE_ID},
)
data = response.json()
if data.get("status") == "success":
try:
data = response.json()
except ValueError:
data = {}
if response.status_code != 200:
print(f"HTTP {response.status_code}")
print(f" Body: {safe_text(response.text)}")
elif data.get("status") == "success":
limit_info = data.get("limit_info", {})
print(f"✅ Thành công!")
print(f" Used: {limit_info.get('used')}/{limit_info.get('limit')}")
print(f" Remaining: {limit_info.get('remaining')}")
used = limit_info.get("used")
limit = limit_info.get("limit")
remaining = limit_info.get("remaining")
print("OK: allowed")
print(f" Used: {used}/{limit}")
print(f" Remaining: {remaining}")
else:
print(f"❌ Bị chặn!")
print(f" Error: {data.get('error_code')}")
print(f" Message: {data.get('message')}")
print("BLOCKED")
print(f" Error: {safe_text(data.get('error_code'))}")
print(f" Message: {safe_text(data.get('message'))}")
print(f" Require login: {data.get('require_login')}")
limit_info = data.get("limit_info", {})
if limit_info:
print(f" Used: {limit_info.get('used')}/{limit_info.get('limit')}")
if not data:
print(f" Body: {safe_text(response.text)}")
print()
print("=" * 50)
print("TEST HOÀN TẤT!")
print("TEST COMPLETE")
backend/test_limit_v2.py
View file @ fbd1e348
This diff is collapsed.
backend/test_memos_api.py 0 → 100644
View file @ fbd1e348
This diff is collapsed.
backend/test_prompt_update.py
View file @ fbd1e348
This diff is collapsed.
backend/test_revert_prompt.py
View file @ fbd1e348
This diff is collapsed.
backend/test_sqlite_history.py 0 → 100644
View file @ fbd1e348
import os
import asyncio
import sys
import uuid
from pathlib import Path
from fastapi.testclient import TestClient
# Ensure local/dev friendly toggles before importing app
os.environ["DISABLE_AUTH"] = "true"
os.environ["USE_SQLITE_HISTORY"] = "true"
os.environ["REDIS_CACHE_TURN_ON"] = "false"
ROOT_DIR = Path(__file__).resolve().parent
if str(ROOT_DIR) not in sys.path:
sys.path.insert(0, str(ROOT_DIR))
from server import app # noqa: E402
from common.conversation_manager import get_conversation_manager # noqa: E402
def test_sqlite_conversation_manager_roundtrip():
async def _run():
mgr = await get_conversation_manager()
identity_key = f"tester-{uuid.uuid4()}"
await mgr.clear_history(identity_key)
await mgr.save_conversation_turn(identity_key, "hello", '{"ai_response": "world"}')
history = await mgr.get_chat_history(identity_key)
# Two entries: human then AI (ordered DESC by id)
assert len(history) == 2
assert any(not entry["is_human"] for entry in history)
assert any(entry.get("message") == "world" for entry in history)
asyncio.run(_run())
def test_history_endpoint_returns_data():
client = TestClient(app)
# Using random device to avoid collisions
device_id = f"dev-{uuid.uuid4()}"
resp = client.get("/api/history/any", headers={"device_id": device_id})
assert resp.status_code == 200
payload = resp.json()
assert "data" in payload
assert isinstance(payload["data"], list)
frontend/package-lock.json
View file @ fbd1e348
This diff is collapsed.
frontend/src/App.tsx
View file @ fbd1e348
......@@ -2,14 +2,12 @@ import { useEffect } from "react";
import { Outlet } from "react-router-dom";
import { useInstance } from "./contexts/InstanceContext";
import { MemoFilterProvider } from "./contexts/MemoFilterContext";
import useNavigateTo from "./hooks/useNavigateTo";
import { useUserLocale } from "./hooks/useUserLocale";
import { useUserTheme } from "./hooks/useUserTheme";
import { cleanupExpiredOAuthState } from "./utils/oauth";
const App = () => {
const navigateTo = useNavigateTo();
const { profile: instanceProfile, generalSetting: instanceGeneralSetting } = useInstance();
const { generalSetting: instanceGeneralSetting } = useInstance();
// Apply user preferences reactively
useUserLocale();
......@@ -20,12 +18,7 @@ const App = () => {
cleanupExpiredOAuthState();
}, []);
// Redirect to sign up page if no instance owner
useEffect(() => {
if (!instanceProfile.owner) {
navigateTo("/auth/signup");
}
}, [instanceProfile.owner, navigateTo]);
// Auth is disabled for now; skip signup redirect.
useEffect(() => {
if (instanceGeneralSetting.additionalStyle) {
......
frontend/src/components/ChatbotPanel.tsx 0 → 100644
View file @ fbd1e348
This diff is collapsed.
frontend/src/components/ChatbotWidget.tsx 0 → 100644
View file @ fbd1e348
This diff is collapsed.
frontend/src/components/Navigation.tsx
View file @ fbd1e348
This diff is collapsed.
frontend/src/components/UserMenu.tsx
View file @ fbd1e348
This diff is collapsed.
frontend/src/connect.ts
View file @ fbd1e348
This diff is collapsed.
frontend/src/layouts/RootLayout.tsx
View file @ fbd1e348
This diff is collapsed.
frontend/src/main.tsx
View file @ fbd1e348
This diff is collapsed.
frontend/src/router/index.tsx
View file @ fbd1e348
This diff is collapsed.
frontend/src/router/routes.ts
View file @ fbd1e348
This diff is collapsed.
frontend/src/utils/auth-redirect.ts
View file @ fbd1e348
This diff is collapsed.
frontend/vite.config.mts
View file @ fbd1e348
This diff is collapsed.
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment