# 🚀 DOING: Auto-Improve Prompt — Tự động cải tiến Prompt

## 📁 Folders & Files liên quan
```
backend/
├── agent/
│   ├── feedback_agent/
│   │   ├── langfuse_puller.py        ← MODIFY (thay mock → query autotrain.db)
│   │   └── learning_loop.py          ← MODIFY (thay mock → LLM thật via create_llm)
│   └── prompt_module/
│       ├── 01_core.txt ... 07_*.txt  ← READ/WRITE (prompt source + deploy target)
│       └── push_modules_to_langfuse.py ← OK (không đổi)
├── api/
│   ├── autotrain/
│   │   ├── autotrain_route.py        ← MODIFY (_ai_generate_rule → LLM thay keyword)
│   │   └── auto_pipeline_route.py    ← 🆕 NEW (pipeline orchestrator)
│   └── prompt_optimizer/
│       └── prompt_optimizer_route.py  ← OK (judge logic tái dùng, không đổi)
└── database/
    └── autotrain.db                   ← MODIFY (thêm bảng pipeline_runs)
```

## 📌 Context
- **Status:** 🚧 In Progress

## 🎯 Goal
Closed-loop pipeline: Bad feedback → AI phân tích → Sinh prompt rule → Judge đánh giá → Deploy → Regression test. **Thay 100% mock bằng LLM thật**, lưu pipeline history vào SQLite, đọc prompt từ `prompt_module/*.txt` local.

## ⚠️ Active Trade-offs
- Storage: SQLite local (dùng chung `database/autotrain.db` đã có sẵn)
- Prompt source: `agent/prompt_module/*.txt` (7 files, đọc trực tiếp)
- LLM: dùng `create_llm(DEFAULT_MODEL)` có sẵn (không thêm dependency mới)
- Feedback source: fallback từ autotrain feedbacks table (đã có) thay vì bắt buộc Langfuse SDK

---

## 🔍 Hiện trạng — 3 module rời rạc, đầy mock

### Module A: `feedback_agent/` — ❌ 100% MOCK
```python
# langfuse_puller.py — line 13:
# TODO: Khởi tạo db connect / langfuse client thật ở đây khi có Auth Key

# learning_loop.py — line 40:
# TODO: Tích hợp langchain_google_genai hoặc API Gemini thật tại đây...
```
- `langfuse_puller.py`: return hardcoded 2 feedbacks
- `learning_loop.py`: return hardcoded negative_matches JSON

### Module B: `api/prompt_optimizer/` — ✅ ~80% Done
- `/run-test`: gọi chatbot API lấy response thật ✅
- `/judge`: AI Judge 5 tiêu chí ✅
- `/optimize`: AI rewrite prompt ✅
- ❌ Thiếu closed-loop endpoint

### Module C: `api/autotrain/` — ⚠️ 60% Done
- CRUD feedbacks/rules/deploy: ✅
- `_ai_generate_rule()`: ❌ **pattern-matching cứng** — chỉ check keyword "nóng/lùn/emoji"
- ❌ Không dùng LLM

### ❌ Vấn đề: 3 module KHÔNG kết nối
```
feedback_agent (MOCK) ──╮
                        ├── Không có orchestrator
prompt_optimizer (80%) ──╯
autotrain (60% MOCK) ──── Standalone
```

---

## 🛠 Tao sẽ làm gì & làm như nào

### 🏗️ Pipeline sau khi hoàn thiện:

```
┌─────────────────────────────────────────────────────────┐
│  AUTO-IMPROVE PIPELINE (1 endpoint trigger hết)         │
│                                                         │
│  Step 1: Pull bad feedbacks                             │
│  ┌───────────────────────────────────────┐              │
│  │ autotrain.db → feedbacks (status=     │              │
│  │ 'pending') HOẶC Langfuse API (nếu có)│              │
│  └────────────────┬──────────────────────┘              │
│                   ▼                                     │
│  Step 2: AI Generate Rules (LLM thật)                   │
│  ┌───────────────────────────────────────┐              │
│  │ create_llm(DEFAULT_MODEL)             │              │
│  │ Input: feedback + current prompt      │              │
│  │ Output: {rule_type, new_rule,         │              │
│  │         reasoning, confidence}        │              │
│  └────────────────┬──────────────────────┘              │
│                   ▼                                     │
│  Step 3: Judge quality (prompt_optimizer/judge)          │
│  ┌───────────────────────────────────────┐              │
│  │ Test rule bằng cách chạy chatbot      │              │
│  │ → Judge đánh giá 5 tiêu chí          │              │
│  │ → Score ≥ 0.7 → auto-approve         │              │
│  └────────────────┬──────────────────────┘              │
│                   ▼                                     │
│  Step 4: Deploy (ghi vào prompt_module/*.txt)            │
│  ┌───────────────────────────────────────┐              │
│  │ Backup file cũ → append rule mới     │              │
│  │ Log vào autotrain.db deployments      │              │
│  └────────────────┬──────────────────────┘              │
│                   ▼                                     │
│  Step 5: Verify (regression test + user simulator)       │
│  ┌───────────────────────────────────────┐              │
│  │ Chạy regression_test với chatbot URL  │              │
│  │ (Optional) Trigger simulator #16      │              │
│  │ Log results vào pipeline_runs table   │              │
│  └───────────────────────────────────────┘              │
└─────────────────────────────────────────────────────────┘
```

### 📐 Pattern theo project:
- DB: mở rộng `database/autotrain.db` (thêm bảng `pipeline_runs`)
- LLM: `from common.llm_factory import create_llm` (có sẵn, 0 dependency mới)
- Prompt files: đọc/ghi trực tiếp `agent/prompt_module/02_rules.txt` etc.
- Connection: `sqlite3.connect()` sync (y hệt autotrain_route.py)

---

## 📋 Execution Checklist

### Phase 1: Fix `feedback_agent` — Thay MOCK bằng LLM (~20m)
- [x] Task 1.1: `langfuse_puller.py` — Thay mock `fetch_bad_feedbacks()`:
  - Primary: query `autotrain.db` → `SELECT * FROM feedbacks WHERE status='pending'` (data đã có sẵn từ autotrain!)
  - Optional fallback: nếu có `LANGFUSE_SECRET_KEY` trong env → dùng Langfuse API
- [x] Task 1.2: `learning_loop.py` — Thay mock `run_optimization_logic()`:
  - Dùng `create_llm(DEFAULT_MODEL, json_mode=True)` (có sẵn trong project)
  - Input: feedbacks + đọc prompt từ `prompt_module/*.txt`
  - Output: JSON rules đã validate
- [x] Task 1.3: Thêm output validation — parse JSON, reject nếu `new_rule` < 10 chars hoặc `confidence` < 0.3

### Phase 2: Fix `autotrain` — Thay pattern-matching bằng LLM (~15m)
- [x] Task 2.1: `autotrain_route.py` → `_ai_generate_rule()`:
  - Thay if/else keyword matching bằng LLM call
  - Prompt: "Bạn là Fashion QA Agent. Dựa trên feedback: {feedback_text} về query: {query}, sinh 1 rule fix..."
  - Giữ pattern-matching cũ làm fallback nếu LLM timeout/fail
- [x] Task 2.2: Auto-approve rule nếu confidence ≥ 0.85
- [x] Task 2.3: Deploy target mở rộng — ngoài `fashion_rules.json`, thêm support deploy vào `prompt_module/*.txt`

### Phase 3: Pipeline Orchestrator (~25m)
- [x] Task 3.1: Thêm bảng `pipeline_runs` vào `autotrain.db`:
  ```sql
  CREATE TABLE pipeline_runs (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    run_id TEXT UNIQUE,
    started_at TIMESTAMP,
    completed_at TIMESTAMP,
    status TEXT DEFAULT 'running',  -- running/success/failed/rolled_back
    feedbacks_processed INTEGER,
    rules_generated INTEGER,
    rules_approved INTEGER,
    rules_deployed INTEGER,
    regression_passed BOOLEAN,
    summary_json TEXT,
    prompt_version_before TEXT,
    prompt_version_after TEXT
  )
  ```
- [x] Task 3.2: Tạo `api/autotrain/auto_pipeline_route.py`:
  - `POST /autotrain/pipeline/run` — trigger full pipeline (5 steps)
  - `GET /autotrain/pipeline/history` — list past runs
  - `GET /autotrain/pipeline/history/{run_id}` — chi tiết 1 run
  - `POST /autotrain/pipeline/rollback/{run_id}` — rollback prompt file từ backup
- [x] Task 3.3: Pipeline orchestrator logic:
  ```python
  async def run_pipeline():
      # Step 1: Pull feedbacks từ autotrain.db
      feedbacks = get_pending_feedbacks()
      # Step 2: Generate rules bằng LLM
      for fb in feedbacks:
          rule = await _ai_generate_rule_llm(fb)
      # Step 3: Judge quality
      for rule in rules:
          score = await judge_rule(rule)
          if score >= 0.7: approve(rule)
      # Step 4: Deploy approved rules
      for rule in approved:
          deploy_to_prompt_file(rule)
      # Step 5: Verify
      regression_result = await run_regression()
      if not regression_result.passed:
          rollback_all()
  ```
- [x] Task 3.4: Register router trong `main_router.py`

### Phase 4: Prompt file read/write (~10m)
- [x] Task 4.1: Helper `read_prompt_module(module_name)` — đọc `prompt_module/{module}.txt`
- [x] Task 4.2: Helper `append_rule_to_prompt(module_name, rule_text)` — backup file cũ + append rule
- [x] Task 4.3: Helper `get_prompt_version()` — hash nội dung 7 files → version string
- [x] Task 4.4: Helper `rollback_prompt(module_name)` — restore từ backup

### Phase 5: Kết nối với Doings #16 (Optional) (~5m)
- [x] Task 5.1: Sau pipeline deploy → trigger `run_simulation()` nếu simulator available
- [x] Task 5.2: Log conversion rate trước/sau vào `pipeline_runs.summary_json`

### Phase 6: Testing (~15m)
- [x] Task 6.1: Test `_ai_generate_rule()` mới — submit 3 feedbacks → verify LLM trả JSON hợp lệ
- [x] Task 6.2: Test pipeline endpoint — full flow feedback → rule → deploy
- [x] Task 6.3: Test rollback — deploy → force fail → rollback → verify file restored
- [x] Task 6.4: Verify `autotrain.db` tables đúng schema

## 🔄 Sub-Doings
_None yet_

## ✅ Completion Gate
- [x] All tasks [x] | [x] Sub-Doings resolved | [x] Smoke test passed | [x] Trade-offs confirmed

---
_Started: 2026-05-10 | Updated: 2026-05-10_
