# Auto-Train System — Cookbook

## 1. Overview

**Auto-Train** là hệ thống tự động cập nhật prompt rules từ user feedback. Đây là pipeline end-to-end:

```
User Feedback → AI Analysis → Rule Suggestion → Admin Review → Deploy → Production
```

---

## 2. Architecture Flow

```mermaid
graph LR
    A[User submits feedback] --> B[API: POST /autotrain/feedback]
    B --> C[Store in SQLite]
    C --> D[Background worker]
    D --> E[AI generates rule]
    E --> F[Store as pending]
    F --> G[Admin reviews]
    G --> H{Approve?}
    H -->|Yes| I[Deploy to production]
    H -->|No| J[Reject]
    I --> K[Next query uses new rule]
```

---

## 3. Database Schema

### `feedbacks` table
```json
{
  "id": "INT PK",
  "feedback_id": "TEXT UNIQUE (FB-XXXX)",
  "user_id": "TEXT",
  "query": "TEXT",
  "ai_response": "TEXT",
  "feedback_text": "TEXT",
  "feedback_type": "bug|suggestion|praise",
  "status": "pending|processing|processed|rejected"
}
```

### `generated_rules` table
```json
{
  "id": "INT PK",
  "rule_id": "TEXT UNIQUE (RUL-XXXX)",
  "feedback_id": "TEXT FK",
  "module_name": "fashion_rules.json",
  "rule_type": "prompt_update|system_message",
  "old_rule": "TEXT",
  "new_rule": "TEXT",
  "reasoning": "TEXT",
  "confidence": "0.0-1.0",
  "status": "pending|approved|rejected|deployed"
}
```

---

## 4. API Endpoints

| Method | Endpoint | Purpose |
|--------|----------|---------|
| POST | `/autotrain/feedback` | Submit feedback |
| GET | `/autotrain/feedback` | List feedbacks |
| POST | `/autotrain/generate/{feedback_id}` | Generate rule from feedback |
| GET | `/autotrain/rules` | List rules |
| POST | `/autotrain/rules/{rule_id}/approve` | Approve rule |
| POST | `/autotrain/rules/{rule_id}/deploy` | Deploy rule |
| GET | `/autotrain/status` | System status |
| POST | `/autotrain/run-all` | Process all pending |

---

## 5. Mock AI Rule Generation

Current implementation uses **pattern matching**:

```python
def _ai_generate_rule(feedback):
    fb_text = feedback['feedback_text'].lower()

    if "nóng" in fb_text and "áo khoác" in query:
        return {
            "old_rule": "- Luôn ưu tiên áo khoác/dày",
            "new_rule": "- Khi trời nóng (>30°): KHÔNG gợi ý áo khoác dày",
            "reasoning": "Weather mismatch correction"
        }
    # ... more patterns
```

**Production**: Replace with LLM call:
```python
response = claude.messages.create(
    model="claude-3",
    messages=[{
        "role": "user",
        "content": f"Given feedback: {feedback}, suggest a prompt rule..."
    }]
)
```

---

## 6. Deployment Flow (Production)

Currently `deploy_rule()` only logs to DB. Production implementation:

```python
async def deploy_rule(rule_id: str):
    # 1. Fetch rule
    rule = await get_rule(rule_id)

    # 2. Load target file (fashion_rules.json or prompt_v2.txt)
    target_path = CONFIG_DIR / rule['module_name']
    content = json.load(open(target_path))

    # 3. Apply rule update
    content['rules'].append({
        "rule": rule['new_rule'],
        "added_from": rule['feedback_id'],
        "confidence": rule['confidence']
    })

    # 4. Write back (with backup)
    backup(target_path)
    json.dump(content, open(target_path, 'w'), indent=2)

    # 5. Reload chatbot config
    await reload_chatbot_config()

    # 6. Log deployment
    await log_deployment(rule_id)

    return {"status": "deployed"}
```

---

## 7. Testing Checklist

- [ ] Submit feedback → DB stores correctly
- [ ] Generate rule → Mock AI produces relevant rule
- [ ] Approve rule → Status changes to 'approved'
- [ ] Deploy → Status changes to 'deployed', file updated (production)
- [ ] `run-all` → Processes all pending feedbacks
- [ ] Concurrent deploy → No race conditions
- [ ] Invalid rule_id → 404 error

---

## 8. Extending the System

### Add new AI pattern
```python
# In _ai_generate_rule():
if "pattern_keyword" in fb_text:
    return {
        "rule_type": "prompt_update",
        "new_rule": "- Your rule here",
        "reasoning": "Your reasoning",
        "confidence": 0.85
    }
```

### Add new module type
```python
# Update generated_rules.module_name values
ALLOWED_MODULES = [
    "fashion_rules.json",
    "prompt_v2.txt",
    "stylist_guidelines.md"
]
```

### Hook to real LLM
Replace `_ai_generate_rule()` with:
```python
async def _ai_generate_rule(feedback):
    prompt = build_prompt(feedback)
    resp = await anthropic_client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=1000,
        messages=[{"role": "user", "content": prompt}]
    )
    return parse_llm_response(resp.content[0].text)
```

---

## 9. Monitoring & Alerting

Recommended metrics to add:

| Metric | Threshold | Alert |
|--------|-----------|-------|
| Pending feedbacks > 100 | Warning | Slack |
| Rule generation error rate > 5% | Critical | PagerDuty |
| Deployed rules per day > 50 | Info | — |
| Avg approval time > 24h | Warning | Email |

---

## 10. Security Considerations

- **Authentication**: Add JWT auth to all endpoints (currently open)
- **Authorization**: Only admin can approve/deploy
- **Input validation**: Pydantic already validates; add profanity filter for feedback_text
- **Rate limiting**: Prevent spam feedback (10 per user/hour)
- **Audit log**: Track who deployed what and when (already in deployments table)

---

## 11. Future Enhancements

1. **A/B Testing**: Deploy rules to 10% of users first
2. **Rollback**: One-click revert to previous rule version
3. **Confidence Threshold**: Auto-approve rules with confidence > 0.9
4. **Feedback Clustering**: Group similar feedback before rule generation
5. **Webhook Notifications**: Notify admin on new rule suggestion
6. **Version Control**: Store rule history in Git, not just DB

---

## 12. Quick Start

```bash
# Start server
uvicorn api.main:app --reload

# Submit feedback
curl -X POST http://localhost:8000/autotrain/feedback \
  -H "Content-Type: application/json" \
  -d '{
    "query": "tìm áo khoác cho trời 35 độ",
    "ai_response": "Mình gợi ý áo khoác nỉ này...",
    "feedback_text": "Trời nóng quá, AI còn tư vấn áo khoác",
    "feedback_type": "bug"
  }'

# Generate rule
curl -X POST http://localhost:8000/autotrain/generate/FB-1001

# Approve
curl -X POST http://localhost:8000/autotrain/rules/RUL-1001/approve

# Deploy
curl -X POST http://localhost:8000/autotrain/rules/RUL-1001/deploy

# Check status
curl http://localhost:8000/autotrain/status
```
