120 lines
4.4 KiB
Markdown
120 lines
4.4 KiB
Markdown
# AI Coding Agent Instructions
|
|
|
|
## Project Overview
|
|
This is an OpenPose-based image generation server that integrates with ComfyUI for AI image generation. The system processes pose coordinates to generate skeleton images and sends them to ComfyUI for final image generation.
|
|
|
|
## Architecture & Key Components
|
|
|
|
### Core Service Flow
|
|
1. **Flask API** (`app.py`) receives pose coordinates via REST endpoints
|
|
2. **OpenPose Generation** (`openpose_gen.py`) converts coordinates to skeleton images
|
|
3. **ComfyUI Integration** uploads images and queues generation workflows
|
|
4. **Skeleton Library** (`skeleton_lib.py`) handles pose drawing with different formats (COCO, Body25)
|
|
|
|
### Critical Dependencies
|
|
- **ComfyUI Server**: External service at `localhost:8188` for AI image generation
|
|
- **OpenCV**: For image processing and skeleton rendering
|
|
- **Flask**: REST API server
|
|
- **requests-toolbelt**: For multipart file uploads to ComfyUI
|
|
|
|
## Development Patterns
|
|
|
|
### Coordinate System Convention
|
|
- Input coordinates are flat arrays: `[x1, y1, confidence1, x2, y2, confidence2, ...]`
|
|
- Use `coordinates_to_keypoints()` to convert to `Keypoint` objects
|
|
- Support both single pose (`/gen_image`) and multi-pose (`/gen_group_pic`) workflows
|
|
|
|
### File Naming & Counters
|
|
- Output images use incremental counters: `body_pose_output0000.png`, `body_pose_output0001.png`
|
|
- Each function maintains its own static counter using `hasattr()` pattern
|
|
- Circular queue naming for ComfyUI uploads with hash-based names
|
|
|
|
### MCP Integration
|
|
- Use `context7` defined in `.vscode/mcp.json` for coding tasks
|
|
- Use `python-language-server` MCP for Python coding assistance
|
|
- Use `analyzer` MCP for code analysis tasks
|
|
|
|
### ComfyUI Workflow Integration
|
|
- Workflow templates stored as JSON: `fencerAPI.json`, `group_pic.json`
|
|
- Modify seed values for randomization: `prompt["3"]["inputs"]["seed"] = random.randint(0, 10000000000)`
|
|
- Reference uploaded images by name in workflow nodes: `prompt["17"]["inputs"]['image'] = openpose_image_name`
|
|
|
|
### Error Handling Pattern
|
|
```python
|
|
if not coordinates or not canvas_size:
|
|
return jsonify({"status": "error", "message": "Missing data"}), 422
|
|
```
|
|
|
|
## Key Functions to Understand
|
|
|
|
### `save_bodypose()` / `save_bodypose_mulit()`
|
|
- Converts coordinates to skeleton images using CV2
|
|
- Creates output directory if missing
|
|
- Returns image path for ComfyUI upload
|
|
|
|
### `upload_image_circular_queue()`
|
|
- Manages unique image names per user/session using SHA256 hash
|
|
- Implements circular queue to prevent infinite file accumulation
|
|
- Essential for ComfyUI integration
|
|
|
|
### `queue_prompt()`
|
|
- Sends workflow JSON to ComfyUI `/prompt` endpoint
|
|
- Triggers actual AI image generation
|
|
|
|
## Development Workflow
|
|
|
|
### Testing API Endpoints
|
|
```bash
|
|
# Single pose generation
|
|
curl -X POST -H "Content-Type: application/json" \
|
|
-d '{"coordinates": [x1,y1,conf1,...], "canvas_size": [width,height], "pid": "user123"}' \
|
|
http://localhost:5000/gen_image
|
|
```
|
|
|
|
### Running the Server
|
|
```bash
|
|
python app.py # Starts Flask in debug mode on localhost:5000
|
|
```
|
|
|
|
### Skeleton Format Support
|
|
- **COCO format**: 18 keypoints (default for single poses)
|
|
- **Body25 format**: 25 keypoints (used in `main()` function)
|
|
- Use corresponding `limbSeq` and `colors` arrays from `skeleton_lib.py`
|
|
|
|
## Integration Points
|
|
|
|
### ComfyUI Server Requirements
|
|
- Must be running on `localhost:8188`
|
|
- Requires `/upload/image` and `/prompt` endpoints
|
|
- Workflow JSON files must match ComfyUI node structure
|
|
|
|
### Output Directory Structure
|
|
```
|
|
output/ # Generated skeleton images
|
|
embeddings/ # ComfyUI embeddings and models
|
|
script_examples/ # API usage examples
|
|
```
|
|
|
|
## Common Modifications
|
|
|
|
## Adding New API Endpoints
|
|
1. Define new Flask route in `app.py`
|
|
2. Create corresponding handler function
|
|
3. Follow existing patterns for input validation, image generation, and ComfyUI queuing
|
|
|
|
### Adding New Pose Formats
|
|
1. Define new `limbSeq` and `colors` in `skeleton_lib.py`
|
|
2. Update coordinate conversion in `coordinates_to_keypoints()`
|
|
3. Modify canvas drawing in `save_bodypose()`
|
|
|
|
### New ComfyUI Workflows
|
|
1. Export workflow from ComfyUI as JSON
|
|
2. Save in project root (e.g., `new_workflow.json`)
|
|
3. Create API function following `gen_fencer_prompt()` pattern
|
|
4. Add Flask endpoint in `app.py`
|
|
|
|
### Debugging ComfyUI Integration
|
|
- Check ComfyUI server status at `http://localhost:8188`
|
|
- Verify uploaded images in ComfyUI interface
|
|
- Monitor workflow queue for errors
|
|
- Use `script_examples/` for isolated testing |