Update README.md

README.md

@@ -18,7 +18,7 @@ ONNX/RKNN2部署Florence-2视觉多模态大模型!
 
 ## 使用方法
 
-1. 克隆项目到本地
+1. 克隆项目到本地(开发板上)
 
 2. 安装依赖
 
@@ -26,20 +26,14 @@ ONNX/RKNN2部署Florence-2视觉多模态大模型!
 pip install transformers onnxruntime pillow numpy<2 rknn-toolkit-lite2
 ```
 
-3. 修改项目路径
-分词器和预处理配置仍然需要使用原项目中的文件. 将onnx/onnxrun.py或onnx/rknnrun.py中的对应路径修改为项目所在路径. 
-```python
-AutoProcessor.from_pretrained(
-    "path/to/Florence-2-base-ft-ONNX-RKNN2",
-    trust_remote_code=True
-)
-```
-
-4. 运行
+3. 运行
 ```bash
-python onnx/onnxrun.py # 或 python onnx/rknnrun.py
+cd onnx
+python ./run.py ./test.jpg "<MORE_DETAILED_CAPTION>"
 ```
 
+你可以修改`run.py`最上方的import来切换使用ONNX或RKNN推理。
+
 ## RKNN模型转换
 
 你需要提前安装rknn-toolkit2 v2.3.2或更高版本. 
@@ -56,6 +50,7 @@ python convert.py all
 - ~~在同分辨率下推理精度相比onnxruntime有显著下降, 已确定问题出在vision encoder部分. 如果对精度要求高, 这部分可以换成onnxruntime~~ (已解决, 不过现在貌似仍然会丢失一点精度)
 - ~~vision encoder推理需要1.5秒, 占比最大, 但其中大半时间都在做Transpose, 也许还可以优化~~ (已优化)
 - decode阶段因为kvcache的长度不断变化, 貌似无法简单的使用NPU推理. 不过用onnxruntime应该也足够了
+- 理论上可以使用rkllm推理decoder, 但因为rkllm缺少固定位置编码的支持暂时无法实现。参考：https://github.com/airockchip/rknn-llm/issues/296
 
 ## 参考
 - [microsoft/Florence-2-base-ft](https://huggingface.co/microsoft/Florence-2-base-ft)
@@ -65,57 +60,48 @@ python convert.py all
 
 # English README
 
-# Florence-2-base-ft-ONNX-RKNN2
-
-ONNX/RKNN2 deployment for Florence-2 visual-language multimodal large model!
+Deploy Florence-2 vision multi-modal large model with ONNX/RKNN2!
 
-- Inference speed (RKNN2): RK3588 inference with a 768x768 image, using the `<MORE_DETAILED_CAPTION>` instruction, takes ~4 seconds in total.
-- Memory usage (RKNN2): Approximately 2GB
+- **Inference Speed (RKNN2):** On RK3588, inferring a 768x768 image with the `<MORE_DETAILED_CAPTION>` instruction takes ~4 seconds in total.
+- **Memory Usage (RKNN2):** About 2GB.
 
 ## Usage
 
-1. Clone the project locally
-
-2. Install dependencies
+1. Clone the project locally (on your development board).
 
+2. Install dependencies:
 ```bash
 pip install transformers onnxruntime pillow numpy<2 rknn-toolkit-lite2
 ```
 
-3. Modify project paths
-The tokenizer and preprocessing configurations still need to use files from the original project. Modify the corresponding paths in onnx/onnxrun.py or onnx/rknnrun.py to the project's location.
-```python
-AutoProcessor.from_pretrained(
-    "path/to/Florence-2-base-ft-ONNX-RKNN2",
-    trust_remote_code=True
-)
-```
-
-4. Run
+3. Run:
 ```bash
-python onnx/onnxrun.py # or python onnx/rknnrun.py
+cd onnx
+python ./run.py ./test.jpg "<MORE_DETAILED_CAPTION>"
 ```
 
+You can modify the import at the top of `run.py` to switch between ONNX and RKNN inference.
+
 ## RKNN Model Conversion
 
-You need to install rknn-toolkit2 v2.3.2 or higher in advance.
+You need to install rknn-toolkit2 v2.3.2 or a higher version beforehand.
 
 ```bash
 cd onnx
 python convert.py all
 ```
 
-Note: The RKNN model does not support arbitrary input lengths at runtime, so you need to determine the input shape in advance. You can modify `vision_size`, `vision_tokens`, and `prompt_tokens` in convert.py to change the input shape.
+Note: RKNN models do not support dynamic input shapes at runtime, so you need to define the input shape in advance. You can modify `vision_size`, `vision_tokens`, and `prompt_tokens` in `convert.py` to change the input shape.
 
-## Existing Issues (rknn)
+## Known Issues (rknn)
 
-- ~~It is known that converting the vision encoder with an input resolution >= 640x640 will result in a `Buffer overflow!` error and conversion failure. Therefore, the current input is 512x512, which will reduce inference quality.~~ (Solved.)
-- ~~Inference accuracy is significantly lower compared to onnxruntime at the same resolution. The problem has been identified in the vision encoder part. If high accuracy is required, this part can be replaced with onnxruntime.~~ (Solved, although there is still some precision loss present.)
-- ~~Vision encoder inference takes 1.5 seconds, accounting for the largest proportion, but most of this time is spent on Transpose op, which may be further optimized.~~ (Solved.)
-- The decode phase seems unable to simply use NPU inference because the length of kvcache keeps changing. However, using onnxruntime should be sufficient.
+- ~~When converting the vision encoder, an input resolution of >=640x640 would cause a `Buffer overflow!` error and fail the conversion. Therefore, a 512x512 input was used, which degraded inference quality.~~ (Resolved)
+- ~~Inference precision was significantly lower compared to onnxruntime at the same resolution. The issue was identified in the vision encoder part. For higher precision requirements, onnxruntime could be used for this part.~~ (Resolved, though there might still be a slight loss of precision)
+- ~~Vision encoder inference took 1.5 seconds, accounting for the largest portion of time, with most of it spent on Transpose operations. There might be room for optimization.~~ (Optimized)
+- In the decode stage, due to the constantly changing length of the kvcache, it seems that NPU inference cannot be simply used. However, onnxruntime should be sufficient.
+- Theoretically, rkllm could be used for decoder inference, but it's not currently feasible due to the lack of support for fixed positional encoding in rkllm. See: https://github.com/airockchip/rknn-llm/issues/296
 
 ## References
-
 - [microsoft/Florence-2-base-ft](https://huggingface.co/microsoft/Florence-2-base-ft)
 - [onnx-community/Florence-2-base-ft](https://huggingface.co/onnx-community/Florence-2-base-ft)
-- [florence2-webgpu](https://huggingface.co/spaces/Xenova/florence2-webgpu)
+- [florence2-webgpu](https://huggingface.co/spaces/Xenova/florence2-webgpu)