Update submission guidelines: new review policy and v2 schema

.gradio/certificate.pem

@@ -0,0 +1,31 @@
+-----BEGIN CERTIFICATE-----
+MIIFazCCA1OgAwIBAgIRAIIQz7DSQONZRGPgu2OCiwAwDQYJKoZIhvcNAQELBQAw
+TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
+cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTUwNjA0MTEwNDM4
+WhcNMzUwNjA0MTEwNDM4WjBPMQswCQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJu
+ZXQgU2VjdXJpdHkgUmVzZWFyY2ggR3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBY
+MTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAK3oJHP0FDfzm54rVygc
+h77ct984kIxuPOZXoHj3dcKi/vVqbvYATyjb3miGbESTtrFj/RQSa78f0uoxmyF+
+0TM8ukj13Xnfs7j/EvEhmkvBioZxaUpmZmyPfjxwv60pIgbz5MDmgK7iS4+3mX6U
+A5/TR5d8mUgjU+g4rk8Kb4Mu0UlXjIB0ttov0DiNewNwIRt18jA8+o+u3dpjq+sW
+T8KOEUt+zwvo/7V3LvSye0rgTBIlDHCNAymg4VMk7BPZ7hm/ELNKjD+Jo2FR3qyH
+B5T0Y3HsLuJvW5iB4YlcNHlsdu87kGJ55tukmi8mxdAQ4Q7e2RCOFvu396j3x+UC
+B5iPNgiV5+I3lg02dZ77DnKxHZu8A/lJBdiB3QW0KtZB6awBdpUKD9jf1b0SHzUv
+KBds0pjBqAlkd25HN7rOrFleaJ1/ctaJxQZBKT5ZPt0m9STJEadao0xAH0ahmbWn
+OlFuhjuefXKnEgV4We0+UXgVCwOPjdAvBbI+e0ocS3MFEvzG6uBQE3xDk3SzynTn
+jh8BCNAw1FtxNrQHusEwMFxIt4I7mKZ9YIqioymCzLq9gwQbooMDQaHWBfEbwrbw
+qHyGO0aoSCqI3Haadr8faqU9GY/rOPNk3sgrDQoo//fb4hVC1CLQJ13hef4Y53CI
+rU7m2Ys6xt0nUW7/vGT1M0NPAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNV
+HRMBAf8EBTADAQH/MB0GA1UdDgQWBBR5tFnme7bl5AFzgAiIyBpY9umbbjANBgkq
+hkiG9w0BAQsFAAOCAgEAVR9YqbyyqFDQDLHYGmkgJykIrGF1XIpu+ILlaS/V9lZL
+ubhzEFnTIZd+50xx+7LSYK05qAvqFyFWhfFQDlnrzuBZ6brJFe+GnY+EgPbk6ZGQ
+3BebYhtF8GaV0nxvwuo77x/Py9auJ/GpsMiu/X1+mvoiBOv/2X/qkSsisRcOj/KK
+NFtY2PwByVS5uCbMiogziUwthDyC3+6WVwW6LLv3xLfHTjuCvjHIInNzktHCgKQ5
+ORAzI4JMPJ+GslWYHb4phowim57iaztXOoJwTdwJx4nLCgdNbOhdjsnvzqvHu7Ur
+TkXWStAmzOVyyghqpZXjFaH3pO3JLF+l+/+sKAIuvtd7u+Nxe5AW0wdeRlN8NwdC
+jNPElpzVmbUq4JUagEiuTDkHzsxHpFKVK7q4+63SM1N95R1NbdWhscdCb+ZAJzVc
+oyi3B43njTOQ5yOf+1CceWxG1bQVs5ZufpsMljq4Ui0/1lvh+wjChP4kqKOJ2qxq
+4RgqsahDYVvTH9w7jXbyLeiNdd8XM2w9U/t7y0Ff/9yi0GE44Za4rF2LN9d11TPA
+mRGunUHBcnWEvgJBQl9nJEiU0Zsnvgc/ubhPgXRR4Xq37Z0j4r7g1SgEEzwxA57d
+emyPxgcYxn/eR44/KJ4EBs+lVDR3veyJm+kXQ99b21/+jh5Xos1AnX5iItreGCc=
+-----END CERTIFICATE-----

content.py

@@ -19,7 +19,7 @@ Our goal is to conduct a rigorous assessment of the current state of web agents.
 
 When using our benchmark or submitting results, please first carefully review the important notes to ensure proper usage and obtain reliable evaluation results and follow the "Submission Guideline".
 
-**We usually need about one week to review the results. If your results require urgent verification, please let us know in advance. Thank you for your understanding.**
+**We have outsourced human evaluation, so submissions can now be reviewed within a few days. For pricing and review details, see [this link](https://github.com/OSU-NLP-Group/Online-Mind2Web/blob/main/data/schema_v2/HUMAN_EVALUATION_PRICING.md).**
 
 ### ⚠ Important Notes for Reliable Evaluation:
 - **Start from the specified websites, not Google Search**: To enable fair comparisons, please ensure that each task starts from the specified website in our benchmark. Starting from Google Search or alternative websites can lead agents to use different websites to solve the task, resulting in varying difficulty levels and potentially skewed evaluation results.
@@ -29,16 +29,18 @@ When using our benchmark or submitting results, please first carefully review th
 To obtain more reliable automatic evaluation results, the action representation should be as detailed as possible, including only factual actions and excluding any agent outputs. Here is an example [script](https://github.com/OSU-NLP-Group/Online-Mind2Web/blob/main/src/clean_html.py) to process the element's HTML as the action representation. It can preserve valuable information while filtering out irrelevant attributes.
 
 **Please do not use it as training data for your agent.**
-
-## News
-- [11/03/2025] We’ve updated 36 tasks that are no longer valid or involve websites with CAPTCHA verification. Please check out the updated tasks! (We added a date suffix to the updated task IDs to distinguish them from the previous versions.)
 """
 
 SUBMISSION_TEXT = """
 ## Submissions
-Participants are invited to submit your agent's trajectory to test. The submissions will be evaluated based on our auto-eval.
+Participants are invited to submit your agent's trajectory for evaluation.
+### Review policy
+- **Auto-eval**: We provide free review for auto-eval submissions.
+- **Human eval**: Human evaluation is outsourced for submission review. For pricing and review details, see [this link](https://github.com/OSU-NLP-Group/Online-Mind2Web/blob/main/data/schema_v2/HUMAN_EVALUATION_PRICING.md).
+- **Academic submissions**: For submissions from academia, the Online-Mind2Web team can still provide free evaluation. Turnaround is usually longer than outsourced human evaluation, so please notify us **1-2 weeks in advance**.
+
 ### Format of submission
-Submissions must include a sequence of images (i.e., screenshots in the trajectory) and a result.json file for each task. The JSON file should contain the fields: "Task", "Task_id", and "action_history". You can refer to an example of the submission files.
+All submissions must follow the **v2 submission schema** (`online-mind2web-v2`). Each task directory should contain a `result.json` (conforming to the v2 schema) and a `trajectory/` folder with per-step screenshots. Please refer to the "Submission Guideline" tab for details.
 """
 
 EVALUATION_DETAILS = """
@@ -72,21 +74,31 @@ CITATION_BUTTON_TEXT = r"""
 """
 
 SUBMIT_INTRODUCTION = """
-You should use the script provided in our GitHub [repository](https://github.com/OSU-NLP-Group/Online-Mind2Web) to obtain automatic evaluation results on your own and submit them along with all trajectories to enhance transparency. 
+You should use the script provided in our GitHub [repository](https://github.com/OSU-NLP-Group/Online-Mind2Web) to obtain automatic evaluation results on your own and submit them along with all trajectories to enhance transparency.
 To ensure the authenticity and reliability of the reported results, we will also verify the auto-eval results.
 If you have conducted your own human evaluation, please also attach your human-eval results. We will spot-check these before adding them to the human-eval table.
 
+## Review policy
+- **Auto-eval**: We provide free review for auto-eval submissions.
+- **Human eval**: Human evaluation is outsourced for submission review. For pricing and review details, see [this link](https://github.com/OSU-NLP-Group/Online-Mind2Web/blob/main/data/schema_v2/HUMAN_EVALUATION_PRICING.md).
+- **Academic submissions**: For submissions from academia, the Online-Mind2Web team can still provide free evaluation. Turnaround is usually longer than outsourced human evaluation, so please notify us **1-2 weeks in advance**.
+
 ## ⚠ Important Notes for Reliable Evaluation:
-- **Start from the specified websites, not Google Search**:To enable fair comparisons, please ensure that each task starts from the specified website in our benchmark. Starting from Google Search or alternative websites can lead agents to use different websites to solve the task, resulting in varying difficulty levels and potentially skewed evaluation results.
+- **Start from the specified websites, not Google Search**: To enable fair comparisons, please ensure that each task starts from the specified website in our benchmark. Starting from Google Search or alternative websites can lead agents to use different websites to solve the task, resulting in varying difficulty levels and potentially skewed evaluation results.
 - **Include only factual actions, not agent outputs**: The action history should contain only the factual actions taken by the agent to complete the task (e.g., Clicking elements and Typing text). Do not include the final response or any other agent's outputs, as they may contain hallucinated content and result in a high rate of false positives.
 - **Use o4-mini for WebJudge**: WebJudge powered by o4-mini demonstrates a higher alignment with human judgment, achieving an average agreement rate of 85.7% and maintaining a narrow success rate gap of just 3.8%. Therefore, please use o4-mini as the backbone for automatic evaluation.
 
-## ⚠ Please submit the trajectory file with the following format:
+## ⚠ Please submit the trajectory file using the v2 schema (`online-mind2web-v2`):
+
+All submissions must use the **v2 submission schema**. In v2, each step is a self-contained object that bundles its action, thought, screenshot, and URL together, making it straightforward for human evaluators to review each step with all relevant context in one place.
+
+For the full schema specification, action dictionary, and validation rules, see the [v2 schema README](https://github.com/OSU-NLP-Group/Online-Mind2Web/tree/main/data/schema_v2/README.md).
+
 The result of each task is stored in a folder named as its `task_id`, containing:
-- `trajectory/`: Stores screenshots of each step.
-- `result.json`: Task metadata and action history.
+- `trajectory/`: Per-step screenshots (one per step, e.g. `0000.png`, `0001.png`, ...).
+- `result.json`: Task metadata and action history following the v2 schema.
 
-Here is an [example](https://github.com/OSU-NLP-Group/Online-Mind2Web/tree/main/data/example/fb7b4f784cfde003e2548fdf4e8d6b4f) of the format.
+Here is an [example](https://github.com/OSU-NLP-Group/Online-Mind2Web/tree/main/data/example/example_v2/1b867afecf072cb877ebfa4069263746) of the v2 format.
 
 **Structure:**
 ```
@@ -94,31 +106,52 @@ main_directory/
 └── task_id/
     ├── result.json
     └── trajectory/
-        ├── 0_screenshot.png
-        ├── 1_screenshot.png
+        ├── 0000.png
+        ├── 0001.png
         └── ...
 ```
-**`result.json` format:**
+**`result.json` format (v2):**
 ```json
 {
-    "task_id": 123,
-    "task": "abc",
-    "action_history": ["abc", "xyz", "..."]
+    "schema_version": "online-mind2web-v2",
+    "task": "Display the figure comparing unemployment trends among women in Illinois and Michigan.",
+    "task_id": "1b867afecf072cb877ebfa4069263746",
+    "agent_final_answer": "The figure is displayed on the current page.",
+    "reference_length": 12,
+    "action_history": [
+        {
+            "step": 0,
+            "screenshot": "0000.png",
+            "url": "https://example.com/",
+            "action": "page -> NAVIGATE -> Initial navigation to the provided URL",
+            "action_status": null,
+            "thought": "Navigating to initial URL that was provided."
+        },
+        {
+            "step": 1,
+            "screenshot": "0001.png",
+            "url": "https://example.com/data",
+            "action": "CLICK coords(902, 204) -> click the 'Explore Data' button | SUCCESS",
+            "action_status": "SUCCESS",
+            "thought": "Opening the Explore Data menu to find the target metric."
+        }
+    ]
 }
 ```
+
 **`human_result.json` format:**
 ```json
 [
     {
-        "task_id": 123,
+        "task_id": "abc123",
         "task": "abc",
         "human_label": 0 or 1 (failure or success)
     },
     {
-        "task_id": 456,
+        "task_id": "def456",
         "task": "def",
         "human_label": 0 or 1 (failure or success)
-    },
+    }
 ]
 ```