RAG 数据集及评价标准

{
	input: How does Spark Streaming enable real-time data processing?
	answers: ['Spark Streaming extends ...... ']
  	context: "Whole Book......"
	length: 131651
	context_id: 7bcef8714a477fd61fc8fb0d499b2cc3
	_id: b2fd8d9c6d1499d521d778ce3d6d06fa
	label: cs
	meta: {'title': 'Machine Learning With Spark', 'authors': 'Nick Pentreath'}
}

 User: A tech journalist looking for insights and trends in the tech industry
 Task: Understanding how tech leaders view the role of policy and regulation
 Questions:
 1. Which episodes deal primarily with tech policy and government regulation?
 2. How do guests perceive the impact of privacy laws on technology development?
 3. Do any guests discuss the balance between innovation and ethical considerations?
 4. What are the suggested changes to current policies mentioned by the guests?
 5. Are collaborations between tech companies and governments discussed and how?

总问题数量: 90447

问题类型分布:
  - comparison: 17456 (19.3%)
  - bridge: 72991 (80.7%)
  
难度分布:
  - medium: 56814 (62.8%)
  - hard: 15661 (17.3%)
  - easy: 17972 (19.9%)
  
问题长度分布:
  - 0-10: 0 (0.0%)
  - 11-20: 8 (0.0%)
  - 21-30: 96 (0.1%)
  - 31-40: 664 (0.7%)
  - 41-50: 3352 (3.7%)
  - 51-70: 18602 (20.6%)
  - 71-100: 31161 (34.5%)
  - 100+: 36564 (40.4%) 100左右最多
  
supporting_facts统计:
  - 几乎都是2，其次3、4，其他极少
  
context统计:
  - 几乎每个问题的上下文数量都在9-10

{
  "_id": "5a8b57f25542995d1e6f1371",  # 问题编号
  "answer": "yes",  # 回答（简短）
  "question": "Were Scott Derrickson and Ed Wood of the same nationality?",  # 问题
  "supporting_facts": [  # 黄金段落所在文档的标题以及句子的编号
    ["Scott Derrickson", 0],
    ["Ed Wood", 0]
  ],
  "context": [  # 相关的文档，文档内包含多个段落
    [
      "Ed Wood (film)",
      [
        "Ed Wood is a 1994 American biographical period comedy-drama film directed and produced by Tim Burton, and starring Johnny Depp as cult filmmaker Ed Wood.",
        " The film concerns the period in Wood's life when he made his best-known films as well as his relationship with actor Bela Lugosi, played by Martin Landau.",
        " Sarah Jessica Parker, Patricia Arquette, Jeffrey Jones, Lisa Marie, and Bill Murray are among the supporting cast."
      ]
    ],
    [
      "Scott Derrickson",
      [
        "Scott Derrickson (born July 16, 1966) is an American director, screenwriter and producer.",
        " He lives in Los Angeles, California.",
        " He is best known for directing horror films such as \"Sinister\", \"The Exorcism of Emily Rose\", and \"Deliver Us From Evil\", as well as the 2016 Marvel Cinematic Universe installment, \"Doctor Strange.\""
      ]
    ]
  ],
  "type": "comparison",  # 问题类型
  "level": "hard"  # 问题等级
}

总问题数量: 7405

问题类型分布:
  - comparison: 1487 (20.1%)
  - bridge: 5918 (79.9%)

难度分布:
  - hard: 7405 (100.0%)
  
supporting_facts统计:
  - 几乎都是2，其次3、4，其他极少
  - 平均每个问题的支持性事实数量: 2.43
  - 最多支持性事实数量: 8
  - 最少支持性事实数量: 2

context统计:
  - 10篇文章，2篇相关，8篇不相关

问题类型分布:
  - comparison: 1487 (20.1%)
  - bridge: 5918 (79.9%)

难度分布:
  - hard: 7405 (100.0%)

问题长度分布:
  - 0-10: 0 (0.0%)
  - 11-20: 0 (0.0%)
  - 21-30: 0 (0.0%)
  - 31-40: 49 (0.7%)
  - 41-50: 282 (3.8%)
  - 51-70: 1612 (21.8%)
  - 71-100: 2919 (39.4%)
  - 100+: 2543 (34.3%)

supporting_facts数量分布:
  - 包含 2 个supporting_facts的问题: 4990 (67.4%)
  - 包含 3 个supporting_facts的问题: 1774 (24.0%)
  - 包含 5 个supporting_facts的问题: 80 (1.1%)
  - 包含 4 个supporting_facts的问题: 537 (7.3%)
  - 包含 7 个supporting_facts的问题: 9 (0.1%)
  - 包含 6 个supporting_facts的问题: 14 (0.2%)
  - 包含 8 个supporting_facts的问题: 1 (0.0%)

context数量分布:
  - 10篇文章，属于是open-domain的开放域问答任务（没有人为设置负样本？）